A practical guide to stemming in Elasticsearch and OpenSearch in 2026. Why the Porter stemmer's aggressive defaults hurt precision once you add a vector retrieval leg, when light stemming is the right choice for hybrid search, and how to handle multilingual corpora.

Stemming in Elasticsearch and OpenSearch: Why Light Beats Aggressive in the Hybrid Search Era

Most Elasticsearch and OpenSearch indices in production still ship with the english analyzer or a porter_stem filter wired into the text field, often because someone copied a mapping in 2015 and it survived three migrations. That default was a reasonable call when BM25 had to do all the recall work on its own. In a hybrid search setup with a dense vector retrieval leg, it is the wrong default. The vector leg already absorbs morphological and semantic recall; aggressive stemming on the BM25 leg now mostly contributes false positives that the fusion step cannot undo.

This post walks through what stemming does in the Lucene-based stack, why the Porter stemmer's aggressive defaults are a worse fit today than they were a decade ago, how the hybrid search architecture in both engines reshapes the trade-off, and what to do about your existing mappings.

What Stemming Actually Does in Elasticsearch and OpenSearch

Stemming is a token filter that reduces inflected and derived forms of a word to a common stem so that BM25 can match across them - "running", "ran", and "runs" all collapse to a shared token so a query for one finds documents containing the others. In Elasticsearch and OpenSearch, stemming runs as one stage in an analyzer chain: character filters → tokenizer → token filters (which is where the stemmer lives).

The Lucene-based stack offers a tiered menu of stemmers, from nearly-do-nothing to dictionary-driven:

Aggressiveness Filter / analyzer What it does
Minimal minimal_english Strips plurals, little else
Light light_english (alias for kstem), light_german, light_french, light_italian, light_spanish, light_finnish, light_russian, arabic Plurals + common inflections
Moderate kstem Krovetz: algorithmic + built-in dictionary
Aggressive porter_stem, stemmer/english (Porter), stemmer/porter2 (Snowball English) Suffix-stripping rules; collapses many derived forms
Dictionary hunspell Per-locale .dic/.aff files; closer to true lemmatization
Segmentation (not stemming) analysis-kuromoji, analysis-nori, analysis-smartcn Word-segment CJK; morphology is moot

A few details that bite engineers in practice. The english analyzer uses the Porter stemmer by default, not the lighter kstem. The light_english filter is an alias for kstem under the hood. And the built-in german, french, italian, and spanish analyzers default to the light variants - Elastic made the conservative choice for those languages but not for English, mostly for backwards compatibility. The Elasticsearch stemmer filter reference lists every variant; the language analyzer page shows what each built-in analyzer pipes together.

The Porter Stemmer's Famous Failures

The canonical example comes from Manning, Raghavan, and Schütze's Introduction to Information Retrieval, §2.2.4: Porter collapses operate, operating, operates, operation, operative, operatives, operational all to oper. The book's verdict is blunt - "a sentence with the words operate and system is not a good match for the query operating and system" - and lays out the general rule that "stemming increases recall while harming precision."

A second family of failures: universal, universe, university, universities all collapse to univers. Three distinct modern meanings, one stem. A query for university rankings now matches articles about Universal Studios and the heat death of the universe. The third one is even more embarrassing: news stems to new, so a query for news matches every document that mentions anything new. That last one is documented as an open complaint on the Elasticsearch tracker since 2015 (elastic/elasticsearch#11541), and it is the reason Martin Porter himself recommended using Porter2 (Snowball English) instead - the original Porter is kept around mostly for academic comparison.

Brand and proper-noun damage is the failure mode that costs e-commerce and content publishers the most. Operative (the ad-tech company), Universal (the studio), Novel (a thousand products), Microsoft Surface matching surface documents - none of these survive Porter cleanly, and most teams discover the breakage only when a marketing manager forwards a screenshot.

Even before vectors entered the picture, Krovetz (kstem) and the light stemmers were the more defensible choices for any corpus where precision mattered as much as recall. What changed with hybrid search is that the recall argument for Porter mostly evaporated.

Why Hybrid Search Changes the Calculus

Hybrid search in both engines combines a lexical leg (BM25) with a dense vector leg, fusing the two result lists. Elasticsearch exposes this through the retriever API with RRF; OpenSearch ships a hybrid query plus a normalization-processor, and added RRF in 2.19 via the score-ranker processor. RRF's appeal is that it needs no tuning: each leg contributes 1 / (k + rank) per document, with k=60 by default.

The vector leg is morphology-agnostic. Running, runs, and ran all sit close in embedding space, and so do paraphrases, near-synonyms, and translingual variants. That is the recall job Porter was being asked to do. The vector leg now does it better, and it does it without conflating operate and operational into the same token.

That reshuffles what the BM25 leg is for. Its remaining specialties are the things vectors are weak at: exact matches, rare terms, SKUs, model numbers, brand names, jargon, novel proper nouns, code identifiers, and any token the embedding model under-weights because it was rare in training. Aggressive stemming directly damages every one of those jobs. When Porter collapses operative into oper and BM25 ranks a sea of operation documents above the actual Operative hit, RRF cannot rescue the precision - it only sees ranks, not why a document ranked. False positives from the BM25 leg poison the fusion exactly when you most needed BM25's exactness to win.

The pragmatic conclusion: on the BM25 leg of a hybrid pipeline, prefer light stemming (kstem / light_english) for the morphological forgiveness you actually need - plurals, possessives, gerunds - without conflating semantically distinct words. Let the vector leg own the rest.

A Practical Multi-Field Pattern

A workable production mapping indexes the same text three ways and lets the query layer pick what to use:

{
    "mappings": {
      "properties": {
        "title": {
          "type": "text",
          "analyzer": "light_english_analyzer",
          "fields": {
            "exact":  { "type": "keyword" },
            "heavy":  { "type": "text", "analyzer": "english" }
          }
        },
        "title_embedding": { "type": "dense_vector", "dims": 1024 }
      }
    },
    "settings": {
      "analysis": {
        "analyzer": {
          "light_english_analyzer": {
            "tokenizer": "standard",
            "filter": ["lowercase", "kstem"]
          }
        }
      }
    }
  }

The BM25 leg of your hybrid query targets title (light stemming) with a boost on title.exact (no stemming at all) for high-precision matches on brand and SKU tokens. The title.heavy field with the aggressive english analyzer is kept as a low-weight fallback - useful for documents where light stemming misses a real morphological variant. The vector leg goes through title_embedding and is unaffected by analyzer choice.

In Elasticsearch, the retriever stitches it together:

{
    "retriever": {
      "rrf": {
        "retrievers": [
          { "standard": { "query": { "multi_match": {
              "query": "operative campaign management",
              "fields": ["title^1.0", "title.exact^3.0", "title.heavy^0.3"]
          }}}},
          { "knn": { "field": "title_embedding", "query_vector": [/* ... */], "k": 50, "num_candidates": 200 } }
        ],
        "rank_window_size": 100,
        "rank_constant": 60
      }
    }
  }

In OpenSearch the equivalent uses a hybrid query with a search pipeline configured to apply RRF or min-max normalization. Same idea, different syntax. The key is that the BM25 leg's analyzer choice is now a precision lever, not a recall lever - tune it accordingly.

A common mistake here: copying the same heavy-stemming analyzer onto the field you feed into the embedding model. Vector models do their own subword tokenization (BPE, WordPiece, SentencePiece) and want clean text. Lowercasing and Unicode normalization are usually enough; running text through a stemmer before embedding throws away signal the model would have used.

Light Stemming Across Languages

English is the easy case. The interesting decisions are everywhere else.

German: stemming matters less than compound splitting. Donaudampfschiff will never match a query for Donau with any stemmer, but the hyphenation_decompounder with a hyphenation pattern file plus a German word list does the job. Pair that with light_german and you have a respectable German pipeline. Heavy stemming adds little.

French, Spanish, Italian, Portuguese: the built-in french, spanish, italian, portuguese analyzers default to light stemmers and are a fine baseline. Multilingual embedding models like BGE-M3 and the E5 family are strong enough here that the BM25 leg can stay light without losing recall.

Arabic: Larkey et al.'s Light Stemming for Arabic Information Retrieval showed back in 2007 that the light10 algorithm outperformed root-extraction stemmers like Khoja on TREC; Lucene's Arabic light stemmer is descended from that work. With a dense leg on top, light remains the right call. Root extraction is too aggressive and loses too many distinctions.

Finnish, Turkish, Hungarian: agglutinative morphology that compounds aggressively. Embedding models in these languages are noticeably weaker than English-or-German-trained ones, so the BM25 leg has to pull more weight. Snowball stemmers (finnish, turkish, hungarian) and Hunspell with locale dictionaries earn their keep here even in hybrid setups. This is the genuine exception to "go light."

Chinese, Japanese, Korean: morphology is largely moot. What matters is segmentation - turning a character stream into word-like tokens. Use analysis-kuromoji for Japanese, analysis-nori for Korean, and analysis-smartcn for Chinese. The stemmer question does not apply.

When You Still Want Aggressive Stemming

The hybrid-search-changes-everything argument has limits. Aggressive stemming or full lemmatization is still the right call in several cases:

  1. No vector layer. Pure lexical search on legacy or air-gapped systems is back to the original Porter trade-off: recall starvation hurts more than precision loss.
  2. Very short documents. Titles, product names, tweets - any corpus where the average document has fewer than 20 tokens needs every recall boost it can get.
  3. Heavy-morphology languages with weak embeddings. Finnish, Turkish, Hungarian, some Slavic languages.
  4. Reproducibility against academic benchmarks. TREC, BEIR, and MS MARCO baselines are usually published against specific analyzer chains.
  5. Genuine lemmatization is wanted. Hunspell, spaCy, or Stanza in an external enrichment pipeline gives you proper lemmas (better → good, mice → mouse) that no algorithmic stemmer can produce. The cost is a bigger ingestion pipeline.

For everything else with a vector leg in the mix, lighter is the safer default.

Migrating Off Porter Without Breaking Production

Production search teams rarely have the luxury of a clean re-architecture. A safe migration path:

  1. Build a labeled query set. Pull the top few hundred queries from your search logs, judge the top 10 results for each (or pay a labeling vendor to). This is the only way to know whether a change helped or hurt.
  2. Index a parallel field. Add a title_light field with light_english next to your existing title field. Reindex in place - no need to rebuild the index from scratch.
  3. A/B at query time. Route a fraction of traffic to a query targeting title_light instead of title. Compare NDCG@10 and click-through rates against your judged set.
  4. Audit synonym files. If you have a synonyms file that was authored against aggressive stems, some entries will quietly stop matching. oper, operate, operation was probably written because Porter collapsed them anyway; with light stemming you may need explicit synonym entries.
  5. Watch the long tail. Aggregate failure cases come from rare queries. Set up a small dashboard of zero-result queries before and after the cutover.
  6. Cut over the BM25 leg first. Vectors are usually downstream of a separate embedding service - leave that alone for the first migration and only revisit if the eval set asks for it.

Our precision and recall primer covers the evaluation side in more depth, and the search modernization playbook walks through the broader hybrid architecture this analyzer change fits into.

Key Takeaways

  • Stemming is a recall tool for BM25. Once a dense vector leg shoulders the morphological recall job, aggressive stemming becomes mostly a precision liability.
  • The Porter stemmer is too aggressive for most modern hybrid pipelines. kstem / light_english is a better default. Use Snowball Porter2 instead of original Porter if you must stay heavy.
  • The BM25 leg in a hybrid setup is a precision leg. Light stemming preserves the exact-match and rare-term jobs that vectors are weak at.
  • RRF cannot undo BM25 false positives. It only sees ranks, not match quality, so the BM25 leg has to earn its place by being right.
  • Multilingual decisions still vary. Light stemming is the right default for English, German (with decompounding), Romance languages, and Arabic. Heavy morphology languages with weak embeddings remain the genuine exception. CJK is about segmentation, not stemming.
  • Migrate with measurement, not vibes. Parallel fields, judged query sets, and zero-result dashboards beat anecdotes every time.

If you are working through an analyzer redesign on Elasticsearch or OpenSearch and want a second pair of eyes on the search relevance math, get in touch.