top of page

The Limits of Full-Text Search in E-Commerce: Stemming, Compounds, Synonyms & 8 More Walls

Writer: Ondrej Nespor
Ondrej Nespor
5 days ago
20 min read

Written by Ondřej Nešpor, CTO at Raventic


The query "black leather boots under €100" split into five index tokens: "black" and "leather" match literally, "boots" needs a stemmer, and "under" and "100" are dropped or meaningless so the price constraint is lost.

Almost every e-commerce search setup starts with full-text search: an inverted index, a tokenizer, maybe a stemmer, and a scoring algorithm like BM25 to turn term frequencies into a relevance score. It’s fast, predictable, lightweight, and has decades of production mileage.



Why we're writing this — Raventic builds search and product discovery for e-commerce catalogs. So the "permanent maintenance tax" in this chapter isn't a survey of other people's problems - inflection tables, decompounders, diacritic collisions, root-shift rules - it's the exact stack we set out to stop hand-maintaining. We're writing this series to explain, precisely, why lexical search behaves the way it does, and what that implies for anyone building or buying a search solution.



Series navigation

Part 1:   The Limits of Full-Text Search in E-Commerce (you’re here)

Part 2:   The Limits of Semantic Search, and the Best You Can Do Within Them

Part 3:   The Limits of Hybrid Search

Part 4:   The Constraint Satisfaction Paradigm in Ecommerce Search



TL;DR

  • Full-text search (an inverted index plus BM25) is fast, predictable, and auditable - and every serious stack, Alibaba, Walmart, and Amazon included, keeps it. The question was never whether to drop the lexical index, but what to build on top of it.


  • It matches characters, not meaning, so it fails in a fixed cycle: literal matching hits a wall → the industry patches it with a dictionary, rule, or scoring dial → that patch becomes a permanent maintenance tax or quietly costs precision. This chapter walks eleven of those walls.


  • Two load-bearing ones most tuning guides skip: decompounding (a German store is simply broken without it - splitting beats stemming by 15–25% for German and Swedish) and Slavic internal root shifts (pes → psa, vůz → vozu) that no suffix stemmer resolves - a hand-built Czech stemmer barely beats indexing raw 4-grams (MAP 0.3068 vs. 0.3057).


  • Every patch (stemmer, decompounder, synonym list, match percentage, merchandising rule) is a manual stand-in for knowledge the engine doesn't have, rebuilt for every language, catalog, and slang shift. Pushing lexical search to its limit (Macy's 7-pass cascade) doesn't add understanding; it just codifies the trade-offs in a giant state machine.


  • Full-text isn't the wrong engine - it's an incomplete one. Keep it for what it's unbeatable at (exact match, sub-millisecond filtering, hard inventory rules, auditable results) and hand the semantic jobs elsewhere. Part 2 asks whether a single model can replace the whole hand-built stack.


Scope clarification - this isn’t about exact-match lookups. SKUs, part numbers, and product codes are trivial string matches. When we talk about “search” here, we mean natural-language discovery: shoppers searching in unstructured terms and expecting the system to understand what they want.


This is where full-text search hits a wall. Not because of bad engineering, but because we’re asking a rigid, literal string-matching engine to solve semantic intent. Every limitation follows a predictable cycle: full-text hits a structural boundary, the industry patches it, and that patch introduces manual maintenance or hurts precision.


The same cycle repeats for all eleven full-text failure modes in this article: each patch buys a fix but adds permanent maintenance or costs precision, which surfaces the next limitation.

Plugging these leaks is never a one-time fix. It’s a permanent tax — a constant cycle of curating dictionaries, updating synonym lists, and tweaking scoring dials to match changing catalogs and search behavior.


None of this means the inverted index is useless. Every failure mode we’ll look at happens because the engine is doing exactly what it was designed to do — it’s just being fed inputs it wasn’t built to interpret. Production setups at Alibaba, Walmart, and Amazon all keep the lexical index. The question isn’t whether to throw it away, but what we need to build on top of it.


Query understanding: the pre-retrieval layer

In production, search pipelines rarely send raw user queries straight to an inverted index. Instead, an upstream query understanding (QU) layer acts as a gatekeeper - classifying intent (e.g., transactional vs. informational) and rewriting the text before the database even sees it.


Take Amazon’s research on query rewriting. “Tail” queries (rare, long, or oddly phrased searches) lack historical log data, so raw lexical indexes struggle to match them. Amazon solves this by training seq2seq models to rewrite these tail queries into popular “head” queries. This highlights why simple lexical tricks like synonym tables and stemming fail on the long tail: they depend on token patterns and historical frequencies that rare queries simply don’t have.


A single token change can also flip a query’s meaning entirely. Home Depot’s query logs analyzed in the JointMap paper show this sensitivity perfectly:



"30 in. 5.8 cu. ft. gas range installation kit" → a buyer looking for a product

"30 in. 5.8 cu. ft. gas range installation" → someone looking for a service


One dropped token (kit) flips the intent. In their dataset, non-commercial service queries made up only 1.5% of total volume, but at Home Depot’s scale, that meant 33 million queries a year.


This illustrates the wide gap between low-level engine capabilities and what users actually do. Search engines like Elasticsearch, OpenSearch, Solr, or Typesense offer powerful Domain-Specific Languages (DSLs) with field-level boosting (title^3), range filters (price:[* TO 50]), and boolean logic. But shoppers type "black nike shoes under 50", not brand:nike AND color:black AND price:[* TO 50]. The engine’s low-level dials are useless without an upstream layer that extracts entities and numeric ranges from unstructured text. Full-text search is an incredibly fast execution engine, but it requires an external translation layer to bridge the gap between human language and structural database queries.


If you want to solve these limits, you don’t rewrite the inverted index. You wrap it in machine learning systems: intent classifiers, entity extractors, query rewriting models, and click-graph translators.


Production limitations: order of operational impact


1. Morphological rigidity

Raw full-text indexes match literal characters, not concepts. A search for "boty" (Czech for “shoes”) won’t match items with "botami" (with shoes) or "botách" (in shoes) because the tokenizer generates entirely different hashes for them. In highly inflected languages like Czech or German, where nouns have a dozen or more case and number forms, literal string matching ruins search recall out of the box. German takes this a step further with compound words, which inflection rules can’t help with (we’ll look at that in section 3).


The result is a constant maintenance headache: every new product title, category, or attribute requires linguistic coverage for every possible grammatical form a user might search for.


2. The limits of stemming

To fix this, search engines use stemming to collapse inflections ("boty", "botami", "botách") down to a single root. Dictionary-based stemmers (like Hunspell) rely on lookup tables. While they work well for simple plurals, dictionary stemming has major structural flaws:


  1. Static dictionaries vs. infinite slang: Dictionaries are finite and static. Real user vocabularies are dynamic and infinite. Out-of-vocabulary (OOV) slang, brands, and neologisms slip right past the stemmer.


  2. Prefix stripping collapses meaning: In Slavic languages, prefixes don’t just change grammar - they change the entire meaning of a word (derivational morphology). Simple suffix-stripping rules risk chopping off prefixes and blending completely unrelated terms together.


  3. SaaS black boxes: If you use a managed SaaS search platform, you rarely get access to the underlying dictionary files. Fixing an OOV gap usually means opening a support ticket and hoping their SLAs cover it.


  4. Shifting word roots (internal alternations): Rule-based stemming assumes the root of a word stays the same while only the endings change. In many languages, the root itself mutates - usually in the middle vowels.


The fourth issue is a silent killer for search recall. Czech, for example, has two major classes of internal alternations that no suffix-stripping rule can resolve:


Vowel shifts: vůz (car) inflects to vozu or vozy. dům (house) becomes domu or domy. kůže (leather) derives kožený (made of leather). The ů to o shift is a historic sound change from the 14th–16th centuries (ó > uo > ů), meaning the mutation occurs right in the middle of the stem rather than at the edge (Marešová, 2008; NESČ).


The disappearing vowel: pes (dog) inflects to psa, psi, and psů. Here, the -e- disappears entirely. This pattern traces back to Proto-Slavic jers and remains a core feature of the modern language (Havlík, 1889).


In practice, this causes real problems. In the standard Czech Hunspell dictionary (cs_CZ), words like psa, psi, psů, psech, psí, domu, and vozu are listed as separate base words rather than inflected forms of pes or dům. Hunspell fails to connect them. If a user searches for pes, they won’t find products containing psí (dog-related).


Rule-based stemmers try to handle this by hard-coding every variation. Dolamic and Savoy’s Czech stemmer uses 52 case-ending rules plus five manual root normalization rules to handle these vowel shifts. Yet, their benchmarks show that a light Czech stemmer only gets a Mean Average Precision (MAP) of 0.3068, while simply matching raw character 4-grams (no stemming at all) gets 0.3057.


Diagram of the Czech words "pes" (dog) and "vůz" (car): the stem mutates internally - a vowel disappears (pes → psa) or shifts mid-word (vůz → vozu) - where no suffix-stripping stemmer can reach.



A hand-crafted linguistic stemmer barely outperforms throwing out morphological rules entirely and indexing arbitrary slices of characters: MAP 0.3068 (light stemmer) vs. 0.3057 (raw 4-grams).


This isn’t an isolated edge case. Ševčíková (2018) found that these internal mutations affect almost all vowels and consonants. Among the 500 most common Czech base words, 20% experience a shift relative to their root, and over 54% of their derived forms are affected. As the paper concludes, the resulting patterns are highly irregular. Irregularity is exactly what hard-coded rule tables struggle to scale with.


Algorithmic stemmers (like Porter) skip dictionaries entirely and use deterministic rules to chop off endings. But this introduces classic information retrieval errors:


  • Over-stemming: Stripping too much and merging unrelated words (e.g., matching "organization" and "organ" to the same root).


  • Under-stemming: Stripping too little and failing to group variations of the same word.


While the Porter stemmer is the industry default, it is notoriously brittle and hard to tune (CS4300, Cornell). Its rules were engineered for English and translate poorly to highly inflected languages (GeeksforGeeks, 2023).


As inflection complexity rises, rule counts explode. Milošević (2012) points out that while English Porter stemmers need around 63 rules, South Slavic languages require three to four times as many. Because of this complexity, standard frameworks like Snowball only support a couple of Slavic languages natively.


Bar chart: a hand-built light Czech stemmer scores 0.3068 Mean Average Precision versus 0.3057 for indexing raw character 4-grams with no linguistics at all (a gap of just +0.0011).


Agglutinative languages like Finnish show the ultimate limits of dictionary-based stemming. Hunspell’s format caps consecutive suffixes at two, but Finnish regularly chains three or more (case endings, possessive markers, and clitic markers). As Finnish localization docs highlight, Hunspell simply cannot represent Finnish grammar, regardless of dictionary size. Finnish production setups have to replace Hunspell entirely with specialized Finite-State Transducer (FST) systems like Voikko or Omorfi.


3. Decompounding: the critical piece most guides skip

Almost every search tuning guide talks about stemming. Barely any mention decompounding, even though it’s a hard requirement for acceptable search quality in European languages like German, Dutch, Danish, Swedish, Norwegian, Finnish, and Greek. All of these languages join compound words into a single word (Baroni, Matiasek & Trost, 2002).


German is the classic example. It’s famous for words like Rindfleischetikettierungsüberwachungsaufgabenübertragungsgesetz (a former law name), which Elasticsearch’s blog calls “a nightmare for search engines unprepared to handle compounding”. The process is straightforward concatenation: Rind (cow) + Fleisch (meat) = Rindfleisch (beef). You can chain nouns indefinitely.


Stemming can’t help you here. Stemming fixes the edges of a word (running → run). A compound word has an internal matching issue: Lederjacke (leather jacket) and Jacke (jacket) look like completely different strings to a tokenizer. No affix rule, lookup table, or stemmer will make a query for Jacke find a product named Lederjacke.


And you can’t just list all compounds in a dictionary — compounding is a creative, open-ended process. Baroni et al. found that 47% of word types in their German corpus were compounds, yet 83% of those compounds appeared five times or fewer. Their takeaway is key: compounding produces a massive volume of unique words that can’t possibly be pre-listed in a dictionary.


The search penalty is steep. Hollink et al. (2004) measured search accuracy (MAP) and found that compound splitting boosted retrieval by 12.2% for German, 18.7% for Finnish, 6% for Swedish, and 4% for Dutch. More importantly, splitting beat stemming by 15.5% for German and 25.3% for Swedish.


Language

MAP gain from compound splitting

Splitting vs. stemming

German

+12.2%

+15.5% better than stemming

Finnish

+18.7%

/

Swedish

+6%

+25.3% better than stemming

Dutch

+4%

/


For these languages, the feature most guides ignore is actually far more important than the stemming they all cover. Other studies (Braschler and Ripplinger, 2004; Airio, 2006) confirm that decompounding is the single most effective way to boost search quality outside of English.


Engines do support decompounding, but as a separate, complex module. Lucene has a compound analysis package exposed in Elasticsearch as two filters: dictionary_decompounder (which checks subwords against a list) and hyphenation_decompounder (which uses XML hyphenation rules to find splits and checks them against a dictionary). Algolia supports decompounding as a per-language toggle for only six languages.


But setting up decompounding is a painful chore:


  • You must supply your own word lists. Neither Elasticsearch filter comes with a default dictionary. The gold standard for German, Uwe Schindler’s german-decompounder, contains about 14,500 curated words - specifically, the building blocks of compounds, not the compounds themselves.


  • Hyphenation patterns are hard to license. Lucene can’t ship the hyphenation files directly because of license conflicts, so you have to track them down and download them yourself from the OFFO project.


  • Accuracy depends entirely on the quality of your list. Lucene’s docs state that output quality is tied directly to your dictionary, and that dictionary-only matching is much slower than using hyphenation rules.


Decompounding also introduces a precision problem: compound words aren’t always just the sum of their parts. Handschuh is a glove, not a “hand-shoe”. Algolia points this out as a word you should never split, highlighting the classic trade-off: a larger dictionary increases search recall but degrades precision.



Hollink et al. found similar issues: German Bahnhof (train station) splits into Bahn (rail) and Hof (yard); Dutch brandstof (fuel) becomes brand (fire) and stof (dust). These splits introduce words that are barely related to the actual product, causing search results to drift. An aggressive dictionary will split Kaffeetasse (coffee cup) and find a fake subword fee (fairy). Elasticsearch 8.17 had to add a no_sub_matches flag just to block these kinds of errors.


As Krotova et al. (2020) write, figuring out whether a word is literal or idiomatic is a matter of degree. It requires understanding meaning, which is the one thing a tokenizer lacks.


The pattern here is identical to stemming. You patch a fundamental gap in string matching by throwing a hand-curated dictionary at it. You then have to maintain that dictionary forever against an open-ended vocabulary, trading recall for precision. The decompounder is load-bearing — a German storefront is fundamentally broken without it — but it doesn’t solve the core problem: we still need something upstream that understands Lederjacke is a jacket.


4. Synonymy and vocabulary mismatch

Stemming groups different forms of the same word ("shoes" / "shoe"). Synonymy tries to connect completely different words that mean the same concept ("couch" / "sofa", "hoodie" / "sweatshirt").



The Vocabulary Mismatch Problem — a classic study by Furnas et al. (1987) showed that two people choose the exact same word for a common object less than 20% of the time. Relying on a single target term leads to an 80% to 90% search failure rate. Other search research (Moldovan et al.) found that vocabulary mismatch alone is responsible for over a quarter of all search failures.


Vocabulary mismatch causes unrecoverable recall loss. If a product page doesn’t share a literal term with the user’s query, it gets filtered out immediately during candidate retrieval (see also Sease.io’s breakdown of document expansion). No downstream machine learning or re-ranking model can rescue a document that was never retrieved in the first place.


Since almost every product in an e-commerce catalog can be described in multiple ways, keeping up with synonyms requires constantly mapping a shifting vocabulary against a growing catalog. It’s a permanent operational overhead. Ship bare full-text and synonym tuning stops being a task and becomes a lifestyle.


To see why this overhead is so stubborn, try picking any noun from an e-commerce catalog - sofa, sneakers, or hoodie - and give yourself thirty seconds. You will easily list two or three synonyms (couch, trainers, sweatshirt) without breaking a sweat.


But that is the trap. The exercise feels trivial because, as humans, we do not just produce a flat list of words; we silently rank how strong each synonym is and intuitively sense where it would misfire in context. Encoding that complex human judgment at catalog scale, across every supported language, and keeping it accurate as inventory and slang drift, is an entirely different problem.


As search engine providers like Algolia have noted, shipping a generic, out-of-the-box synonym dictionary per language is ultimately counterproductive because synonymy is a matter of degree rather than binary, fixed set membership. For example, a generic dictionary might treat “substitute” as a synonym for “alternative,” but in an e-commerce catalog, this can easily misfire against brand names or product titles. Similarly, mapping tee to t-shirt is highly effective in general apparel, but on a sports site, it introduces severe noise by matching golf tees. Many words are also polysemous (such as crane matching both construction equipment and birds) and can only be resolved through context.


5. The limits of statistical scoring (TF-IDF / BM25)

Traditional scoring algorithms like BM25 look at term frequency ($tf$) and inverse document frequency ($idf$). This math works beautifully for long-form documents (like Wikipedia articles), but breaks down on e-commerce product titles and short descriptions:


  • No natural frequency: E-commerce metadata is sparse. Product titles don’t have natural term frequency distributions that BM25 expects.


  • Keyword stuffing: Merchants frequently abuse BM25 by stuffing redundant keywords into product descriptions to game search rankings (KuaiSearch, 2026).


Because traditional scoring relies on simple word counts, it suffers from the same semantic blindness as the index itself. Both candidate retrieval and relevance scoring fail when they are forced to rely purely on literal token overlap.


6. Term roles, gradience, and non-lexical concepts

Full-text matching treats every word in a query as equally important, completely ignoring its semantic role:


1) Role asymmetry: In the query "yellow hoodie", "hoodie" is the core noun (the product category) and "yellow" is just a modifier (the attribute). If a search engine returns a green hoodie, that’s a minor color mismatch. If it returns a yellow scarf, it’s a total category failure. Traditional search engines let you boost certain fields (title over description), but raw string matching treats every word in a field the same way. Pinning down which word is the noun and which is the attribute requires upstream query parsing to map terms to specific fields.


2) No continuous gradients: Color and style live on a spectrum. If a yellow hoodie is out of stock, a great search engine should degrade gracefully to similar colors like mustard, gold, or orange. But full-text search is binary: a word either matches or it doesn’t. It has no concept of semantic distance.


3) No concept of “cheap” or “affordable”: Searches like "cheap shoes" or "affordable hoodie" refer to price ranges or inventory status, not literal keywords. You rarely see "cheap" written in product descriptions. While search engines can filter by price ranges (price <= 50), translating the concept of “cheap” into a numeric range requires an upstream layer to parse the query and build the correct database filters.


Three product cards: a green hoodie and a yellow scarf each match one of the two query tokens for "yellow hoodie", yet one is a usable near-miss and the other the wrong category; below, a yellow-to-rust colour ramp that full-text search cannot perceive.

These limits have a massive impact on actual user experience. Baymard Institute benchmarks show that non-relevance sorting degrades search results on 90% of e-commerce sites, and 36% of storefronts have product-list design and feature flaws severe enough to actively harm users’ ability to find and select products. Large platforms like JD.com are actively researching generative, multi-task models that produce facets dynamically because traditional indexes simply can’t surface the right attributes on their own.


7. The AND vs. OR dilemma

Query logic in full-text engines forces you into a trade-off:


Strategy

Gains

Failure mode

Strict AND

Maximizes precision

Ruins recall — a single typo, missing synonym, or un-stemmed word drops the product entirely

Loose OR

Maximizes recall

Hurts precision — relies on BM25 to push relevant items up, which fails because the engine doesn’t understand the words

minimum_should_match

Tunable middle ground

Turns search quality into an endless game of tweaking percentages for different query lengths

This dilemma exists entirely because the engine is role-blind: if it knew the difference between the main noun and a modifier, it wouldn’t need a clunky global percentage rule.


8. The limits of edit distance (typo tolerance)

To handle typos, search engines use Levenshtein edit distance, often allowing more edits for longer words. But this geometric approach has no semantic context:


  • Orthographic collisions: In Czech, "lodičky" (pumps/shoes) and "hodinky" (watches) differ by just two letter changes (l to h, č to n). A standard edit-distance rule sees two changes on a 7-letter word as safe, causing watches to show up under shoe searches.


  • Spelling vs. meaning: As noted in NLP tutorials, "cat" and "feline" have a huge edit distance despite meaning the exact same thing, while "cat" and "cot" are just one letter apart but mean completely different things.


Research on weighted edit distance (Samuelsson, 2017) confirms that simple character math can’t distinguish a real typo from a completely different word. Proposed fixes (arXiv:1805.11611) try to weight edit steps using vector embeddings - which is basically admitting that character-level edit distance needs a neural model to work properly.


Scatter plot of character edit distance versus meaning: a real typo (hoodie ↔ hoodei) sits at 2 edits, but so do unrelated words (lodičky ↔ hodinky, pas ↔ pás), while synonyms (cat ↔ feline) are 6 edits apart - no distance threshold separates typos from different words.

9. Diacritic normalization vs. precision loss

In languages with diacritics (like Czech: á, č, ď, é, ě, í, ň, ó, ř, š, ť, ú, ů, ý, ž), mobile users rarely bother to type accent marks. If you try to absorb this missing detail by turning up typo tolerance, you get even more search collisions across your catalog.


The standard workaround is ASCII folding — stripping all diacritics from both the product data and the user’s query. But diacritics often distinguish completely different words:


pas (passport) vs. pás (belt / strap)

Stripping accents collapses these distinct words into the same token, trading away precision to get better recall.


10. The visual blind spot

Visual qualities — like patterns, textures, cuts, and silhouettes — are invisible to text search. If a visual feature isn’t explicitly written down in the product metadata, no amount of stemming, synonyms, or scoring tweaks can find it.


The industry’s solution — using computer vision models to auto-tag images — is a clear admission of full-text’s visual blind spot: it cannot capture visual style without outsourcing the job to multi-modal neural models.


11. The language barrier

Full-text search requires a perfect literal match between the query language and the catalog language. While most sites handle multilingual traffic by translating their catalogs, dense multilingual models can match concepts across languages natively — which we’ll cover in Part 2.


The superpower: determinism and auditability


Despite all these issues, full-text search has one massive advantage: it is completely predictable.


Because matching is built on clear, discrete token rules rather than fuzzy, high-dimensional math, every single result is auditable. An engineer can look at a query and explain exactly why a product showed up based on term matches, document frequencies, synonym rules, or stemming.


This explainability is why enterprise architectures keep lexical indexes even when they adopt neural search. Alibaba’s Taobao engine (MGDSPR, 2021) uses an inverted index layer specifically to enforce strict business rules within its neural retrieval pipeline. Similarly, Walmart’s research team (2024) points out that inverted indexes remain essential for speed, scale, and predictability.




The rigidity that causes full-text search to fail is also what makes it so easy to debug. Lexical bugs are deterministic — you can trace them to a specific rule and fix them, even if that fix requires manual effort.


That’s why these eleven failure modes aren’t a reason to throw away the inverted index. They are a specification of what the index does well and what needs to be handled elsewhere. Instant exact matching, sub-millisecond filtering, hard inventory checks, and clear debug trails are incredibly hard to replicate with neural models. We don’t want to replace the lexical index — we want to hand off the jobs it was never designed to do.


Overriding relevance: merchandising rules and “relevance debt”


To survive the failures of lexical search in production, commercial platforms (Adobe Commerce, Searchspring, Meilisearch, Boost Commerce) rely heavily on manual merchandising engines. Merchandisers spend hours manually pinning popular products, boosting attributes, or setting up promotional banners.


But when rules collide, engines have to implement complex precedence chains (like Boost Commerce’s priority list):


Hide > Banner > Pin > Filter > Demote > Boost

As search architecture blogs highlight (Wizzy, 2026), stacking these manual overrides creates severe relevance debt. Local band-aids pile up without any central control, creating an incredibly brittle system where fixing one search term breaks three others. Industry vendors like Coveo openly admit that manually combing logs to manage synonyms and merchandising rules is an endless, unscalable chore.


Case study: the cascading lexical waterfall


The absolute limit of pure full-text engineering is perfectly illustrated by Macy’s multi-pass search patent (USPTO Patent #9,449,098 B2).


To squeeze every drop of capability out of Solr/Lucene, Macy’s built a 7-stage fallback cascade:

  1. Exact product ID / UPC match.

  2. Exact match across pre-defined field combinations (e.g., brand + color + type).

  3. Exact match against implicit refinements from step 2.

  4. All-terms match across any field.

  5. Tokenized any-term search.

  6. All-terms match with active spell-correction (triggered if total results $<10$).

  7. Partial token match merged with prior candidates.


Macy's seven-pass search cascade: each fallback pass (exact ID, field combinations, all-terms, tokenized, spell-corrected, partial match) fires only when the one above returns nothing, all feeding a single result set.

This architecture is the peak of full-text engineering. It handles the AND/OR dilemma by stepping down a multi-pass threshold, and simulates term roles by hard-coding specific field combinations. But it doesn’t solve the core limits of full-text — it just codifies them. Settings like stemming_mode and spellingAccuracy remain hardcoded XML properties that require continuous manual tuning.


Macy’s design shows that pushing a lexical engine to its limit doesn’t magically make it understand semantic intent. It just creates a massive, complex state machine to manage the trade-offs.



Summary


Full-text search is an incredibly fast, predictable engine for catalog lookup. But relying purely on literal character matching creates massive problems with grammar, compounding, synonyms, word roles, typos, and visual attributes. Patching these holes with pure text search requires a mountain of dictionaries, custom rule overrides, and manual configurations.


The pattern is clear: every patch we’ve looked at is a manual linguistic or statistical band-aid standing in for knowledge the search engine doesn’t have. The stemmer defines what counts as the same word. The decompounder defines where a word splits. The synonym list defines what counts as the same concept. The match percentage defines how much we can ignore. The merchandising rule overrides the engine when it fails.


Each of these tools is load-bearing and necessary — a German store without a decompounder or a Czech store without root-shifting rules is simply broken. But together, they represent a collection of separate approximations of a single missing capability, rebuilt over and over for every language, catalog, and slang shift.



Full-text search isn’t the wrong engine. It’s just an incomplete one, and we are smuggling in the missing pieces one manual dictionary at a time.


In Part 2, we’ll look at the obvious next step. If every one of these patches is trying to approximate knowledge that a model could simply learn, can we replace the whole stack? We’ll explore whether neural representations and subword tokenizers can replace the hand-built stemmers and decompounders we’ve detailed here.



Next in this series - Part 2 asks the obvious question: if every patch here is standing in for knowledge a model could just learn, can neural search replace the whole stack? Subscribe to get notified when it goes live, or if you're weighing how much lexical tuning your own catalog really needs, get in touch with our team.



Sources




 
 
bottom of page