Record what nobody can find, and a spelling corrector that refuses to guess

Two halves of one question - how to handle a misspelled search - deliberately
kept separate, because only one of them should be turned on today.

WHAT NOBODY CAN FIND IS NOT RECORDED ANYWHERE. A search returning zero results
is the only evidence of the gap between what people look for and what is there,
and it vanished. Logged now at INFO with a stable prefix, so a week of it greps
into a list. That list is what should decide whether correction is worth wiring
in: multi-word matching and cross-field matching both changed in the last day,
so a good share of what used to fail may already be found. The failures left
over might be typos, or vocabulary nobody has entered, or records that genuinely
do not exist - and each wants a different answer.

THE CORRECTOR IS BUILT AND NOT WIRED IN. shopdb/core/services/spellfix.py, with
tests, ready to attach in about ten lines once there is evidence about what to
attach it to.

NOT SOUNDEX, which was the obvious candidate. MySQL has it and SQLite does not,
and the suite runs on SQLite while production runs MySQL - so a SOUNDEX() in a
query is either an error in every test or a production path no test executes.
It is also wrong for this data: soundex is English-name phonetics, four
characters wide, and it DISCARDS DIGITS, so CSF16 and CSF17 hash identically.
Half of what people search here is an identifier.

So: character distance in Python, same behaviour on both dialects.

THE VOCABULARY IS THE DATA. No dictionary holds Genspect, Telesis, Keyence or
wax-trace. Terms come from the columns being searched, which also means a vendor
added this morning is correctable this morning.

TWO GUARDS, and they are the point rather than a detail. Digits must match
EXACTLY: CSF16 to CSF17 is one edit and a different bay, so anything carrying
digits is either right or not correctable - while cfs16 to csf16 still works,
because the guard is on the digits and not on identifiers wholesale. And the
first character must match, since typos land mid-word far more often than on the
first key, which costs almost no recall.

Distance is Damerau-Levenshtein so a transposition costs one edit rather than
two - Keyecne for Keyence is the commonest error there is, and plain Levenshtein
pushes it past the threshold on short words. Allowed distance scales with
length. A tie returns NOTHING: two equally good candidates means there is no
answer, and offering either implies a confidence that is not there.

It suggests; it never rewrites. Silently searching for something else is how
somebody orders the wrong cartridge.

13 tests, weighted toward the refusals, because those are the cases where being
wrong costs something.

tests/test_core/test_spellfix.py

@@ -0,0 +1,109 @@
+"""Correcting a misspelled search word without wrecking an identifier.
+
+Pure arithmetic over a word list, so it is tested that way. The cases that
+matter most are the ones where it must REFUSE: a wrong suggestion is acted on,
+and here that means the wrong bay or the wrong cartridge.
+"""
+
+from shopdb.core.services.spellfix import (
+    Vocabulary, damerau_levenshtein, max_distance,
+)
+
+SHOPFLOOR = [
+    'Keyence vision system', 'Genspect', 'Telesis part marker',
+    'wax-trace', 'Zebra ZT411', 'calibration', 'Haas Automation',
+    'CSF16', 'CSF17', 'MT-1042', 'FC733SY3', 'printer', 'grinder',
+]
+
+
+def vocab():
+    return Vocabulary(SHOPFLOOR)
+
+
+# ------------------------------------------------------------------ distance
+
+def test_a_transposition_costs_one_edit_not_two():
+    """The commonest typing error. Plain Levenshtein charges two, which pushes
+    a real typo past the threshold on a short word."""
+    assert damerau_levenshtein('keyecne', 'keyence') == 1
+
+
+def test_distance_allowed_scales_with_length():
+    assert max_distance('bay') == 1
+    assert max_distance('grinder') == 2
+    assert max_distance('calibration') == 3
+
+
+# ------------------------------------------------------------------ suggests
+
+def test_it_corrects_vocabulary_no_dictionary_would_hold():
+    assert vocab().suggest('keyecne') == 'keyence'
+    assert vocab().suggest('genspct') == 'genspect'
+    assert vocab().suggest('telisis') == 'telesis'
+
+
+def test_a_word_that_is_already_right_is_left_alone():
+    assert vocab().suggest('keyence') is None
+    assert vocab().suggest('printer') is None
+
+
+def test_a_word_nothing_resembles_gets_no_guess():
+    """A wrong suggestion costs more than no suggestion."""
+    assert vocab().suggest('helicopter') is None
+
+
+# ------------------------------------------------------- identifier guards
+
+def test_an_asset_number_is_never_corrected_to_a_different_one():
+    """THE case this exists for. CSF16 to CSF17 is one edit and a different bay.
+
+    Both are in the vocabulary, so without the digit guard the near miss is
+    exactly the kind of thing edit distance loves to 'fix'.
+    """
+    assert vocab().suggest('csf18') is None
+    assert vocab().suggest('mt-1043') is None
+
+
+def test_a_typo_in_the_letters_of_an_identifier_still_corrects():
+    """The guard is on the DIGITS, not on identifiers as a whole: the digits
+    have to match, and here they do."""
+    assert vocab().suggest('cfs16') == 'csf16'
+
+
+def test_the_first_character_must_match():
+    """Typos land in the middle and at the end. Requiring the first key costs
+    little recall and removes most of the nonsense."""
+    assert vocab().suggest('teyence') is None
+
+
+def test_a_tie_produces_no_suggestion():
+    """Two equally good answers means there is no answer. CSF16 and CSF17 are
+    both one edit from CSF1x, and offering either implies a confidence that is
+    not there - but the digit guard rejects both first, so build a case where
+    only the tie is in play."""
+    tied = Vocabulary(['barn', 'bard'])
+    assert tied.suggest('bare') is None
+
+
+# ---------------------------------------------------------------- queries
+
+def test_a_multi_word_query_corrects_only_what_is_wrong():
+    assert vocab().suggest_query('keyecne calibration') == 'keyence calibration'
+
+
+def test_a_query_that_is_entirely_fine_returns_none():
+    assert vocab().suggest_query('keyence calibration') is None
+
+
+def test_short_words_are_left_alone():
+    """Below three characters there is not enough word to correct."""
+    assert vocab().suggest('cs') is None
+    assert vocab().suggest_query('cs') is None
+
+
+def test_the_vocabulary_comes_from_the_data():
+    """A vendor added this morning is correctable this morning."""
+    fresh = vocab()
+    assert fresh.suggest('renishw') is None
+    fresh.add('Renishaw')
+    assert fresh.suggest('renishw') == 'renishaw'