Make the legacy import safe to resume, and aimable without editing it

Three defects broke the loader's own documented resume guarantee. All three
were silent: nothing raised, the run finished cheerfully, and fewer rows
arrived than anyone counted.

THE PAGE SIZE WAS CLAMPED, SO THE 409-RESOLVE SEARCHED THE FIRST 100 ROWS

_upsert asked for ?per_page=10000 and read one response. The server clamps any
page size to MAX_PAGE_SIZE, which is 100 - not an error, not a warning, just
100 rows. Anything past that resolved to None, so the id never reached the
crosswalk and the run quietly dropped a model's link on every machine that used
it, or an application's installs.

The target already holds more than 100 models and more than 100 applications,
so this fired on any re-run or resume - exactly the case the loader documents
as safe.

_resolve_existing now tries the exact natural-key filter the API has exposed
all along (?modelnumber= on models, ?appname= on applications, ?vendor= on
vendors) and then pages properly, bounded. An endpoint without such a filter
ignores the unknown parameter and the paged walk finds the row.

stage_metrology repeated the same mistake in its own way: it skipped 409s
entirely, so a resumed run left the tool out of the crosswalk and the Controls
edge unmade - metrology PCs driving nothing. stage_printers already had the
resolve branch; that stage was written later and missed it.

THE CROSSWALK WAS NOT BOUND TO THE DATABASE IT DESCRIBED

idmap.json maps this site's legacy ids to assetids in ONE database, and those
assetids mean nothing anywhere else. Keeping it across a drop-and-rebuild left
warranties and printer assignments attaching to whatever asset now holds that
id: wrong data, no error. The runbook's two "drop and rebuild" steps never
mentioned the file.

The loader now writes a uuid marker into the TARGET's own settings table, so it
dies with the data, and stamps it into the crosswalk. A crosswalk built against
a different database is refused before a row is written. The database name
alone would not do - a rebuild keeps the name. A file written before this guard
is adopted rather than refused, so no existing crosswalk is stranded.

Also adds --idmap, so one checkout can drive more than one target.

THE SOURCE DATABASES ARE NAMED, NOT HARDCODED

'shopdb_src' appeared at 29 call sites, so aiming the loader at the prodscratch
restore meant editing run.py. Those names are now LOGICAL and resolved in
Source.db() from SOURCE_DB_MAIN / SOURCE_DB_USB / SOURCE_DB_EMPLOYEES, with
host, port, user and password overridable too. The call sites are untouched -
resolving centrally avoids missing one, and the literals now read as
documentation of which source a stage draws from. The run prints every source
it redirected, so "it imported nothing" can be told apart from "it read the
wrong database".

This is what stood between the loader and the prodscratch run.

TWO SMALLER ONES

_source_password() raised IndexError rather than its intended message whenever
the DATABASE_URL user is not root, which it will not be on a real server.

main() returned 1 on failure into a caller that discarded it. A run that logged
200 endpoint errors exited 0 and read as a success in a deploy log; it now
exits non-zero.

TESTS AND DOCS

The loader had no tests at all. Sixteen now cover the resolve, the crosswalk
binding and the source aliasing - the logic that can run without a MySQL
source, which is the part that was wrong. Verified they are not vacuous by
running the old algorithm against the same fixture: it returns None where the
new one returns the row.

The README claimed 15 stages "complete" with counts from 2026-07-13 that two
later routing changes invalidated, and the runbook told operators to verify a
fresh import against them. Both corrected, with the three gaps NOT fixed here
recorded plainly: warranties, notifications and knowledgebase POST without a
duplicate check so resuming across them doubles rows; there is no --dry-run or
target guard; and the verify stage counts target assets against source machines,
which is apples to oranges.

Full suite 2178 passed.

tests/test_site_import_loader.py

@@ -0,0 +1,164 @@
+"""The WJ reference loader's resolve, crosswalk binding and source aliasing.
+
+The loader had no tests at all, and three of its defects were the silent kind:
+they did not raise, they imported fewer rows and finished cheerfully. These
+cover the logic that can be exercised without a MySQL source, which is exactly
+the part that was wrong.
+
+Not covered here: the stage functions themselves. They need the classic source
+schema, so they belong in a fixture-backed integration test against a scratch
+database, not in the unit suite.
+"""
+import json
+import os
+
+import pytest
+
+from scripts.site_imports.wjf import run as loader
+from scripts.site_imports.wjf.harness import IdMap, Source, StaleIdMapError
+
+
+class FakeHarness:
+    """Records the paths requested and replays canned list responses."""
+
+    def __init__(self, pages):
+        self.pages = pages
+        self.requested = []
+
+    def get(self, path):
+        self.requested.append(path)
+        return 200, self.pages.get(path, {'items': []})
+
+
+class TestResolveExisting:
+    def test_finds_a_row_beyond_the_first_page(self):
+        """The bug: per_page is clamped to 100, so row 101 was invisible.
+
+        The old code asked for per_page=10000, read one response, and returned
+        None for anything the server did not fit in its first 100 rows. The
+        target already holds more than 100 models and more than 100
+        applications, so this fired on every resume.
+        """
+        page1 = {'items': [{'modelnumber': 'M%03d' % i, 'modelnumberid': i}
+                           for i in range(100)]}
+        page2 = {'items': [{'modelnumber': 'LATE', 'modelnumberid': 999}]}
+        harness = FakeHarness({
+            '/api/models?modelnumber=LATE&per_page=100': {'items': []},
+            '/api/models?per_page=100&page=1': page1,
+            '/api/models?per_page=100&page=2': page2,
+        })
+        found = loader._resolve_existing(
+            harness, '/api/models', 'modelnumber', 'LATE', 'modelnumberid')
+        assert found == 999
+
+    def test_tries_the_exact_filter_first(self):
+        harness = FakeHarness({
+            '/api/models?modelnumber=WIDGET&per_page=100':
+                {'items': [{'modelnumber': 'WIDGET', 'modelnumberid': 7}]},
+        })
+        assert loader._resolve_existing(
+            harness, '/api/models', 'modelnumber', 'WIDGET', 'modelnumberid') == 7
+        # One request: it must not walk every page when the filter answered.
+        assert len(harness.requested) == 1
+
+    def test_values_needing_encoding_do_not_break_the_filter(self):
+        harness = FakeHarness({})
+        loader._resolve_existing(harness, '/api/vendors', 'vendor',
+                                 'Acme & Co/Ltd', 'vendorid')
+        assert 'Acme%20%26%20Co/Ltd' in harness.requested[0] or \
+               'Acme%20%26%20Co%2FLtd' in harness.requested[0]
+
+    def test_matching_ignores_case_and_padding(self):
+        harness = FakeHarness({
+            '/api/vendors?vendor=acme&per_page=100':
+                {'items': [{'vendor': '  ACME ', 'vendorid': 3}]},
+        })
+        assert loader._resolve_existing(
+            harness, '/api/vendors', 'vendor', 'acme', 'vendorid') == 3
+
+    def test_missing_row_returns_none_without_spinning(self):
+        harness = FakeHarness({})
+        assert loader._resolve_existing(
+            harness, '/api/models', 'modelnumber', 'NOPE', 'modelnumberid') is None
+        # Empty first page ends it: the filter attempt plus one page.
+        assert len(harness.requested) == 2
+
+    def test_a_short_page_ends_the_walk(self):
+        harness = FakeHarness({
+            '/api/models?per_page=100&page=1': {'items': [{'modelnumber': 'A', 'modelnumberid': 1}]},
+        })
+        assert loader._resolve_existing(
+            harness, '/api/models', 'modelnumber', 'ZZZ', 'modelnumberid') is None
+        assert len(harness.requested) == 2
+
+    def test_a_bare_list_response_is_accepted(self):
+        harness = FakeHarness({
+            '/api/x?name=a&per_page=100': [{'name': 'a', 'id': 5}],
+        })
+        assert loader._resolve_existing(harness, '/api/x', 'name', 'a', 'id') == 5
+
+    def test_an_existing_query_string_keeps_its_separator(self):
+        harness = FakeHarness({})
+        loader._resolve_existing(harness, '/api/models?category=pc',
+                                 'modelnumber', 'M1', 'modelnumberid')
+        assert harness.requested[0].startswith('/api/models?category=pc&modelnumber=')
+
+
+class TestIdMapBinding:
+    def test_a_crosswalk_from_another_database_is_refused(self, tmp_path):
+        """The silent corruption this guard exists for.
+
+        Keeping idmap.json across a drop-and-rebuild leaves legacy ids pointing
+        at assetids from the OLD database, so warranties and printer
+        assignments attach to whatever asset holds that id now. No error, wrong
+        data.
+        """
+        path = tmp_path / 'idmap.json'
+        path.write_text(json.dumps({'_target': 'aaaa', 'asset': {'1': 10}}))
+        with pytest.raises(StaleIdMapError) as caught:
+            IdMap(str(path), target='bbbb')
+        assert 'different database' in str(caught.value)
+
+    def test_the_matching_target_loads(self, tmp_path):
+        path = tmp_path / 'idmap.json'
+        path.write_text(json.dumps({'_target': 'aaaa', 'asset': {'1': 10}}))
+        ids = IdMap(str(path), target='aaaa')
+        assert ids.get('asset', 1) == 10
+
+    def test_an_unstamped_crosswalk_is_adopted_not_refused(self, tmp_path):
+        # Written before the guard existed. Refusing would strand a crosswalk
+        # that is probably fine; adopting binds it for every run after.
+        path = tmp_path / 'idmap.json'
+        path.write_text(json.dumps({'asset': {'1': 10}}))
+        ids = IdMap(str(path), target='ccc')
+        assert ids.get('asset', 1) == 10
+        assert json.loads(path.read_text())['_target'] == 'ccc'
+
+    def test_the_bookkeeping_key_is_not_counted_as_a_namespace(self, tmp_path):
+        path = tmp_path / 'idmap.json'
+        ids = IdMap(str(path), target='ccc')
+        ids.put('asset', 1, 10)
+        assert ids.count('asset') == 1
+        assert ids.count('_target') == 0
+
+    def test_a_fresh_crosswalk_needs_no_stamp_to_start(self, tmp_path):
+        ids = IdMap(str(tmp_path / 'new.json'), target='ddd')
+        ids.put('asset', 5, 50)
+        assert ids.get('asset', 5) == 50
+
+
+class TestSourceAliasing:
+    def test_logical_names_default_to_the_scratch_databases(self, monkeypatch):
+        for var in ('SOURCE_DB_MAIN', 'SOURCE_DB_USB', 'SOURCE_DB_EMPLOYEES'):
+            monkeypatch.delenv(var, raising=False)
+        assert Source.resolve('shopdb_src') == 'shopdb_src'
+        assert Source.resolve('cmmc_usb_src') == 'cmmc_usb_src'
+        assert Source.resolve('wjf_employees_src') == 'wjf_employees_src'
+
+    def test_the_environment_repoints_the_loader(self, monkeypatch):
+        """What unblocks the prodscratch run without editing run.py."""
+        monkeypatch.setenv('SOURCE_DB_MAIN', 'prodscratch')
+        assert Source.resolve('shopdb_src') == 'prodscratch'
+
+    def test_an_unknown_name_passes_through(self, monkeypatch):
+        assert Source.resolve('something_else') == 'something_else'