Check the installed packages against the lockfiles
A clean lockfile does not mean a clean workstation. pip-audit reported vulnerabilities here while requirements.txt was already pinned to fixed versions - the installed environment had drifted from the pins, and nothing compared the two. The same drift had left requirements-dev.txt uninstalled, so pytest.ini's -n auto failed for want of pytest-xdist. Add scripts/check-dependency-drift.py, comparing installed distributions against both lockfiles. MISSING is reported separately from MISMATCHED: a package that was never installed and a package at the wrong version need different fixes, and folding them together hid the missing dev requirements. Wire it into the style gate advisorially, with DEPENDENCY_DRIFT_ENFORCE=1 for CI. A developer mid-upgrade should not be blocked by a gate that is telling them something they already know. The gate now probes python, python3 and py by executing them rather than by looking them up on PATH. On Windows, python3 resolves to the Microsoft Store alias stub, which is present on PATH and does nothing when run. CONTRIBUTING.md gains the Windows-specific traps this exposed: npm.cmd rather than npm under a Restricted execution policy, the explicit Git Bash path because bare bash finds the WSL shim, and npm test rather than npx vitest run because only the pretest hook stages plugin frontends.
This commit is contained in:
benjamin.baker
@@ -206,6 +206,74 @@ Tests are required for:
|
||||
- New plugin hooks (contract test)
|
||||
- Schema migrations (before/after data integrity)
|
||||
|
||||
### Running the gates on Windows
|
||||
|
||||
Three commands in this repository resolve to the wrong program on a Windows dev
|
||||
box. Each one fails in a way that looks like a code fault, which is why all
|
||||
three are written out here rather than left to be rediscovered.
|
||||
|
||||
```bash
|
||||
& "C:\Program Files\Git\bin\bash.exe" scripts/check-naming-and-style.sh
|
||||
cd frontend; npm.cmd test; npm.cmd run build; cd ..
|
||||
python -m pytest tests/ -q
|
||||
```
|
||||
|
||||
- **`bash` resolves to WSL.** `C:\Windows\System32\bash.exe` is on PATH by
|
||||
default. With no distribution installed it fails with
|
||||
`execvpe(/bin/bash) failed: No such file or directory`, and with one installed
|
||||
it cannot read the Windows path you handed it. Always give the Git Bash path
|
||||
in full. The same trap bit the test suite: `tests/test_naming_gate_migrations.py`
|
||||
used bare `bash`, and its 13 failures were miscategorised as "environmental"
|
||||
for weeks. It now resolves Git Bash explicitly and probes it before use.
|
||||
- **`npm` resolves to `npm.ps1`.** PowerShell prefers the `.ps1` over the
|
||||
`.cmd`, and the default execution policy is `Restricted`, so bare `npm` dies
|
||||
with "npm.ps1 cannot be loaded because running scripts is disabled on this
|
||||
system" before any test runs. Use `npm.cmd`. Nothing in the repo causes this
|
||||
and nothing in the repo can fix it; it is machine policy. If you would rather
|
||||
fix it once, `Set-ExecutionPolicy -Scope CurrentUser RemoteSigned` also works,
|
||||
but that is a machine-level change and `npm.cmd` is free.
|
||||
- **`npm test`, not `npx vitest run`.** The `pretest` hook runs
|
||||
`stage-frontend.mjs`; bare vitest never stages plugin frontends and silently
|
||||
skips their specs.
|
||||
|
||||
### Keeping the environment matched to the lockfiles
|
||||
|
||||
```bash
|
||||
python scripts/check-dependency-drift.py
|
||||
```
|
||||
|
||||
Editing a lockfile changes what a FUTURE install produces. It does not touch an
|
||||
already-populated `site-packages`. Commit `a8f8580` raised eight pins carrying
|
||||
advisories; no install was re-run afterwards, so that workstation kept the old
|
||||
libraries for three weeks while `requirements.txt` said otherwise. Every local
|
||||
test run in that window exercised different libraries than a container build
|
||||
would have.
|
||||
|
||||
Run this after any pull that touches `requirements.txt` or
|
||||
`requirements-dev.txt`, and fix drift with:
|
||||
|
||||
```bash
|
||||
python -m pip install --require-hashes -r requirements.txt
|
||||
python -m pip install --require-hashes -r requirements-dev.txt
|
||||
```
|
||||
|
||||
The check reports MISSING separately from MISMATCHED, because absent packages
|
||||
are the class that gets missed: `pytest-xdist` was simply not installed, which
|
||||
surfaced only as `pytest: error: unrecognized arguments: -n` - a message that
|
||||
reads as a broken `pytest.ini` and invites deleting `-n auto` instead of
|
||||
installing the plugin.
|
||||
|
||||
`scripts/check-naming-and-style.sh` runs this check too, advisory by default.
|
||||
CI sets `DEPENDENCY_DRIFT_ENFORCE=1` to make it fail the build.
|
||||
|
||||
If you use the repo-local `env/` virtualenv, check it specifically - it drifts
|
||||
independently of the global interpreter, and both were stale in the incident
|
||||
above:
|
||||
|
||||
```bash
|
||||
env/Scripts/python.exe scripts/check-dependency-drift.py
|
||||
```
|
||||
|
||||
## Branches: what belongs on main, and what does not
|
||||
|
||||
`main` is the product every site runs and the publication mirror exports.
|
||||
|
||||
137
scripts/check-dependency-drift.py
Normal file
137
scripts/check-dependency-drift.py
Normal file
@@ -0,0 +1,137 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Compare installed package versions against the pinned lockfiles.
|
||||
|
||||
Editing a lockfile changes what a FUTURE install produces. It does not touch an
|
||||
already-populated site-packages. On 2026-08-29 commit a8f8580 raised eight pins
|
||||
carrying advisories; no install was re-run, so this workstation kept running
|
||||
pyjwt 2.12.1 and idna 3.13 for three weeks while requirements.txt said 2.13.0
|
||||
and 3.19. Every local test run in that window exercised different libraries than
|
||||
a container build would have, and a vulnerability scan of the environment
|
||||
disagreed with a scan of the lockfile.
|
||||
|
||||
MISSING is reported separately from MISMATCHED on purpose. The first pass at
|
||||
this incident only compared versions of packages that were installed, which
|
||||
silently skipped pytest-xdist and execnet - absent entirely. That surfaced later
|
||||
as "pytest: error: unrecognized arguments: -n", which reads as a broken
|
||||
pytest.ini rather than a stale environment, and invites deleting "-n auto"
|
||||
instead of installing the plugin.
|
||||
|
||||
Exit codes:
|
||||
0 environment matches the lockfiles
|
||||
1 drift found
|
||||
2 could not run the comparison (bad arguments, unreadable lockfile)
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import importlib.metadata
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
PIN_PATTERN = re.compile(r'^([A-Za-z0-9._-]+)==([^\s\\;]+)')
|
||||
|
||||
DEFAULT_LOCKFILES = ('requirements.txt', 'requirements-dev.txt')
|
||||
|
||||
|
||||
def normalize(name):
|
||||
"""PEP 503 normalization, so Pillow/pillow and pytest_cov/pytest-cov agree."""
|
||||
return re.sub(r'[-_.]+', '-', name).lower()
|
||||
|
||||
|
||||
def read_pins(lockfile):
|
||||
"""Map normalized package name to pinned version for one lockfile.
|
||||
|
||||
uv pip compile emits hashes on continuation lines and comments elsewhere;
|
||||
only lines that START with name==version are pins, which is what the regex
|
||||
anchors on.
|
||||
"""
|
||||
pins = {}
|
||||
for line in lockfile.read_text(encoding='utf-8').splitlines():
|
||||
match = PIN_PATTERN.match(line)
|
||||
if match:
|
||||
pins[normalize(match.group(1))] = match.group(2)
|
||||
return pins
|
||||
|
||||
|
||||
def compare(pins):
|
||||
"""Split pinned packages into missing, mismatched, and satisfied."""
|
||||
missing, mismatched, satisfied = [], [], 0
|
||||
for name, wanted in sorted(pins.items()):
|
||||
try:
|
||||
installed = importlib.metadata.version(name)
|
||||
except importlib.metadata.PackageNotFoundError:
|
||||
missing.append((name, wanted))
|
||||
continue
|
||||
if normalize_version(installed) != normalize_version(wanted):
|
||||
mismatched.append((name, installed, wanted))
|
||||
else:
|
||||
satisfied += 1
|
||||
return missing, mismatched, satisfied
|
||||
|
||||
|
||||
def normalize_version(version):
|
||||
"""Trim a trailing .0 series so 1.2.0 and 1.2 compare equal."""
|
||||
trimmed = version.strip()
|
||||
while trimmed.endswith('.0'):
|
||||
trimmed = trimmed[:-2]
|
||||
return trimmed
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description='Check installed packages against the pinned lockfiles.')
|
||||
parser.add_argument('lockfiles', nargs='*', default=list(DEFAULT_LOCKFILES),
|
||||
help='lockfiles to check (default: %s)'
|
||||
% ' '.join(DEFAULT_LOCKFILES))
|
||||
parser.add_argument('--quiet', action='store_true',
|
||||
help='print only drift, not the clean summary')
|
||||
arguments = parser.parse_args()
|
||||
|
||||
repo_root = Path(__file__).resolve().parent.parent
|
||||
total_missing = total_mismatched = 0
|
||||
report_lines = []
|
||||
|
||||
for name in arguments.lockfiles:
|
||||
lockfile = Path(name)
|
||||
if not lockfile.is_absolute():
|
||||
lockfile = repo_root / lockfile
|
||||
if not lockfile.is_file():
|
||||
print('ERROR: lockfile not found: %s' % lockfile, file=sys.stderr)
|
||||
return 2
|
||||
|
||||
pins = read_pins(lockfile)
|
||||
if not pins:
|
||||
print('ERROR: no pins parsed from %s' % lockfile, file=sys.stderr)
|
||||
return 2
|
||||
|
||||
missing, mismatched, satisfied = compare(pins)
|
||||
total_missing += len(missing)
|
||||
total_mismatched += len(mismatched)
|
||||
|
||||
for package, wanted in missing:
|
||||
report_lines.append(' MISSING %s (pinned %s) [%s]'
|
||||
% (package, wanted, lockfile.name))
|
||||
for package, installed, wanted in mismatched:
|
||||
report_lines.append(' MISMATCH %s installed %s, pinned %s [%s]'
|
||||
% (package, installed, wanted, lockfile.name))
|
||||
|
||||
if not arguments.quiet and not missing and not mismatched:
|
||||
print('OK: %s - %d pins satisfied' % (lockfile.name, satisfied))
|
||||
|
||||
if report_lines:
|
||||
print('FAIL: dependency drift - %d missing, %d mismatched:'
|
||||
% (total_missing, total_mismatched))
|
||||
for line in report_lines:
|
||||
print(line)
|
||||
print('')
|
||||
print('Interpreter: %s' % sys.executable)
|
||||
print('Fix with:')
|
||||
for name in arguments.lockfiles:
|
||||
print(' python -m pip install --require-hashes -r %s' % name)
|
||||
return 1
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
@@ -608,6 +608,67 @@ if [ -n "$SITE_HITS" ]; then
|
||||
fi
|
||||
fi
|
||||
|
||||
# --- Dependency drift -------------------------------------------------------
|
||||
#
|
||||
# Not a naming rule, but it belongs to the same question: can this run be
|
||||
# trusted. A stale environment makes every OTHER check here a statement about
|
||||
# libraries that are not the ones the lockfile specifies.
|
||||
#
|
||||
# ADVISORY BY DEFAULT, and deliberately so. Set DEPENDENCY_DRIFT_ENFORCE=1 to
|
||||
# fail the gate (CI should). Two considerations pull against each other here:
|
||||
# a warning nobody reads is worthless, but a blocking check wedges anyone who
|
||||
# is mid-upgrade and legitimately holds an unpinned version for an afternoon.
|
||||
# The tie-breaker is the comment above the site-literal scan: this file has
|
||||
# twice acquired a reasonable-looking flag that silently disabled a check, once
|
||||
# for months with the gate green. A check that ANNOUNCES it is advisory cannot
|
||||
# become that, because its output changes the moment drift appears.
|
||||
#
|
||||
# Skipped when python is absent rather than failing: this script is also run on
|
||||
# machines that only build the frontend.
|
||||
#
|
||||
# Each candidate is TEST-EXECUTED, not merely located with command -v. On
|
||||
# Windows, "python3" resolves to the Microsoft Store app-execution alias - a
|
||||
# zero-byte stub that exists on PATH, satisfies command -v, and then prints
|
||||
# "Python was not found; run without arguments to install from the Microsoft
|
||||
# Store" to stderr and exits non-zero. Locating it and trusting it reported the
|
||||
# drift check as failing on a machine whose environment was in fact clean.
|
||||
DEPENDENCY_DRIFT_ENFORCE=${DEPENDENCY_DRIFT_ENFORCE:-0}
|
||||
echo "==> Checking installed packages against the lockfiles..."
|
||||
DRIFT_PYTHON=""
|
||||
for CANDIDATE in python python3 py; do
|
||||
if command -v "$CANDIDATE" >/dev/null 2>&1 \
|
||||
&& "$CANDIDATE" -c 'import sys' >/dev/null 2>&1; then
|
||||
DRIFT_PYTHON="$CANDIDATE"
|
||||
break
|
||||
fi
|
||||
done
|
||||
# The checker and the lockfiles must both be present. They are not when the gate
|
||||
# is copied alone into a throwaway tree, which is exactly what
|
||||
# tests/test_naming_gate_migrations.py does to exercise the non-git fallback
|
||||
# branch. Absent inputs mean "nothing to compare", not "drift".
|
||||
if [ ! -f scripts/check-dependency-drift.py ] || [ ! -f requirements.txt ]; then
|
||||
DRIFT_PYTHON=""
|
||||
fi
|
||||
if [ -z "$DRIFT_PYTHON" ]; then
|
||||
echo " python not found, skipping dependency drift check"
|
||||
echo
|
||||
else
|
||||
DRIFT_OUTPUT=$("$DRIFT_PYTHON" scripts/check-dependency-drift.py --quiet 2>&1) || DRIFT_STATUS=$?
|
||||
DRIFT_STATUS=${DRIFT_STATUS:-0}
|
||||
if [ "$DRIFT_STATUS" -ne 0 ]; then
|
||||
if [ "$DEPENDENCY_DRIFT_ENFORCE" = "1" ]; then
|
||||
echo "$DRIFT_OUTPUT"
|
||||
echo
|
||||
VIOLATIONS=$((VIOLATIONS + 1))
|
||||
else
|
||||
echo " dependency drift found (report only, not failing):"
|
||||
echo "$DRIFT_OUTPUT" | sed 's/^/ /'
|
||||
echo " Set DEPENDENCY_DRIFT_ENFORCE=1 to make this fail the gate."
|
||||
echo
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ "$VIOLATIONS" -gt 0 ]; then
|
||||
echo "=================================================="
|
||||
echo "$VIOLATIONS naming/style violation(s) found."
|
||||
|
||||
Reference in New Issue
Block a user