Prove the gates ran, and ship the spec the app serves
Two faults with one cause: docs/ is excluded from publication because the prose
becomes the wiki, and two things under docs/ are not prose.
shopdb/core/api/docs.py reads docs/openapi.json AT RUNTIME to serve /api/docs.
Excluding it meant every site installed from the published repo shipped the
ReDoc page, loaded it, and got {"error":"openapi.json not generated"}. The spec
now publishes - it is a product file that happens to live under docs/. The prose
still goes to the wiki and nowhere else, and docs/api-inventory.json stays out,
because nothing reads it at runtime.
The second fault is the reason this was worth chasing. Three gate modules skip
themselves when the file they read is absent, which is correct in a published
checkout and silently wrong on GitHub, which runs against exactly that checkout.
The modules skipped, CI reported green, and an outside branch added nine
undocumented routes and passed. A gate that stops running is worse than no gate,
because nobody looks at it again.
scripts/check-gates-ran.sh asserts each gate ran, with two lists: the ones that
must run everywhere, and the ones needing files publication deliberately leaves
behind - required here, allowed to be absent there. A `gates` job runs it in CI.
Proven in both directions: green in each checkout, and red when the file a gate
reads is removed.
Also purge instance/plugins.json from the publication tree. It was never
published - it is untracked there - but an old copy had sat in that tree for
months, protected from --delete by its own exclude, and it made the published
repo fail its own suite locally: 13 plugins enabled where the code expects 16.
A publication tree that cannot pass its own tests cannot be verified before
shipping. That is the exact trap the comment at the top of the script warns
about, which had claimed a second victim.
Also: concurrency on the workflow, so a second push supersedes the first rather
than queueing behind it holding a self-hosted runner.
This commit is contained in:
101
scripts/check-gates-ran.sh
Executable file
101
scripts/check-gates-ran.sh
Executable file
@@ -0,0 +1,101 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Prove the gates actually RAN.
|
||||
#
|
||||
# Several checks skip themselves when the file they read is absent:
|
||||
#
|
||||
# tests/test_api_inventory_parity.py skipif docs/api-inventory.json missing
|
||||
# tests/test_openapi_spec.py skipif docs/openapi.json missing
|
||||
# tests/test_docs_publishable.py skipif docs/ missing
|
||||
#
|
||||
# That is correct in a published checkout, where docs/ becomes the wiki and
|
||||
# there is genuinely nothing to check. It was silently wrong on GitHub, which
|
||||
# runs against exactly that checkout: the modules skipped, CI reported green,
|
||||
# and an outside branch added nine routes documented nowhere and passed.
|
||||
#
|
||||
# So the list is split. Some gates must run everywhere; the docs-dependent ones
|
||||
# must run HERE, where the files exist, and are allowed to be absent there.
|
||||
#
|
||||
# A gate that stops running is worse than no gate: nobody looks at it again.
|
||||
# This asserts each named module reported at least one PASS and no SKIPS.
|
||||
#
|
||||
# Usage:
|
||||
# bash scripts/check-gates-ran.sh # runs the gates itself
|
||||
# bash scripts/check-gates-ran.sh report.txt # reads an existing -rs report
|
||||
set -uo pipefail
|
||||
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
# Gates that must run in EVERY checkout, published or internal. A skip here is
|
||||
# always a fault.
|
||||
GATES_EVERYWHERE=(
|
||||
"tests/test_openapi_spec.py"
|
||||
"tests/test_core/test_public_read_surface.py"
|
||||
"tests/test_core/test_authz.py"
|
||||
"tests/test_plugin_migrations.py"
|
||||
)
|
||||
|
||||
# Gates that need files the publication export deliberately leaves behind -
|
||||
# docs/ becomes the wiki, so a published checkout genuinely has nothing to check
|
||||
# and skipping is correct there. In THIS repo, skipping means something broke.
|
||||
GATES_INTERNAL_ONLY=(
|
||||
"tests/test_api_inventory_parity.py"
|
||||
"tests/test_docs_publishable.py"
|
||||
)
|
||||
|
||||
# A published checkout is one with no docs/ prose. openapi.json alone does not
|
||||
# make it internal: that file ships because the app serves it at runtime.
|
||||
if compgen -G "docs/*.md" > /dev/null; then
|
||||
GATES=("${GATES_EVERYWHERE[@]}" "${GATES_INTERNAL_ONLY[@]}")
|
||||
echo "==> Internal checkout: every gate must run"
|
||||
else
|
||||
GATES=("${GATES_EVERYWHERE[@]}")
|
||||
echo "==> Published checkout: docs-dependent gates are expected to be absent"
|
||||
fi
|
||||
|
||||
PYTHON="${PYTHON:-python}"
|
||||
[ -x venv/bin/python ] && PYTHON=venv/bin/python
|
||||
|
||||
report="${1:-}"
|
||||
if [ -z "$report" ]; then
|
||||
report="$(mktemp)"
|
||||
# -rs lists every skip with its reason, which is the thing being checked.
|
||||
"$PYTHON" -m pytest "${GATES[@]}" -q -rs > "$report" 2>&1 || true
|
||||
fi
|
||||
|
||||
echo "==> Checking that every gate ran"
|
||||
|
||||
failed=0
|
||||
for gate in "${GATES[@]}"; do
|
||||
if [ ! -f "$gate" ]; then
|
||||
echo " MISSING $gate - the file is not in this checkout"
|
||||
failed=1
|
||||
continue
|
||||
fi
|
||||
if grep -q "SKIPPED.*$(basename "$gate")" "$report"; then
|
||||
reason=$(grep "SKIPPED.*$(basename "$gate")" "$report" | head -1 | sed 's/.*-- //')
|
||||
echo " SKIPPED $gate"
|
||||
echo " reason: ${reason:-not given}"
|
||||
failed=1
|
||||
continue
|
||||
fi
|
||||
echo " ran $gate"
|
||||
done
|
||||
|
||||
if grep -qE "^[0-9]+ failed|error" "$report"; then
|
||||
echo
|
||||
echo "Gate tests FAILED (not merely skipped):"
|
||||
grep -E "^FAILED|^ERROR" "$report" | sed 's/^/ /'
|
||||
failed=1
|
||||
fi
|
||||
|
||||
echo
|
||||
if [ "$failed" -ne 0 ]; then
|
||||
echo "FAIL: a gate did not run. A skipped gate reports green and checks nothing."
|
||||
echo " If a docs-dependent gate skipped in THIS repo, the file it reads"
|
||||
echo " was not generated: run scripts/gen_openapi.py and"
|
||||
echo " scripts/gen_project_map.py. If one skipped in the PUBLISHED repo,"
|
||||
echo " the export dropped a file the gate needs - see tools/export-github.sh."
|
||||
exit 1
|
||||
fi
|
||||
echo "All gates ran."
|
||||
Reference in New Issue
Block a user