fa6b99bf91
Closes the three defects in the imported restore path, at Ben's request. Each was proven against a real stack before and after: two backup sets with different data and different instance files, then a restore of the OLDER one, which is the case that was broken. A restore no longer depends on a tab staying open. The backup sidecar has no Flask, so it loads the dump and leaves a sentinel for the api container to run the migrations. The only thing that ever acted on that sentinel was the poll loop of the modal that queued the restore, so closing the tab - or letting a token expire - during the minutes a load takes left the data at the old revision under new code, every query on a newer column failing, and nothing retrying. A watcher in the api container now claims and runs it. The endpoint no longer runs the upgrade at all; it only makes sure something is pending, which is also how a failed upgrade is retried. That takes Alembic out of the request path, where it could outrun the gunicorn worker timeout and be killed mid-chain, leaving a half-applied revision the next attempt trips over. The claim is a rename, because rename(2) is atomic: several workers and a browser poll can race the same sentinel and exactly one wins. A failed upgrade parks itself as .upgrade-failed rather than re-arming, since putting it back would rerun the same broken chain on every poll forever. Instance files come from the set being restored. The restic call asked for 'latest' regardless of which set was named, so restoring anything but the newest paired that set's DATABASE with the NEWEST snapshot's uploads and plugins.json - a mismatch nothing downstream could detect. Now scoped by the set's own tag, which backup.sh has always written. The tree is replaced rather than restored over, so files created after the snapshot are gone afterwards: a restore has to put the site back to the snapshot, not merge into it. Done as a staged swap rather than 'restic restore --delete --target /', because that aims a delete pass at the container root and the cost of getting its filter wrong is the whole filesystem. Nothing destructive happens before a copy is taken. The dump carries DROP TABLE per table with no transaction around it, and the instance tree is replaced wholesale, so a wrong set or a failure part-way through left nothing to go back to. restore.sh now writes a pre-restore-<stamp>/ set - dump, instance tarball and a README saying what it is and how to load it - and refuses to touch anything if that copy cannot be written. Two more found while proving the above, both of which stopped the overlay working at all: * Every script under scripts/backup/ was committed non-executable, while every other .sh in the repo is 755. The compose entrypoints are /opt/backup/watch.sh and backup.sh from a read-only bind mount, so the backup and watcher containers could not start from a clean checkout on any machine, and nothing could fix it at runtime. * Dockerfile.backup could not build without the corporate CA staged. A plain COPY of the gitignored cert fails outright; the trailing-glob form does too, because BuildKit fails a COPY whose pattern matches nothing - which is exactly the case being handled. Copying the directory and testing for the file inside it is the form that works both ways. Verified by building the image with no cert present. Suite 2600 passed / 9 skipped. The finalizer has unit tests covering the atomic claim, the parked failure and the endpoint no longer running the upgrade. Scope: product
54 lines
2.7 KiB
Docker
54 lines
2.7 KiB
Docker
# Backup sidecar image: mysql:8.4 (for a version-matched mysqldump) plus restic
|
|
# (for deduped, encrypted, destination-agnostic file snapshots) and a small
|
|
# Python for the manifest writer.
|
|
#
|
|
# WHY mysql:8.4 as the base: the dump tool must match the server major/minor so
|
|
# the SQL it emits restores cleanly onto the same engine. Reusing the exact
|
|
# image the db service runs removes version drift as a failure mode.
|
|
FROM mysql:8.4
|
|
|
|
# Track 1 local build unblock: trust the corporate proxy (Zscaler/GE) root CA so
|
|
# the build-time fetches below (Oracle yum, github.com, PyPI) succeed behind TLS
|
|
# interception. This is a local convenience only; the shipping build vendors
|
|
# these deps instead of fetching them (pinned restic + SHA256, vendored
|
|
# bzip2/python3/alembic or a base image that carries them) so the release image
|
|
# needs no outbound calls and no embedded cert. The cert is gitignored (*.pem)
|
|
# and must be staged at scripts/backup/zscaler-root-ca.pem before building.
|
|
#
|
|
# Copied via the directory rather than by name, because the cert is OPTIONAL: it
|
|
# is gitignored, so a plain `COPY scripts/backup/zscaler-root-ca.pem` fails the
|
|
# build outright and nobody outside a TLS-intercepting network - another site,
|
|
# or our own CI - could build this image at all. A trailing glob does not fix
|
|
# that either: BuildKit fails a COPY whose pattern matches nothing, which is
|
|
# precisely the case being handled. Copying a directory that always exists and
|
|
# testing for the file inside it is the form that works both ways.
|
|
COPY scripts/backup/ /tmp/buildctx/
|
|
RUN if [ -f /tmp/buildctx/zscaler-root-ca.pem ]; then \
|
|
echo "staging corporate root CA"; \
|
|
cp /tmp/buildctx/zscaler-root-ca.pem /etc/pki/ca-trust/source/anchors/; \
|
|
update-ca-trust; \
|
|
else \
|
|
echo "no corporate root CA staged - using the public trust store"; \
|
|
fi \
|
|
&& rm -rf /tmp/buildctx
|
|
|
|
# restic from the distro is often stale; pull the pinned release binary so the
|
|
# snapshot format and retention flags are predictable across sites.
|
|
ARG RESTIC_VERSION=0.17.3
|
|
RUN microdnf install -y bzip2 python3 python3-pip ca-certificates \
|
|
&& curl -fsSL -o /tmp/restic.bz2 \
|
|
"https://github.com/restic/restic/releases/download/v${RESTIC_VERSION}/restic_${RESTIC_VERSION}_linux_amd64.bz2" \
|
|
&& bunzip2 /tmp/restic.bz2 \
|
|
&& install -m 0755 /tmp/restic /usr/local/bin/restic \
|
|
&& rm -f /tmp/restic \
|
|
&& microdnf clean all
|
|
|
|
# alembic so the manifest writer can read the current schema head. pip does not
|
|
# read the system trust store, so point it at the extracted system CA bundle
|
|
# (now including the proxy root) to get through TLS interception.
|
|
RUN python3 -m pip install --no-cache-dir \
|
|
--cert /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem \
|
|
alembic
|
|
|
|
ENTRYPOINT ["/opt/backup/backup.sh"]
|