Audit log retention: a way to purge, off by default

The audit log is append-only and nothing ever removed a row, so it grows for as
long as a site runs. This site reached tens of thousands of pages, which is past
the point of being useful: nobody pages through that to find anything.

Retention is OPT-IN and the default keeps everything. An upgrade that silently
started deleting a site's audit history would be indefensible, so
audit_retentiondays defaults to 0 and nothing is removed until somebody sets a
number. The Site & Facility card shows the log's size and exactly how many
entries a purge would delete BEFORE anything is pressed.

Deleting happens in committed batches. That is not an optimisation: one DELETE
over a million rows builds a single enormous transaction and holds locks across
the table while it runs, which on this server means blocking the collectors
posting inventory. The API caps its batch count so the request cannot outlive a
proxy timeout, and a run that stops early reports how much is left rather than
claiming it finished - a purge reporting success over 900k remaining rows would
be the same silent half-failure this codebase keeps getting bitten by.

`flask audit stats` and `flask audit purge` for the server, the latter reporting
by default and deleting only with --apply, with no batch cap because it is not
behind a proxy and a purge that stops halfway leaves the operator guessing.

audit.purge is a separate permission from audit.view. Reading the log and
destroying it are different acts, and plenty of people need only the first.

Chunking selects a page of ids and deletes those rather than using
DELETE ... LIMIT, which is MySQL-only while the suite runs on SQLite.

shopdb/core/api/auditlogs.py

@@ -4,7 +4,8 @@ from flask import Blueprint, request
 from flask_jwt_extended import jwt_required
 
 from shopdb.core.models import AuditLog
-from shopdb.utils.responses import success_response
+from shopdb.extensions import db
+from shopdb.utils.responses import success_response, error_response, ErrorCodes
 from shopdb.utils.authz import require_permission
 
 auditlogs_bp = Blueprint('auditlogs', __name__)
@@ -194,3 +195,74 @@ def db_func_count_by(column):
     ).group_by(column).all()
 
     return {r[0]: r[1] for r in results}
+
+
+@auditlogs_bp.route('/retention', methods=['GET'])
+@jwt_required()
+@require_permission('audit.view')
+def get_retention():
+    """Size of the log and what the configured retention would remove.
+
+    Drives the Site & Facility card: an admin should see the number of entries a
+    purge deletes BEFORE pressing anything.
+    """
+    from shopdb.core.services import auditretention
+    days = auditretention.configureddays()
+    return success_response(auditretention.stats(days or None))
+
+
+@auditlogs_bp.route('/purge', methods=['POST'])
+@jwt_required()
+@require_permission('audit.purge')
+def purge_auditlogs():
+    """Delete entries older than the retention window.
+
+    Separate permission from audit.view: reading the log and destroying it are
+    not the same act, and plenty of people need the first.
+
+    Batch-capped, because this runs inside a request that a proxy will time out.
+    An incomplete purge returns complete=false and the number still to go rather
+    than reporting a finished job - the caller re-posts, or runs
+    `flask audit purge --apply` on the server, which has no cap.
+    """
+    from flask import request as flask_request
+    from shopdb.core.services import auditretention
+
+    body = flask_request.get_json(silent=True) or {}
+    days = body.get('days')
+    if days is None:
+        days = auditretention.configureddays()
+    try:
+        days = int(days)
+    except (TypeError, ValueError):
+        days = 0
+    if days <= 0:
+        return error_response(
+            ErrorCodes.VALIDATION_ERROR,
+            'No retention period is configured. Set one in Site & Facility '
+            'settings before purging.')
+
+    dryrun = bool(body.get('dryrun'))
+    result = auditretention.purge(
+        days, dryrun=dryrun,
+        maxbatches=None if dryrun else auditretention.MAXBATCHES_REQUEST)
+
+    if dryrun:
+        message = '{:,} entry(s) would be deleted'.format(result['matched'])
+    elif result['complete']:
+        message = 'Deleted {:,} entry(s)'.format(result['deleted'])
+    else:
+        message = ('Deleted {:,} entry(s); {:,} still older than the cutoff. '
+                   'Run it again to continue.').format(
+                       result['deleted'], result['remaining'])
+
+    # The purge is itself an audited act, and this row postdates the cutoff so
+    # it survives the purge that created it.
+    if not dryrun and result['deleted']:
+        from shopdb.api import audit_log
+        audit_log(action='deleted', entitytype='AuditLog',
+                  entityname='purge older than {} days'.format(days),
+                  changes={'deleted': result['deleted'],
+                           'cutoff': result['cutoff']})
+
+    return success_response(result, message=message)

shopdb/core/api/settings.py

@@ -522,6 +522,13 @@ def build_default_settings():
             'category': 'site',
             'description': 'Public base URL of this site INCLUDING any subpath it is served under, e.g. https://shopdb.example.net or https://host/shopdb. Consumers append the in-app route to this and add nothing themselves, so a subpath left off here produces links that 404. Blank = use the browsing origin.'
         },
+        {
+            'key': 'audit_retentiondays',
+            'value': '0',
+            'valuetype': 'integer',
+            'category': 'site',
+            'description': 'Delete audit log entries older than this many days. 0 (default) keeps everything forever. The audit log is append-only and grows for as long as the site runs, so a long-lived site eventually needs a limit here; nothing is deleted until one is set, and the settings page shows how many entries a purge would remove before it runs.'
+        },
         {
             'key': 'facility_name',
             'value': '',