USB check-in / check-out audit, as a report rather than a paged list

CMMC removable-media tracking is an audit obligation, and answering "who had
this drive, and when" meant paging through the check-in/out list by hand.
GET /api/usb/reports/audit returns the events - a take and a return are separate
rows, because a count cannot answer the question an audit actually asks - with
summary figures riding along for the people who only want the shape of it, and
a CSV for the people who want it in a spreadsheet.

The window is named rather than a day count: today, last month, six months, a
year, all time. Nobody should have to work out that a year is 365.

Both directory modes answer in one voice. The external path reads the
cmmc_usb checkinoutlog; self-hosted expands one usbcheckouts row into its two
ends, so a returned drive is distinguishable from one still out, and the window
filter runs in SQL against either end of the loan - a drive taken before the
window and returned inside it is a check-in the window has to show. Rows come
back from the mode-specific half; the CSV, the summary and the response shape
are built once in the route.

Ordering is total in both modes. Timestamp alone ties for two events in the same
second, and the same query then returns them in a different order each run,
which reads as the log being unstable.

CHANGELOG.md

@@ -12,6 +12,18 @@ ADR-007 and ADR-002.
 
 ### Added
 
+- **USB check-in / check-out audit report.** CMMC removable-media tracking is an
+  audit obligation and there was no report for it - answering "who had this
+  drive, and when" meant paging through the check-in/out list by hand. The
+  window is named rather than a day count: today, last month, six months, a
+  year, or all time. Events rather than totals, because that is the question an
+  audit actually asks, with a CSV export for the people who want it in a
+  spreadsheet. Works against the external `cmmc_usb` log or the self-hosted
+  tables; in self-hosted mode one stored loan expands into a take and a return,
+  so a returned drive is distinguishable from one still out.
+
+### Fixed
+
 - **USB kiosk: check a drive out and back in without a login.** The people who
   actually take USB drives are standing at the cabinet, and most of them have no
   ShopDB account - so check-out and check-in were gated behind a login that was

docs/PROJECT-MAP.md

@@ -88,6 +88,6 @@ Manifest-less directories under `plugins/` are core frontend surface and always
 
 ## Size
 
-- test functions defined: **1355** (parametrised cases collect higher)
+- test functions defined: **1366** (parametrised cases collect higher)
-- documented API paths: **304** (`docs/openapi.json`, regenerate with `scripts/gen_openapi.py`)
+- documented API paths: **305** (`docs/openapi.json`, regenerate with `scripts/gen_openapi.py`)
 

docs/api-inventory.json

@@ -2842,6 +2842,14 @@
         "params": "body: badge",
         "purpose": "Drives checked out to OTHER people, with holder names, so somebody can return one left behind. Behind its own step rather than on the main kiosk screen, and 404 when usb_kiosk_allow_return_for_others is off - there is no point naming holders of drives the kiosk will not let anyone return",
         "example": "curl -X POST http://localhost:5001/api/usb/kiosk/others -H 'Content-Type: application/json' -d '{\"badge\":\"270015376\"}'"
+      },
+      {
+        "method": "GET",
+        "path": "/api/usb/reports/audit",
+        "auth": "jwt + usb.view",
+        "params": "query: period (day|month|sixmonths|year|all, default month) or days=<n> (0 = all time); format=csv for a download",
+        "purpose": "Removable-media audit: every check-out and check-in in the window, newest first, with who, which drive, when and which locker - plus summary counts. Rows rather than totals, because an audit asks who had a specific drive on a specific day. Works against the external cmmc_usb log or the self-hosted tables; in self-hosted mode one stored loan expands into a take and a return so a returned drive is distinguishable from one still out",
+        "example": "curl 'http://localhost:5001/api/usb/reports/audit?period=year' -H \"Authorization: Bearer $TOKEN\""
       }
     ]
   },

docs/openapi.json

@@ -16473,6 +16473,44 @@
     }
    }
   },
+  "/api/usb/reports/audit": {
+   "get": {
+    "tags": [
+     "plugin-usb"
+    ],
+    "summary": "Removable-media audit: every check-out and check-in in the window, newest first, with who, which drive, when and which...",
+    "description": "Removable-media audit: every check-out and check-in in the window, newest first, with who, which drive, when and which locker - plus summary counts. Rows rather than totals, because an audit asks who had a specific drive on a specific day. Works against the external cmmc_usb log or the self-hosted tables; in self-hosted mode one stored loan expands into a take and a return so a returned drive is distinguishable from one still out\n\n**Auth:** jwt + usb.view\n\n**Params:** query: period (day|month|sixmonths|year|all, default month) or days=<n> (0 = all time); format=csv for a download\n\n**Example:**\n```\ncurl 'http://localhost:5001/api/usb/reports/audit?period=year' -H \"Authorization: Bearer $TOKEN\"\n```",
+    "security": [
+     {
+      "bearerAuth": []
+     }
+    ],
+    "responses": {
+     "200": {
+      "description": "Success. Body is the success_response envelope: {status, data, meta}.",
+      "content": {
+       "application/json": {
+        "$ref": "#/components/schemas/SuccessEnvelope"
+       }
+      }
+     },
+     "default": {
+      "description": "Error. Body is the error envelope; the code and message are nested under data.error.",
+      "content": {
+       "application/json": {
+        "$ref": "#/components/schemas/ErrorEnvelope"
+       }
+      }
+     },
+     "401": {
+      "description": "Missing or invalid credentials."
+     },
+     "403": {
+      "description": "Authenticated, but not permitted."
+     }
+    }
+   }
+  },
   "/api/printers/types": {
    "get": {
     "tags": [

plugins/usb/api/routes.py

@@ -22,9 +22,9 @@ cmmc_usb schema used here:
 
 import logging
 import re
-from datetime import datetime
+from datetime import datetime, timedelta
 
-from flask import Blueprint, request
+from flask import Blueprint, request, Response
 from flask_jwt_extended import jwt_required
 
 from shopdb.api import (
@@ -1068,3 +1068,125 @@ def kiosk_checkin():
 
     return success_response({'device_id': device_id},
                             message='{} checked in'.format(device_id))
+
+
+# =============================================================================
+# Reports
+# =============================================================================
+# CMMC removable-media tracking is an audit obligation, and until now the only
+# way to answer "who had which drive, when" was to read the check-in/out log
+# page by page. This is the same question over a window: a day for "what
+# happened this shift", a month or six for a periodic review, a year or all time
+# for an audit.
+
+# The windows the UI offers, in days. 0 means everything ever recorded.
+AUDITPERIODS = {'day': 1, 'month': 30, 'sixmonths': 182, 'year': 365, 'all': 0}
+
+
+def _auditdays():
+    """Days of history to report. Accepts ?period= or a raw ?days=."""
+    period = (request.args.get('period') or '').strip().lower()
+    if period in AUDITPERIODS:
+        return AUDITPERIODS[period]
+    days = request.args.get('days', type=int)
+    return max(0, days) if days is not None else 30
+
+
+def _auditcsv(rows, columns, filename):
+    """CSV download; generate_csv is not on the contract surface."""
+    import csv
+    import io
+    output = io.StringIO()
+    writer = csv.writer(output)
+    writer.writerow(columns)
+    for row in rows:
+        writer.writerow([row.get(column, '') for column in columns])
+    return Response(output.getvalue(), mimetype='text/csv',
+                    headers={'Content-Disposition':
+                             'attachment; filename={}'.format(filename)})
+
+
+AUDITCOLUMNS = ['timestamp', 'action', 'device_id', 'device_desc', 'badge',
+                'name', 'locker_location']
+
+
+@usb_bp.route('/reports/audit', methods=['GET'])
+@jwt_required()
+@require_permission('usb.view')
+def report_audit():
+    """Every check-out and check-in in the window, newest first.
+
+    Rows, not just totals: an audit asks who had a specific drive on a specific
+    day, and a count cannot answer that. Summary figures ride along for the
+    people who only want the shape of it.
+    """
+    days = _auditdays()
+    if _usb_selfhosted():
+        rows = selfhosted.audit_rows(days)
+        return _auditresponse(rows, days)
+
+    cutoff = None
+    if days > 0:
+        cutoff = (datetime.utcnow() - timedelta(days=days)).strftime(
+            '%Y-%m-%d %H:%M:%S')
+
+    conn = cmmc_usb_connection()
+    try:
+        with conn.cursor() as cur:
+            sql = ('SELECT l.timestamp, l.action, l.device_id, l.badge_number, '
+                   'l.locker_location, d.device_desc, '
+                   'u.first_name, u.last_name '
+                   'FROM checkinoutlog l '
+                   'LEFT JOIN devices d ON d.device_id = l.device_id '
+                   'LEFT JOIN users u ON u.badge_number = l.badge_number ')
+            params = []
+            if cutoff:
+                sql += 'WHERE l.timestamp >= %s '
+                params.append(cutoff)
+            # A total order: timestamp alone ties for two events in the same
+            # second and the same query then returns them in a different order
+            # each run, which reads as the log being unstable.
+            sql += 'ORDER BY l.timestamp DESC, l.log_id DESC'
+            cur.execute(sql, tuple(params))
+            rows = []
+            for row in cur.fetchall() or []:
+                name = '{} {}'.format(row.get('first_name') or '',
+                                      row.get('last_name') or '').strip()
+                stamp = row.get('timestamp')
+                rows.append({
+                    'timestamp': stamp.isoformat() if hasattr(stamp, 'isoformat')
+                                 else (stamp or ''),
+                    'action': row.get('action') or '',
+                    'device_id': row.get('device_id') or '',
+                    'device_desc': row.get('device_desc') or '',
+                    'badge': row.get('badge_number') or '',
+                    'name': name,
+                    'locker_location': row.get('locker_location') or '',
+                })
+    finally:
+        conn.close()
+
+    return _auditresponse(rows, days)
+
+
+def _auditresponse(rows, days):
+    """One voice for both directory modes: same CSV, same JSON, same summary."""
+    if request.args.get('format') == 'csv':
+        return _auditcsv(rows, AUDITCOLUMNS, 'usb-audit.csv')
+    return success_response({
+        'columns': AUDITCOLUMNS,
+        'rows': rows,
+        'days': days,
+        'summary': _auditsummary(rows),
+    })
+
+
+def _auditsummary(rows):
+    checkouts = sum(1 for row in rows if row['action'] == 'check-out')
+    return {
+        'events': len(rows),
+        'checkouts': checkouts,
+        'checkins': sum(1 for row in rows if row['action'] == 'check-in'),
+        'devices': len({row['device_id'] for row in rows if row['device_id']}),
+        'people': len({row['badge'] for row in rows if row['badge']}),
+    }

plugins/usb/api/selfhosted.py

@@ -16,7 +16,7 @@ History rows are synthesized from usbcheckouts (each row = a check-out event and
 if returned, a check-in event).
 """
 
-from datetime import datetime, timezone
+from datetime import datetime, timedelta, timezone
 
 from shopdb.api import (
     db, success_response, error_response, ErrorCodes,
@@ -452,3 +452,47 @@ def _kiosk_flag(key, default):
     if row is None or row.value is None or row.value == '':
         return default
     return str(row.value).strip().lower() in ('true', '1', 'yes', 'on')
+
+
+def audit_rows(days):
+    """Check-out / check-in events over a window (self-hosted mode).
+
+    One usbcheckouts row holds BOTH ends of a loan, so it expands into two
+    events - the take and, when it has come back, the return. Reporting the row
+    once would make a returned drive indistinguishable from one still out.
+
+    Rows only: the caller owns the CSV, the summary and the response shape, so
+    both directory modes answer in exactly one voice.
+    """
+    cutoff = None
+    if days > 0:
+        cutoff = datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=days)
+
+    query = (db.session.query(USBCheckout, USBDevice)
+             .join(USBDevice, USBDevice.usbdeviceid == USBCheckout.usbdeviceid))
+    if cutoff is not None:
+        # Either end of the loan can fall in the window: a drive taken before it
+        # and returned inside it is a check-in the window has to show.
+        query = query.filter(db.or_(USBCheckout.checkouttime >= cutoff,
+                                    USBCheckout.checkintime >= cutoff))
+
+    rows = []
+    for checkout, device in query.all():
+        base = {
+            'device_id': device.serialnumber,
+            'device_desc': device.label or device.productname or '',
+            'badge': checkout.sso or '',
+            'name': checkout.checkoutname or '',
+            'locker_location': device.storagelocation or '',
+        }
+        if checkout.checkouttime and (cutoff is None or checkout.checkouttime >= cutoff):
+            rows.append(dict(base, timestamp=checkout.checkouttime.isoformat(),
+                             action='check-out'))
+        if checkout.checkintime and (cutoff is None or checkout.checkintime >= cutoff):
+            rows.append(dict(base, timestamp=checkout.checkintime.isoformat(),
+                             action='check-in'))
+
+    # Newest first, and a total order so two events in the same second do not
+    # swap places between identical runs.
+    rows.sort(key=lambda row: (row['timestamp'], row['device_id']), reverse=True)
+    return rows

plugins/usb/plugin.py

@@ -70,6 +70,26 @@ class USBPlugin(BasePlugin):
             'docs': 'plugins/usb/README.md',
         }
 
+    def get_reports(self) -> List[Dict]:
+        """Removable-media audit.
+
+        CMMC tracking is an audit obligation, and the only way to answer "who
+        had which drive, when" was to page through the check-in/out list by
+        hand. The window covers a shift, a periodic review, or everything ever
+        recorded.
+        """
+        return [
+            {
+                'id': 'usb-audit',
+                'name': 'USB Check-in / Check-out Audit',
+                'description': 'Every take and return over a chosen window - '
+                               'day, month, six months, year or all time - with '
+                               'who, which drive and when',
+                'category': 'usage',
+                'endpoint': '/api/usb/reports/audit',
+            },
+        ]
+
     def get_settings_cards(self) -> List[Dict]:
         """Somewhere to actually turn the kiosk on.
 

tests/test_plugins/test_usb_reports.py

@@ -0,0 +1,139 @@
+"""USB check-in / check-out audit report.
+
+CMMC removable-media tracking is an audit obligation, and the question an audit
+asks is "who had this drive, when" - which a total cannot answer. So this
+reports EVENTS over a window, with the window named rather than expressed as a
+day count nobody should have to work out.
+"""
+
+from datetime import datetime, timedelta, timezone
+
+import pytest
+
+from shopdb.core.models import Setting
+
+
+def _now():
+    return datetime.now(timezone.utc).replace(tzinfo=None)
+
+
+@pytest.fixture
+def usbdata(db):
+    """Self-hosted USB with a loan returned today and one from long ago."""
+    from plugins.usb.models import USBDevice, USBCheckout
+
+    db.session.add(Setting(key='usb_directory_mode', value='selfhosted',
+                           valuetype='string', category='site'))
+    recent = USBDevice(serialnumber='USB-1', label='Blue Kingston')
+    old = USBDevice(serialnumber='USB-2', label='Red SanDisk')
+    db.session.add_all([recent, old])
+    db.session.flush()
+
+    # Taken and returned today: two events.
+    db.session.add(USBCheckout(usbdeviceid=recent.usbdeviceid, sso='270015376',
+                               checkoutname='Dana Reyes',
+                               checkouttime=_now() - timedelta(hours=3),
+                               checkintime=_now() - timedelta(hours=1)))
+    # Taken 200 days ago and never returned: one event, outside most windows.
+    db.session.add(USBCheckout(usbdeviceid=old.usbdeviceid, sso='270099999',
+                               checkoutname='Sam Okafor',
+                               checkouttime=_now() - timedelta(days=200)))
+    db.session.commit()
+    return db
+
+
+def _report(client, headers, **params):
+    query = '&'.join(f'{k}={v}' for k, v in params.items())
+    response = client.get(f'/api/usb/reports/audit?{query}', headers=headers)
+    assert response.status_code == 200, response.get_json()
+    return response.get_json()['data']
+
+
+def test_a_loan_produces_both_a_take_and_a_return(client, usbdata, auth_headers):
+    """One stored row holds both ends of a loan.
+
+    Reporting it once would make a returned drive look identical to one still
+    out, which is the opposite of what the log is for.
+    """
+    data = _report(client, auth_headers, period='month')
+    actions = sorted(row['action'] for row in data['rows'])
+
+    assert actions == ['check-in', 'check-out']
+    assert data['summary']['checkouts'] == 1
+    assert data['summary']['checkins'] == 1
+
+
+def test_the_window_actually_bounds_the_rows(client, usbdata, auth_headers):
+    month = _report(client, auth_headers, period='month')
+    year = _report(client, auth_headers, period='year')
+
+    assert {row['device_id'] for row in month['rows']} == {'USB-1'}
+    assert {row['device_id'] for row in year['rows']} == {'USB-1', 'USB-2'}
+
+
+def test_all_time_keeps_everything(client, usbdata, auth_headers):
+    data = _report(client, auth_headers, period='all')
+    assert data['days'] == 0
+    assert data['summary']['events'] == 3
+
+
+def test_today_is_a_window_too(client, usbdata, auth_headers):
+    """A shift handover asks what happened today, not this month."""
+    data = _report(client, auth_headers, period='day')
+    assert data['days'] == 1
+    assert data['summary']['events'] == 2
+
+
+@pytest.mark.parametrize('period,days', [('day', 1), ('month', 30),
+                                         ('sixmonths', 182), ('year', 365),
+                                         ('all', 0)])
+def test_every_named_window_maps_to_a_day_count(client, usbdata, auth_headers,
+                                                period, days):
+    assert _report(client, auth_headers, period=period)['days'] == days
+
+
+def test_an_unknown_period_falls_back_rather_than_failing(client, usbdata,
+                                                          auth_headers):
+    """A bad query string should not 500 a report somebody is mid-audit on."""
+    assert _report(client, auth_headers, period='fortnight')['days'] == 30
+
+
+def test_rows_are_newest_first(client, usbdata, auth_headers):
+    rows = _report(client, auth_headers, period='all')['rows']
+    stamps = [row['timestamp'] for row in rows]
+    assert stamps == sorted(stamps, reverse=True)
+
+
+def test_it_says_who_and_which_drive(client, usbdata, auth_headers):
+    rows = _report(client, auth_headers, period='month')['rows']
+    row = rows[0]
+    assert row['name'] == 'Dana Reyes'
+    assert row['badge'] == '270015376'
+    assert row['device_id'] == 'USB-1'
+    assert row['device_desc'] == 'Blue Kingston'
+
+
+def test_it_exports_as_csv(client, usbdata, auth_headers):
+    response = client.get('/api/usb/reports/audit?period=all&format=csv',
+                          headers=auth_headers)
+    assert response.status_code == 200
+    assert response.mimetype == 'text/csv'
+    body = response.get_data(as_text=True)
+    assert body.splitlines()[0].startswith('timestamp,action,device_id')
+    assert 'Dana Reyes' in body
+
+
+def test_reading_the_audit_needs_permission(client, usbdata, member_headers):
+    """A role-less member cannot read who had which drive."""
+    response = client.get('/api/usb/reports/audit?period=all',
+                          headers=member_headers)
+    assert response.status_code == 403
+
+
+def test_the_report_is_registered_so_it_appears_in_the_catalogue(client, usbdata,
+                                                                 auth_headers):
+    from plugins.usb.plugin import USBPlugin
+    report = USBPlugin().get_reports()[0]
+    assert report['id'] == 'usb-audit'
+    # The endpoint is what the reports page fetches; a wrong one renders blank.
+    assert report['endpoint'] == '/api/usb/reports/audit'