Let the row refuse the write both kiosks were deciding in Python

Wave 1 of the remediation plan for the 2026-09-03 feature test run. Two kiosks,
one shape of defect, one critical consequence.

Both kiosks read the state, decided in Python, and then wrote. Nothing stops two
requests interleaving between the read and the write, so both see the same
pre-state and both proceed. Reproduced on MySQL 5.6 over HTTP, five concurrent
takes of 5 from a bin holding 5: all five were accepted and quantityonhand
landed on -20, because each request wrote its own arithmetic. The ledger had
recorded every one of them, so the cached quantity and the ledger disagreed
permanently, which is the number the shop floor actually reads. The USB kiosk
had the same shape: concurrent scans of one drive all succeeded and inserted
several open custody rows, so one drive was recorded out to two people and the
documented 409 never fired.

The comparison now lives in the WHERE clause, where the database settles it at
write time. printedparts._move_quantity issues
UPDATE ... WHERE quantityonhand >= :n and treats rowcount 0 as the refusal,
raising InsufficientStock, which take and adjust turn back into the message they
already gave - carrying the quantity the database holds now, not the stale one
the request read. The low-stock alert derives its before-value from what the
write produced for the same reason. Both USB paths claim the device with
ischeckedout (self-hosted) or status (external cmmc_usb) in the WHERE clause;
the external one claims BEFORE writing the log line, so a refused claim leaves
nothing to undo. Check-in is guarded the same way.

After the fix, the same five concurrent takes: one accepted, four refused,
quantityonhand 0 and the ledger sum 0.

Check-in also closes EVERY open row for the device rather than the newest.
One device can only be out once, so a second open row is corruption, and
closing only the newest left the older ones open forever - the device then read
as returned while its custody history said otherwise.

Separately, usbcheckouts now records who handed a drive back
(checkinsso/checkinname, migration usb0004checkinidentity). Returning for a
colleague is a supported flow that the kiosk asks to confirm, but the row it
wrote attributed the check-in to the HOLDER, so the custody history and the
audit report both named somebody who was not there. Nullable and deliberately
not backfilled: copying the holder's badge into old rows would manufacture
evidence for the exact claim this stops making.

Four tests. Both guards verified by planting: strip the condition out of either
UPDATE and its test fails.

Scope: product
This commit is contained in:
cproudlock
2026-09-03 13:58:26 -04:00
parent 377cc69264
commit aa8028f96e
8 changed files with 362 additions and 23 deletions

View File

@@ -257,6 +257,52 @@ from ..models import PrintedItemTransaction
from ..services.badges import BadgeError, resolve_badge
class InsufficientStock(Exception):
"""A decrement was refused because the row no longer holds that many.
Carries the quantity the DATABASE holds, not the stale one the caller read,
so the message the operator sees is the truth at the moment of the write.
"""
def __init__(self, available):
super().__init__('insufficient stock')
self.available = available
def _move_quantity(item, quantitychange):
"""Move quantityonhand by quantitychange in ONE statement, and let the row
itself refuse a decrement it cannot cover.
This is the guard, not the caller's earlier `quantity > item.quantityonhand`
check. That check reads, decides in Python, and writes - so two kiosk takes
that interleave both read the same on-hand figure and both pass it. Five
concurrent takes of 5 from a bin holding 5 all succeeded, each wrote its own
5 - 5 = 0, and quantityonhand ended up permanently out of step with the
ledger that recorded all five.
Expressing it as `UPDATE ... WHERE quantityonhand >= :n` makes the database
do the comparing at write time, where it is atomic. A row that no longer
holds enough matches nothing, rowcount is 0, and the take is refused.
Returns the resulting quantity.
"""
query = PrintedItem.query.filter(
PrintedItem.printeditemid == item.printeditemid)
if quantitychange < 0:
query = query.filter(PrintedItem.quantityonhand >= -quantitychange)
moved = query.update(
{PrintedItem.quantityonhand:
PrintedItem.quantityonhand + quantitychange},
synchronize_session=False)
if not moved:
# Read back what is actually there so the refusal can say so.
db.session.rollback()
available = db.session.query(PrintedItem.quantityonhand).filter(
PrintedItem.printeditemid == item.printeditemid).scalar() or 0
raise InsufficientStock(available)
db.session.refresh(item)
return item.quantityonhand
def _ledger_write(item, transactiontype, quantitychange, sso, name, reason=None,
revision=None):
"""Append a ledger row and move the cached quantity in ONE commit.
@@ -266,9 +312,17 @@ def _ledger_write(item, transactiontype, quantitychange, sso, name, reason=None,
alert when this write CROSSES the item's threshold downward - crossing
(not being below) is the natural debounce: one alert per depletion, and
restocking above the threshold rearms it.
Raises InsufficientStock when the row cannot cover the decrement. The
caller has already refused the obvious case; this is the one that only
shows up under concurrency.
"""
quantitybefore = item.quantityonhand
item.quantityonhand += quantitychange
# Derived from what the write ACTUALLY produced, not from what this request
# read earlier. Under concurrency the two differ, and the threshold
# crossing has to be judged on the real numbers or the alert fires for the
# wrong take (or not at all).
quantityafter = _move_quantity(item, quantitychange)
quantitybefore = quantityafter - quantitychange
db.session.add(PrintedItemTransaction(
printeditemid=item.printeditemid,
transactiontype=transactiontype,
@@ -427,7 +481,13 @@ def adjust_item(item_id: int):
except BadgeError as badge_error:
return error_response(ErrorCodes.VALIDATION_ERROR, str(badge_error),
http_code=422)
_ledger_write(item, 'adjust', quantitychange, sso, name, reason=reason)
try:
_ledger_write(item, 'adjust', quantitychange, sso, name, reason=reason)
except InsufficientStock as shortfall:
return error_response(
ErrorCodes.VALIDATION_ERROR,
f'Adjustment would drive stock below zero '
f'(on hand: {shortfall.available})')
return success_response(item.to_dict(), message='Stock adjusted')
@@ -579,7 +639,16 @@ def kiosk_take():
except (ValueError, TypeError):
revision = None
_ledger_write(item, 'take', -quantity, sso, name, revision=revision)
try:
_ledger_write(item, 'take', -quantity, sso, name, revision=revision)
except InsufficientStock as shortfall:
# Another kiosk took the last of it between this request's read and its
# write. Same message as the pre-check, with the number the database
# actually holds now.
return error_response(
ErrorCodes.VALIDATION_ERROR,
f'Only {shortfall.available} on hand - take fewer or see the '
f'parts team')
return success_response(item.to_dict(),
message=f'Took {quantity}, {item.quantityonhand} left')