ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Make the API usable by an agent, and stop the contract going stale

Browse Source 
Someone writing a script against ShopDB had two options: read a 705 KB spec, or
guess. The spec is the reference, not an index, and handing an agent the whole
thing means either truncating it or spending a context window on 479 operations
nobody asked about.

Three additions, each feeding the next.

/api/docs/operations.txt is the index: every operation on one line with its
method, path, tag, auth and summary, at around 70 KB for all 486. A caller can
hold the entire API at once, find the four operations it needs, and only then
fetch their schemas.

/api/docs/openapi.json?tag=... narrows the spec to one of the 41 tag groups.
Components are kept intact deliberately: pruning the schema graph to match would
leave references resolving to nothing, and a generator that fails confusingly is
worse than a file that is larger than strictly necessary.

tools/mcp/ is an MCP server built on both. Five tools rather than 483, because a
tool per operation is unusable - clients degrade past a few dozen and models
start picking the wrong one. It reads everything from the live instance at call
time, so enabling a plugin adds operations it can already reach with no change
here. The token is the security boundary and the README says so plainly: every
operation the credential permits is reachable, so exploratory work wants a
read-scoped token.

The rest of this is about the contract not drifting, which is what makes any of
the above trustworthy.

Two CI jobs:

- api-contract runs the parity gates and then regenerates the spec and fails if
  anything changed. Those gates already existed and already worked - they caught
  both routes added here, unprompted - but nothing ran them except by hand, and
  the committed spec had silently drifted by three operations, including two
  setup endpoints added the day before.
- migrations-mysql is the job the header comment has been claiming for some
  time while not existing. It described "the REAL multi-site deploy path" and
  "what catches an Alembic regression on MySQL before it ships", and there was
  no such job. That is worse than no comment: it is the false confidence that
  let a migration ship which could not build a schema on a fresh MySQL database.
  The suite runs on SQLite, and SQLite accepts a foreign key onto a table that
  does not exist.

CONTRIBUTING.md now says to run gen_openapi.py --refresh after changing a route,
and that a route the sweep cannot classify belongs in PUBLIC_ENDPOINTS with a
reason rather than hand-edited into the inventory, which the next refresh would
overwrite. The new index route is declared there alongside the spec it indexes.
This commit is contained in:
cproudlock
2026-09-20 12:59:49 -04:00
parent 6c00b10618
commit 762d1b7210
11 changed files with 894 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

86
tools/mcp/README.md Normal file
View File

@@ -0,0 +1,86 @@
# ShopDB MCP server
Exposes a ShopDB instance to an LLM client (Claude Code, Claude Desktop, or any
other MCP client) so it can discover and call the API directly, instead of
someone generating a client by hand from the spec.
## The token is the security boundary
**Read this before pointing it at production.**
Every operation the credential is allowed to perform is reachable through this
server, including writes and deletes. There is no separate allow-list here: the
scope of the personal access token decides what can happen, and nothing else
does.
That is a deliberate choice, and it puts the decision where it can be enforced
server-side rather than in a client that a model is talking to. It also means:
- For anything exploratory, create a **read-scoped token**. A read-scoped token
cannot be talked into a write, whatever the model decides to call.
- Give it the narrowest scopes that cover the task, not your admin session.
- `DELETE` against an asset database is irreversible. The tool descriptions say
so, which makes a model markedly more careful, but a token that permits it is
the only thing standing in the way.
Create a token under Settings > API Tokens. `shopdb_whoami` reports what the
token can do; call it first.
## Install
pip install -e tools/mcp
Then register it with your client. For Claude Code:
claude mcp add shopdb \
--env SHOPDB_URL=https://shopdb.example.net/shopdb \
--env SHOPDB_TOKEN=<token> \
-- shopdb-mcp
`SHOPDB_URL` must include the mount path if the site is served under one
(`/shopdb`), because that is what the instance's own spec declares.
| Variable | Required | Meaning |
|---|---|---|
| `SHOPDB_URL` | yes | Base URL including any mount path |
| `SHOPDB_TOKEN` | yes | Personal access token |
| `SHOPDB_VERIFY_TLS` | no | `0` to accept a self-signed certificate. Defaults to verifying |
## Tools
| Tool | Purpose |
|---|---|
| `shopdb_whoami` | Instance, identity, scopes, operation count. Call first |
| `shopdb_search_operations` | Find operations by keyword or tag. Start here |
| `shopdb_describe_operation` | Parameters, request body and responses for one operation |
| `shopdb_call` | Call any operation |
| `shopdb_llms_guide` | The instance's own orientation guide |
## Why five tools and not 483
ShopDB publishes 483 operations. One tool each would be unusable: MCP clients
degrade badly past a few dozen tools and models start choosing wrong ones. So
the surface is small and generic, and the instance's own published index is what
makes it navigable - `search` reads `/api/docs/operations.txt` (about 70 KB for
every operation) and `describe` fetches only the tag group it needs rather than
the 700 KB spec.
Nothing is generated ahead of time. Enabling a plugin adds operations this
server can already reach, with no change here and no reinstall.
## Staying current
The server reads everything from the live instance, so it describes whatever
that instance actually serves. The staleness risk is upstream: `docs/openapi.json`
is generated and committed, and if it drifts, the instance serves a stale spec
and this server faithfully reports stale information.
That is guarded by the `api-contract` CI job and by
`python scripts/gen_openapi.py --refresh`. See CONTRIBUTING.md.
## Limitations
- Air-gapped sites need `mcp` vendored; it is not bundled here.
- No streaming or subscriptions - every tool is a request and a response.
- Responses over 100 KB are truncated with a note. Paginate instead
(`perpage` is silently capped at 100 by the API).

16
tools/mcp/pyproject.toml Normal file
View File

@@ -0,0 +1,16 @@
[project]
name = "shopdb-mcp"
version = "0.1.0"
description = "MCP server for a ShopDB instance"
requires-python = ">=3.10"
dependencies = ["mcp>=1.2"]
[project.scripts]
shopdb-mcp = "shopdb_mcp:main"
[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"
[tool.setuptools.packages.find]
include = ["shopdb_mcp*"]

8
tools/mcp/shopdb_mcp/__init__.py Normal file
View File

@@ -0,0 +1,8 @@
"""MCP server exposing a ShopDB instance to an LLM client."""
__all__ = ['main']
def main():
from .server import main as _main
_main()

191
tools/mcp/shopdb_mcp/client.py Normal file
View File

@@ -0,0 +1,191 @@
"""HTTP access to a ShopDB instance, and the operation index it publishes.
Everything the server knows about the API is fetched from the instance at
runtime: the operation index, the schemas, the mount path. Nothing is generated
ahead of time and nothing is pinned, so a site running an older or newer build
is described accurately rather than according to whatever was true when this
package was written.
"""
from __future__ import annotations
import json
import os
import re
import threading
import time
from dataclasses import dataclass
from typing import Any
from urllib import error, parse, request
# How long a fetched index or spec stays usable before it is re-read. Long
# enough that a burst of tool calls does not re-download a 700 KB document,
# short enough that enabling a plugin shows up without restarting the server.
CACHE_SECONDS = 300
DEFAULT_TIMEOUT = 30
# Verbs that change or destroy data. Named so the tool descriptions can say so:
# a model told that DELETE is irreversible behaves better than one left to infer
# it from the verb.
WRITE_METHODS = {'POST', 'PUT', 'PATCH', 'DELETE'}
DESTRUCTIVE_METHODS = {'DELETE'}
class ShopdbError(RuntimeError):
"""An error worth showing the caller verbatim."""
@dataclass
class Operation:
method: str
path: str
tag: str
auth: str
summary: str
def matches(self, needle: str) -> bool:
blob = f'{self.method} {self.path} {self.tag} {self.summary}'.lower()
return all(word in blob for word in needle.lower().split())
class ShopdbClient:
"""Talks to one ShopDB instance with one credential."""
def __init__(self, base_url: str | None = None, token: str | None = None,
verify_tls: bool | None = None, timeout: int = DEFAULT_TIMEOUT):
self.base_url = (base_url or os.environ.get('SHOPDB_URL') or '').rstrip('/')
self.token = token or os.environ.get('SHOPDB_TOKEN') or ''
if verify_tls is None:
# Sites commonly run a self-signed certificate on an internal name,
# so this has to be settable - but it defaults to verifying, and the
# caller has to say otherwise deliberately.
verify_tls = (os.environ.get('SHOPDB_VERIFY_TLS', '1').strip().lower()
not in ('0', 'false', 'no'))
self.verify_tls = verify_tls
self.timeout = timeout
self._lock = threading.Lock()
self._cache: dict[str, tuple[float, Any]] = {}
if not self.base_url:
raise ShopdbError(
'SHOPDB_URL is not set. Point it at the instance, including any '
'mount path, for example https://shopdb.example.net/shopdb')
# ---- plumbing --------------------------------------------------------
def _opener(self):
if self.verify_tls:
return request.build_opener()
import ssl
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
return request.build_opener(request.HTTPSHandler(context=ctx))
def request(self, method: str, path: str, params: dict | None = None,
body: Any = None) -> tuple[int, Any]:
"""One HTTP call. Returns (status, parsed-body-or-text)."""
url = self.base_url + '/' + path.lstrip('/')
if params:
# Drop empty values rather than sending key= , which several list
# endpoints treat as a filter on the empty string.
clean = {k: v for k, v in params.items() if v not in (None, '')}
if clean:
url += ('&' if '?' in url else '?') + parse.urlencode(clean, doseq=True)
data = None
headers = {'Accept': 'application/json'}
if self.token:
headers['Authorization'] = f'Bearer {self.token}'
if body is not None:
data = json.dumps(body).encode('utf-8')
headers['Content-Type'] = 'application/json'
req = request.Request(url, data=data, headers=headers, method=method.upper())
try:
with self._opener().open(req, timeout=self.timeout) as response:
return response.status, _decode(response.read(),
response.headers.get('Content-Type'))
except error.HTTPError as exc:
# An error body from this API is the useful part (it carries the
# error code and message), so return it rather than raising.
return exc.code, _decode(exc.read(), exc.headers.get('Content-Type'))
except error.URLError as exc:
raise ShopdbError(f'could not reach {self.base_url}: {exc.reason}') from exc
def _cached(self, key: str, fetch):
with self._lock:
hit = self._cache.get(key)
if hit and (time.time() - hit[0]) < CACHE_SECONDS:
return hit[1]
value = fetch()
with self._lock:
self._cache[key] = (time.time(), value)
return value
# ---- what the instance publishes about itself ------------------------
def operations(self) -> list[Operation]:
"""The published operation index, parsed.
Read from /api/docs/operations.txt rather than the full spec: the index
is around 70 KB against 700 KB, and listing what exists does not need
the schemas.
"""
def fetch():
status, text = self.request('GET', '/api/docs/operations.txt')
if status != 200:
raise ShopdbError(
f'operation index unavailable (HTTP {status}). This instance '
'may predate /api/docs/operations.txt.')
return _parse_index(text if isinstance(text, str) else str(text))
return self._cached('operations', fetch)
def spec(self, tag: str | None = None) -> dict:
"""The OpenAPI spec, optionally narrowed to one tag group."""
def fetch():
path = '/api/docs/openapi.json'
status, payload = self.request('GET', path, params={'tag': tag} if tag else None)
if status != 200:
raise ShopdbError(f'spec unavailable (HTTP {status}): {payload}')
return payload
return self._cached(f'spec:{tag or "*"}', fetch)
def find_operation(self, method: str, path: str) -> dict | None:
"""The spec entry for one operation, fetched by its tag to stay small."""
method = method.lower()
for op in self.operations():
if op.method.lower() == method and op.path == path:
spec = self.spec(op.tag)
return (spec.get('paths', {}).get(path, {}) or {}).get(method)
return None
_INDEX_LINE = re.compile(
r'^(?P<method>[A-Z]+)\s+(?P<path>\S+)\s+\[(?P<tag>[^\]]*)\]\s+'
r'\((?P<auth>[^)]*)\)\s*(?P<summary>.*)$')
def _parse_index(text: str) -> list[Operation]:
out = []
for line in text.splitlines():
line = line.strip()
if not line or line.startswith('#'):
continue
m = _INDEX_LINE.match(line)
if m:
out.append(Operation(m.group('method'), m.group('path'),
m.group('tag'), m.group('auth'),
m.group('summary').strip()))
return out
def _decode(raw: bytes, content_type: str | None):
text = raw.decode('utf-8', errors='replace')
if content_type and 'json' in content_type.lower():
try:
return json.loads(text)
except json.JSONDecodeError:
return text
return text

226
tools/mcp/shopdb_mcp/server.py Normal file
View File

@@ -0,0 +1,226 @@
"""MCP server for a ShopDB instance.
WHY IT IS SHAPED LIKE THIS. ShopDB publishes 483 operations. Exposing one tool
per operation would be useless: MCP clients degrade badly past a few dozen tools
and a model starts picking the wrong one. So the surface here is small and
generic - find an operation, read its contract, call it - and the instance's own
published index is what makes that navigable. Adding a plugin adds operations
that this server can already reach, with no change here.
SECURITY MODEL: the token IS the boundary. Every operation the credential is
allowed to perform is reachable, including writes and deletes. That is a
deliberate choice, and it means the scope of the personal access token decides
the blast radius, not this code. Create a read-scoped token for anything
exploratory. See README.md.
"""
from __future__ import annotations
import json
import os
from typing import Any
# The SDK renamed FastMCP to MCPServer in 2.0. The two have the same shape for
# what this server uses (a .tool() decorator and .run()), so support both rather
# than forcing a pin: a site may already have either installed.
try:
from mcp.server.mcpserver import MCPServer as _Server # mcp >= 2
except ImportError: # pragma: no cover
from mcp.server.fastmcp import FastMCP as _Server # mcp 1.x
from .client import (DESTRUCTIVE_METHODS, WRITE_METHODS, ShopdbClient,
ShopdbError)
mcp = _Server('shopdb')
_client: ShopdbClient | None = None
def client() -> ShopdbClient:
global _client
if _client is None:
_client = ShopdbClient()
return _client
def _fail(message: str) -> str:
"""Errors come back as text the model can read and act on.
Raising would surface a stack trace the model cannot do anything with; a
sentence telling it what went wrong and what to try next is more useful.
"""
return f'ERROR: {message}'
@mcp.tool()
def shopdb_whoami() -> str:
"""Check which ShopDB instance this server targets and what the token can do.
Call this first. It confirms the instance is reachable, the credential is
valid, and - importantly - what the token is scoped to, which determines
which other calls will succeed.
"""
try:
c = client()
except ShopdbError as exc:
return _fail(str(exc))
try:
status, body = c.request('GET', '/api/auth/me')
except ShopdbError as exc:
return _fail(str(exc))
if status == 401:
return _fail('token rejected (HTTP 401). Check SHOPDB_TOKEN.')
if status != 200:
return _fail(f'HTTP {status} from /api/auth/me: {body}')
user = (body or {}).get('data') or {}
try:
count = len(c.operations())
except ShopdbError as exc:
count = f'unknown ({exc})'
return json.dumps({
'instance': c.base_url,
'username': user.get('username'),
'roles': user.get('roles'),
'permissions': len(user.get('permissions') or []),
'operations_published': count,
'tls_verified': c.verify_tls,
}, indent=2)
@mcp.tool()
def shopdb_search_operations(query: str = '', tag: str = '',
limit: int = 40) -> str:
"""Find API operations by keyword. START HERE to discover what exists.
Searches method, path, tag and summary. `query` words are ANDed, so
"printer supplies" matches an operation mentioning both. `tag` narrows to
one group (for example plugin-printers, core-assets).
Returns one line per operation. Use shopdb_describe_operation for the
parameters and body schema of anything you intend to call.
"""
try:
ops = client().operations()
except ShopdbError as exc:
return _fail(str(exc))
if tag:
ops = [o for o in ops if o.tag == tag]
if query:
ops = [o for o in ops if o.matches(query)]
if not ops:
tags = sorted({o.tag for o in client().operations()})
return (f'No operations matched query={query!r} tag={tag!r}.\n'
f'Available tags: {", ".join(tags)}')
lines = [f'{o.method} {o.path} [{o.tag}] ({o.auth}) {o.summary}'
for o in ops[:limit]]
more = len(ops) - len(lines)
if more > 0:
lines.append(f'... and {more} more; narrow the query or raise limit')
return '\n'.join(lines)
@mcp.tool()
def shopdb_describe_operation(method: str, path: str) -> str:
"""Full contract for one operation: parameters, request body, responses.
`path` must be the templated form exactly as shopdb_search_operations
printed it, for example /api/machines/{machineid} - not a filled-in URL.
Read this before calling anything you have not called before. It also warns
when the operation writes or deletes.
"""
try:
op = client().find_operation(method, path)
except ShopdbError as exc:
return _fail(str(exc))
if op is None:
return _fail(f'no operation {method.upper()} {path}. '
'Use shopdb_search_operations to find the exact path.')
out = {
'method': method.upper(),
'path': path,
'summary': op.get('summary'),
'description': op.get('description'),
'parameters': op.get('parameters'),
'requestBody': op.get('requestBody'),
'responses': {k: {'description': v.get('description')}
for k, v in (op.get('responses') or {}).items()},
}
warning = ''
if method.upper() in DESTRUCTIVE_METHODS:
warning = ('\n\nWARNING: this DELETES data and cannot be undone. '
'Confirm with the user before calling it.')
elif method.upper() in WRITE_METHODS:
warning = '\n\nNote: this MODIFIES data.'
return json.dumps(out, indent=2, default=str) + warning
@mcp.tool()
def shopdb_call(method: str, path: str, params: dict[str, Any] | None = None,
body: dict[str, Any] | None = None) -> str:
"""Call any ShopDB operation. Use shopdb_describe_operation first.
`path` may contain filled-in ids (/api/machines/42). `params` become the
query string; `body` is sent as JSON.
Every response is the envelope {status, data, meta} - the payload you want
is under "data", not at the top level. List endpoints paginate with page and
perpage, and perpage is silently capped at 100, so asking for 500 returns
100 without an error.
This can write and delete if the token allows it. For DELETE, confirm with
the user first.
"""
try:
c = client()
except ShopdbError as exc:
return _fail(str(exc))
verb = method.upper()
try:
status, payload = c.request(verb, path, params=params, body=body)
except ShopdbError as exc:
return _fail(str(exc))
if status == 401:
return _fail('HTTP 401. The token is missing, expired or rejected.')
if status == 403:
return _fail('HTTP 403. The token lacks the scope or role for this '
'operation - check shopdb_whoami.')
if status == 404 and isinstance(payload, str):
return _fail(f'HTTP 404 and a non-JSON body, which usually means the '
f'path is wrong rather than the record missing: {path}')
text = payload if isinstance(payload, str) else json.dumps(payload, indent=2,
default=str)
if len(text) > 100_000:
text = text[:100_000] + '\n... truncated; narrow with params or paginate'
return f'HTTP {status}\n{text}'
@mcp.tool()
def shopdb_llms_guide() -> str:
"""The instance's own orientation guide for agents.
Short. Covers the response envelope, authentication and how the rest of the
documentation is laid out. Worth reading once at the start of a session.
"""
try:
status, text = client().request('GET', '/api/docs/llms.txt')
except ShopdbError as exc:
return _fail(str(exc))
if status != 200:
return _fail(f'guide unavailable (HTTP {status})')
return text if isinstance(text, str) else str(text)
def main() -> None:
if not os.environ.get('SHOPDB_URL'):
raise SystemExit(
'SHOPDB_URL is not set.\n'
' SHOPDB_URL=https://shopdb.example.net/shopdb\n'
' SHOPDB_TOKEN=<personal access token>\n'
'See tools/mcp/README.md.')
mcp.run()
if __name__ == '__main__':
main()