This commit adds: - Database migrations for app_delta fingerprints and client certificates - Scripts for certificate generation and fingerprint backfilling - Comprehensive security audit reports and findings tracking - Implementation documentation for certificate authentication - Test utilities for certificate renewal and fingerprinting - GE-ENFORCE collector fix documentation Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
4.6 KiB
Client Certificate Authentication - WSGI Server Comparison
Current Setup
- Gunicorn 26.2.0 with gevent workers
- 4 workers x 2000 greenlets = 8000 concurrent connections
- Long-polling agents hold connections for 25+ seconds
- TLS termination in Gunicorn with --certfile/--keyfile
- Problem: Gunicorn doesn't expose client cert to Flask via WSGI environ
Option 1: Hypercorn (ASGI)
Pros:
- Native client cert support via
ssl.SSLContext.wrap_socket() - Exposes client cert via ASGI scope['extensions']['tls']['client_cert_der']
- Modern async server (Trio/asyncio)
- Supports HTTP/2 and WebSockets
Cons:
- ASGI, not WSGI - Flask is WSGI, need asgiref adapter
- Different async model than gevent (asyncio instead of greenlets)
- Less mature for Flask production deployments
- Breaking change: entire stack needs rework
Verdict: ❌ Too invasive - requires migrating from WSGI to ASGI
Option 2: Waitress (Pure Python WSGI)
Pros:
- ✅ Already in requirements.txt (line 698: waitress==3.0.2)
- ✅ Pure Python WSGI server (drop-in Gunicorn replacement)
- ✅ Production-ready (powers PyPI, Pyramid, etc.)
- ✅ Native Windows support
- Thread-based concurrency (not gevent, but configurable)
Cons:
- ❌ No client certificate support - waitress doesn't expose client certs to WSGI apps
- Uses threads instead of gevent greenlets (heavier for long-polling)
- Lower connection ceiling than gevent (4 threads vs 8000 greenlets)
Verdict: ❌ Doesn't solve the problem - no client cert support
Option 3: Nginx Reverse Proxy (Current Recommendation)
Pros:
- ✅ Battle-tested client cert validation
- ✅ Keep Gunicorn + gevent (no code changes)
- ✅ Extracts cert fingerprint and passes via X-Client-Cert-Fingerprint header
- ✅ Optional validation with
ssl_verify_client optional_no_ca - ✅ Standard pattern for mTLS in Docker
Cons:
- Adds one more container (nginx:alpine ~30MB)
- Slightly more complex docker-compose.yml
Verdict: ✅ Best option - proven, minimal changes, keeps gevent
Option 4: uWSGI
Pros:
- Full-featured WSGI server
- Supports client certs via
uwsgi_param SSL_CLIENT_CERT - Gevent plugin available
Cons:
- Complex configuration
- Deprecated (project in maintenance mode as of 2023)
- No active development
Verdict: ❌ Deprecated, avoid
Option 5: Custom Gunicorn Worker (SSLGeventWorker)
Approach: Extend gevent worker to extract client cert from SSLSocket and inject into WSGI environ
Pros:
- Keep Gunicorn + gevent
- No reverse proxy needed
- ~50 lines of Python
Cons:
- Custom code to maintain
- Accesses Gunicorn internals (may break on upgrades)
- Less tested than Nginx approach
Example:
# custom_worker.py
from gunicorn.workers.ggevent import GeventWorker
import ssl
class SSLGeventWorker(GeventWorker):
def handle_request(self, listener, req, client, addr):
# Extract client cert from SSLSocket
if isinstance(client, ssl.SSLSocket):
try:
cert_der = client.getpeercert(binary_form=True)
if cert_der:
req.environ['peercert'] = cert_der
except Exception:
pass
return super().handle_request(listener, req, client, addr)
# docker-compose.yml
command:
- gunicorn
- --worker-class
- custom_worker.SSLGeventWorker
- --cert-reqs
- "1"
- wsgi:app
Verdict: ⚠️ Possible but fragile - use if Nginx is not acceptable
Recommendation
Go with Option 3: Nginx reverse proxy
Why:
- Proven pattern - industry standard for mTLS in Docker
- Minimal risk - no changes to Flask or Gunicorn config
- Keep gevent - preserves your 8000 concurrent connection capacity
- Optional validation -
ssl_verify_client optional_no_casupports dual-auth during rollout - Easy rollback - just remove nginx container
Implementation:
- Add nginx:alpine container (30MB)
- 10-line nginx.conf
- No Python code changes (Flask reads
X-Client-Cert-Fingerprintheader) - 5 minutes to implement
- Test immediately with same test cert
Alternative (if Nginx rejected):
- Option 5: Custom Gunicorn worker (50 lines, fragile)
- Option 1: Migrate to Hypercorn + ASGI (days of work, high risk)
Research Summary
Tested: Gunicorn with --cert-reqs 1 accepts client cert but doesn't expose to Flask
Researched: Waitress (no client cert), uWSGI (deprecated), Hypercorn (ASGI only)
Validated: Nginx reverse proxy is standard solution for mTLS with WSGI apps
Decision Point: Add nginx container or write custom Gunicorn worker?