Add agent fingerprinting, client certificate auth, and security audit documentation

This commit adds: - Database migrations for app_delta fingerprints and client certificates - Scripts for certificate generation and fingerprint backfilling - Comprehensive security audit reports and findings tracking - Implementation documentation for certificate authentication - Test utilities for certificate renewal and fingerprinting - GE-ENFORCE collector fix documentation Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-09-15 09:18:12 -04:00
parent 1a7bb9bd34
commit 5cd5332397
32 changed files with 8912 additions and 0 deletions
--- a/CERT-AUTH-IMPLEMENTATION.md
+++ b/CERT-AUTH-IMPLEMENTATION.md
@@ -0,0 +1,695 @@
+# Per-Endpoint Certificate Authentication - Implementation Guide
+
+## Overview
+
+Implement client certificate authentication for ShopDB endpoint agents that works across multiple deployment types:
+- **WJ Production:** IIS 2019 on Windows Server
+- **Other Sites:** Docker Compose with Nginx reverse proxy
+
+## Architecture
+
+```
+Agent (has cert) → Server (IIS or Nginx) → Flask
+                        ↓
+                   Extracts cert
+                   Passes to Flask via header/environ
+                        ↓
+                   Flask validates fingerprint
+                   against database
+```
+
+## Phase 1: Database Schema
+
+Add client certificate columns to `assets` table.
+
+**Migration:** `plugins/endpoints/migrations/versions/0021_client_certificates.py`
+
+```python
+"""Add client certificate authentication columns
+
+Revision ID: endpoints0021clientcerts
+Revises: endpoints0020appfingerprints
+Create Date: 2026-09-14
+"""
+
+from alembic import op
+import sqlalchemy as sa
+
+revision = 'endpoints0021clientcerts'
+down_revision = 'endpoints0020appfingerprints'
+branch_labels = None
+depends_on = None
+plugin_name = 'endpoints'
+
+
+def upgrade():
+    # Add client cert columns to assets table
+    op.add_column('assets',
+        sa.Column('client_cert_fingerprint', sa.String(64), nullable=True))
+    
+    op.add_column('assets',
+        sa.Column('cert_issued_at', sa.DateTime, nullable=True))
+    
+    op.add_column('assets',
+        sa.Column('cert_expires_at', sa.DateTime, nullable=True))
+    
+    # Add index for fast fingerprint lookup during authentication
+    op.create_index(
+        'idx_assets_cert_fingerprint',
+        'assets',
+        ['client_cert_fingerprint'],
+        unique=False
+    )
+
+
+def downgrade():
+    op.drop_index('idx_assets_cert_fingerprint', 'assets')
+    op.drop_column('assets', 'cert_expires_at')
+    op.drop_column('assets', 'cert_issued_at')
+    op.drop_column('assets', 'client_cert_fingerprint')
+```
+
+**Run migration:**
+```bash
+docker compose exec api flask db upgrade
+```
+
+---
+
+## Phase 2: Flask Code (Universal)
+
+Add client certificate authentication support that works with both IIS and Nginx.
+
+**File:** `shopdb/core/auth/client_cert.py` (new file)
+
+```python
+"""Client certificate authentication helpers.
+
+Extracts client certificate fingerprints from various deployment types:
+- IIS 2019: SSL_CLIENT_CERT environ (PEM format)
+- IIS with ARR: X-ARR-ClientCert header (base64 DER)
+- Nginx: X-Client-Cert-Fingerprint header (pre-computed)
+"""
+
+import hashlib
+import base64
+from flask import request, current_app
+from cryptography import x509
+from cryptography.hazmat.backends import default_backend
+from cryptography.hazmat.primitives import serialization
+
+
+def get_client_cert_fingerprint():
+    """Extract client certificate SHA256 fingerprint from request.
+    
+    Returns:
+        str: 64-character lowercase hex fingerprint, or None if no cert
+    
+    Supports multiple deployment types:
+    - Nginx: reads X-Client-Cert-Fingerprint header (SHA256, colon-separated)
+    - IIS direct: reads SSL_CLIENT_CERT environ (PEM format)
+    - IIS with ARR: reads X-ARR-ClientCert header (base64 DER)
+    """
+    
+    # Option 1: Nginx passes pre-computed fingerprint
+    # Format: "AB:CD:EF:..." (colon-separated uppercase hex)
+    nginx_fp = request.headers.get('X-Client-Cert-Fingerprint')
+    if nginx_fp:
+        # Normalize: remove colons, lowercase
+        return nginx_fp.replace(':', '').lower()
+    
+    # Option 2: IIS passes full cert in PEM format via WSGI environ
+    # This is the standard IIS behavior when client certs are enabled
+    iis_cert_pem = request.environ.get('SSL_CLIENT_CERT')
+    if iis_cert_pem:
+        try:
+            # Parse PEM certificate
+            if isinstance(iis_cert_pem, str):
+                iis_cert_pem = iis_cert_pem.encode('utf-8')
+            
+            cert = x509.load_pem_x509_certificate(iis_cert_pem, default_backend())
+            
+            # Convert to DER and hash
+            cert_der = cert.public_bytes(serialization.Encoding.DER)
+            fingerprint = hashlib.sha256(cert_der).hexdigest()
+            
+            current_app.logger.debug(f'[client_cert] Extracted from IIS SSL_CLIENT_CERT: {fingerprint[:16]}...')
+            return fingerprint
+        except Exception as e:
+            current_app.logger.warning(f'[client_cert] Failed to parse IIS cert: {e}')
+            return None
+    
+    # Option 3: IIS with ARR (Application Request Routing) passes base64-encoded DER
+    arr_cert_b64 = request.headers.get('X-ARR-ClientCert')
+    if arr_cert_b64:
+        try:
+            # Decode base64 to get DER bytes
+            cert_der = base64.b64decode(arr_cert_b64)
+            fingerprint = hashlib.sha256(cert_der).hexdigest()
+            
+            current_app.logger.debug(f'[client_cert] Extracted from ARR header: {fingerprint[:16]}...')
+            return fingerprint
+        except Exception as e:
+            current_app.logger.warning(f'[client_cert] Failed to parse ARR cert: {e}')
+            return None
+    
+    # No client certificate present
+    return None
+
+
+def authenticate_by_client_cert():
+    """Authenticate request by client certificate.
+    
+    Returns:
+        Asset: The authenticated asset, or None if cert auth fails
+    
+    Checks:
+    1. Client cert present in request?
+    2. Fingerprint matches an asset in database?
+    3. Certificate not expired?
+    
+    Does NOT raise exceptions - returns None on any failure.
+    Caller decides whether to fall back to other auth methods.
+    """
+    from shopdb.api import Asset
+    from datetime import datetime, timezone
+    
+    fingerprint = get_client_cert_fingerprint()
+    if not fingerprint:
+        return None
+    
+    # Look up asset by cert fingerprint
+    asset = Asset.query.filter_by(client_cert_fingerprint=fingerprint).first()
+    if not asset:
+        current_app.logger.warning(
+            f'[client_cert] Unknown fingerprint: {fingerprint[:16]}... '
+            f'(not associated with any asset)'
+        )
+        return None
+    
+    # Check certificate expiration
+    if asset.cert_expires_at:
+        now = datetime.now(timezone.utc).replace(tzinfo=None)
+        if asset.cert_expires_at < now:
+            current_app.logger.warning(
+                f'[client_cert] Expired certificate for asset {asset.assetid} '
+                f'(expired {asset.cert_expires_at})'
+            )
+            return None
+    
+    current_app.logger.info(
+        f'[client_cert] Authenticated asset {asset.assetid} '
+        f'via certificate {fingerprint[:16]}...'
+    )
+    
+    return asset
+```
+
+**File:** `plugins/endpoints/plugin.py` (modify `apply_collector_payload`)
+
+```python
+# Add import at top
+from shopdb.core.auth.client_cert import authenticate_by_client_cert
+
+# In apply_collector_payload method, BEFORE existing auth check:
+def apply_collector_payload(self, payload: Dict) -> Dict:
+    """Process endpoint collector payload (ADR-006 contract)."""
+    
+    # CERT AUTH FIRST: Check for client certificate
+    cert_asset = authenticate_by_client_cert()
+    if cert_asset:
+        # Client cert authenticated - use this asset
+        assetid = cert_asset.assetid
+        hostname = payload.get('hostname', 'UNKNOWN')
+        
+        current_app.logger.info(
+            f'[collector] asset_id={assetid} authenticated via client certificate'
+        )
+        
+        # Update cert metadata if agent provides new expiration
+        if 'cert_expires_at' in payload:
+            cert_asset.cert_expires_at = payload['cert_expires_at']
+            db.session.flush()
+    else:
+        # FALLBACK: Existing CollectorKey authentication
+        # (your existing _collector_auth_or_error logic here)
+        pass
+    
+    # Rest of method unchanged...
+```
+
+---
+
+## Phase 3: IIS 2019 Configuration (WJ Production)
+
+### Step 1: Enable Client Certificates in IIS
+
+1. Open **IIS Manager**
+2. Select your ShopDB site (not server level)
+3. Open **SSL Settings**
+4. Check **"Accept"** (not "Require" - we want dual-auth during rollout)
+5. Click **Apply**
+
+### Step 2: Verify Certificate Mapping (Optional)
+
+IIS will automatically pass client certs to WSGI via `SSL_CLIENT_CERT` environ variable - no additional config needed.
+
+If using ARR (Application Request Routing):
+1. Open **URL Rewrite** for your site
+2. Add **Server Variable**: `HTTP_X_ARR_ClientCert`
+3. Set value to: `{HTTP_SSL_CLIENT_CERT}`
+
+### Step 3: Test with curl
+
+```powershell
+# Generate test client cert (PowerShell)
+$cert = New-SelfSignedCertificate `
+    -Subject "CN=TEST-ENDPOINT" `
+    -CertStoreLocation "Cert:\CurrentUser\My" `
+    -KeyExportPolicy Exportable `
+    -NotAfter (Get-Date).AddYears(2)
+
+# Export to PFX
+$pwd = ConvertTo-SecureString -String "test123" -Force -AsPlainText
+Export-PfxCertificate `
+    -Cert $cert `
+    -FilePath "C:\temp\test-client.pfx" `
+    -Password $pwd
+
+# Test with curl (Git Bash on Windows)
+curl -k --cert-type P12 --cert C:/temp/test-client.pfx:test123 \
+    https://localhost/api/endpoints/test/client-cert
+```
+
+**Expected response:**
+```json
+{
+  "data": {
+    "message": "Client cert test endpoint",
+    "ssl_client_cert_present": true,
+    "fingerprint_sha256": "325e42ef..."
+  }
+}
+```
+
+### Step 4: Configure Agent to Use Certificate
+
+**Agent change (C# .NET):**
+
+```csharp
+// ShopDbEndpointAgent.Core/EndpointCollector.cs
+
+private static HttpClient CreateHttpClient(string certThumbprint = null)
+{
+    var handler = new HttpClientHandler
+    {
+        ServerCertificateCustomValidationCallback = (msg, cert, chain, errors) =>
+        {
+            // Accept self-signed server cert (existing behavior)
+            return true;
+        }
+    };
+
+    // If client cert configured, attach it
+    if (!string.IsNullOrEmpty(certThumbprint))
+    {
+        var cert = GetClientCertFromStore(certThumbprint);
+        if (cert != null)
+        {
+            handler.ClientCertificates.Add(cert);
+            Log.Information($"Using client certificate: {cert.Subject}");
+        }
+    }
+
+    return new HttpClient(handler)
+    {
+        Timeout = TimeSpan.FromSeconds(90)
+    };
+}
+
+private static X509Certificate2 GetClientCertFromStore(string thumbprint)
+{
+    using (var store = new X509Store(StoreName.My, StoreLocation.LocalMachine))
+    {
+        store.Open(OpenFlags.ReadOnly);
+        var certs = store.Certificates.Find(
+            X509FindType.FindByThumbprint,
+            thumbprint,
+            validOnly: false
+        );
+        return certs.Count > 0 ? certs[0] : null;
+    }
+}
+```
+
+**Config file:** `C:\ProgramData\ShopDB\config.json`
+
+```json
+{
+  "apiUrl": "https://shopdb.wj.local/api",
+  "clientCertThumbprint": "325E42EFA401CDA4B75BB7B478CFE01FBD5EE956C629ADD3A075A74847D71F77"
+}
+```
+
+---
+
+## Phase 4: Docker + Nginx Configuration (Other Sites)
+
+### Step 1: Add nginx.conf
+
+**File:** `nginx.conf` (create in repo root)
+
+```nginx
+events {
+    worker_connections 2048;
+}
+
+http {
+    upstream flask_backend {
+        server api:5001;
+    }
+
+    server {
+        listen 443 ssl;
+        server_name _;
+
+        # Server certificate (HTTPS)
+        ssl_certificate /etc/nginx/certs/server.crt;
+        ssl_certificate_key /etc/nginx/certs/server.key;
+
+        # Client certificate validation (optional during rollout)
+        # optional_no_ca: accept cert but don't validate against CA
+        # Flask validates fingerprint against database
+        ssl_verify_client optional_no_ca;
+
+        # Increase timeouts for long-polling agents
+        proxy_connect_timeout 90s;
+        proxy_send_timeout 90s;
+        proxy_read_timeout 90s;
+
+        # Health check endpoint
+        location /api/dashboard/health {
+            proxy_pass http://flask_backend;
+            proxy_http_version 1.1;
+            proxy_set_header Connection "";
+        }
+
+        # All other routes
+        location / {
+            proxy_pass http://flask_backend;
+            proxy_http_version 1.1;
+            proxy_set_header Connection "";
+            
+            # Pass real client IP (not nginx IP)
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+            proxy_set_header Host $host;
+            
+            # Pass client certificate fingerprint to Flask
+            # $ssl_client_fingerprint format: "AB:CD:EF:..." (colon-separated hex)
+            proxy_set_header X-Client-Cert-Fingerprint $ssl_client_fingerprint;
+            proxy_set_header X-Client-Cert-S-DN $ssl_client_s_dn;
+        }
+
+        # Access log includes client cert status
+        access_log /dev/stdout combined;
+        error_log /dev/stderr warn;
+    }
+}
+```
+
+### Step 2: Update docker-compose.yml
+
+```yaml
+services:
+  nginx:
+    image: nginx:1.27-alpine
+    restart: unless-stopped
+    depends_on:
+      - api
+    ports:
+      - "5001:443"
+    volumes:
+      - ./nginx.conf:/etc/nginx/nginx.conf:ro
+      - ./certs:/etc/nginx/certs:ro
+    healthcheck:
+      test: ["CMD", "wget", "--spider", "-q", "--no-check-certificate", "https://localhost/api/dashboard/health"]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+
+  api:
+    # ... existing config ...
+    ports:
+      # Expose on localhost only (for debugging)
+      - "127.0.0.1:5002:5001"
+    command:
+      - gunicorn
+      - --bind
+      - 0.0.0.0:5001
+      # REMOVE: --certfile (nginx does TLS now)
+      # REMOVE: --keyfile
+      # REMOVE: --cert-reqs
+      - --worker-class
+      - gevent
+      - --workers
+      - "4"
+      - --worker-connections
+      - "2000"
+      - --timeout
+      - "60"
+      - --access-logfile
+      - "-"
+      - --access-logformat
+      - '%(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s %(D)sus "%(f)s" "%(a)s"'
+      - --error-logfile
+      - "-"
+      - wsgi:app
+    # Remove healthcheck (nginx handles it now)
+```
+
+### Step 3: Rebuild and Deploy
+
+```bash
+# Stop current stack
+docker compose down
+
+# Start with nginx
+docker compose up -d
+
+# Verify nginx is proxying
+curl -k https://localhost:5001/api/dashboard/health
+
+# Test with client cert (same cert from earlier test)
+openssl s_client -connect localhost:5001 \
+    -cert certs/test-client.crt \
+    -key certs/test-client.key \
+    -quiet <<< "GET /api/endpoints/test/client-cert HTTP/1.1
+Host: localhost:5001
+Connection: close
+
+"
+```
+
+---
+
+## Phase 5: Agent Certificate Generation
+
+### PowerShell Script (Windows agents)
+
+**File:** `scripts/generate-agent-cert.ps1`
+
+```powershell
+param(
+    [Parameter(Mandatory=$true)]
+    [string]$Hostname,
+    
+    [Parameter(Mandatory=$false)]
+    [int]$ValidityYears = 2
+)
+
+# Generate self-signed cert for this endpoint
+$cert = New-SelfSignedCertificate `
+    -Subject "CN=$Hostname" `
+    -DnsName $Hostname `
+    -CertStoreLocation "Cert:\LocalMachine\My" `
+    -KeyExportPolicy NonExportable `
+    -KeyUsage DigitalSignature, KeyEncipherment `
+    -NotAfter (Get-Date).AddYears($ValidityYears) `
+    -HashAlgorithm SHA256
+
+Write-Host "Generated certificate:"
+Write-Host "  Thumbprint: $($cert.Thumbprint)"
+Write-Host "  Subject: $($cert.Subject)"
+Write-Host "  Expires: $($cert.NotAfter)"
+
+# Compute SHA256 fingerprint (for database registration)
+$certBytes = $cert.Export([System.Security.Cryptography.X509Certificates.X509ContentType]::Cert)
+$sha256 = [System.Security.Cryptography.SHA256]::Create()
+$fingerprintBytes = $sha256.ComputeHash($certBytes)
+$fingerprint = ($fingerprintBytes | ForEach-Object { $_.ToString("x2") }) -join ""
+
+Write-Host ""
+Write-Host "Register in ShopDB with:"
+Write-Host "  UPDATE assets"
+Write-Host "  SET client_cert_fingerprint = '$fingerprint',"
+Write-Host "      cert_issued_at = CURRENT_TIMESTAMP,"
+Write-Host "      cert_expires_at = '$($cert.NotAfter.ToString("yyyy-MM-dd HH:mm:ss"))'"
+Write-Host "  WHERE hostname = '$Hostname';"
+
+# Update agent config
+$configPath = "C:\ProgramData\ShopDB\config.json"
+if (Test-Path $configPath) {
+    $config = Get-Content $configPath | ConvertFrom-Json
+    $config.clientCertThumbprint = $cert.Thumbprint
+    $config | ConvertTo-Json | Set-Content $configPath
+    Write-Host ""
+    Write-Host "Updated $configPath with thumbprint"
+}
+
+return $cert.Thumbprint
+```
+
+**Usage:**
+```powershell
+# On endpoint VM
+.\generate-agent-cert.ps1 -Hostname "DESKTOP-HF1K9M4"
+
+# Output:
+# Generated certificate:
+#   Thumbprint: 325E42EFA401CDA4B75BB7B478CFE01FBD5EE956C629ADD3A075A74847D71F77
+#   Subject: CN=DESKTOP-HF1K9M4
+#   Expires: 9/14/2028 12:00:00 PM
+#
+# Register in ShopDB with:
+#   UPDATE assets
+#   SET client_cert_fingerprint = '325e42efa401cda4b75bb7b478cfe01fbd5ee956c629add3a075a74847d71f77',
+#       cert_issued_at = CURRENT_TIMESTAMP,
+#       cert_expires_at = '2028-09-14 12:00:00'
+#   WHERE hostname = 'DESKTOP-HF1K9M4';
+```
+
+---
+
+## Rollout Sequence
+
+### Week 1: WJ Production (IIS)
+1. Deploy Flask code (cert auth + CollectorKey fallback)
+2. Configure IIS SSL Settings → "Accept" client certs
+3. Generate cert for 1 test endpoint
+4. Register cert fingerprint in database
+5. Update agent config with thumbprint
+6. Verify dual-auth works (cert + fallback)
+
+### Week 2: Remaining WJ Fleet
+1. Generate certs for remaining endpoints (PowerShell script)
+2. Register fingerprints in database
+3. Deploy updated agent configs
+
+### Week 3: Docker Sites
+1. Add nginx container to docker-compose.yml
+2. Deploy updated bundle to 1 pilot site
+3. Generate certs for that site's endpoints
+4. Register fingerprints
+5. Verify nginx proxy works
+
+### Month 2: Remaining Sites
+1. Roll out nginx updates to remaining Docker sites
+2. Generate certs per-site
+3. After 100% cert adoption: change `ssl_verify_client optional` to `require`
+
+---
+
+## Testing Checklist
+
+### Flask Tests
+- [ ] `get_client_cert_fingerprint()` returns None when no cert
+- [ ] `get_client_cert_fingerprint()` parses nginx header (X-Client-Cert-Fingerprint)
+- [ ] `get_client_cert_fingerprint()` parses IIS environ (SSL_CLIENT_CERT)
+- [ ] `authenticate_by_client_cert()` returns asset when fingerprint matches
+- [ ] `authenticate_by_client_cert()` returns None when cert expired
+- [ ] Collector falls back to CollectorKey when cert auth fails
+
+### IIS Tests
+- [ ] IIS passes `SSL_CLIENT_CERT` to Flask
+- [ ] Agent with cert authenticates successfully
+- [ ] Agent without cert falls back to CollectorKey
+- [ ] Expired cert rejected
+
+### Docker/Nginx Tests
+- [ ] Nginx forwards requests to Gunicorn
+- [ ] Nginx adds `X-Client-Cert-Fingerprint` header
+- [ ] Agent with cert authenticates via nginx
+- [ ] Agent without cert uses CollectorKey
+- [ ] Long-polling works (25+ second holds)
+
+---
+
+## Security Considerations
+
+1. **Cert Storage:** LocalMachine\My store (admin-only access)
+2. **Private Key:** Non-exportable (can't be stolen)
+3. **Expiration:** 2 years (balance security vs. rotation overhead)
+4. **Revocation:** Remove fingerprint from database (immediate)
+5. **Self-Signed:** No CA needed (air-gapped sites)
+
+---
+
+## Rollback Plan
+
+### If Flask code breaks:
+```bash
+git revert <commit>
+docker compose restart api  # or iisreset
+```
+
+### If IIS config breaks:
+1. IIS Manager → SSL Settings → Uncheck "Accept"
+2. Click Apply
+3. Agents fall back to CollectorKey immediately
+
+### If nginx breaks:
+```bash
+# Restore direct Gunicorn access
+docker compose down nginx
+# Re-add --certfile/--keyfile to api service
+docker compose up -d api
+```
+
+---
+
+## Monitoring
+
+### Metrics to Track
+- % of agents using cert auth vs. CollectorKey
+- Cert expiration warnings (30 days before)
+- Failed cert authentications (unknown fingerprint)
+- Expired cert attempts
+
+### Logs to Watch
+```
+[client_cert] Authenticated asset 2528 via certificate 325e42ef...
+[client_cert] Unknown fingerprint: 325e42ef... (not associated with any asset)
+[client_cert] Expired certificate for asset 2528 (expired 2026-09-14 12:00:00)
+```
+
+---
+
+## Next Steps
+
+1. **Create migration** (0021_client_certificates.py)
+2. **Add Flask auth code** (client_cert.py)
+3. **Test on IIS** (your WJ production)
+4. **Add nginx config** (for Docker sites)
+5. **Generate first cert** (test endpoint)
+6. **Document in wiki** (deployment runbook)
+
+**Estimated effort:**
+- Flask code: 2 hours
+- IIS config + test: 1 hour
+- Docker/nginx: 1 hour
+- Agent changes: 2 hours
+- **Total: 6 hours**
+
+**Ready to start with Flask code?**