Add agent fingerprinting, client certificate auth, and security audit documentation

This commit adds: - Database migrations for app_delta fingerprints and client certificates - Scripts for certificate generation and fingerprint backfilling - Comprehensive security audit reports and findings tracking - Implementation documentation for certificate authentication - Test utilities for certificate renewal and fingerprinting - GE-ENFORCE collector fix documentation Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-09-15 09:18:12 -04:00
parent 1a7bb9bd34
commit 5cd5332397
32 changed files with 8912 additions and 0 deletions
--- a/CERT-AUTH-SECURITY.md
+++ b/CERT-AUTH-SECURITY.md
@@ -0,0 +1,450 @@
+# Client Certificate Authentication Security Documentation
+
+## Overview
+
+ShopDB endpoint agent uses mTLS (mutual TLS) for per-endpoint authentication. This document covers security considerations, deployment requirements, and operational procedures.
+
+**Last Updated:** 2026-09-14  
+**Status:** Production-ready with documented limitations
+
+---
+
+## Security Model
+
+### Authentication Flow
+
+1. **Agent Startup**: Loads client certificate from LocalMachine\My by thumbprint
+2. **HTTPS Connection**: Presents certificate to nginx during TLS handshake
+3. **Nginx Validation**: Accepts certificate (no CA validation - self-signed OK)
+4. **Fingerprint Extraction**: Nginx computes SHA1, passes to Flask via header
+5. **Flask Authentication**: Looks up asset by fingerprint, checks revocation/expiration
+6. **Authorization**: Verifies cert's asset matches payload's hostname asset
+7. **Fallback**: If cert auth fails, falls back to collector API key
+
+### Dual-Auth During Rollout
+
+- **Phase 1-3**: Both cert AND API key accepted (backward compatible)
+- **Phase 4**: API key disabled via `plugin.endpoints.envkeyfallback=false`
+- **Rollback**: Re-enable API key fallback instantly if cert auth breaks
+
+---
+
+## Known Limitations
+
+### 1. SHA1 Fingerprints (nginx limitation)
+
+**Issue:** Nginx `$ssl_client_fingerprint` outputs SHA1 (40 hex chars), not SHA256.
+
+**Why SHA1?**
+- Nginx does not expose SHA256 fingerprint via built-in variables
+- Would require custom nginx module or full certificate parsing in Flask
+
+**Is SHA1 Acceptable?**
+- ✅ YES for certificate fingerprinting (not signing)
+- ✅ Acceptable in air-gapped environments with:
+  - Short-lived certificates (2 years max)
+  - Certificate revocation support (cert_revoked_at column)
+  - Self-signed certificates only (no external CA trust)
+
+**SHA1 Collision Risk:**
+- Computationally expensive to generate collision (estimated $100K+ in 2026)
+- Would require attacker to generate cert with same SHA1 as legitimate endpoint
+- Air-gapped environment prevents external certificate injection
+- Revocation support allows instant kill switch if compromised
+
+**Migration to SHA256:**
+```
+Option 1: Custom nginx module to expose SHA256
+Option 2: Migrate to IIS (exposes full PEM cert for server-side hashing)
+Option 3: Pass full certificate to Flask for SHA256 computation (bandwidth cost)
+```
+
+### 2. Private Key Permissions (tray access)
+
+**Issue:** PowerShell script grants `BUILTIN\Users` read access to private key.
+
+**Why?** Tray app (running as interactive user) needs cert access for manual agent runs.
+
+**Risk:** ANY interactive user on the machine can extract the private key.
+
+**Mitigations:**
+1. **Best:** Remove tray dependency (tray calls service API, service uses cert)
+2. **Acceptable:** Grant only to specific user account running tray
+3. **Current:** Document that tray deployment requires trusted console users only
+
+**When Acceptable:**
+- Shop floor PCs with single trusted operator per machine
+- Kiosks where all users are authorized operators
+- Environments where console access = trusted
+
+**NOT Acceptable:**
+- Multi-user workstations with untrusted users
+- Shared administrator workstations
+- Development machines
+
+### 3. No Certificate Revocation List (CRL)
+
+**Issue:** No automated CRL distribution. Revocation is manual SQL UPDATE.
+
+**Workaround:**
+```sql
+-- Revoke certificate (takes effect immediately)
+UPDATE assets
+SET cert_revoked_at = CURRENT_TIMESTAMP,
+    cert_revocation_reason = 'Compromised - laptop stolen - TICKET-456'
+WHERE client_cert_fingerprint = 'abc123...';
+```
+
+**Future:** Web UI for certificate management (provision, revoke, monitor expiration).
+
+---
+
+## Security Features Implemented
+
+### ✅ Constant-Time Fingerprint Comparison
+
+Prevents timing attacks on fingerprint validation:
+
+```python
+if not hmac.compare_digest(asset.client_cert_fingerprint, fingerprint):
+    return None  # Constant time, no early exit
+```
+
+### ✅ Format Validation
+
+Rejects malformed fingerprints before database lookup:
+
+```python
+# Must be 40 hex chars (SHA1) or 64 hex chars (SHA256)
+if len(normalized) not in (40, 64):
+    return None
+if not re.match(r'^[0-9a-f]+$', normalized):
+    return None
+```
+
+### ✅ Certificate Revocation Support
+
+Admin can revoke compromised certificates instantly:
+
+```sql
+-- Revoke
+UPDATE assets SET cert_revoked_at = NOW(), cert_revocation_reason = '...' WHERE ...;
+-- Un-revoke (if cert was revoked by mistake)
+UPDATE assets SET cert_revoked_at = NULL, cert_revocation_reason = NULL WHERE ...;
+```
+
+### ✅ Expiration Enforcement
+
+Expired certificates rejected automatically:
+
+```python
+if asset.cert_expires_at < now:
+    return None  # Must regenerate and re-register
+```
+
+### ✅ Authorization Check
+
+Certificate authenticates asset X, but payload reports for asset Y → DENIED:
+
+```python
+if g.cert_authenticated_asset_id != assetid:
+    raise PermissionError(...)  # Prevents cross-asset impersonation
+```
+
+### ✅ Audit Logging
+
+All authentication attempts logged to AuditLog:
+
+- `auth.cert.success` - Successful authentication
+- `auth.cert.unknown` - Unknown fingerprint
+- `auth.cert.revoked` - Attempt to use revoked cert
+- `auth.cert.expired` - Attempt to use expired cert
+
+---
+
+## Deployment Requirements
+
+### Phase 1: Server Deployment
+
+1. **Deploy nginx with mTLS support:**
+   ```yaml
+   # docker-compose.yml
+   services:
+     nginx:
+       image: nginx:alpine
+       volumes:
+         - ./nginx.conf:/etc/nginx/nginx.conf
+         - ./certs:/etc/nginx/certs
+   ```
+
+2. **Run database migrations:**
+   ```bash
+   # Adds client_cert_fingerprint, cert_issued_at, cert_expires_at, 
+   # cert_revoked_at, cert_revocation_reason columns
+   flask db upgrade
+   ```
+
+3. **Verify fallback still works:**
+   ```bash
+   # Existing agents continue using API key
+   curl -H "X-API-Key: $COLLECTOR_API_KEY" https://shopdb/api/collector/endpoints
+   ```
+
+### Phase 2: Certificate Generation (per endpoint)
+
+1. **Run PowerShell script on endpoint:**
+   ```powershell
+   .\generate-agent-cert.ps1 -ValidityYears 2
+   ```
+
+2. **Script outputs SQL to register certificate:**
+   ```sql
+   UPDATE assets
+   SET client_cert_fingerprint = 'abc123...',
+       cert_issued_at = CURRENT_TIMESTAMP,
+       cert_expires_at = '2028-09-14 12:00:00',
+       cert_revoked_at = NULL,
+       cert_revocation_reason = NULL
+   WHERE assetnumber = 'G5QX1GT3ESF';
+   ```
+
+3. **Verify agent config updated:**
+   ```json
+   {
+     "baseurl": "https://shopdb.company.com",
+     "apikey": "collector_key_here",
+     "clientCertThumbprint": "ABC123...",
+     "cadenceseconds": 900
+   }
+   ```
+
+4. **Restart agent service:**
+   ```powershell
+   Restart-Service ShopDbEndpointAgent
+   ```
+
+5. **Verify cert auth in logs:**
+   ```
+   [client_cert] Authenticated asset 1234 (number=G5QX1GT3ESF) via certificate abc123...
+   ```
+
+### Phase 3: Fleet Rollout
+
+**Automation Required:**
+- PowerShell script to generate certs across fleet
+- CSV export of asset_number,fingerprint,expires_at
+- SQL bulk INSERT script
+- Group Policy or deployment tool to push agent config updates
+
+**Validation:**
+- Monitor `auth.cert.success` audit log entries
+- Dashboard showing cert expiration dates
+- Alert on endpoints still using API key after N days
+
+### Phase 4: Retire API Key Fallback
+
+**Only after 100% cert coverage verified:**
+
+```sql
+UPDATE settings
+SET value = 'false'
+WHERE key = 'plugin.endpoints.envkeyfallback';
+```
+
+**Revoke old collector API key tokens:**
+```python
+# In Flask admin UI or API
+revoke_service_tokens('collector.ingest.endpoints')
+```
+
+**CRITICAL:** Test rollback plan before this step!
+
+---
+
+## Operational Procedures
+
+### Certificate Rotation
+
+**When to rotate:**
+- 30 days before expiration (automated alert recommended)
+- After suspected compromise
+- After employee termination (if they had console access)
+
+**How to rotate:**
+1. Generate new certificate (keep old one installed temporarily)
+2. Update database with new fingerprint
+3. Update agent config with new thumbprint
+4. Restart agent
+5. Verify new cert working
+6. Remove old cert from certificate store
+
+### Certificate Revocation
+
+**Immediate revocation (compromised/lost endpoint):**
+
+```sql
+UPDATE assets
+SET cert_revoked_at = CURRENT_TIMESTAMP,
+    cert_revocation_reason = 'Laptop stolen - INCIDENT-789'
+WHERE assetnumber = 'G5QX1GT3ESF';
+```
+
+**Effect:** Agent's next request denied, falls back to API key if enabled.
+
+### Monitoring
+
+**Key metrics to monitor:**
+
+1. **Cert Expiration:**
+   ```sql
+   SELECT assetnumber, cert_expires_at,
+          DATEDIFF(cert_expires_at, NOW()) AS days_remaining
+   FROM assets
+   WHERE cert_expires_at IS NOT NULL
+     AND cert_expires_at < DATE_ADD(NOW(), INTERVAL 30 DAY)
+   ORDER BY cert_expires_at;
+   ```
+
+2. **Auth Failure Rate:**
+   ```sql
+   SELECT action, COUNT(*) AS count
+   FROM auditlog
+   WHERE action LIKE 'auth.cert.%'
+     AND createdat > NOW() - INTERVAL 24 HOUR
+   GROUP BY action;
+   ```
+
+3. **Endpoints Still Using API Key:**
+   ```sql
+   SELECT e.hostname, e.lastreportedat
+   FROM endpointstates e
+   JOIN assets a ON a.assetid = e.assetid
+   WHERE a.client_cert_fingerprint IS NULL
+     AND e.lastreportedat > NOW() - INTERVAL 1 HOUR;
+   ```
+
+---
+
+## Troubleshooting
+
+### Agent reports but logs show API key auth (not cert)
+
+**Cause:** Client certificate not configured or not found.
+
+**Fix:**
+```powershell
+# Verify cert exists
+Get-ChildItem Cert:\LocalMachine\My | Where-Object { $_.Subject -like "*$env:COMPUTERNAME*" }
+
+# Verify config
+Get-Content C:\ProgramData\ShopDB\config.json | ConvertFrom-Json
+
+# Verify cert has private key
+$cert = Get-Item "Cert:\LocalMachine\My\$thumbprint"
+$cert.HasPrivateKey  # Should be True
+```
+
+### "Revoked certificate" in logs but cert not revoked
+
+**Cause:** Database cert_revoked_at column set incorrectly.
+
+**Fix:**
+```sql
+UPDATE assets
+SET cert_revoked_at = NULL,
+    cert_revocation_reason = NULL
+WHERE assetnumber = 'G5QX1GT3ESF';
+```
+
+### "Unknown fingerprint" in logs
+
+**Cause:** Certificate fingerprint not registered in database.
+
+**Fix:**
+```powershell
+# Get fingerprint from cert
+$cert = Get-Item "Cert:\LocalMachine\My\$thumbprint"
+$certBytes = $cert.Export([System.Security.Cryptography.X509Certificates.X509ContentType]::Cert)
+$sha1 = [System.Security.Cryptography.SHA1]::Create()
+$fp = ($sha1.ComputeHash($certBytes) | ForEach-Object { $_.ToString("x2") }) -join ""
+
+# Register in database
+UPDATE assets SET client_cert_fingerprint = '$fp' WHERE assetnumber = 'G5QX1GT3ESF';
+```
+
+### Agent fails after cert rotation
+
+**Cause:** Old cert removed before new cert registered in database.
+
+**Fix:**
+1. Reinstall old cert temporarily
+2. Register new cert fingerprint in database
+3. Update agent config to new thumbprint
+4. Restart agent
+5. Remove old cert
+
+---
+
+## Security Best Practices
+
+### ✅ DO
+
+- Generate certificates with 2-year maximum validity
+- Rotate certificates 30 days before expiration
+- Revoke certificates immediately when endpoint compromised
+- Monitor audit logs for revoked/expired cert attempts
+- Document certificate issuance and revocation in tickets
+- Test rollback to API key before disabling fallback
+- Use short-lived certificates in untrusted environments
+
+### ❌ DON'T
+
+- Grant `BUILTIN\Users` access to private keys on multi-user machines
+- Skip hostname validation in cert generation script
+- Deploy to production without testing cert revocation
+- Disable API key fallback without verified 100% cert coverage
+- Use expired certificates (regenerate immediately)
+- Share certificates between endpoints
+- Export private keys to network shares
+
+---
+
+## Future Enhancements
+
+1. **Web UI for Certificate Management**
+   - Upload CSRs or paste fingerprints
+   - View expiration dashboard
+   - One-click revocation
+   - Automated renewal reminders
+
+2. **SHA256 Migration**
+   - Custom nginx module for SHA256
+   - Or migrate to IIS deployment
+   - Dual-column support (SHA1 + SHA256) during migration
+
+3. **Automated Certificate Rotation**
+   - Agent auto-generates new cert 30 days before expiration
+   - Server auto-accepts new fingerprint after validating old cert
+   - Zero-touch renewal
+
+4. **Per-User Certificate ACLs**
+   - Grant tray access to specific user account only
+   - Service creates runas token for tray HTTP requests
+
+---
+
+## References
+
+- **ADR-006:** Collector Contract (dual-auth model)
+- **ADR-016:** Credential Delivery (provisioning workflow)
+- **AGENT-CONTRACT.md:** Wire protocol and authentication
+- **nginx.conf:** mTLS configuration and SHA1 limitation
+- **generate-agent-cert.ps1:** Certificate generation script
+
+---
+
+## Contact
+
+For security questions or to report vulnerabilities:
+- **Internal:** File ticket in JIRA under "Security" component
+- **Emergency:** Page on-call security engineer via PagerDuty