Files
shopdb-flask/.github/GE-ENFORCE-COLLECTOR-FIX.html
Proudlock 5cd5332397
Some checks failed
CI / backend (push) Has been cancelled
CI / naming (push) Has been cancelled
CI / gates (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / lean-build (push) Has been cancelled
Add agent fingerprinting, client certificate auth, and security audit documentation
This commit adds:
- Database migrations for app_delta fingerprints and client certificates
- Scripts for certificate generation and fingerprint backfilling
- Comprehensive security audit reports and findings tracking
- Implementation documentation for certificate authentication
- Test utilities for certificate renewal and fingerprinting
- GE-ENFORCE collector fix documentation

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-09-15 09:18:12 -04:00

749 lines
25 KiB
HTML

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>GE-Enforce Collector Fix - API Key Deployment via Intune</title>
<style>
body {
font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
line-height: 1.6;
max-width: 1200px;
margin: 0 auto;
padding: 20px;
background: #f5f5f5;
}
.container {
background: white;
padding: 30px;
border-radius: 8px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
}
h1 {
color: #003087;
border-bottom: 3px solid #003087;
padding-bottom: 10px;
}
h2 {
color: #005eb8;
margin-top: 30px;
border-left: 4px solid #005eb8;
padding-left: 15px;
}
h3 {
color: #0077c8;
}
.alert {
padding: 15px;
margin: 20px 0;
border-radius: 4px;
border-left: 4px solid;
}
.alert-danger {
background: #f8d7da;
border-color: #dc3545;
color: #721c24;
}
.alert-warning {
background: #fff3cd;
border-color: #ffc107;
color: #856404;
}
.alert-success {
background: #d4edda;
border-color: #28a745;
color: #155724;
}
.alert-info {
background: #d1ecf1;
border-color: #17a2b8;
color: #0c5460;
}
pre {
background: #2b2b2b;
color: #f8f8f2;
padding: 15px;
border-radius: 4px;
overflow-x: auto;
font-family: 'Consolas', 'Monaco', monospace;
font-size: 14px;
}
code {
background: #f4f4f4;
padding: 2px 6px;
border-radius: 3px;
font-family: 'Consolas', 'Monaco', monospace;
color: #c7254e;
}
pre code {
background: transparent;
color: #f8f8f2;
padding: 0;
}
.step {
background: #f8f9fa;
border-left: 4px solid #007bff;
padding: 15px;
margin: 15px 0;
border-radius: 4px;
}
.step-number {
display: inline-block;
background: #007bff;
color: white;
width: 30px;
height: 30px;
line-height: 30px;
text-align: center;
border-radius: 50%;
margin-right: 10px;
font-weight: bold;
}
table {
width: 100%;
border-collapse: collapse;
margin: 20px 0;
}
th, td {
border: 1px solid #ddd;
padding: 12px;
text-align: left;
}
th {
background: #003087;
color: white;
}
tr:nth-child(even) {
background: #f9f9f9;
}
.badge {
display: inline-block;
padding: 4px 8px;
border-radius: 3px;
font-size: 12px;
font-weight: bold;
}
.badge-primary {
background: #007bff;
color: white;
}
.badge-success {
background: #28a745;
color: white;
}
.badge-warning {
background: #ffc107;
color: black;
}
.toc {
background: #f8f9fa;
border: 1px solid #dee2e6;
padding: 20px;
border-radius: 4px;
margin: 20px 0;
}
.toc ul {
list-style: none;
padding-left: 20px;
}
.toc a {
color: #007bff;
text-decoration: none;
}
.toc a:hover {
text-decoration: underline;
}
@media print {
body {
background: white;
}
.container {
box-shadow: none;
}
pre {
border: 1px solid #ccc;
}
}
</style>
</head>
<body>
<div class="container">
<h1>GE-Enforce Collector API Key Fix</h1>
<p><strong>Issue Date:</strong> Identified September 14, 2026</p>
<p><strong>Last Check-in:</strong> August 28, 2026 at 9:55 PM</p>
<p><strong>Affected Systems:</strong> All GE-Enforce client machines</p>
<div class="alert alert-danger">
<strong>CRITICAL:</strong> GE-Enforce profiles stopped checking in on August 28, 2026.
Collector API token was likely revoked. Immediate action required to restore service.
</div>
<!-- Table of Contents -->
<div class="toc">
<h3>Table of Contents</h3>
<ul>
<li><a href="#problem">1. Problem Summary</a></li>
<li><a href="#diagnosis">2. Diagnosis</a></li>
<li><a href="#quick-fix">3. Quick Fix (Database)</a></li>
<li><a href="#intune-deployment">4. Intune DSC Deployment</a>
<ul>
<li><a href="#option1">Option 1: PowerShell Script (Recommended)</a></li>
<li><a href="#option2">Option 2: Configuration Profile</a></li>
<li><a href="#option3">Option 3: Full DSC Configuration</a></li>
</ul>
</li>
<li><a href="#token-generation">5. Generate New API Token</a></li>
<li><a href="#testing">6. Testing</a></li>
<li><a href="#monitoring">7. Monitoring</a></li>
<li><a href="#rollback">8. Rollback Plan</a></li>
</ul>
</div>
<!-- Problem Summary -->
<h2 id="problem">1. Problem Summary</h2>
<h3>What Happened</h3>
<p>On <strong>August 28, 2026</strong>, a major code deployment introduced a new collector authentication system
that supports per-plugin managed tokens. As part of this change:</p>
<ul>
<li>Plugins can now disable the <code>.env</code> API key fallback via database setting</li>
<li>GE-Enforce collector token was likely revoked during deployment</li>
<li>All GE-Enforce clients stopped checking in after 9:55 PM on August 28</li>
</ul>
<h3>Technical Details</h3>
<p><strong>Code Change:</strong> Commit <code>80bb7b5ba1</code> - "Merge wilm: endpoints plugin, agent, policies"</p>
<p><strong>Affected File:</strong> <code>shopdb/core/api/collector.py</code></p>
<div class="step">
<strong>Before August 28:</strong>
<pre><code>if api_key == expected_key:
return None # Authorized</code></pre>
</div>
<div class="step">
<strong>After August 28:</strong>
<pre><code>if expected_key and api_key == expected_key \
and not _env_fallback_disabled(pluginname): # NEW CHECK
return None # Authorized</code></pre>
</div>
<!-- Diagnosis -->
<h2 id="diagnosis">2. Diagnosis</h2>
<h3>Check Production Database</h3>
<pre><code>-- Check if .env fallback is disabled
SELECT * FROM settings
WHERE key = 'plugin.geenforce.envkeyfallback';
-- Expected result if disabled: value = 'false'
-- Expected result if enabled: no rows or value = 'true'
-- Check last check-in
SELECT MAX(receivedat) as last_checkin,
COUNT(DISTINCT hostname) as unique_hosts
FROM manifestenforcementreports
WHERE iscurrent = 1;
-- Check managed tokens
SELECT * FROM apitokens
WHERE scope LIKE '%geenforce%'
ORDER BY createdat DESC;</code></pre>
<h3>Check IIS Logs</h3>
<p>Look for collector requests failing:</p>
<pre><code>POST /api/collector/geenforce
Status: 401 Unauthorized
Error: "Invalid API key" or "Token revoked"</code></pre>
<!-- Quick Fix -->
<h2 id="quick-fix">3. Quick Fix (Database)</h2>
<div class="alert alert-warning">
<strong>Emergency Fix:</strong> Re-enable .env API key fallback to restore service immediately.
</div>
<div class="step">
<span class="step-number">1</span>
<strong>Connect to production database</strong>
</div>
<div class="step">
<span class="step-number">2</span>
<strong>Re-enable .env fallback</strong>
<pre><code>-- Option A: Delete the setting (defaults to enabled)
DELETE FROM settings
WHERE key = 'plugin.geenforce.envkeyfallback';
-- Option B: Explicitly set to true
INSERT INTO settings (key, value, valuetype)
VALUES ('plugin.geenforce.envkeyfallback', 'true', 'boolean')
ON DUPLICATE KEY UPDATE value = 'true';</code></pre>
</div>
<div class="step">
<span class="step-number">3</span>
<strong>Verify clients start checking in</strong>
<pre><code>-- Wait 15-30 minutes, then check
SELECT MAX(receivedat) as last_checkin
FROM manifestenforcementreports
WHERE receivedat > NOW() - INTERVAL 1 HOUR;</code></pre>
</div>
<div class="alert alert-info">
<strong>Note:</strong> This assumes your <code>.env</code> file has a valid <code>COLLECTOR_API_KEY</code>
and all GE-Enforce clients are configured with that key.
</div>
<!-- Intune Deployment -->
<h2 id="intune-deployment">4. Intune DSC Deployment</h2>
<p>Deploy the API key to all GE-Enforce client machines via Intune.</p>
<h3>GE-Enforce Registry Configuration</h3>
<table>
<tr>
<th>Registry Path</th>
<th>Value Name</th>
<th>Value Type</th>
<th>Purpose</th>
</tr>
<tr>
<td><code>HKLM:\SOFTWARE\GE\ShopDB</code></td>
<td><code>BaseUrl</code></td>
<td>REG_SZ</td>
<td>ShopDB server URL</td>
</tr>
<tr>
<td><code>HKLM:\SOFTWARE\GE\ShopDB</code></td>
<td><code>ApiToken</code></td>
<td>REG_SZ</td>
<td>Collector API key</td>
</tr>
</table>
<!-- Option 1 -->
<h3 id="option1">Option 1: PowerShell Script <span class="badge badge-success">RECOMMENDED</span></h3>
<p><strong>Deployment Path:</strong> Intune Portal > Devices > Scripts > Add Windows 10 script</p>
<div class="step">
<span class="step-number">1</span>
<strong>Create PowerShell script</strong>
<p>Save as <code>Deploy-GEEnforceAPIKey.ps1</code>:</p>
</div>
<pre><code>&lt;#
.SYNOPSIS
Deploy GE-Enforce collector API key via Intune
.DESCRIPTION
Writes the ShopDB API token to registry for GE-Enforce client.
Run as SYSTEM, no user context needed.
.NOTES
Deploy via: Intune > Devices > Scripts > Add
Run as: SYSTEM
Platform: Windows 10/11
#>
$ErrorActionPreference = 'Stop'
# ============================================
# CONFIGURATION - UPDATE THESE VALUES
# ============================================
$ShopdbBaseUrl = "https://shopdb.your-company.com"
$ShopdbApiToken = "shopdb_pat_YOUR_NEW_TOKEN_HERE" # Replace with actual token
# ============================================
# REGISTRY CONFIGURATION
# ============================================
$RegPath = 'HKLM:\SOFTWARE\GE\ShopDB'
try {
Write-Output "Starting GE-Enforce API key deployment..."
# Create registry key if it doesn't exist
if (-not (Test-Path $RegPath)) {
New-Item -Path $RegPath -Force | Out-Null
Write-Output "Created registry path: $RegPath"
}
# Set BaseUrl
Set-ItemProperty -Path $RegPath -Name 'BaseUrl' -Value $ShopdbBaseUrl -Type String
Write-Output "Set BaseUrl: $ShopdbBaseUrl"
# Set ApiToken (REG_SZ - plain text, registry ACL protects it)
Set-ItemProperty -Path $RegPath -Name 'ApiToken' -Value $ShopdbApiToken -Type String
# Verify token was written
$savedToken = Get-ItemProperty -Path $RegPath -Name 'ApiToken' -ErrorAction Stop
if ($savedToken.ApiToken -eq $ShopdbApiToken) {
Write-Output "Set ApiToken: [REDACTED - first 20 chars: $($ShopdbApiToken.Substring(0, 20))...]"
Write-Output "SUCCESS: GE-Enforce API configuration deployed"
exit 0
} else {
throw "Token verification failed"
}
} catch {
Write-Error "FAILED: $_"
Write-Output "Registry path: $RegPath"
Write-Output "Error details: $($_.Exception.Message)"
exit 1
}</code></pre>
<div class="step">
<span class="step-number">2</span>
<strong>Configure Intune deployment</strong>
<ul>
<li><strong>Name:</strong> Deploy GE-Enforce Collector API Key</li>
<li><strong>Description:</strong> Updates ShopDB API token for GE-Enforce clients</li>
<li><strong>Script location:</strong> Upload <code>Deploy-GEEnforceAPIKey.ps1</code></li>
<li><strong>Run this script using logged-on credentials:</strong> No (run as SYSTEM)</li>
<li><strong>Enforce script signature check:</strong> No</li>
<li><strong>Run script in 64-bit PowerShell:</strong> Yes</li>
</ul>
</div>
<div class="step">
<span class="step-number">3</span>
<strong>Assign to devices</strong>
<ul>
<li><strong>Pilot Group:</strong> Deploy to test group first (5-10 machines)</li>
<li><strong>Production:</strong> All devices with GE-Enforce installed</li>
<li><strong>Schedule:</strong> Run once (not recurring)</li>
</ul>
</div>
<!-- Option 2 -->
<h3 id="option2">Option 2: Configuration Profile <span class="badge badge-primary">ENTERPRISE</span></h3>
<p><strong>Deployment Path:</strong> Intune Portal > Devices > Configuration profiles > Create > Settings catalog</p>
<div class="step">
<span class="step-number">1</span>
<strong>Create Settings Catalog profile</strong>
<ul>
<li><strong>Name:</strong> GE-Enforce Collector Configuration</li>
<li><strong>Platform:</strong> Windows 10 and later</li>
</ul>
</div>
<div class="step">
<span class="step-number">2</span>
<strong>Add Settings</strong>
<p><strong>Setting 1: BaseUrl</strong></p>
<ul>
<li>Category: Administrative Templates > Registry</li>
<li>Key Path: <code>HKLM\SOFTWARE\GE\ShopDB</code></li>
<li>Value Name: <code>BaseUrl</code></li>
<li>Value Type: String</li>
<li>Value Data: <code>https://shopdb.your-company.com</code></li>
</ul>
<p><strong>Setting 2: ApiToken</strong></p>
<ul>
<li>Category: Administrative Templates > Registry</li>
<li>Key Path: <code>HKLM\SOFTWARE\GE\ShopDB</code></li>
<li>Value Name: <code>ApiToken</code></li>
<li>Value Type: String</li>
<li>Value Data: <code>shopdb_pat_YOUR_TOKEN_HERE</code></li>
</ul>
</div>
<div class="step">
<span class="step-number">3</span>
<strong>Assign and deploy</strong>
</div>
<!-- Option 3 -->
<h3 id="option3">Option 3: Full DSC Configuration <span class="badge badge-primary">ADVANCED</span></h3>
<p>For Azure Automation DSC environments:</p>
<pre><code>Configuration GEEnforceCollector {
param(
[Parameter(Mandatory)]
[string]$BaseUrl,
[Parameter(Mandatory)]
[string]$ApiToken
)
Import-DscResource -ModuleName 'PSDesiredStateConfiguration'
Node 'localhost' {
Registry ShopDBBaseUrl {
Key = 'HKLM:\SOFTWARE\GE\ShopDB'
ValueName = 'BaseUrl'
ValueData = $BaseUrl
ValueType = 'String'
Ensure = 'Present'
}
Registry ShopDBApiToken {
Key = 'HKLM:\SOFTWARE\GE\ShopDB'
ValueName = 'ApiToken'
ValueData = $ApiToken
ValueType = 'String'
Ensure = 'Present'
DependsOn = '[Registry]ShopDBBaseUrl'
}
Script VerifyGEEnforce {
GetScript = { @{ Result = Test-Path 'HKLM:\SOFTWARE\GE\ShopDB' } }
TestScript = {
$token = Get-ItemProperty -Path 'HKLM:\SOFTWARE\GE\ShopDB' -Name 'ApiToken' -ErrorAction SilentlyContinue
return ($null -ne $token)
}
SetScript = { Write-Verbose "Registry already configured by dependencies" }
DependsOn = '[Registry]ShopDBApiToken'
}
}
}
# Compile configuration
GEEnforceCollector -BaseUrl "https://shopdb.your-company.com" `
-ApiToken "shopdb_pat_YOUR_TOKEN_HERE" `
-OutputPath "C:\DSC\GEEnforce"
# Upload to Azure Automation and assign to nodes</code></pre>
<!-- Token Generation -->
<h2 id="token-generation">5. Generate New API Token</h2>
<h3>Option A: Use Existing .env Key <span class="badge badge-success">QUICKEST</span></h3>
<p>On your production IIS server, check <code>.env</code> file:</p>
<pre><code># Look for:
COLLECTOR_API_KEY=shopdb_pat_abc123def456...
# Or plugin-specific:
COLLECTOR_API_KEY_GEENFORCE=shopdb_pat_xyz789...</code></pre>
<p>Use this token in your Intune deployment script.</p>
<h3>Option B: Generate New Managed Token <span class="badge badge-primary">RECOMMENDED</span></h3>
<div class="step">
<span class="step-number">1</span>
<strong>Connect to production server</strong>
</div>
<div class="step">
<span class="step-number">2</span>
<strong>Generate token via Flask CLI</strong>
<pre><code># SSH/RDP to IIS server
cd C:\path\to\shopdb-flask
# Generate new token
flask token create collector.geenforce "GE-Enforce Collector - Intune" `
--scope "collector.ingest.geenforce" `
--userid 1
# Output will be:
# Token created: shopdb_pat_abc123def456...
# Copy this token for Intune deployment</code></pre>
</div>
<div class="step">
<span class="step-number">3</span>
<strong>Verify token in database</strong>
<pre><code>SELECT tokenprefix, scope, description, isactive, createdat
FROM apitokens
WHERE scope LIKE '%geenforce%'
ORDER BY createdat DESC
LIMIT 1;</code></pre>
</div>
<!-- Testing -->
<h2 id="testing">6. Testing</h2>
<h3>Manual Registry Test (Before Intune Deployment)</h3>
<p>Test on a single pilot machine first:</p>
<pre><code># Run as Administrator on a test machine
Set-ItemProperty -Path 'HKLM:\SOFTWARE\GE\ShopDB' `
-Name 'BaseUrl' -Value 'https://shopdb.your-company.com'
Set-ItemProperty -Path 'HKLM:\SOFTWARE\GE\ShopDB' `
-Name 'ApiToken' -Value 'shopdb_pat_YOUR_TOKEN_HERE'
# Verify registry
Get-ItemProperty -Path 'HKLM:\SOFTWARE\GE\ShopDB'
# Run GE-Enforce manually to test
& "C:\Program Files\GE-Enforce\Invoke-ShopdbEnforce.ps1"
# Check for errors in output</code></pre>
<h3>Verify Check-in on Server</h3>
<pre><code>-- Wait 15-30 minutes after deployment, then check database
SELECT hostname, scopename, status, lastcheckin, receivedat
FROM manifestenforcementreports
WHERE receivedat > NOW() - INTERVAL 1 HOUR
ORDER BY receivedat DESC;</code></pre>
<h3>Intune Deployment Status</h3>
<p>Monitor deployment in Intune portal:</p>
<ul>
<li><strong>Devices > Scripts</strong> - Check device status</li>
<li><strong>Look for:</strong> Green checkmark = Success, Red X = Failed</li>
<li><strong>View logs:</strong> Click device > View output</li>
</ul>
<!-- Monitoring -->
<h2 id="monitoring">7. Monitoring</h2>
<h3>Create Alert for Missing Check-ins</h3>
<pre><code>-- Run this query on a schedule (every 6 hours)
-- Alert if any results returned
SELECT
'geenforce' as collector,
MAX(receivedat) as last_checkin,
TIMESTAMPDIFF(HOUR, MAX(receivedat), NOW()) as hours_since,
COUNT(DISTINCT hostname) as total_hosts
FROM manifestenforcementreports
WHERE iscurrent = 1
HAVING hours_since > 24;
-- Expected: 0 rows (all clients checking in)
-- Alert if: > 24 hours since last check-in</code></pre>
<h3>Dashboard Query</h3>
<pre><code>-- GE-Enforce fleet health dashboard
SELECT
DATE(receivedat) as report_date,
COUNT(DISTINCT hostname) as unique_hosts,
COUNT(*) as total_reports,
SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END) as successful,
SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END) as failed
FROM manifestenforcementreports
WHERE receivedat > NOW() - INTERVAL 7 DAY
GROUP BY DATE(receivedat)
ORDER BY report_date DESC;</code></pre>
<!-- Rollback -->
<h2 id="rollback">8. Rollback Plan</h2>
<div class="alert alert-warning">
<strong>If deployment fails:</strong> Follow these steps to restore service.
</div>
<h3>Step 1: Re-enable .env Fallback (Immediate)</h3>
<pre><code>DELETE FROM settings
WHERE key = 'plugin.geenforce.envkeyfallback';</code></pre>
<h3>Step 2: Deploy Old Token via Intune</h3>
<p>Re-deploy the original API key from <code>.env</code> file using same Intune script.</p>
<h3>Step 3: Verify Recovery</h3>
<pre><code>SELECT MAX(receivedat) as last_checkin
FROM manifestenforcementreports
WHERE receivedat > NOW() - INTERVAL 1 HOUR;</code></pre>
<h3>Step 4: Review Logs</h3>
<p>Check Intune device output logs for errors:</p>
<ul>
<li>Access denied to registry</li>
<li>Token format invalid</li>
<li>Script execution failure</li>
</ul>
<!-- Summary -->
<h2>Summary Checklist</h2>
<table>
<tr>
<th>Step</th>
<th>Action</th>
<th>Owner</th>
<th>Status</th>
</tr>
<tr>
<td>1</td>
<td>Verify problem in production database</td>
<td>Admin</td>
<td>☐</td>
</tr>
<tr>
<td>2</td>
<td>Re-enable .env fallback (emergency fix)</td>
<td>Admin</td>
<td>☐</td>
</tr>
<tr>
<td>3</td>
<td>Generate new API token (or get from .env)</td>
<td>Admin</td>
<td>☐</td>
</tr>
<tr>
<td>4</td>
<td>Test manually on 1 pilot machine</td>
<td>Engineer</td>
<td>☐</td>
</tr>
<tr>
<td>5</td>
<td>Create Intune PowerShell script</td>
<td>Engineer</td>
<td>☐</td>
</tr>
<tr>
<td>6</td>
<td>Deploy to pilot group (5-10 machines)</td>
<td>Admin</td>
<td>☐</td>
</tr>
<tr>
<td>7</td>
<td>Verify pilot check-ins in database</td>
<td>Engineer</td>
<td>☐</td>
</tr>
<tr>
<td>8</td>
<td>Deploy to all GE-Enforce clients</td>
<td>Admin</td>
<td>☐</td>
</tr>
<tr>
<td>9</td>
<td>Monitor check-ins for 24 hours</td>
<td>Engineer</td>
<td>☐</td>
</tr>
<tr>
<td>10</td>
<td>Create monitoring alert</td>
<td>Engineer</td>
<td>☐</td>
</tr>
</table>
<div class="alert alert-success">
<strong>Success Criteria:</strong> All GE-Enforce clients checking in within 1 hour of deployment,
no 401 errors in IIS logs, monitoring alert configured.
</div>
<hr style="margin: 40px 0;">
<p><small>
<strong>Document Version:</strong> 1.0<br>
<strong>Created:</strong> September 14, 2026<br>
<strong>Repository:</strong> <code>flask-shopdb/.github/</code><br>
<strong>Related Changes:</strong> Commit <code>80bb7b5ba1</code> (August 28, 2026)
</small></p>
</div>
</body>
</html>