Engineer image: upload the Engineering Layer from the web app

New Engineering Layer tab on the engineer image page. A site uploads
Engineering_Install_v<ver>.exe (keeps prereqs) or a .zip of the whole
extra_engineering payload. The upload is streamed to disk, checked (MZ
header, no unsafe zip paths) and swapped in; the replaced payload is
archived (two kept) with a Restore button. Install-EngineeringLayer.ps1
stays repo-owned and now installs the newest Engineering_Install_v*.exe.

KB: Part 4 becomes Site Content (enrollment package, Engineering Layer,
Blancco site settings, Shop Floor apps) plus a new Adding Shop Floor
Apps section for preinstall.json/postppkg.json. WJ examples removed.
This commit is contained in:
cproudlock
2026-10-16 20:48:25 -04:00
parent 04db6d6605
commit b9a4d0230d
9 changed files with 775 additions and 9 deletions

View File

@@ -420,7 +420,7 @@
}
.checkbox-list li::before {
content: "☐";
content: "\2610";
position: absolute;
left: 0;
color: var(--ge-blue-primary);
@@ -548,9 +548,10 @@
<li><a href="#usb">Part 1 - Build the Installer USB</a></li>
<li><a href="#install">Part 2 - Install the Server</a></li>
<li><a href="#images">Part 3 - Load the Images</a></li>
<li><a href="#enroll">Part 4 - Enrollment Package</a></li>
<li><a href="#enroll">Part 4 - Site Content</a></li>
<li><a href="#test">Part 5 - Image a Test PC</a></li>
<li><a href="#sitevars">Site Settings</a></li>
<li><a href="#apps">Adding Shop Floor Apps</a></li>
<li><a href="#options">Script Options</a></li>
<li><a href="#troubleshooting">Troubleshooting</a></li>
</ul>
@@ -713,8 +714,41 @@ powershell -ExecutionPolicy Bypass -File .\scripts\Build-PxeUsb.ps1 -Interactive
<hr class="section-divider">
<!-- PART 4 -->
<h2 id="enroll">Part 4 - Enrollment Package</h2>
<p>Shop Floor PCs enroll with a provisioning package. Open <strong>Tools &gt; Enrollment packages</strong>, click <strong>Upload SFLD package</strong> and upload the current <span class="command-inline">GCCH_Prod_SFLD_v&lt;version&gt;.ppkg</span>. The package you upload becomes the active one (shown on that page); upload a newer version the same way when one is released.</p>
<h2 id="enroll">Part 4 - Site Content</h2>
<p>The build folder carries no site content. Add these in the web interface; each one can be replaced later the same way.</p>
<div class="step">
<span class="step-number">Step 1</span>
<h3>Enrollment package (Shop Floor)</h3>
<p>Shop Floor PCs enroll with a provisioning package. Open <strong>Tools &gt; Enrollment packages</strong>, click <strong>Upload SFLD package</strong> and upload the current <span class="command-inline">GCCH_Prod_SFLD_v&lt;version&gt;.ppkg</span>. The package you upload becomes the active one (shown on that page); upload a newer version the same way when one is released.</p>
</div>
<div class="step">
<span class="step-number">Step 2</span>
<h3>Engineering Layer (Engineer)</h3>
<p>Engineer PCs install the Engineering Layer at first logon. Without it they still image, but with no engineering applications. Open <strong>Images &gt; Engineer</strong>, tab <strong>Engineering Layer</strong>, and upload one of:</p>
<ul>
<li>the whole payload as a <span class="command-inline">.zip</span>, with the folders <span class="command-inline">engineering_layer\</span> (holding <span class="command-inline">Engineering_Install_v&lt;version&gt;.exe</span>) and <span class="command-inline">prereqs\</span> (VC++ and .NET 3.5) at its top, or inside one <span class="command-inline">extra_engineering\</span> folder. Use this the first time.</li>
<li>only a newer <span class="command-inline">Engineering_Install_v&lt;version&gt;.exe</span>. It replaces the installer and keeps the prerequisites.</li>
</ul>
<p>The page shows the installer version in use and the status <span class="highlight">Ready</span> once both parts are present. The two previous uploads are kept under <strong>Previous versions</strong>, with a <strong>Restore</strong> button. The Engineer image must be imported (Part 3) first.</p>
</div>
<div class="step">
<span class="step-number">Step 3</span>
<h3>Blancco site settings</h3>
<p>Open <strong>Updates &gt; Blancco releases</strong> and fill in <strong>Site settings</strong>:</p>
<ul>
<li><strong>Business location</strong>: your site, as it should appear on every erasure report.</li>
<li><strong>Wi-Fi network (SSID)</strong> and <strong>Wi-Fi password</strong>: the internet-only Wi-Fi Blancco joins to license each erasure in the GE Blancco cloud. Usually <span class="command-inline">INTERNETACCESS</span>; the password differs between sites, so get yours from the site network team.</li>
</ul>
<p>Click <strong>Save</strong>. The settings go into the active Blancco release at once and into every release staged later.</p>
</div>
<div class="step">
<span class="step-number">Step 4</span>
<h3>Shop Floor apps (optional)</h3>
<p>A Shop Floor PC gets the provisioning package and the ShopFloor auto-logon and lockdown, and nothing else. To install your own applications on it, see <a href="#apps">Adding Shop Floor Apps</a>.</p>
</div>
<hr class="section-divider">
<!-- PART 5 -->
@@ -749,7 +783,7 @@ powershell -ExecutionPolicy Bypass -File .\scripts\Build-PxeUsb.ps1 -Interactive
<h3>Follow the format</h3>
<ul>
<li>One setting per line: <span class="command-inline">name: value</span>, with a space after the colon.</li>
<li>Put text in double quotes: <span class="command-inline">site_name: "West Jefferson"</span>.</li>
<li>Put text in double quotes: <span class="command-inline">site_name: "Example Site"</span>.</li>
<li>Lists go in square brackets: <span class="command-inline">corp_admin_cidrs: ["10.12.34.0/24", "10.12.40.0/24"]</span>.</li>
<li>A line starting with <span class="command-inline">#</span> is a note and is ignored. Remove the <span class="command-inline">#</span> to turn a setting on.</li>
<li>Start every setting at the beginning of the line. No tabs.</li>
@@ -765,9 +799,9 @@ powershell -ExecutionPolicy Bypass -File .\scripts\Build-PxeUsb.ps1 -Interactive
<h3>The settings</h3>
<table>
<tr><th>Setting</th><th>What it does</th><th>When to set it</th><th>Example</th></tr>
<tr><td><span class="command-inline">site_name</span></td><td>The site name stamped on Blancco erasure reports and the monthly CMMC evidence packs. Not set: <em>Unnamed site</em>. Letters, digits, spaces and . , - ( ) &amp; / only.</td><td><strong>Always</strong></td><td><span class="command-inline">"West Jefferson"</span></td></tr>
<tr><td><span class="command-inline">site_name</span></td><td>The site name stamped on Blancco erasure reports and the monthly CMMC evidence packs. Not set: <em>Unnamed site</em>. Letters, digits, spaces and . , - ( ) &amp; / only.</td><td><strong>Always</strong></td><td><span class="command-inline">"Example Site"</span></td></tr>
<tr><td><span class="command-inline">site_timezone</span></td><td>Time zone of the server clock, of every time in the web interface, and of the evidence-pack month boundaries.</td><td>If the site is not on US Eastern time</td><td><span class="command-inline">"America/Chicago"</span></td></tr>
<tr><td><span class="command-inline">clonezilla_smb_password</span></td><td>Password of the share Clonezilla backs up to and restores from. The boot menu carries it, so it protects the share on the imaging network only. 8-64 characters from A-Z a-z 0-9 . _ ~ -</td><td><strong>Always</strong> (the default is the same at every site)</td><td><span class="command-inline">"WJ-Clone-2026"</span></td></tr>
<tr><td><span class="command-inline">clonezilla_smb_password</span></td><td>Password of the share Clonezilla backs up to and restores from. The boot menu carries it, so it protects the share on the imaging network only. 8-64 characters from A-Z a-z 0-9 . _ ~ -</td><td><strong>Always</strong> (the default is the same at every site)</td><td><span class="command-inline">"ExSite-Clone-2026"</span></td></tr>
<tr><td><span class="command-inline">corp_admin_cidrs</span></td><td>The GE subnets allowed to reach the web interface (port 9009) and SSH. Not set: all of GE. The imaging network always has access.</td><td>Recommended once the server is on the GE network</td><td><span class="command-inline">["10.12.34.0/24"]</span></td></tr>
<tr><td><span class="command-inline">pxe_nic_mac</span><br><span class="command-inline">corp_nic_mac</span></td><td>Pin the imaging port and the GE port by MAC address instead of picking the imaging port at install. Then both cables may be in from the start.</td><td>Only if the automatic pick is not wanted</td><td><span class="command-inline">"34:c8:d6:b1:10:10"</span></td></tr>
<tr><td><span class="command-inline">pxe_nic_allowed_switches</span></td><td>A <em>managed</em> imaging switch sends LLDP/CDP like a GE switch, and the server would refuse to serve on it. Its name (or MAC) here marks it as the imaging switch. Comma separated.</td><td>Only if the imaging switch is managed</td><td><span class="command-inline">"IMG-SW-01"</span></td></tr>
@@ -829,6 +863,79 @@ clonezilla_smb_password: "ExSite-Clone-2026"
<hr class="section-divider">
<h2 id="apps">Adding Shop Floor Apps</h2>
<p>A Shop Floor PC installs two app lists, both empty when the server is installed:</p>
<table>
<tr><th>List</th><th>When it installs</th><th>Use it for</th></tr>
<tr><td><span class="command-inline">preinstall.json</span></td><td>At first logon, in the same run as the provisioning package, before the PC restarts</td><td>Apps that must be present before the PC enrolls</td></tr>
<tr><td><span class="command-inline">postppkg.json</span></td><td>Once, after the restart that follows the provisioning package (as SYSTEM)</td><td>Everything else</td></tr>
</table>
<p>Both are in the <span class="command-inline">pre-install</span> folder of the enrollment share. The first list shows as a step on <strong>Operate &gt; Imaging floor</strong>; each PC keeps its logs in <span class="command-inline">C:\Logs\PreInstall\install.log</span> and <span class="command-inline">C:\Logs\PostPpkg\install.log</span>.</p>
<div class="step">
<span class="step-number">Step 1</span>
<h3>Open the share</h3>
<p>From a PC on the imaging switch, open <span class="command-inline">\\172.16.9.1\enrollment\pre-install</span> and sign in as <span class="command-inline">pxe-upload</span> (password <span class="command-inline">pxe</span>, the account <span class="command-inline">Upload-Image.ps1</span> uses).</p>
</div>
<div class="step">
<span class="step-number">Step 2</span>
<h3>Copy the installer</h3>
<p>Make a folder per app under <span class="command-inline">installers\</span> and copy the installer into it, for example <span class="command-inline">installers\exampleapp\ExampleApp.msi</span>.</p>
</div>
<div class="step">
<span class="step-number">Step 3</span>
<h3>Add it to a list</h3>
<p>Open the list in Notepad and add one entry per app to <span class="command-inline">Applications</span>. Each list contains an <span class="command-inline">_example</span> entry to copy; it is ignored. Apps install in list order, and an app already detected on the PC is skipped, so a PC can be re-imaged safely.</p>
</div>
<table>
<tr><th>Field</th><th>Meaning</th><th>Required</th></tr>
<tr><td><span class="command-inline">Name</span></td><td>Shown in the log</td><td>Yes</td></tr>
<tr><td><span class="command-inline">Installer</span></td><td>Path under <span class="command-inline">installers\</span>. In JSON, write every <span class="command-inline">\</span> as <span class="command-inline">\\</span></td><td>Yes</td></tr>
<tr><td><span class="command-inline">Type</span></td><td><span class="command-inline">MSI</span> (run with msiexec /i) or <span class="command-inline">EXE</span> (run directly)</td><td>Yes</td></tr>
<tr><td><span class="command-inline">InstallArgs</span></td><td>Silent-install switches, e.g. <span class="command-inline">/qn /norestart</span> for an MSI</td><td>Usually</td></tr>
<tr><td><span class="command-inline">DetectionMethod</span></td><td><span class="command-inline">Registry</span> (a registry key, optionally one value) or <span class="command-inline">File</span> (a file or folder exists)</td><td>Recommended</td></tr>
<tr><td><span class="command-inline">DetectionPath</span></td><td>The registry key (<span class="command-inline">HKLM:\\SOFTWARE\\...</span>) or file path that proves the app is installed</td><td>With DetectionMethod</td></tr>
<tr><td><span class="command-inline">DetectionName</span><br><span class="command-inline">DetectionValue</span></td><td>Registry only: a value under the key, and the data it must have</td><td>No</td></tr>
<tr><td><span class="command-inline">PCTypes</span></td><td>Limit the app to these PC types, e.g. <span class="command-inline">["gea-shopfloor-standard"]</span>. Not set: every Shop Floor PC</td><td>No</td></tr>
<tr><td><span class="command-inline">LogFile</span></td><td>EXE only: the installer's own log, copied into the install log when it fails</td><td>No</td></tr>
</table>
<h3>Example</h3>
<div class="code-block">
<button class="copy-button" onclick="copyCode(this)">Copy</button>
<pre>{
"Version": "1.0",
"Applications": [
{
"Name": "Example App 1.2",
"Installer": "exampleapp\\ExampleApp.msi",
"Type": "MSI",
"InstallArgs": "/qn /norestart",
"DetectionMethod": "Registry",
"DetectionPath": "HKLM:\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\{00000000-0000-0000-0000-000000000000}"
},
{
"Name": "Example Tool 4.0",
"Installer": "exampletool\\setup.exe",
"Type": "EXE",
"InstallArgs": "/S",
"DetectionMethod": "File",
"DetectionPath": "C:\\Program Files\\Example Tool\\tool.exe"
}
]
}</pre>
</div>
<div class="alert alert-warning">
<strong>Test before the floor:</strong>
A broken list or a failed install never stops imaging: it is logged and the PC carries on without that app. Image one test PC after every change and check its install log.
</div>
<p>Reinstalling the server (<span class="command-inline">sudo /opt/first-boot.sh</span>) keeps both lists and the installers.</p>
<hr class="section-divider">
<h2 id="options">Script Options</h2>
<p>Only needed for special cases. Run the script from an elevated PowerShell in <span class="command-inline">C:\pxe-usb</span> with options instead of <span class="command-inline">-Interactive</span>:</p>
<div class="code-block">

View File

@@ -41,7 +41,10 @@ param(
# imaging - the unattend installs PowerShell 7 from C:\PreInstall\installers.
[string]$StagingRoot = 'C:\PreInstall\engineering',
[string]$LogDir = 'C:\Logs\Engineering',
[string]$Installer = 'engineering_layer\Engineering_Install_v3.1.exe',
# Relative to StagingRoot. Empty: the newest
# engineering_layer\Engineering_Install_v<version>.exe (uploaded on the
# PXE webapp, engineer image > Engineering Layer).
[string]$Installer = '',
[switch]$StageOnly
)
@@ -117,6 +120,18 @@ if (Test-Path -LiteralPath $marker) {
}
# --- stage the payload out of C:\Deploy -------------------------------------
function Find-Installer([string]$Dir) {
Get-ChildItem -LiteralPath (Join-Path $Dir 'engineering_layer') -Filter 'Engineering_Install_v*.exe' -File -ErrorAction SilentlyContinue |
Where-Object { $_.Name -match '^Engineering_Install_v(\d+(\.\d+)*)\.exe$' } |
Sort-Object { $v = ($_.Name -replace '^Engineering_Install_v', '') -replace '\.exe$', ''; if ($v -notmatch '\.') { $v += '.0' }; [version]$v } -Descending |
Select-Object -First 1
}
if (-not $Installer) {
$found = Find-Installer $StagingRoot
if (-not $found) { $found = Find-Installer $Source }
$Installer = if ($found) { 'engineering_layer\' + $found.Name } else { 'engineering_layer\Engineering_Install_v3.1.exe' }
}
Write-EngLog "Installer: $Installer"
$stagedExe = Join-Path $StagingRoot $Installer
if (Test-Path -LiteralPath $stagedExe) {

View File

@@ -71,6 +71,7 @@ from views import dell_updates as dell_updates_views
from views import images as images_views
from views import images_drivers as images_drivers_views
from views import images_unattend as images_unattend_views
from views import engineering_layer as engineering_layer_views
from views import ipxe as ipxe_views
app = Flask(__name__)
@@ -97,6 +98,7 @@ app.register_blueprint(bprt_sync_views.bp)
app.register_blueprint(images_views.bp)
app.register_blueprint(images_drivers_views.bp)
app.register_blueprint(images_unattend_views.bp) # Unattend review/save/undo
app.register_blueprint(engineering_layer_views.bp) # Engineer image: Engineering Layer upload
# ---------------------------------------------------------------------------

View File

@@ -0,0 +1,287 @@
"""The Engineering Layer payload of the engineer image.
The gea-engineer unattend stages and runs
Deploy/Applications/extra_engineering/Install-EngineeringLayer.ps1 (the
playbook deploys it from the repo). It installs, from beside itself:
engineering_layer/Engineering_Install_v<version>.exe (~530 MB)
prereqs/redistrib/..., prereqs/sxs/*.cab (VC++ / .NET 3.5)
A new site has none of that; this module lets it be uploaded from the
engineer image page (Engineering Layer tab):
* Engineering_Install_v<ver>.exe replaces the installer, keeps prereqs/
* a .zip with engineering_layer/ (and usually prereqs/) at its top level,
or under one extra_engineering/ folder replaces both
The upload streams to disk, is checked (an MZ executable named
Engineering_Install_v<ver>.exe; a zip with no absolute or .. paths), and
only then swapped in: the current engineering_layer/ and prereqs/ move to
UPDATES_DIR/englayer/archive/<timestamp>/ (the two newest are kept) and the
new ones are renamed into place. Install-EngineeringLayer.ps1 is never
touched (it belongs to the repo); it installs the newest
Engineering_Install_v*.exe it finds.
"""
import fcntl
import os
import re
import shutil
import tempfile
import zipfile
from datetime import datetime
import config
from services import updates_common as uc
IMAGE = "gea-engineer"
EXE = re.compile(r"^Engineering_Install_v(\d+(?:\.\d+)*)\.exe$", re.IGNORECASE)
PARTS = ("engineering_layer", "prereqs")
KEEP_ARCHIVES = 2
def root():
return os.path.join(config.SAMBA_SHARE, IMAGE, "Deploy", "Applications", "extra_engineering")
def _state(*parts):
return uc.state_dir("englayer", *parts)
def _version_key(v):
return tuple(int(x) for x in v.split("."))
def _installers(folder):
out = []
try:
names = os.listdir(folder)
except OSError:
return out
for n in names:
m = EXE.match(n)
p = os.path.join(folder, n)
if m and os.path.isfile(p):
st = os.stat(p)
out.append({"name": n, "version": m.group(1), "size": st.st_size, "mtime": st.st_mtime})
return sorted(out, key=lambda x: _version_key(x["version"]), reverse=True)
def _tree_size(path):
total, count = 0, 0
for dirpath, _, files in os.walk(path):
for f in files:
try:
total += os.path.getsize(os.path.join(dirpath, f))
count += 1
except OSError:
pass
return total, count
def status():
r = root()
inst = _installers(os.path.join(r, "engineering_layer"))
pre = os.path.join(r, "prereqs")
pre_size, pre_files = _tree_size(pre) if os.path.isdir(pre) else (0, 0)
archives = []
adir = _state("archive")
for name in sorted(os.listdir(adir), reverse=True):
p = os.path.join(adir, name)
if os.path.isdir(p):
a_inst = _installers(os.path.join(p, "engineering_layer"))
archives.append({"name": name, "version": a_inst[0]["version"] if a_inst else None,
"prereqs": os.path.isdir(os.path.join(p, "prereqs"))})
return {
"root": r,
"image_exists": os.path.isdir(os.path.join(config.SAMBA_SHARE, IMAGE)),
"installers": inst,
"active": inst[0] if inst else None,
"prereqs": {"present": pre_files > 0, "files": pre_files, "size": pre_size,
"sxs": os.path.isdir(os.path.join(pre, "sxs")),
"redistrib": os.path.isdir(os.path.join(pre, "redistrib"))},
"script": os.path.isfile(os.path.join(r, "Install-EngineeringLayer.ps1")),
"archives": archives,
}
class _Lock:
def __enter__(self):
self.fd = os.open(os.path.join(_state(), ".lock"), os.O_CREAT | os.O_RDWR, 0o644)
try:
fcntl.flock(self.fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
os.close(self.fd)
raise RuntimeError("another Engineering Layer upload is being applied") from None
return self
def __exit__(self, *exc):
fcntl.flock(self.fd, fcntl.LOCK_UN)
os.close(self.fd)
return False
def _check_name(name):
name = os.path.basename(str(name or "")).strip()
if EXE.match(name) or name.lower().endswith(".zip"):
return name
raise ValueError("upload Engineering_Install_v<version>.exe, or a .zip of the "
"extra_engineering payload (engineering_layer/ and prereqs/)")
def _receive(stream, name, length):
"""Stream to UPDATES_DIR/englayer/uploads/ (never held in memory)."""
folder = _state("uploads")
if length:
free = shutil.disk_usage(folder).free
if 2 * length > free - (1 << 30): # the upload plus its unpacked copy
raise RuntimeError(f"not enough disk space for {length // (1 << 20)} MB "
f"({free // (1 << 20)} MB free)")
fd, part = tempfile.mkstemp(dir=folder, prefix=".upload.", suffix=".part")
written = 0
try:
with os.fdopen(fd, "wb") as f:
while True:
block = stream.read(1 << 20)
if not block:
break
f.write(block)
written += len(block)
if length and written != length:
raise RuntimeError(f"upload incomplete: {written} of {length} bytes")
if not written:
raise RuntimeError("empty upload")
return part, written
except BaseException:
if os.path.exists(part):
os.remove(part)
raise
def _new_from_exe(part, name, work):
with open(part, "rb") as f:
if f.read(2) != b"MZ":
raise ValueError(f"{name} is not a Windows executable")
layer = os.path.join(work, "engineering_layer")
os.makedirs(layer)
shutil.move(part, os.path.join(layer, name))
return ("engineering_layer",)
def _new_from_zip(part, work):
try:
zf = zipfile.ZipFile(part)
except zipfile.BadZipFile:
raise ValueError("the .zip file is damaged or not a zip") from None
with zf:
names = [i.filename.replace("\\", "/") for i in zf.infolist()]
for n in names:
if n.startswith("/") or ":" in n.split("/")[0] or ".." in n.split("/"):
raise ValueError(f"unsafe path in the zip: {n}")
# Accept the payload at the top, or inside one extra_engineering/ folder.
prefix = ""
if not any(n.split("/")[0] in PARTS for n in names):
tops = {n.split("/")[0] for n in names if n}
if len(tops) == 1:
prefix = tops.pop() + "/"
exes = [n for n in names if n.startswith(prefix + "engineering_layer/")
and EXE.match(n.rsplit("/", 1)[-1])]
if not exes:
raise ValueError("the zip has no engineering_layer/Engineering_Install_v<version>.exe")
found = set()
for info in zf.infolist():
n = info.filename.replace("\\", "/")
if not n.startswith(prefix):
continue
rel = n[len(prefix):]
top = rel.split("/")[0]
if top not in PARTS:
continue # Install-EngineeringLayer.ps1 etc.: the repo owns those
found.add(top)
dest = os.path.join(work, *rel.split("/"))
if info.is_dir():
os.makedirs(dest, exist_ok=True)
continue
os.makedirs(os.path.dirname(dest), exist_ok=True)
with zf.open(info) as src, open(dest, "wb") as out:
shutil.copyfileobj(src, out, 1 << 20)
with open(os.path.join(work, exes[0][len(prefix):]), "rb") as f:
if f.read(2) != b"MZ":
raise ValueError(f"{exes[0]} is not a Windows executable")
return tuple(p for p in PARTS if p in found)
def _fix_modes(path):
for dirpath, dirs, files in os.walk(path):
os.chmod(dirpath, 0o755)
for f in files:
os.chmod(os.path.join(dirpath, f), 0o644)
def upload(stream, name, length=None):
"""Receive one upload and swap it in. Returns a summary dict."""
name = _check_name(name)
if not os.path.isdir(os.path.join(config.SAMBA_SHARE, IMAGE)):
raise RuntimeError(f"no {IMAGE} image on this server yet; import it first")
part, size = _receive(stream, name, length)
work = tempfile.mkdtemp(dir=_state("uploads"), prefix=".new.")
try:
if EXE.match(name):
parts = _new_from_exe(part, name, work)
else:
parts = _new_from_zip(part, work)
_fix_modes(work)
r = root()
os.makedirs(r, exist_ok=True)
stamp = datetime.now().strftime("%Y%m%d-%H%M%S")
archive = os.path.join(_state("archive"), stamp)
with _Lock():
moved = []
for p in parts:
cur = os.path.join(r, p)
if os.path.exists(cur):
os.makedirs(archive, exist_ok=True)
shutil.move(cur, os.path.join(archive, p))
moved.append(p)
for p in parts:
shutil.move(os.path.join(work, p), os.path.join(r, p))
_prune()
inst = _installers(os.path.join(r, "engineering_layer"))
return {"name": name, "size": size, "replaced": list(parts),
"archived": stamp if moved else None,
"version": inst[0]["version"] if inst else None}
finally:
shutil.rmtree(work, ignore_errors=True)
if os.path.exists(part):
os.remove(part)
def _prune():
adir = _state("archive")
old = sorted((d for d in os.listdir(adir) if os.path.isdir(os.path.join(adir, d))), reverse=True)
for d in old[KEEP_ARCHIVES:]:
shutil.rmtree(os.path.join(adir, d), ignore_errors=True)
def restore(stamp):
"""Put an archived payload back (the current one is archived in turn)."""
if not re.fullmatch(r"\d{8}-\d{6}", str(stamp or "")):
raise ValueError("no such archive")
src = os.path.join(_state("archive"), stamp)
if not os.path.isdir(src):
raise ValueError("no such archive")
r = root()
now = datetime.now().strftime("%Y%m%d-%H%M%S")
back = os.path.join(_state("archive"), now)
with _Lock():
parts = [p for p in PARTS if os.path.isdir(os.path.join(src, p))]
for p in parts:
cur = os.path.join(r, p)
if os.path.exists(cur):
os.makedirs(back, exist_ok=True)
shutil.move(cur, os.path.join(back, p))
for p in parts:
shutil.move(os.path.join(src, p), os.path.join(r, p))
shutil.rmtree(src, ignore_errors=True)
_prune()
return parts

View File

@@ -31,6 +31,9 @@ IMAGE_TABS = (
{"id": "drivers", "label": "Drivers", "icon": "bi-hdd-network", "endpoint": "images_drivers.page"},
{"id": "unattend", "label": "Unattend", "icon": "bi-file-earmark-code", "endpoint": "unattend_editor"},
{"id": "os", "label": "OS and Packages", "icon": "bi-windows", "endpoint": "images.os_packages"},
# Only on the images listed in "images" (the header skips it elsewhere).
{"id": "englayer", "label": "Engineering Layer", "icon": "bi-box-seam",
"endpoint": "engineering_layer.page", "images": ("gea-engineer",)},
)
TAB_IDS = tuple(t["id"] for t in IMAGE_TABS)

View File

@@ -97,7 +97,7 @@
</div>
<nav aria-label="Image sections"><ul class="image-tabs">
{% for t in image_tabs %}
{% for t in image_tabs if not t.images or image_type in t.images %}
<li><a href="{{ image_tab_url(image_type, t.id) }}"{% if t.id == _tab %} aria-current="page"{% endif %}>
{{ t.label }}
{% if t.id == 'drivers' %}<span class="tab-count" title="Entries in HardwareDriver.json">{{ _hdr.driver_count }}</span>

View File

@@ -0,0 +1,137 @@
{% extends "base.html" %}
{% import '_ui.html' as ui %}
{% block title %}{{ friendly_name }} - Engineering Layer{% endblock %}
{# Engineer image > Engineering Layer: what the engineer unattend will
install (extra_engineering/), an upload that replaces it, and the two
previous versions to restore. services/engineering_layer.py. #}
{% block extra_head %}
<style>
.sec-h { display: flex; flex-wrap: wrap; align-items: flex-end; justify-content: space-between; gap: 8px 16px; padding: 18px 24px 12px; }
.sec-h .sub { margin-top: 2px; }
.cur { padding: 20px 24px; display: flex; flex-direction: column; gap: 10px; }
.cur-top { display: flex; flex-wrap: wrap; align-items: center; gap: 8px 14px; }
.el-facts { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 14px 20px; margin: 4px 0 0; }
.el-facts dt { font-size: 13px; color: var(--muted); font-weight: 400; }
.el-facts dd { margin: 2px 0 0; font-size: 15px; }
.el-upl { display: flex; flex-wrap: wrap; gap: 10px 12px; align-items: flex-end; padding: 6px 24px 18px; }
.el-upl .fld { flex: 1 1 22rem; min-width: 0; }
.el-upl label { display: block; font-size: 13px; color: var(--muted); margin-bottom: 4px; }
.el-prog { padding: 0 24px 18px; }
.el-prog .bar { height: 8px; border-radius: 4px; background: var(--line); overflow: hidden; margin-top: 6px; }
.el-prog .bar > div { height: 100%; background: var(--accent, #2563eb); width: 0; }
.el-prog .bar > div.bad { background: var(--err, #dc2626); }
.el-row { display: grid; grid-template-columns: minmax(0, 1fr) auto auto; gap: 6px 14px; align-items: center;
padding: 11px 24px; border-top: 1px solid var(--line); font-size: 14px; }
.el-row form { margin: 0; }
@media (max-width: 760px) {
.el-facts { grid-template-columns: minmax(0, 1fr); }
.el-row { grid-template-columns: minmax(0, 1fr); padding: 11px 16px; }
.el-upl, .el-prog, .sec-h { padding-left: 16px; padding-right: 16px; }
.cur { padding: 18px 16px; }
}
</style>
{% endblock %}
{% block content %}
{% include "_image_header.html" %}
<div class="ui-stack">
<section class="panel cur" aria-labelledby="el-h">
<div class="cur-top">
<div>
<h2 id="el-h" class="sec" style="font-size: 15px; color: var(--muted)">Engineering Layer</h2>
<div class="sub">Installed by the engineer image at first logon (Install-EngineeringLayer.ps1), from <span class="mono">Deploy\Applications\extra_engineering</span>. It installs the newest <span class="mono">Engineering_Install_v*.exe</span> it finds.</div>
</div>
<span style="margin-left: auto">
{% if not st.image_exists %}{{ ui.chip('Engineer image not imported', 'warn') }}
{% elif st.active and st.prereqs.present %}{{ ui.chip('Ready', 'ok') }}
{% elif st.active %}{{ ui.chip('No prereqs', 'warn') }}
{% else %}{{ ui.chip('Not uploaded', 'warn') }}{% endif %}
</span>
</div>
<dl class="el-facts">
<div><dt>Installer</dt><dd>{% if st.active %}<span class="mono">{{ st.active.name }}</span><div class="sub">{{ st.active.size | filesizeformat }}, {{ st.active.mtime | local_dt }}</div>{% else %}<span class="muted">none</span>{% endif %}</dd></div>
<div><dt>Prerequisites (VC++ / .NET 3.5)</dt><dd>{% if st.prereqs.present %}{{ st.prereqs.files }} files, {{ st.prereqs.size | filesizeformat }}{% else %}<span class="muted">none</span>{% endif %}</dd></div>
<div><dt>Setup script</dt><dd>{% if st.script %}present{% else %}<span class="muted">missing (the playbook deploys it)</span>{% endif %}</dd></div>
</dl>
{% if st.installers | length > 1 %}<div class="notice">More than one installer is in the folder ({{ st.installers | map(attribute='name') | join(', ') }}); the newest is used.</div>{% endif %}
</section>
<section class="panel ui-list" id="upload" aria-labelledby="up-h">
<div class="sec-h"><div><h2 id="up-h" class="sec">Upload</h2>
<div class="sub">Either the installer alone (<span class="mono">Engineering_Install_v&lt;version&gt;.exe</span>, keeps the prerequisites), or a .zip of the whole payload with <span class="mono">engineering_layer\</span> and <span class="mono">prereqs\</span> at its top (or inside one <span class="mono">extra_engineering\</span> folder). The current files are kept as a previous version.</div></div></div>
<form id="elUpload" method="post" action="{{ url_for('engineering_layer.upload') }}" class="el-upl">
<div class="fld">
<label for="elFile">Installer (.exe) or payload (.zip)</label>
<input type="file" class="field" id="elFile" name="payload" accept=".exe,.zip" required>
</div>
<button type="submit" class="ui-btn pri"{% if not st.image_exists %} disabled title="Import the engineer image first"{% endif %}>Upload</button>
</form>
<div id="elProg" class="el-prog" style="display: none;" role="status">
<span id="elText">Uploading...</span>
<div class="bar"><div id="elBar"></div></div>
</div>
</section>
{% if st.archives %}
<section class="panel ui-list" aria-labelledby="arc-h">
<div class="sec-h"><div><h2 id="arc-h" class="sec">Previous versions</h2>
<div class="sub">The two most recent replaced payloads. Restoring swaps the current one into this list.</div></div></div>
{% for a in st.archives %}
<div class="el-row">
<span><span class="mono">{{ a.name }}</span> <span class="sub">installer {{ ('v' ~ a.version) if a.version else 'none' }}{% if a.prereqs %}, with prerequisites{% endif %}</span></span>
<span></span>
<form method="post" action="{{ url_for('engineering_layer.restore', stamp=a.name) }}"
onsubmit="return confirm('Restore the Engineering Layer from {{ a.name }}? The current files are kept as a previous version.');">
<input type="hidden" name="_csrf_token" value="{{ csrf_token() }}">
<button type="submit" class="ui-btn sm">Restore</button>
</form>
</div>
{% endfor %}
</section>
{% endif %}
</div>
{% endblock %}
{% block extra_scripts %}
<script>
(function () {
var csrf = document.querySelector('meta[name="csrf-token"]').content;
var form = document.getElementById('elUpload');
form.addEventListener('submit', function (ev) {
var input = document.getElementById('elFile');
if (!input.files.length || !window.XMLHttpRequest) { ev.preventDefault(); return; }
ev.preventDefault();
var file = input.files[0];
var box = document.getElementById('elProg');
var bar = document.getElementById('elBar');
var text = document.getElementById('elText');
var xhr = new XMLHttpRequest();
xhr.open('POST', form.action + '?name=' + encodeURIComponent(file.name));
xhr.setRequestHeader('X-CSRF-Token', csrf);
xhr.setRequestHeader('Content-Type', 'application/octet-stream');
xhr.upload.onprogress = function (e) {
if (!e.lengthComputable) { return; }
bar.style.width = (100 * e.loaded / e.total).toFixed(1) + '%';
text.textContent = 'Uploading ' + file.name + ': ' + Math.round(e.loaded / 1048576) + ' of ' +
Math.round(e.total / 1048576) + ' MB';
if (e.loaded === e.total) { text.textContent = 'Checking and unpacking ' + file.name + '...'; }
};
xhr.onload = function () {
var r = {};
try { r = JSON.parse(xhr.responseText); } catch (e) {}
if (xhr.status === 200 && r.ok) { location.reload(); return; }
text.textContent = 'Upload failed: ' + (r.error || ('HTTP ' + xhr.status));
bar.classList.add('bad');
form.querySelector('button[type=submit]').disabled = false;
};
xhr.onerror = function () { text.textContent = 'Upload failed: connection lost.'; bar.classList.add('bad'); };
box.style.display = '';
form.querySelector('button[type=submit]').disabled = true;
xhr.send(file);
});
})();
</script>
{% endblock %}

View File

@@ -0,0 +1,157 @@
"""Engineer image > Engineering Layer upload.
Run from the webapp folder:
PYTHONPATH=.:<deps> python -m unittest tests.test_engineering_layer -v
"""
import io
import os
import re
import shutil
import sys
import tempfile
import time
import unittest
import zipfile
_T = tempfile.mkdtemp(prefix="pxe-englayer-test-")
for _k, _sub in {"SAMBA_SHARE": "samba", "CLONEZILLA_SHARE": "cz", "BLANCCO_REPORTS": "br",
"ENROLLMENT_SHARE": "enr", "WEB_ROOT": "www", "IMAGING_DIR": "img", "UPDATES_DIR": "upd",
"TFTP_ROOT": "tftp", "BIOS_DIR": "bios", "UPLOAD_DIR": "upl", "SAMBA_LOG_DIR": "smblog",
"STARTNET_BACKUPS_DIR": "snb"}.items():
os.environ.setdefault(_k, os.path.join(_T, _sub))
os.makedirs(os.environ[_k], exist_ok=True)
for _k in ("AUDIT_LOG", "HARDWARE_STATE", "BLANCCO_STATE", "DNSMASQ_LEASES", "APACHE_ACCESS_LOG",
"DNSMASQ_SYSLOG", "NIC_STATUS_FILE", "NIC_CONF", "DNSMASQ_NIC_CONF", "DNSMASQ_CONF", "CLONEZILLA_STATE"):
os.environ.setdefault(_k, os.path.join(_T, _k.lower()))
WEBAPP = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, WEBAPP)
os.chdir(WEBAPP)
import config # noqa: E402
import app as A # noqa: E402
from services import engineering_layer as EL # noqa: E402
EXE_BYTES = b"MZ" + b"\x00" * 1000
def make_zip(entries):
buf = io.BytesIO()
with zipfile.ZipFile(buf, "w") as z:
for name, data in entries.items():
z.writestr(name, data)
return buf.getvalue()
def payload_zip(version="3.1", prefix=""):
return make_zip({
f"{prefix}engineering_layer/Engineering_Install_v{version}.exe": EXE_BYTES,
f"{prefix}prereqs/sxs/netfx3.cab": b"cab",
f"{prefix}prereqs/redistrib/v14/vc_redist.x64.exe": b"MZ..",
f"{prefix}Install-EngineeringLayer.ps1": b"# from the zip - must be ignored",
})
class EngineeringLayer(unittest.TestCase):
def setUp(self):
for d in (config.SAMBA_SHARE, config.UPDATES_DIR):
shutil.rmtree(d, ignore_errors=True)
os.makedirs(d)
os.makedirs(os.path.join(config.SAMBA_SHARE, "gea-engineer", "Deploy"))
os.makedirs(EL.root())
with open(os.path.join(EL.root(), "Install-EngineeringLayer.ps1"), "w") as f:
f.write("# repo copy")
def up(self, name, data):
return EL.upload(io.BytesIO(data), name, len(data))
def test_zip_installs_both_parts_and_keeps_the_script(self):
res = self.up("payload.zip", payload_zip("3.1"))
self.assertEqual(res["version"], "3.1")
st = EL.status()
self.assertEqual(st["active"]["name"], "Engineering_Install_v3.1.exe")
self.assertTrue(st["prereqs"]["present"] and st["prereqs"]["sxs"] and st["prereqs"]["redistrib"])
with open(os.path.join(EL.root(), "Install-EngineeringLayer.ps1")) as f:
self.assertEqual(f.read(), "# repo copy")
def test_zip_under_extra_engineering_folder(self):
self.up("extra_engineering.zip", payload_zip("3.2", prefix="extra_engineering/"))
self.assertEqual(EL.status()["active"]["version"], "3.2")
def test_exe_replaces_installer_and_keeps_prereqs(self):
self.up("payload.zip", payload_zip("3.1"))
res = self.up("Engineering_Install_v3.2.exe", EXE_BYTES)
st = EL.status()
self.assertEqual([i["name"] for i in st["installers"]], ["Engineering_Install_v3.2.exe"])
self.assertTrue(st["prereqs"]["present"])
self.assertEqual(res["replaced"], ["engineering_layer"])
self.assertEqual(st["archives"][0]["version"], "3.1")
def test_rejections(self):
with self.assertRaisesRegex(ValueError, "upload Engineering_Install"):
self.up("setup.exe", EXE_BYTES)
with self.assertRaisesRegex(ValueError, "not a Windows executable"):
self.up("Engineering_Install_v3.1.exe", b"PK not an exe")
with self.assertRaisesRegex(ValueError, "no engineering_layer"):
self.up("p.zip", make_zip({"prereqs/x.cab": b"x"}))
with self.assertRaisesRegex(ValueError, "unsafe path"):
self.up("p.zip", make_zip({"engineering_layer/Engineering_Install_v3.1.exe": EXE_BYTES,
"engineering_layer/../../evil.txt": b"x"}))
with self.assertRaisesRegex(ValueError, "damaged"):
self.up("p.zip", b"not a zip")
self.assertIsNone(EL.status()["active"]) # nothing changed
self.assertEqual(os.listdir(EL._state("uploads")), []) # no leftovers
def test_no_engineer_image_refused(self):
shutil.rmtree(os.path.join(config.SAMBA_SHARE, "gea-engineer"))
with self.assertRaisesRegex(RuntimeError, "import it first"):
self.up("Engineering_Install_v3.1.exe", EXE_BYTES)
def test_archives_pruned_to_two_and_restore(self):
for v in ("3.1", "3.2", "3.3", "3.4"):
self.up(f"Engineering_Install_v{v}.exe", EXE_BYTES)
time.sleep(1.05) # archive names are per second
st = EL.status()
self.assertEqual(st["active"]["version"], "3.4")
self.assertEqual([a["version"] for a in st["archives"]], ["3.3", "3.2"])
EL.restore(st["archives"][1]["name"])
st = EL.status()
self.assertEqual(st["active"]["version"], "3.2")
self.assertIn("3.4", [a["version"] for a in st["archives"]])
with self.assertRaises(ValueError):
EL.restore("../../etc")
class EngineeringLayerPages(unittest.TestCase):
def setUp(self):
shutil.rmtree(config.SAMBA_SHARE, ignore_errors=True)
for img in ("gea-engineer", "gea-standard"):
os.makedirs(os.path.join(config.SAMBA_SHARE, img, "Deploy", "Control"))
self.c = A.app.test_client()
def test_tab_only_on_engineer(self):
eng = self.c.get("/images/gea-engineer").get_data(as_text=True)
std = self.c.get("/images/gea-standard").get_data(as_text=True)
self.assertIn("Engineering Layer", eng)
self.assertNotIn("engineering-layer", std)
r = self.c.get("/images/gea-standard/engineering-layer")
self.assertEqual(r.status_code, 302)
def test_page_and_upload_route(self):
r = self.c.get("/images/gea-engineer/engineering-layer")
self.assertEqual(r.status_code, 200)
html = r.get_data(as_text=True)
tok = re.search(r'name="csrf-token" content="([^"]+)"', html).group(1)
r = self.c.post("/images/gea-engineer/engineering-layer/upload?name=Engineering_Install_v3.5.exe",
data=EXE_BYTES, headers={"X-CSRF-Token": tok, "Content-Type": "application/octet-stream"})
self.assertEqual(r.status_code, 200, r.get_data(as_text=True))
self.assertTrue(r.get_json()["ok"])
html = self.c.get("/images/gea-engineer/engineering-layer").get_data(as_text=True)
self.assertIn("Engineering_Install_v3.5.exe", html)
r = self.c.post("/images/gea-engineer/engineering-layer/upload?name=bad.exe",
data=EXE_BYTES, headers={"X-CSRF-Token": tok})
self.assertEqual(r.status_code, 400)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,58 @@
"""Engineer image > Engineering Layer tab: upload the Engineering Layer
installer (or the whole extra_engineering payload) the engineer unattend
installs. Logic in services/engineering_layer.py."""
from flask import Blueprint, flash, jsonify, redirect, render_template, request, url_for
import config
from services import engineering_layer as el
from services.audit import audit
bp = Blueprint("engineering_layer", __name__, url_prefix="/images")
def _back():
return redirect(url_for("engineering_layer.page", image_type=el.IMAGE))
@bp.route("/<image_type>/engineering-layer")
def page(image_type):
if image_type != el.IMAGE:
flash("The Engineering Layer belongs to the engineer image.", "warning")
return redirect(url_for("images.overview", image_type=image_type)) \
if image_type in config.IMAGE_TYPES else redirect(url_for("dashboard"))
return render_template(
"image_engineering_layer.html",
image_type=image_type,
friendly_name=config.FRIENDLY_NAMES.get(image_type, image_type),
active_tab="englayer",
st=el.status(),
)
@bp.route("/gea-engineer/engineering-layer/upload", methods=["POST"])
def upload():
"""Raw body (the page's XHR upload, name in ?name=) streams to disk."""
name = request.args.get("name", "")
try:
res = el.upload(request.stream, name, request.content_length)
except (ValueError, RuntimeError, OSError) as ex:
return jsonify({"ok": False, "error": str(ex)}), 400
audit("ENGINEERING_LAYER_UPLOAD",
f"{res['name']} ({res['size']} bytes): replaced {', '.join(res['replaced'])}; "
f"installer v{res['version']}; previous in archive {res['archived'] or '(none)'}")
flash(f"Engineering Layer updated from {res['name']}: installer v{res['version']}"
+ (f", previous kept as {res['archived']}." if res["archived"] else "."), "success")
return jsonify({"ok": True, **res})
@bp.route("/gea-engineer/engineering-layer/restore/<stamp>", methods=["POST"])
def restore(stamp):
try:
parts = el.restore(stamp)
except (ValueError, RuntimeError, OSError) as ex:
flash(f"Not restored: {ex}", "danger")
return _back()
audit("ENGINEERING_LAYER_RESTORE", f"{stamp}: {', '.join(parts)}")
flash(f"Restored the Engineering Layer from {stamp}; the replaced one is archived.", "success")
return _back()