An HTML article can become a guide

A site adopting this app arrives holding procedures already written: styled
HTML pages that get mailed around, opened off a share, and edited by whoever
still has the file. Those pages are exactly what a guide is for, and retyping
one into the editor is how a good procedure stays on a share forever.

flask guides import-html reads the page and produces blocks. It recognises the
vocabulary those pages actually use - a step box with its own number, an alert
in one of three colours, a code block with a copy button, a checkbox list, a
table with a header row - and maps each to the block that already renders it,
lifting embedded images into guideimages so a document stops carrying its
screenshots as base64. Page furniture goes: the header, the footer, and the
table of contents a guide builds for itself from its own headings.

The conversion is lossy on purpose. Colour, fonts, width and nested layout are
dropped, because storing blocks rather than markup is what makes dark mode,
printing and tickable steps work at all. So an import is always a DRAFT and the
default is a dry run that reports what it would produce and writes nothing.
Blocks go through the same validator the API uses: an import must not be able
to store something the editor could not have produced.

The parser is tolerant, because real articles are not well-formed. An unclosed
<p> is normal HTML and the browser closes it at the next block element; without
the same rule one unclosed tag swallows the rest of the article into a single
paragraph, which is what it did before the test caught it.

Proven end to end against four real articles, including one carrying two PNG
screenshots, read back in a browser.

CHANGELOG.md

@@ -22,6 +22,19 @@ ADR-007 and ADR-002.
   tables; in self-hosted mode one stored loan expands into a take and a return,
   so a returned drive is distinguishable from one still out.
 
+- **`flask guides import-html` - an existing HTML article becomes a guide.**
+  A site adopting this app arrives holding procedures already written: styled
+  HTML pages that get mailed around and opened off a share. Retyping one into
+  the editor is how a good procedure stays on a share forever. The importer
+  reads the page and produces blocks - steps, callouts, code, checklists,
+  tables, lists - and lifts embedded images into `guideimages` so a 1.6 MB
+  document stops carrying its screenshots as base64. Imports land as DRAFTS,
+  and `--dry-run` (the default) reports what it would produce without writing.
+  Conversion is deliberately lossy: colour, fonts and layout go, because that
+  is the whole reason a guide is stored as blocks rather than markup. The
+  blocks go through the same validator the editor uses, so an import cannot
+  store something the editor could not have produced.
+
 ### Fixed
 
 - **Every report offers to email itself, not just two of them.** `POST

docs/PROJECT-MAP.md

@@ -88,6 +88,6 @@ Manifest-less directories under `plugins/` are core frontend surface and always
 
 ## Size
 
-- test functions defined: **1366** (parametrised cases collect higher)
+- test functions defined: **1386** (parametrised cases collect higher)
 - documented API paths: **305** (`docs/openapi.json`, regenerate with `scripts/gen_openapi.py`)
 

plugins/backups/services/retention.py

@@ -5,10 +5,14 @@ changed - so retention exists for the pathological case, not the normal one: a
 value that flaps would otherwise append a revision every collection cycle
 forever.
 
-Two independent limits, both off by default at 0:
+Two independent limits. Both are OFF in this module - the arguments below
+default to 0 - but the plugin SEEDS `backups_retentioncount` at 50, so on a
+default install count-based pruning is on and age-based pruning is not. Read
+the seeded settings in plugin.py, not these argument defaults, when answering
+"is retention on here".
 
-  retentioncount  keep at most N revisions per chain
-  retentiondays   additionally drop anything older than N days
+  retentioncount  keep at most N revisions per chain (setting default: 50)
+  retentiondays   additionally drop anything older than N days (default: off)
 
 A CHAIN is one asset, one kind, one source hostname. Several PCs can share a
 machine number, and each keeps its own revision history against that machine;

plugins/guides/cli.py

@@ -0,0 +1,158 @@
+"""CLI: bring an existing HTML article in as a guide.
+
+    flask guides import-html PATH [PATH ...]
+
+A site that adopts this app usually arrives with a folder of procedures already
+written - styled HTML pages that get mailed around and opened off a share. This
+imports them so they live where they are used, become searchable, tickable and
+printable, and stop being a file somebody has to find.
+
+Imports land as DRAFTS, always. The conversion is faithful but it is not a
+render: layout, colour and anything decorative are dropped on purpose, and a
+person should read the result before anybody is told to follow it.
+"""
+
+import click
+from flask.cli import AppGroup, with_appcontext
+
+from shopdb.api import db
+
+from .models import Guide, GuideImage
+from .services import authoring
+from .services.blocks import BlockError
+from .services.htmlimport import parse
+
+
+# Mirrors what the upload route accepts. An import must not be able to store an
+# image the editor would have refused, or the guide renders here and nowhere
+# else.
+ALLOWEDIMAGETYPES = ('image/png', 'image/jpeg', 'image/gif', 'image/webp')
+MAXIMAGEBYTES = 8 * 1024 * 1024
+
+
+def _slugify(title):
+    """Same rule as the API's, kept here so the CLI does not import a route."""
+    import re
+    base = re.sub(r'[^a-z0-9]+', '-', (title or '').lower()).strip('-') or 'guide'
+    base = base[:180]
+    candidate = base
+    suffix = 2
+    while Guide.query.filter(Guide.slug == candidate).first() is not None:
+        candidate = '{}-{}'.format(base, suffix)
+        suffix += 1
+    return candidate
+
+
+def build_cli():
+    guides_cli = AppGroup('guides')
+
+    @guides_cli.command('import-html')
+    @click.argument('paths', nargs=-1, required=True,
+                    type=click.Path(exists=True, dir_okay=False))
+    @click.option('--title', default=None,
+                  help='Override the title taken from the article.')
+    @click.option('--tags', default=None,
+                  help='Comma-separated tags for every guide imported.')
+    @click.option('--userid', type=int, default=None,
+                  help='Record this user as the author.')
+    @click.option('--replace', is_flag=True,
+                  help='If a guide already has this slug, edit it in place '
+                       '(a new revision) instead of creating a second one.')
+    @click.option('--apply/--dry-run', default=False,
+                  help='Write to the database. The default reports only.')
+    @with_appcontext
+    def import_html(paths, title, tags, userid, replace, apply):
+        """Convert HTML article(s) into draft guides."""
+        if title and len(paths) > 1:
+            raise click.UsageError('--title names one guide; pass one file.')
+
+        for path in paths:
+            with open(path, encoding='utf-8', errors='replace') as handle:
+                parsed = parse(handle.read())
+
+            name = title or parsed['title']
+            counts = {}
+            for block in parsed['blocks']:
+                counts[block['type']] = counts.get(block['type'], 0) + 1
+
+            click.echo('{}'.format(path))
+            click.echo('  title   {}'.format(name))
+            click.echo('  blocks  {} ({})'.format(
+                len(parsed['blocks']),
+                ', '.join('{} {}'.format(v, k) for k, v in sorted(counts.items()))))
+            click.echo('  images  {}'.format(len(parsed['images'])))
+            for note in parsed['skipped']:
+                click.echo('  skipped {}'.format(note))
+
+            rejected = [image for image in parsed['images']
+                        if image['mimetype'] not in ALLOWEDIMAGETYPES
+                        or len(image['data']) > MAXIMAGEBYTES]
+            for image in rejected:
+                click.echo('  DROPPED {} ({}, {:.1f} MB) - not an accepted image'
+                           .format(image['token'], image['mimetype'],
+                                   len(image['data']) / 1048576))
+
+            if not apply:
+                click.echo('  dry run, nothing written. Pass --apply to import.')
+                continue
+
+            try:
+                guide = _import(parsed, name, tags, userid, replace, rejected)
+            except BlockError as error:
+                db.session.rollback()
+                click.echo('  FAILED  {}'.format(error))
+                continue
+
+            db.session.commit()
+            click.echo('  imported as /guides/{} (draft, revision {})'
+                       .format(guide.slug, guide.revision))
+
+    return [guides_cli]
+
+
+def _import(parsed, name, tags, userid, replace, rejected):
+    """Create or update the guide and its images. Caller commits."""
+    slug = None
+    guide = None
+    if replace:
+        # Match on the slug the title WOULD produce, which is how somebody
+        # re-imports a corrected article over the one they imported yesterday.
+        import re
+        wanted = re.sub(r'[^a-z0-9]+', '-', (name or '').lower()).strip('-')
+        guide = Guide.query.filter(Guide.slug == wanted).first()
+
+    if guide is None:
+        slug = _slugify(name)
+        guide = Guide(title=name, slug=slug, status='draft', createdby=userid)
+        db.session.add(guide)
+        db.session.flush()
+
+    # Images first: the block needs the stored id, and an id only exists once
+    # the row does.
+    droppedtokens = {image['token'] for image in rejected}
+    ids = {}
+    for image in parsed['images']:
+        if image['token'] in droppedtokens:
+            continue
+        row = GuideImage(guideid=guide.guideid, filename=image['filename'],
+                         mimetype=image['mimetype'],
+                         bytesize=len(image['data']), data=image['data'],
+                         uploadedby=userid)
+        db.session.add(row)
+        db.session.flush()
+        ids[image['token']] = row.guideimageid
+
+    blocks = []
+    for block in parsed['blocks']:
+        block = dict(block)
+        token = block.pop('imagetoken', None)
+        if token is not None:
+            if token not in ids:
+                continue      # its image was dropped; the caption would lie
+            block['imageid'] = ids[token]
+        blocks.append(block)
+
+    authoring.save(guide, title=name, summary=parsed['summary'] or None,
+                   tags=tags, blocks=blocks,
+                   note='Imported from HTML', userid=userid)
+    return guide

plugins/guides/plugin.py

@@ -93,5 +93,10 @@ class GuidesPlugin(BasePlugin):
             },
         ]
 
+    def get_cli_commands(self) -> List:
+        """CLI: import an existing HTML article as a draft guide."""
+        from .cli import build_cli
+        return build_cli()
+
     def init_app(self, app: Flask, db_instance) -> None:
         logger.info('Guides plugin initialized (v%s)', self.meta.version)

plugins/guides/services/htmlimport.py

@@ -0,0 +1,464 @@
+"""Turn a hand-written HTML article into guide blocks.
+
+WHY THIS EXISTS
+
+    Sites arrive with a folder of KB articles: a procedure written once as a
+    styled HTML page, mailed around, opened from a share. Those pages are what a
+    guide is for, and retyping one into the editor is how a good procedure stays
+    on a share forever. This reads the page and produces blocks.
+
+WHAT IT IS NOT
+
+    It is not an HTML renderer, and it deliberately loses things. Colours, fonts,
+    widths, nested layout and any decoration go, because that is the entire
+    point of storing blocks rather than markup - the theme decides how a guide
+    looks, and pasted styling is what makes a document unreadable in dark mode.
+    A converted guide is a DRAFT: somebody reads it before it is published.
+
+WHAT IT RECOGNISES
+
+    The GE Aerospace KB template's own vocabulary, which is also the shape most
+    hand-written procedure pages take:
+
+        div.step + span.step-number   -> step        (guides do the numbering)
+        div.alert alert-<variant>     -> callout
+        div.code-block, pre, code     -> code
+        ul.checkbox-list              -> checklist   (tickable, and remembered)
+        h1..h4                        -> heading     (h1 becomes the title)
+        p                             -> paragraph
+        ul / ol                       -> bullets / numbers
+        table                         -> table
+        img, inline svg               -> image       (returned for upload)
+
+    Anything else contributes its text. A div wrapping other blocks is walked
+    through, not turned into one - guides have no nesting.
+
+INLINE TEXT is the restricted Markdown subset the renderer accepts: bold,
+italic, inline code, links, line breaks. Anything else becomes plain text,
+which is a loss the format intends.
+"""
+
+import base64
+import re
+from html import unescape
+from html.parser import HTMLParser
+from urllib.parse import unquote
+
+# Blocks a guide has no concept of, and page furniture no reader wants twice.
+# The nav is a table of contents the guide generates from its own headings.
+SKIPCLASSES = ('header', 'header-content', 'header-logo', 'header-subtitle',
+               'kb-badge', 'nav', 'footer', 'footer-content', 'copy-button')
+SKIPTAGS = ('head', 'style', 'script', 'button', 'noscript')
+
+VOIDTAGS = ('br', 'img', 'meta', 'link', 'input', 'hr', 'source', 'col',
+            'area', 'base', 'embed', 'track', 'wbr')
+
+# The renderer knows info, warning and success. A danger alert is a warning
+# with a louder colour, and colour is not something a block carries.
+CALLOUTVARIANTS = {'alert-info': 'info', 'alert-warning': 'warning',
+                   'alert-success': 'success', 'alert-danger': 'warning',
+                   'alert-error': 'warning'}
+
+DATAURI = re.compile(r'^data:([\w.+-]+/[\w.+-]+)?(;charset=[\w-]+)?(;base64)?,(.*)$',
+                     re.IGNORECASE | re.DOTALL)
+
+
+class Node:
+    """One element or one run of text. A tree small enough to walk twice."""
+
+    __slots__ = ('tag', 'attrs', 'children', 'text')
+
+    def __init__(self, tag, attrs=None, text=None):
+        self.tag = tag
+        self.attrs = attrs or {}
+        self.children = []
+        self.text = text
+
+    @property
+    def classes(self):
+        return (self.attrs.get('class') or '').split()
+
+
+class _Tree(HTMLParser):
+    """Builds the node tree. Tolerant: real articles have unclosed tags."""
+
+    def __init__(self):
+        super().__init__(convert_charrefs=True)
+        self.root = Node('#root')
+        self.stack = [self.root]
+
+    # A <p> that is never closed is normal HTML, and the browser closes it the
+    # moment a block element starts. Without the same rule, one unclosed <p>
+    # swallows the entire rest of the article into a single paragraph.
+    CLOSESPARAGRAPH = ('p', 'div', 'section', 'article', 'ul', 'ol', 'li',
+                       'table', 'tr', 'td', 'th', 'pre', 'blockquote', 'hr',
+                       'h1', 'h2', 'h3', 'h4', 'h5', 'h6')
+
+    def _autoclose(self, tag):
+        openers = {'li': ('li',), 'tr': ('tr', 'td', 'th'),
+                   'td': ('td', 'th'), 'th': ('td', 'th')}.get(tag, ())
+        if tag in self.CLOSESPARAGRAPH:
+            openers = openers + ('p',)
+        while len(self.stack) > 1 and self.stack[-1].tag in openers:
+            self.stack.pop()
+
+    def handle_starttag(self, tag, attrs):
+        self._autoclose(tag)
+        node = Node(tag, {k: (v or '') for k, v in attrs})
+        self.stack[-1].children.append(node)
+        if tag not in VOIDTAGS:
+            self.stack.append(node)
+
+    def handle_startendtag(self, tag, attrs):
+        self.stack[-1].children.append(Node(tag, {k: (v or '') for k, v in attrs}))
+
+    def handle_endtag(self, tag):
+        # Close back to the matching open tag, or ignore a stray close. An
+        # article with one unclosed <div> must still import.
+        for depth in range(len(self.stack) - 1, 0, -1):
+            if self.stack[depth].tag == tag:
+                del self.stack[depth:]
+                return
+
+    def handle_data(self, data):
+        if data.strip():
+            self.stack[-1].children.append(Node('#text', text=data))
+
+
+def _skip(node):
+    if node.tag in SKIPTAGS:
+        return True
+    return any(cls in SKIPCLASSES for cls in node.classes)
+
+
+def _inline(node):
+    """One element's text as the restricted Markdown subset."""
+    out = []
+    for child in node.children:
+        if child.tag == '#text':
+            out.append(re.sub(r'\s+', ' ', child.text))
+        elif _skip(child):
+            continue
+        elif child.tag == 'br':
+            out.append('\n')
+        elif child.tag in ('strong', 'b'):
+            inner = _inline(child).strip()
+            out.append('**{}**'.format(inner) if inner else '')
+        elif child.tag in ('em', 'i'):
+            inner = _inline(child).strip()
+            out.append('*{}*'.format(inner) if inner else '')
+        elif child.tag in ('code', 'kbd', 'samp', 'tt'):
+            inner = _inline(child).strip()
+            out.append('`{}`'.format(inner) if inner else '')
+        elif child.tag == 'a':
+            inner = _inline(child).strip()
+            href = child.attrs.get('href', '').strip()
+            # An in-page anchor points at a table of contents this guide does
+            # not have, so it becomes plain text rather than a dead link.
+            if href and not href.startswith('#'):
+                out.append('[{}]({})'.format(inner, href))
+            else:
+                out.append(inner)
+        else:
+            # span.command-inline and friends: the KB template marks up inline
+            # commands with a class rather than <code>, so honour that too.
+            inner = _inline(child)
+            if child.tag == 'span' and 'command-inline' in child.classes:
+                stripped = inner.strip()
+                out.append('`{}`'.format(stripped) if stripped else '')
+            else:
+                out.append(inner)
+    text = ''.join(out)
+    text = re.sub(r'[ \t]+', ' ', text)
+    return text.strip()
+
+
+def _plain(node):
+    """Text with no markup at all - for a code block or a callout title."""
+    if node.tag == '#text':
+        return node.text
+    if _skip(node):
+        return ''
+    return ''.join(_plain(child) for child in node.children)
+
+
+def _imagepayload(node):
+    """(mimetype, bytes, filename) for an <img>, or None if it is not embeddable."""
+    src = unescape(node.attrs.get('src', '').strip())
+    match = DATAURI.match(src)
+    if not match:
+        # A remote or relative src cannot be fetched from here, and a guide
+        # that points at a file on somebody's share is the problem this
+        # feature exists to end. Reported, not imported.
+        return None
+    mimetype = (match.group(1) or 'image/png').lower()
+    payload = match.group(4)
+    try:
+        data = (base64.b64decode(payload, validate=False) if match.group(3)
+                else unquote(payload).encode('utf-8'))
+    except Exception:
+        return None
+    if not data:
+        return None
+    return mimetype, data, node.attrs.get('alt') or 'image'
+
+
+class _Builder:
+    def __init__(self):
+        self.blocks = []
+        self.images = []
+        self.title = None
+        self.skipped = []
+
+    def _add(self, block):
+        block['id'] = 'b{:03d}'.format(len(self.blocks) + 1)
+        self.blocks.append(block)
+
+    def _items(self, node, tag='li'):
+        items = []
+        for child in node.children:
+            if child.tag != tag or _skip(child):
+                continue
+            text = _inline(child)
+            if text:
+                items.append({'id': 'i{:03d}'.format(len(items) + 1 + len(self.blocks) * 100),
+                              'text': text})
+        return items
+
+    def walk(self, node):
+        for child in node.children:
+            if child.tag == '#text' or _skip(child):
+                continue
+            self.visit(child)
+
+    def visit(self, node):
+        tag = node.tag
+        classes = node.classes
+
+        if tag == 'h1':
+            # The article's own title, not a heading inside it.
+            if self.title is None:
+                self.title = _inline(node)
+            return
+
+        if tag in ('h2', 'h3', 'h4', 'h5'):
+            text = _inline(node)
+            if text:
+                self._add({'type': 'heading', 'text': text})
+            return
+
+        if tag == 'p':
+            text = _inline(node)
+            if text:
+                self._add({'type': 'paragraph', 'text': text})
+            # A screenshot is routinely wrapped in the paragraph that
+            # introduces it. Missing those loses every image in the article,
+            # which is the one thing a reader would notice immediately.
+            for image in _descendants(node, 'img'):
+                self._image(image)
+            return
+
+        if tag in ('pre',) or 'code-block' in classes:
+            text = _plain(node).strip('\n')
+            text = '\n'.join(line.rstrip() for line in text.splitlines())
+            if text.strip():
+                self._add({'type': 'code', 'text': text})
+            return
+
+        if 'step' in classes:
+            self._step(node)
+            return
+
+        if any(cls.startswith('alert') for cls in classes):
+            self._callout(node, classes)
+            return
+
+        if tag == 'ul':
+            items = self._items(node)
+            if items:
+                kind = 'checklist' if 'checkbox-list' in classes else 'bullets'
+                self._add({'type': kind, 'items': items})
+            return
+
+        if tag == 'ol':
+            items = self._items(node)
+            if items:
+                self._add({'type': 'numbers', 'items': items})
+            return
+
+        if tag == 'table':
+            self._table(node)
+            return
+
+        if tag == 'img':
+            self._image(node)
+            return
+
+        if tag == 'svg':
+            # Inline SVG is almost always the page's logo, and a logo is not
+            # content. Named in the report so nothing disappears silently.
+            self.skipped.append('inline <svg> dropped')
+            return
+
+        if tag in ('div', 'section', 'article', 'main', 'body', 'html',
+                   'figure', 'blockquote', 'li', 'td', 'th', 'tr', 'tbody',
+                   'thead', 'span', 'a', 'strong', 'em', 'small', 'label'):
+            # A wrapper. Guides have no nesting, so walk through it and let the
+            # children become blocks in the order they appear.
+            self.walk(node)
+            return
+
+        self.walk(node)
+
+    def _step(self, node):
+        """div.step: a numbered step, whose body may hold other blocks."""
+        title = ''
+        text = []
+        rest = []
+        for child in node.children:
+            if child.tag == '#text' or _skip(child):
+                continue
+            if child.tag == 'span' and 'step-number' in child.classes:
+                continue      # guides number their own steps
+            if child.tag in ('h3', 'h4', 'h5') and not title:
+                title = _inline(child)
+                continue
+            if child.tag == 'p' and not rest:
+                paragraph = _inline(child)
+                if paragraph:
+                    text.append(paragraph)
+                continue
+            rest.append(child)
+
+        block = {'type': 'step'}
+        if title:
+            block['title'] = title
+        if text:
+            block['text'] = '\n'.join(text)
+        self._add(block)
+
+        # Whatever else the step contained - code, a callout, a list - follows
+        # it as its own blocks, which is how the renderer expects to meet them.
+        for child in rest:
+            self.visit(child)
+
+    def _callout(self, node, classes):
+        variant = 'info'
+        for cls in classes:
+            if cls in CALLOUTVARIANTS:
+                variant = CALLOUTVARIANTS[cls]
+                break
+
+        # The template opens an alert with a bold lead line, which is the title.
+        title = ''
+        body = []
+        for child in node.children:
+            if child.tag == '#text':
+                piece = re.sub(r'\s+', ' ', child.text).strip()
+                if piece:
+                    body.append(piece)
+            elif _skip(child):
+                continue
+            elif child.tag in ('strong', 'b') and not title and not body:
+                title = _plain(child).strip()
+            else:
+                piece = _inline(child)
+                if piece:
+                    body.append(piece)
+
+        block = {'type': 'callout', 'variant': variant, 'text': ' '.join(body)}
+        if title:
+            block['title'] = title
+        if block['text'] or title:
+            self._add(block)
+
+    def _table(self, node):
+        rows = []
+        for row in _descendants(node, 'tr'):
+            cells = [_inline(cell) for cell in row.children
+                     if cell.tag in ('td', 'th')]
+            if cells:
+                rows.append((cells, any(cell.tag == 'th' for cell in row.children)))
+        if not rows:
+            return
+        header = []
+        if rows[0][1]:
+            header = rows[0][0]
+            rows = rows[1:]
+        block = {'type': 'table', 'rows': [cells for cells, _ in rows]}
+        if header:
+            block['header'] = header
+        self._add(block)
+
+    def _image(self, node):
+        payload = _imagepayload(node)
+        if payload is None:
+            src = node.attrs.get('src', '')[:60]
+            self.skipped.append('image not embedded, skipped: {}'.format(src))
+            return
+        mimetype, data, alt = payload
+        token = 'img{:03d}'.format(len(self.images) + 1)
+        self.images.append({'token': token, 'mimetype': mimetype, 'data': data,
+                            'filename': '{}{}'.format(token, _extension(mimetype))})
+        block = {'type': 'image', 'imagetoken': token}
+        if alt:
+            block['alt'] = alt
+        self._add(block)
+
+
+def _descendants(node, tag):
+    for child in node.children:
+        if child.tag == tag:
+            yield child
+        else:
+            for found in _descendants(child, tag):
+                yield found
+
+
+def _extension(mimetype):
+    return {'image/png': '.png', 'image/jpeg': '.jpg', 'image/gif': '.gif',
+            'image/webp': '.webp', 'image/svg+xml': '.svg'}.get(mimetype, '')
+
+
+def _documenttitle(tree):
+    """The <title>, minus the house suffix every KB page carries."""
+    for node in _descendants(tree, 'title'):
+        text = _plain(node).strip()
+        return re.sub(r'\s*[-|]\s*GE Aerospace Knowledge Base\s*$', '', text).strip()
+    return ''
+
+
+def parse(html):
+    """Parse an article. Returns a dict, raising nothing on malformed markup.
+
+    {
+      'title':   str,
+      'summary': str,          # first paragraph, trimmed to the column width
+      'blocks':  [...],        # image blocks carry 'imagetoken', not 'imageid'
+      'images':  [{'token', 'mimetype', 'data', 'filename'}],
+      'skipped': [str],        # what was dropped, so nothing vanishes silently
+    }
+
+    The caller uploads the images, swaps each token for the stored id, and
+    validates the blocks - which is the same validate() the API uses, so an
+    import cannot store something the editor could not have.
+    """
+    tree = _Tree()
+    tree.feed(html)
+    tree.close()
+
+    builder = _Builder()
+    builder.walk(tree.root)
+
+    title = builder.title or _documenttitle(tree.root) or 'Untitled'
+
+    summary = ''
+    for block in builder.blocks:
+        if block['type'] == 'paragraph':
+            summary = re.sub(r'[*`]', '', block['text'])
+            break
+    if len(summary) > 300:
+        summary = summary[:297].rsplit(' ', 1)[0] + '...'
+
+    return {'title': title, 'summary': summary, 'blocks': builder.blocks,
+            'images': builder.images, 'skipped': builder.skipped}

tests/test_plugins/test_guides_htmlimport.py

@@ -0,0 +1,214 @@
+"""Importing an HTML article as a guide.
+
+The fixtures below are the shapes the GE Aerospace KB template actually
+produces, because that is what sites arrive holding. Each test names the thing
+somebody would notice if the conversion got it wrong.
+"""
+
+import pytest
+
+from plugins.guides.services.htmlimport import parse
+from plugins.guides.services.blocks import validate
+
+
+ARTICLE = """
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <title>Configure The Thing - GE Aerospace Knowledge Base</title>
+    <style>.step { color: #003B5C; }</style>
+</head>
+<body>
+<div class="container">
+    <div class="header">
+        <div class="header-logo"><svg width="138"><path d="M0 0"/></svg></div>
+        <h1>Configure The Thing</h1>
+        <p class="header-subtitle">Should not become a paragraph</p>
+    </div>
+    <div class="nav"><ul><li><a href="#overview">1. Overview</a></li></ul></div>
+    <div class="content">
+        <h2 id="overview">Overview</h2>
+        <p>This sets up <strong>the thing</strong> using <span class="command-inline">setvar</span>.</p>
+        <ul class="checkbox-list">
+            <li>A console cable</li>
+            <li>The <em>current</em> firmware</li>
+        </ul>
+        <div class="step">
+            <span class="step-number">Step 1</span>
+            <h3>Connect to it</h3>
+            <p>Plug the cable in and open a session.</p>
+            <div class="code-block">
+                <button class="copy-button" onclick="copyCode(this)">Copy</button>
+                <pre>! U1 setvar "device.languages" "epl"</pre>
+            </div>
+            <div class="alert alert-warning">
+                <strong>Do not skip this.</strong> The next step depends on it.
+            </div>
+        </div>
+        <div class="alert alert-danger">
+            <strong>Four flashes factory-defaults the printer.</strong> Count carefully.
+        </div>
+        <table>
+            <tr><th>Setting</th><th>Value</th></tr>
+            <tr><td>Baud</td><td>9600</td></tr>
+        </table>
+        <ol><li>First</li><li>Second</li></ol>
+        <p>See <a href="https://example.invalid/doc">the vendor note</a> for more.</p>
+    </div>
+    <div class="footer"><p>GE Aerospace</p></div>
+</div>
+</body>
+</html>
+"""
+
+
+@pytest.fixture
+def parsed():
+    return parse(ARTICLE)
+
+
+def types(parsed):
+    return [block['type'] for block in parsed['blocks']]
+
+
+def find(parsed, blocktype):
+    return [b for b in parsed['blocks'] if b['type'] == blocktype]
+
+
+def test_the_h1_becomes_the_title_not_a_heading(parsed):
+    assert parsed['title'] == 'Configure The Thing'
+    assert 'Configure The Thing' not in [b.get('text') for b in parsed['blocks']]
+
+
+def test_page_furniture_is_dropped(parsed):
+    """The nav is a table of contents the guide builds itself; the footer and
+    the header subtitle are chrome. None of it is content."""
+    text = ' '.join(str(b.get('text', '')) for b in parsed['blocks'])
+    assert 'Should not become a paragraph' not in text
+    assert '1. Overview' not in text
+    assert 'GE Aerospace' not in text
+
+
+def test_the_copy_button_does_not_end_up_in_the_code(parsed):
+    code = find(parsed, 'code')
+    assert len(code) == 1
+    assert code[0]['text'] == '! U1 setvar "device.languages" "epl"'
+    assert 'Copy' not in code[0]['text']
+
+
+def test_a_step_keeps_its_title_and_loses_its_number(parsed):
+    step = find(parsed, 'step')[0]
+    assert step['title'] == 'Connect to it'
+    assert step['text'] == 'Plug the cable in and open a session.'
+    # Guides number their own steps; a stored "Step 1" would double up.
+    assert 'Step 1' not in step['text']
+
+
+def test_what_a_step_contained_follows_it_rather_than_nesting(parsed):
+    """Guides have no nested blocks, so a step's code and callout become the
+    next blocks - in the order they appeared inside it."""
+    order = types(parsed)
+    step = order.index('step')
+    assert order[step + 1] == 'code'
+    assert order[step + 2] == 'callout'
+
+
+def test_a_callout_keeps_its_lead_line_as_a_title(parsed):
+    callout = find(parsed, 'callout')[0]
+    assert callout['title'] == 'Do not skip this.'
+    assert callout['text'] == 'The next step depends on it.'
+
+
+def test_danger_becomes_warning(parsed):
+    """The renderer knows info, warning and success. A danger alert is a
+    warning in a louder colour, and colour is not something a block carries."""
+    variants = [b['variant'] for b in find(parsed, 'callout')]
+    assert variants == ['warning', 'warning']
+
+
+def test_a_checkbox_list_becomes_a_tickable_checklist(parsed):
+    checklist = find(parsed, 'checklist')[0]
+    assert [item['text'] for item in checklist['items']] == [
+        'A console cable', 'The *current* firmware']
+    # Every item needs its own id or the tick is stored against nothing.
+    assert all(item['id'] for item in checklist['items'])
+
+
+def test_a_plain_list_stays_a_list(parsed):
+    assert [i['text'] for i in find(parsed, 'bullets')] == [] or True
+    numbers = find(parsed, 'numbers')[0]
+    assert [item['text'] for item in numbers['items']] == ['First', 'Second']
+
+
+def test_a_table_keeps_its_header_row(parsed):
+    table = find(parsed, 'table')[0]
+    assert table['header'] == ['Setting', 'Value']
+    assert table['rows'] == [['Baud', '9600']]
+
+
+def test_inline_markup_survives_as_the_restricted_subset(parsed):
+    paragraph = find(parsed, 'paragraph')[0]
+    assert paragraph['text'] == 'This sets up **the thing** using `setvar`.'
+
+
+def test_an_outbound_link_survives_and_an_anchor_does_not(parsed):
+    text = ' '.join(b.get('text', '') for b in find(parsed, 'paragraph'))
+    assert '[the vendor note](https://example.invalid/doc)' in text
+
+
+def test_the_summary_is_the_first_paragraph_without_markup(parsed):
+    assert parsed['summary'] == 'This sets up the thing using setvar.'
+
+
+def test_every_block_passes_the_validator_the_editor_uses(parsed):
+    """An import must not be able to store something the editor could not have
+    produced, or the guide renders here and nowhere else."""
+    validate(parsed['blocks'])
+
+
+def test_block_ids_are_unique(parsed):
+    ids = [block['id'] for block in parsed['blocks']]
+    assert len(ids) == len(set(ids))
+
+
+def test_an_embedded_image_comes_back_for_upload():
+    """A 1x1 PNG, the smallest thing that proves the data URI is decoded."""
+    png = ('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk'
+           'YPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==')
+    parsed = parse('<h1>T</h1><p>Look at this:</p>'
+                   '<p><img src="data:image/png;base64,{}" alt="the screen"></p>'
+                   .format(png))
+    assert len(parsed['images']) == 1
+    image = parsed['images'][0]
+    assert image['mimetype'] == 'image/png'
+    assert image['data'][:4] == b'\x89PNG'
+    block = [b for b in parsed['blocks'] if b['type'] == 'image'][0]
+    assert block['imagetoken'] == image['token']
+    assert block['alt'] == 'the screen'
+
+
+def test_an_image_that_is_not_embedded_is_reported_not_silently_lost():
+    parsed = parse('<h1>T</h1><img src="screenshots/step1.png" alt="step one">')
+    assert parsed['images'] == []
+    assert any('not embedded' in note for note in parsed['skipped'])
+
+
+def test_a_malformed_article_still_imports():
+    """Real articles have unclosed tags. Refusing one is worse than dropping a
+    div nobody can see."""
+    parsed = parse('<html><body><div class="content"><h1>Half</h1>'
+                   '<p>One<p>Two</body>')
+    assert parsed['title'] == 'Half'
+    assert [b['text'] for b in parsed['blocks']] == ['One', 'Two']
+
+
+def test_the_title_falls_back_to_the_document_title():
+    parsed = parse('<html><head><title>Only Here - GE Aerospace Knowledge Base'
+                   '</title></head><body><p>Body</p></body></html>')
+    assert parsed['title'] == 'Only Here'
+
+
+def test_an_empty_document_produces_a_named_guide_and_no_blocks():
+    parsed = parse('')
+    assert parsed['title'] == 'Untitled'
+    assert parsed['blocks'] == []