An HTML article can become a guide

A site adopting this app arrives holding procedures already written: styled
HTML pages that get mailed around, opened off a share, and edited by whoever
still has the file. Those pages are exactly what a guide is for, and retyping
one into the editor is how a good procedure stays on a share forever.

flask guides import-html reads the page and produces blocks. It recognises the
vocabulary those pages actually use - a step box with its own number, an alert
in one of three colours, a code block with a copy button, a checkbox list, a
table with a header row - and maps each to the block that already renders it,
lifting embedded images into guideimages so a document stops carrying its
screenshots as base64. Page furniture goes: the header, the footer, and the
table of contents a guide builds for itself from its own headings.

The conversion is lossy on purpose. Colour, fonts, width and nested layout are
dropped, because storing blocks rather than markup is what makes dark mode,
printing and tickable steps work at all. So an import is always a DRAFT and the
default is a dry run that reports what it would produce and writes nothing.
Blocks go through the same validator the API uses: an import must not be able
to store something the editor could not have produced.

The parser is tolerant, because real articles are not well-formed. An unclosed
<p> is normal HTML and the browser closes it at the next block element; without
the same rule one unclosed tag swallows the rest of the article into a single
paragraph, which is what it did before the test caught it.

Proven end to end against four real articles, including one carrying two PNG
screenshots, read back in a browser.

plugins/guides/cli.py

@@ -0,0 +1,158 @@
+"""CLI: bring an existing HTML article in as a guide.
+
+    flask guides import-html PATH [PATH ...]
+
+A site that adopts this app usually arrives with a folder of procedures already
+written - styled HTML pages that get mailed around and opened off a share. This
+imports them so they live where they are used, become searchable, tickable and
+printable, and stop being a file somebody has to find.
+
+Imports land as DRAFTS, always. The conversion is faithful but it is not a
+render: layout, colour and anything decorative are dropped on purpose, and a
+person should read the result before anybody is told to follow it.
+"""
+
+import click
+from flask.cli import AppGroup, with_appcontext
+
+from shopdb.api import db
+
+from .models import Guide, GuideImage
+from .services import authoring
+from .services.blocks import BlockError
+from .services.htmlimport import parse
+
+
+# Mirrors what the upload route accepts. An import must not be able to store an
+# image the editor would have refused, or the guide renders here and nowhere
+# else.
+ALLOWEDIMAGETYPES = ('image/png', 'image/jpeg', 'image/gif', 'image/webp')
+MAXIMAGEBYTES = 8 * 1024 * 1024
+
+
+def _slugify(title):
+    """Same rule as the API's, kept here so the CLI does not import a route."""
+    import re
+    base = re.sub(r'[^a-z0-9]+', '-', (title or '').lower()).strip('-') or 'guide'
+    base = base[:180]
+    candidate = base
+    suffix = 2
+    while Guide.query.filter(Guide.slug == candidate).first() is not None:
+        candidate = '{}-{}'.format(base, suffix)
+        suffix += 1
+    return candidate
+
+
+def build_cli():
+    guides_cli = AppGroup('guides')
+
+    @guides_cli.command('import-html')
+    @click.argument('paths', nargs=-1, required=True,
+                    type=click.Path(exists=True, dir_okay=False))
+    @click.option('--title', default=None,
+                  help='Override the title taken from the article.')
+    @click.option('--tags', default=None,
+                  help='Comma-separated tags for every guide imported.')
+    @click.option('--userid', type=int, default=None,
+                  help='Record this user as the author.')
+    @click.option('--replace', is_flag=True,
+                  help='If a guide already has this slug, edit it in place '
+                       '(a new revision) instead of creating a second one.')
+    @click.option('--apply/--dry-run', default=False,
+                  help='Write to the database. The default reports only.')
+    @with_appcontext
+    def import_html(paths, title, tags, userid, replace, apply):
+        """Convert HTML article(s) into draft guides."""
+        if title and len(paths) > 1:
+            raise click.UsageError('--title names one guide; pass one file.')
+
+        for path in paths:
+            with open(path, encoding='utf-8', errors='replace') as handle:
+                parsed = parse(handle.read())
+
+            name = title or parsed['title']
+            counts = {}
+            for block in parsed['blocks']:
+                counts[block['type']] = counts.get(block['type'], 0) + 1
+
+            click.echo('{}'.format(path))
+            click.echo('  title   {}'.format(name))
+            click.echo('  blocks  {} ({})'.format(
+                len(parsed['blocks']),
+                ', '.join('{} {}'.format(v, k) for k, v in sorted(counts.items()))))
+            click.echo('  images  {}'.format(len(parsed['images'])))
+            for note in parsed['skipped']:
+                click.echo('  skipped {}'.format(note))
+
+            rejected = [image for image in parsed['images']
+                        if image['mimetype'] not in ALLOWEDIMAGETYPES
+                        or len(image['data']) > MAXIMAGEBYTES]
+            for image in rejected:
+                click.echo('  DROPPED {} ({}, {:.1f} MB) - not an accepted image'
+                           .format(image['token'], image['mimetype'],
+                                   len(image['data']) / 1048576))
+
+            if not apply:
+                click.echo('  dry run, nothing written. Pass --apply to import.')
+                continue
+
+            try:
+                guide = _import(parsed, name, tags, userid, replace, rejected)
+            except BlockError as error:
+                db.session.rollback()
+                click.echo('  FAILED  {}'.format(error))
+                continue
+
+            db.session.commit()
+            click.echo('  imported as /guides/{} (draft, revision {})'
+                       .format(guide.slug, guide.revision))
+
+    return [guides_cli]
+
+
+def _import(parsed, name, tags, userid, replace, rejected):
+    """Create or update the guide and its images. Caller commits."""
+    slug = None
+    guide = None
+    if replace:
+        # Match on the slug the title WOULD produce, which is how somebody
+        # re-imports a corrected article over the one they imported yesterday.
+        import re
+        wanted = re.sub(r'[^a-z0-9]+', '-', (name or '').lower()).strip('-')
+        guide = Guide.query.filter(Guide.slug == wanted).first()
+
+    if guide is None:
+        slug = _slugify(name)
+        guide = Guide(title=name, slug=slug, status='draft', createdby=userid)
+        db.session.add(guide)
+        db.session.flush()
+
+    # Images first: the block needs the stored id, and an id only exists once
+    # the row does.
+    droppedtokens = {image['token'] for image in rejected}
+    ids = {}
+    for image in parsed['images']:
+        if image['token'] in droppedtokens:
+            continue
+        row = GuideImage(guideid=guide.guideid, filename=image['filename'],
+                         mimetype=image['mimetype'],
+                         bytesize=len(image['data']), data=image['data'],
+                         uploadedby=userid)
+        db.session.add(row)
+        db.session.flush()
+        ids[image['token']] = row.guideimageid
+
+    blocks = []
+    for block in parsed['blocks']:
+        block = dict(block)
+        token = block.pop('imagetoken', None)
+        if token is not None:
+            if token not in ids:
+                continue      # its image was dropped; the caption would lie
+            block['imageid'] = ids[token]
+        blocks.append(block)
+
+    authoring.save(guide, title=name, summary=parsed['summary'] or None,
+                   tags=tags, blocks=blocks,
+                   note='Imported from HTML', userid=userid)
+    return guide