Validating Generated Interactive Explainers Before They Ship: A Release-Gate Checklist

A client library with no profile for a given model id falls back to its own output ceiling — often far below what a structured design call needs — and truncates the response mid-JSON. Framework fallbacks vary by provider and change without notice, so an inherited default is an unowned failure mode.

Declare an explicit max-output-token value per provider in configuration — the reference pipeline uses a 16,000-token default, overridable by environment variable — rather than inheriting whatever the client library assumes.

Grep the provider configuration: every model profile carries an explicit ceiling, and no call path reaches the provider without one.

A truncated structured response is not transient — retrying the identical call produces the identical truncation. Swallowing it yields malformed JSON deep in the pipeline where the cause is invisible.

Detect the provider's truncation signal and raise a dedicated error class (the reference implementation's ModelOutputTruncated) that the retry machinery explicitly does not retry.

Force a low ceiling in a test run; the run fails loudly at the call site with the truncation error named, not later with a JSON parse error.

Shared code (the common data model all units depend on) breaks every downstream unit at once if it fails to parse, and a design that cites evidence ids absent from the ledger poisons verification later.

Parse-check the design's shared code when it is produced, re-ask the model once on failure, and filter out any cited evidence ids not present in the append-only ledger before the design propagates.

Feed a design with a deliberate syntax error and a fabricated evidence id in a test fixture; the pipeline re-asks once and the phantom id never reaches the writers.

Syntax errors are the cheapest failure class to catch, and catching them per unit gives the repair loop an actionable, line-numbered error instead of a whole-page failure.

Run node --check on each unit's wrapped panel code before launching any browser, and renumber reported error lines relative to the unit's own code so the writer model sees its own line numbers.

Inject a missing parenthesis into one unit; the failure report names that unit, cites a line number valid within that unit's code, and no browser session is spent on it.

A single shared <script> block couples every unit to every other one: one unparseable character anywhere in it means no unit's init function is ever defined, the whole page renders inert, and the validator reports a page-level failure with nothing the repair loop can act on. Correct units ship nothing.

Assemble each unit's code into its own <script> element so a parse failure is scoped to one unit; when a unit's init function is undefined at runtime, report "initLevelN is not defined — its <script> block failed to parse" inside that unit's stage only.

Break unit 2's syntax in a test page; units 1 and 3–5 still initialize and validate, and unit 2's stage carries the attributed parse-failure message.

Syntax checks cannot catch runtime errors, unrendered stages, or dead controls. Per-unit attribution is what makes the repair loop targeted rather than a full regeneration.

Headless-browser validation per unit: activate the unit's tab, check the stage actually rendered (e.g., an <svg> with a minimum descendant count or a nonzero <canvas>), operate a bounded number of controls with per-control timeouts, flag static controls the unit's init source never references, attribute console errors to the active unit, and screenshot each unit for the reviewer. State the known limit in team documentation: this proves absence of failure, not presence of function — a handler that changes nothing still passes.

A test unit with an unwired button is flagged as static; a test unit whose stage renders empty fails with a named cause; screenshots exist per unit in the run artifacts.

One unit blocking the main thread must not consume the budget of the units after it, and "the validator broke" must never be confused with "the validation passed" or silently treated as either.

Enforce a per-unit budget (the reference uses 35 s per level); on overrun, fail that unit with a named cause and open a fresh page for the remaining units. Wrap the whole validator in an outer timeout that never raises: map crash to available=False and timeout to available=True, ok=False. Both states block the gate, for different recorded reasons.

A unit with an infinite loop fails alone with a blocking-main-thread cause while later units still run; killing the validator process mid-run yields available=False and a blocked gate, not a pass.

An unbounded repair loop burns budget on units the model cannot fix, and re-sending passing units risks regressing them.

Cap repair at a fixed round count (the reference uses 2), route each failing unit back to its own writer with its attributed error, and keep an offline repair path that re-runs validate-and-repair against a finished run's artifacts. Keep re-render (presentation only, no model calls, gate status carried unchanged) operationally distinct from re-verification.

Run logs show at most the configured repair rounds, repair requests contain only failing units, and a re-render of a blocked page still carries the blocked status.

When a prompt quotes one concrete page as the example of a format, the model absorbs its subject along with its structure — producing running examples, vocabulary, and palettes belonging to that page rather than to the topic at hand, and to no evidence in the run's own ledger. The organizing principle is that the prompts describe a format, not a subject.

Maintain a curated denylist of words from the page the format was modeled on (the reference's TEMPLATE_WORDS) and run a CI test that fails if any denylisted word appears in generation prompts or schema descriptions.

Adding a denylisted word to a prompt file fails the CI run. Note the known limit: the denylist covers the prompts and schemas, not the output — a writer may legitimately choose a denylisted word where it genuinely fits the topic, and a run that comes out clean is not itself evidence the prompts are neutral.

A single worked example in a prompt makes its discipline the silent default for every future page; multiple unrelated stems prevent any one field from anchoring the format.

Add a CI assertion that prompt examples span several unrelated disciplines (the reference requires stems matching neuroscience, algorithms, cryptography, encryption, and sorting) so no single field reads as the format's home topic.

Removing all but one discipline's stems from the prompts fails the CI test.

Topic leakage is not only lexical. Icons, rail glyphs, and default visual idioms carry a subject too, and page furniture inherited from the exemplar quietly tells every topic what shape it is supposed to be.

Add tests over generated or templated visual chrome (the reference's rail-icon test forbids graph glyphs) mirroring the lexical denylist.

A rail icon drawn from the modeled page's glyph set fails the chrome test in CI.

Model-written prose markup can carry executable payloads, and an unescaped </script sequence inside a unit's code lets one unit terminate its own block and break the per-unit isolation the validator depends on.

Strip active elements, on* event-handler attributes, and script-protocol URLs from model markup at assembly time, and escape </script inside every generated code block.

A test unit containing an inline event handler and a literal </script string assembles into a page where neither survives, and the unit's script block boundary is intact.

Model-written code should have no network access and no reach into the host page; containment bounds blast radius when generation goes wrong. It bounds what a panel can reach — not what it can claim — so this control never substitutes for human review before merge.

Serve the assembled page under a content-security policy that denies network access to panel code, and embed the page in the host via a sandboxed iframe. Derive the exact policy and sandbox attributes from the host's own threat model rather than copying another deployment's values.

A test panel attempting a fetch and a parent-window access fails both under the deployed policy, and the failures are visible in validator console capture.

A scrollHeight-based height report tracks the frame's own height once the host has sized it, so the measurement feeds its own input and ratchets upward without bound — a layout failure that appears only in the embedded context, after validation has passed.

Detect embedding, zero the embedded page's own min-height, report measured height only on meaningful change (the reference threshold is ≥4 px), and clamp the height host-side (the reference clamps to 700–60,000 px).

An embedded test page holds a stable height across repeated resize events instead of growing monotonically.

Prose verification passing tells you nothing about whether the panels work, and a validator that never ran is not a validator that passed. An unvalidated page is unpublishable, not presumed fine — and a gate without enforcement is a dashboard.

Block when verification did not run, extracted zero claims, judged any claim unsupported or contradicted, or cited ids absent from the ledger; add blockers when panels were not validated, the validator was unavailable, or panels failed. On block: import refuses with a nonzero exit, the PR opens as a draft carrying per-unit screenshots for the reviewer, and the page renders a visible blocked notice. Leave partial verdicts non-blocking by design — a wording problem for the reviewer to see. Never treat a gate bypass flag as a normal workflow step.

With the validator disabled in a test run, verification passes but import exits with a refusal code, the PR is a draft, and the blocked notice renders on the page.

Discovering that no browser can launch after paying for research, design, and five writers wastes the entire run's model spend.

Before the first generation call, run a preflight that includes proof a headless browser can actually launch in the execution environment, alongside the usual configuration checks.

Removing the browser binary from a test environment fails the run at preflight with zero model calls issued.

Verification judges prose against the ledger; it never sees panel_js or the numbers a panel computes. The worst named outcome: a level-4 simulation converges to the wrong value, and the page renders, passes validation, scores full marks on its claims, and is wrong. The repair loop fixes code, not pedagogy.

Make pre-merge human review a standing release requirement covering three named obligations: panel numerical correctness, pedagogical quality of the levels, and persona voice consistency — using the per-unit screenshots the gate attaches to the draft PR. Do not present these as automated; both numerical verification of panels and pedagogical scoring are open gaps in the source work.

The merge policy requires a reviewer sign-off record explicitly listing the three obligations before any generated page reaches production.