What Building a Model-Agnostic Research Pipeline Actually Took

Executive summary

This is a report on building and operating a research pipeline that takes a question, researches it against live sources, and produces a citation-backed report — then expands that report into a cluster of derived publications. It runs on hosted model APIs, is designed not to depend on any one of them, and has produced 97 published artifacts to date.

The headline finding is that the vendor-portability problem the architecture is named for turned out to be the cheapest problem in the system. Most published guidance on model-agnostic pipelines centres on the substitution mechanism: a gateway, a unified endpoint, per-request model selection. Our entire provider abstraction is a 99-line module wrapping one library call. It has never broken and has never been the constraint.

The expensive problems were the ones downstream of it:

Knowing whether the output is any good. A research agent that cites sources can cite them wrongly, and the failure is invisible in the prose. We built claim-level verification against an evidence ledger, which works — it once rejected an entire report — but which has a structural blind spot we can now describe precisely.

Knowing what a run actually did. For months a run recorded its token counts and nothing else. It could not say which search backend served its sources, which prompt revision produced it, or whether it had been silently truncated by a wall-clock budget. Two runs that differed in all of those read identically.

Keeping the property you built. Model-agnosticism is not a component you install; it is an invariant across call sites that decays every time someone adds a new one. Half our own system violates it, for entirely ordinary reasons.

What follows is the architecture, seven findings from running it, and the problems still open.

The system

question
  → scope       (reason lane)   research brief + acceptance criteria
  → plan        (reason lane)   subquestions with initial search queries
  → researcher ×N  (parallel, context-isolated)
       search (adapter chain) → select sources (fast lane)
       → fetch (policy-enforced) → extract verbatim evidence (reason lane)
       → append to evidence ledger → typed EvidencePacket
  → coverage    (verifier lane) gaps vs acceptance criteria → optional round 2
  → synthesize  (reason lane)   draft citing ledger ids
  → verify      (verifier lane) extract claims → verdict per claim vs quotes
  → finalize                    citations + Verification / Sources / Methodology

That is one half. The other takes the finished report and expands it into derived artifacts — explainers, glossary entries, checklists, briefs — planned as a dependency DAG, drafted in topological order, and gated on human approval of a per-artifact source brief. The two halves meet at a deliberately dumb interface: a markdown file with YAML frontmatter. Anything that can write that file can enter the pipeline, which is why the research half has been replaced once without the publishing half noticing.

A standard-preset research run costs roughly 15 model calls and 70k tokens.

Finding 1: capability indirection is a small problem

Our provider abstraction is one class with two methods, structured() and text(), over a multi-provider client library. Model ids are provider-prefixed strings — anthropic:claude-opus-4-8, openai:gpt-5.5-pro, google_genai:gemini-3.1-pro-preview, ollama:... — and every graph node calls that interface rather than a vendor SDK.

The prevailing advice would put a gateway here: a deployed service exposing a unified, typically OpenAI-compatible endpoint, with per-request model selection, timeout fallback, and canary routing. Those are real features. They are also features of a shared substitution point — they earn their cost when many services contend for the same backends and credentials, and fallback policy has to stay consistent across teams. For a single-process agent making fifteen calls per run, a gateway adds a network hop, a service to operate, and a new failure mode, in exchange for isolation we already had.

The generalisable version: model-agnosticism is a property of call sites, not a component in a diagram. If every model call already passes through one interface, you have the property. Adding a gateway to a system that already has it does not make it more agnostic — only more distributed. The right question is not "do we have a gateway" but "does anything bypass the interface," and that question is answered by grep, not by architecture review.

We did adopt the fallback pattern — just not for models. Search runs through an ordered adapter chain (Tavily → a provider's server-side web search → a keyless HTML scrape), falling through on error or when every result is rejected by URL policy. That is where redundancy earns its keep, because retrieval backends fail far more often than model APIs do.

Finding 2: route by task role, not per request

The pipeline resolves three model lanes independently:

Lane	Work	Why it is separate
`fast`	Query expansion, source selection	High call volume, low reasoning demand — the expensive model adds nothing
`reason`	Scope, plan, evidence extraction, synthesis	The quality-determining path
`verifier`	Coverage assessment, claim verification	Should be able to disagree with `reason`

Lane defaults derive from whichever provider key is present, so the agent runs with any single vendor configured. The configuration that matters, though, is the heterogeneous one: point the verifier lane at a different provider than the synthesis lane and claim-checking stops being the same model assessing its own output.

This is a more useful axis than per-request model selection. A per-request header is a mechanism for overriding a default. Lanes are a statement about what each model is for, and the separation between the lane that produces work and the lane that checks it is a property worth designing around — one a header cannot express.

Finding 3: budgets belong in code, not prompts

Every resource ceiling is a frozen dataclass consulted by the graph: subquestions, searches per subquestion, results per search, fetches per subquestion, follow-up rounds, page characters, and wall-clock seconds. Three presets — quick, standard, deep — expand to concrete values.

None of this is stated to the model. A budget in a prompt is a request; a budget at the call site is a limit. The wall-clock check is re-tested inside the fetch loop and again at the coverage gate, so a slow run degrades into a thinner report rather than hanging.

That degradation creates its own failure mode: a run that exhausts its wall clock returns less evidence and, unless something records it, says nothing about why. Silent truncation reads exactly like a thorough run on a thin topic. Enforcing budgets in code is necessary; reporting when they bound the work is the other half, and we shipped it late.

Finding 4: freeze the corpus, not the seed

Every extracted quote is appended to a per-run ledger as it arrives, alongside node events and verification verdicts. The report is synthesised from the ledger, and the verifier audits the draft against it.

This was built for provenance. Its more valuable property turned out to be that it separates the non-deterministic part of the pipeline from the deterministic part. Search and fetch are irreducibly non-reproducible — the web changes underneath you. But given a frozen ledger, synthesis and verification can be replayed and diffed.

This matters because the standard reproducibility toolkit is unavailable to us. Seed pinning, greedy decoding, batch-shape control, GPU and kernel pinning, deterministic-algorithm flags — every one of those levers requires owning the inference stack. Behind a hosted API they are not merely expensive, they do not exist. Advice written for self-hosted serving does not degrade gracefully when applied to hosted APIs; it reads as actionable and is not.

The substitute is to reproduce the reasoning rather than the tokens: freeze the retrieved inputs, replay everything downstream, and diff. For a research pipeline that is the property that actually matters, and the ledger provides it at no additional cost because it already exists for provenance.

Finding 5: verification works, and its blind spot is structural

The verifier extracts claims from the draft and returns a verdict for each against the quoted evidence. Two independent defences guard against citation laundering: findings' evidence ids are intersected with ids actually written to the ledger, and at verification time inline citation tokens are recovered from the claim text, unioned with the ids the extractor self-reported, filtered against real ledger ids, and stripped from the display text. A fabricated citation is dropped rather than rendered as authority.

Three runs show what this looks like in practice:

Run	Verdicts
3 July, 18:45	13 unsupported, 0 supported
3 July, 18:51	18 supported, 1 partial
3 July, 21:44	20 supported

The first run failed every claim. The verifier's own notes explain why: "Cites no evidence, though matching quotes exist; without citations the claim is unsupported per instructions." The claims were true and the supporting evidence was in the ledger — the synthesiser had simply not emitted citation tokens. A citation-plumbing bug, scored as total factual failure. Six minutes later, with token recovery in place, the next run scored 18 of 19.

Two things follow. The verifier is not a rubber stamp; it will reject an entire report, which is the behaviour you want and rarely get. And its false-positive rate is demonstrably non-zero and entirely unmeasured — that bug was caught by a human reading output, not by a harness.

The structural limit is more important. Verification here is claim-level and cannot be corpus-level. It asks "do the quoted spans support this sentence," never "is this corpus good enough to support this conclusion." Those come apart badly. A report can score full marks on every claim while resting entirely on a single weak source, because each individual sentence really is supported by the quote beneath it. Nothing in a claim-level pass can notice that the whole corpus is one tutorial and three vendor README files. We have shipped a report in exactly that state, at 20 of 20 supported.

Corpus-level assessment — source diversity, independence, sufficiency against the acceptance criteria — is a different pass, and it does not exist yet.

Finding 6: untrusted content is a tool-layer problem

A research agent fetches arbitrary web pages and feeds them to a model. That is the whole job, and it is also a standing prompt-injection exposure. None of the mitigation lives in a prompt:

URL policy on every request and re-checked on every redirect hop: http/https only, ports 80/443, no credentials in URLs, domain allow/blocklists, and every resolved A/AAAA address must be publicly routable. The DNS-resolution-versus-connect TOCTOU gap is documented rather than hidden; production deployments should pair it with an egress firewall.
Fetched pages are data: control characters stripped, active elements removed, size-capped, and fenced in <untrusted-source> blocks with escape neutralisation.
Subagents return typed packets, never free-form instructions to the supervisor, so a compromised researcher cannot steer the run.

Security rules do ride along in the prompts of every call that sees retrieved content, but they are a courtesy, not a control. A model that has been successfully injected cannot be relied on to enforce its own constraints — which is the entire argument for putting enforcement in the tool wrappers where the model cannot reach it.

Finding 7: observability was the wrong thing to defer

For most of the system's life its observability consisted of a per-run token total. That answered "what did this cost" and nothing else. It could not answer which search backend served the sources — a run served entirely by the keyless fallback is materially weaker than one served by a real search API, and the two were indistinguishable in the output. It could not answer which prompt revision produced a report, whether a run had been truncated, or whether quality moved after a change. The 13-of-13 failure above is exactly the class of bug a trace surfaces in seconds and raw output hides completely.

Both halves are now instrumented, on four principles worth stating because they generalise:

Telemetry gets the same treatment as model access. OpenTelemetry is the interface; the vendor backend is one implementation behind it. An observability layer you cannot swap is a lock-in you adopted while solving lock-in.
One observation per model call attempt. A schema retry that eventually succeeds is invisible if the call is recorded once, and those tokens are real.
Trace whole units of work. Per-call spans answer "what did this request cost." Only a trace spanning the complete task answers "what did this deliverable cost," which is the question anyone actually asks. Our publishing half runs as a chain of spawned processes, so the orchestrator opens a root span and propagates W3C trace context into every child — research, plan, scaffold, and draft land in a single trace.
Record tokens, not prices. Model ids and token counts go to the backend, which prices them. A price table checked into the repository is stale within weeks — we know, because our pinned model ids had already drifted into two different answers across four files before anyone noticed.

One deployment property matters more than it sounds: tracing is off by default and must degrade to a no-op when credentials, packages, or the collector are absent. Observability that can break the pipeline it observes is worse than none. The most valuable test in that subsystem asserts that the tracing-off path — the default, and therefore the one most likely to ship broken — exposes the full interface.

Separately from tracing, every run now writes a manifest: preset and expanded budget, a fingerprint of the prompt set, which retrieval backend served results, verdict distribution, and wall clock against budget. The same data renders into the report's own methodology section, so a published report describes the run that produced it.

Where the property leaks

The research half routes every model call through the provider abstraction. The publishing half — cluster planning, brief scaffolding, per-artifact drafting — instantiates a vendor SDK directly, with hardcoded model defaults.

So the system practises capability indirection in the half that produces one paper, and vendor coupling in the half that produced all 97 published artifacts. If that vendor has an outage, the research agent fails over and the publishing pipeline stops.

This is worth reporting precisely because there is no interesting story behind it. Nobody decided against the abstraction. Those scripts were written first, they worked, and no forcing function ever asked the question. That is how this property is usually lost: not by rejecting it, but by accumulating call sites that never went through it. Which is why the meaningful check is a repository-wide grep for SDK instantiation rather than a diagram — and why it has to cover the peripheral tooling, not just the flagship path.

If you are building one

The advice we would give someone starting, in the order we wish we had done it:

Put every model call behind one interface on day one. Not primarily for portability — because that interface is where budgets, retries, usage accounting, and tracing attach. Retrofitting it means touching every call site.
Size the substitution point to your topology, and write down the trigger to revisit. In-process for one service; a shared library for several; a gateway when teams contend for backends and credentials.
Separate the lane that does the work from the lane that checks it, and make them independently configurable across vendors.
Enforce budgets in code, and report when they bind. The second half is what stops silent truncation reading as a thorough run.
Build the eval harness before you think you need it. This is the one we deferred longest and regret most. Portability without evaluation makes model swaps easy and tells you nothing about whether a swap was good.
Instrument the interface from the start, tracing whole units of work rather than individual calls.
Put untrusted-content enforcement in the tool layer, never in a prompt.

Open problems

Offline evaluation is missing, and it is top of the list. Inline verification checks a report against its own ledger; it cannot tell you whether a prompt change made things better or worse. Three observed runs produced supported-claim rates of 0%, 95%, and 100%, and nothing in the system can say how much of that spread is signal. The fix is specific and cheap: freeze a handful of ledgers, replay synthesis and verification, assert on the verdict distribution. The ledger already supplies the corpus.

Source diversity has no representation in the design. The source-selection prompt says nothing about corroboration or independence, so it selects the top-ranked official pages — which is how a report ends up resting on one tutorial and a set of vendor self-descriptions. There is no rule like "no section rests on a single domain," and no way to express one.

The coverage gate is advisory. The graph has a node whose entire job is to compare gathered evidence against the run's acceptance criteria and trigger a follow-up round. We have shipped reports that stated their criteria were unmet and published anyway. Honest reporting is not the same as a gate.

Verification cannot assess its own corpus, as above. This is the gap most likely to produce a confidently wrong publication.

Provenance

This report was written from the implementation rather than produced by the research agent, so it carries no evidence ledger and no verification appendix. Its claims are grounded in code and run artifacts in our own repository rather than in cited external sources, and it should not be read as carrying the guarantees of an agent-produced paper on this site.

It supersedes an earlier agent-produced report on the same subject, which remains published and marked as superseded.