Hardening a Model-Agnostic LLM Pipeline: Gateway, Evaluation, and Reproducibility Controls

Code that calls vendor SDKs directly reintroduces the lock-in the pipeline exists to eliminate. That single interface is also where budgets, retries, usage accounting, and tracing attach — retrofitting it later means touching every call site.

Define one internal interface for model access and ban direct provider SDK imports outside it via a lint rule or import guard.

A repository grep for provider SDK client instantiations outside the abstraction module returns zero matches, across every service and script in the repo — not just the flagship application.

A networked LLM API gateway buys centralized credentials, cross-team fallback policy, and canarying, at the cost of a service to run, an extra hop, and a new failure mode. Below the scale that needs those, an in-process abstraction delivers identical vendor isolation for none of the cost.

Choose in-process for a single process or team; choose a shared library for several same-language services; choose a gateway when multiple services share backends and credentials. Record the choice and the trigger that would change it.

The architecture record names the current level and the specific condition that would justify moving up. Model-agnosticism is verified by the call-site grep above, not by the presence of a gateway.

Per-request override selects a model; task-role lanes express what each model is for. Separating the lane that produces work from the lane that checks it is what stops verification being self-assessment.

Define lanes by role — high-volume/low-reasoning, quality-determining, and verification — and make each independently configurable, including across vendors.

Pointing the verification lane at a different provider than the synthesis lane is a config change requiring no code edit.

Fallback is worth its complexity where the dependency is flaky. In practice search, retrieval, and scraping backends fail far more often than model APIs, so unconditional model-level failover is often complexity spent in the wrong place.

Measure failure rates per external dependency, then implement ordered fallback chains for the ones that actually fail. Where model failover is warranted, put it behind the substitution point so callers see none of it.

Each fallback chain maps to a measured failure rate. Simulating an outage in staging completes via the fallback path and the trace shows which backend served the request.

If self-hosted models require a different call path than hosted APIs, the pipeline develops two lineages and swaps stop being free.

Register self-hosted servers through the same surface used for hosted providers. Most multi-provider client libraries and gateways support this via OpenAI-compatible endpoints.

A single integration test that varies only the model identifier exercises a hosted and a self-hosted backend, and both pass.

If onboarding a provider requires editing pipeline, evaluation, or observability code, the abstraction has already broken.

Extend the provider list via the abstraction's declared config surface. No pipeline, evaluation, or tracking code should be touched.

A dry-run PR adding a provider modifies only configuration. CI passes without changes elsewhere.

Abstraction and observability components are load-bearing; discovering a license conflict after adoption is expensive.

Read each dependency's LICENSE directly rather than trusting a summary. Record the license string next to the pinned version in the dependency manifest.

Every load-bearing dependency lists an explicit license and pinned version.

Default model ids scattered across services drift. Ours reached two different answers across four files, which is invisible until a release behaves differently depending on which entry point ran.

Resolve every model id from one config module or environment contract. No hardcoded model strings in application code.

A grep for model-id string literals outside the config module returns zero matches.

This is the control teams most often skip and most often regret. Portability without evaluation makes model swaps easy and tells you nothing about whether a swap was good. Inline self-checking cannot detect regression across versions — only a stable corpus can.

Freeze a representative set of inputs and expected properties. Replay the deterministic stages against them and assert on output quality. Run it in CI on every prompt or model change.

Changing a prompt or model id produces a diffable quality report before merge.

Tests bound to a vendor SDK cannot compare across backends and silently block the swaps the pipeline was built to allow.

Point the evaluation harness at the same interface production uses, parameterized by model identifier.

The harness accepts a model id as a parameter and produces a per-model report with no code changes.

Without a defined metric set, "did the swap regress quality?" has no answer.

Choose metrics appropriate to the task, store thresholds in versioned config, and fail the build on regression. Any competent LLM evaluation library will do; having one matters far more than which.

A release evaluation run emits every defined metric and fails the build when a threshold regresses.

A verifier that asks "do the sources support this sentence" cannot ask "are these sources good enough to support this conclusion." A report can score full marks on the first while resting entirely on one weak source. We shipped exactly that report.

Add a separate assessment of source diversity and sufficiency — for example, refusing to let any section rest on a single domain — and report it independently of per-claim verdicts.

Output carries both a per-claim verdict summary and a corpus-quality assessment, and they can disagree.

A release process that only tests the incumbent backend cannot certify a swap.

Add a job that evaluates the primary backend and at least one alternate, and persist both reports.

Every release artifact carries at least two backend evaluation reports.

Per-call spans answer "what did this request cost." Only a trace spanning the complete task answers "what did this deliverable cost," which is the question anyone actually asks. Multi-process pipelines lose this by default.

Open a root span at the entry point and propagate W3C trace context (traceparent) into every spawned process or job.

One trace id resolves every model call made in service of a single completed unit of work.

A price table checked into the repo is stale within weeks, and stale cost data is worse than none because it is quietly wrong.

Attach model id and input/output token counts — including cache reads, which are separately priced and can dominate spend on prompts that resend a large shared context — to each generation span.

Cost per unit of work is answerable from the tracing backend without any repo-side price constant.

A retry that eventually succeeds is invisible if the call is recorded once. Those tokens are spent and the failure rate is real.

Emit one generation observation per attempt, marking retries distinctly.

A forced schema failure in staging produces multiple attempt observations with their own token counts.

Observability that can break the pipeline it observes is worse than none. The tracing-off path is usually the default, so it is the one most likely to ship broken.

Degrade to a no-op when credentials, packages, or the collector are absent. Wrap every backend call. Give the no-op the full interface, not a subset.

A test asserts the pipeline completes with tracing disabled and with an unreachable collector. Both are covered in CI.

Two runs that differ in preset, budget, prompt revision, or which backend served retrieval can read identically in their output. Without stamping, they are not comparable and nobody can tell.

Write a run manifest recording configuration, resource budgets, a fingerprint of the prompt set, which external backends served results, and elapsed time against budget. Surface truncation explicitly.

Any completed run can be reconstructed from its own manifest, and a truncated run says so.

Retrieval is the irreducibly non-reproducible stage — the web changes. Freezing it makes everything downstream replayable, which is the reproducibility property that actually matters.

Append every retrieved input to an immutable per-run record as it arrives, and synthesize from that record rather than from live state.

A past run can be replayed from its frozen record with no network access.

Output is a function of prompts, and an unversioned prompt change is an unattributable quality change.

Record a fingerprint of the prompt set in every run manifest — hashing the prompt source keeps this honest with no bookkeeping.

Every run records a prompt fingerprint, and editing any prompt changes it.

A model name is a moving target; providers update behind stable aliases.

Reference the most specific version identifier the provider exposes and record it per run.

Run manifests carry an exact model version string, not a family alias.

Claiming reproducibility you cannot deliver creates false confidence. Replay reproduces the reasoning, not the tokens.

State in release documentation that outputs are not bit-identical and that replay covers the deterministic stages only.

Reproducibility documentation carries the caveat explicitly.

Seeds and decoding pins are the first layer of determinism — necessary, though not sufficient.

Set temperature=0, top_p=1, greedy decoding, a pinned seed, and speculative decoding disabled. Persist all five in the run manifest.

Every release-blocking run's manifest lists the decoding pins and seed.

A framework upgrade can silently change kernel selection, which changes outputs.

Lock the serving framework in the dependency manifest and document CUDA and framework versions in the environment file.

Reproducing an old run rebuilds the same environment from recorded fields.

GPU generations differ at the kernel level, and batch size and device count shift computation order. Reasoning models are especially vulnerable, where early-token floating-point drift cascades into divergent chains of thought.

Schedule release-blocking runs on a fixed GPU SKU and count, with single-request batches.

Manifests record the hardware profile; scheduling on a different SKU is blocked or flagged.

Deterministic-algorithm flags raise the floor but do not guarantee bit-identical output. Claiming otherwise creates false confidence.

Enable the framework's deterministic-algorithm settings and document that residual nondeterminism is expected.

Configuration shows the flags set and documentation carries the best-effort caveat.

Mixing sampled and greedy runs in one report is uninterpretable.

Per evaluation, pick either sampling with multiple runs reporting mean and error bars, or greedy decoding at full precision with a single run. Record the choice.

Every evaluation report names its protocol and carries the matching statistics.

Silently omitting controls implies coverage that does not exist.

List what this checklist does not cover — orchestration-framework choice, secrets and credential handling, adversarial-input safety, and abstraction-overhead benchmarking — with a named owner for each. Reliability patterns for tool-driven agents belong in tool-use reliability hardening.

Release documentation carries a named-owner entry for each gap.