Designing Generative AI Tutors That Protect Learning, Not Just Practice Performance

A tutor that grounds only in flat document chunks cannot reason about which knowledge component a learner is missing or which misconception is firing. Without explicit objective → KC → misconception → representation → transfer-class structure, hint selection collapses into "say something relevant," which is the regime where helpfulness substitutes for learning.

Author the domain model as a versioned data artifact (YAML, database, or knowledge graph) separate from prompts. Every instructional item, hint, and assessment item references a KC and, where applicable, a known misconception. The orchestrator selects content by KC and misconception, not by embedding similarity alone.

A reviewer can query "show all hints for KC X tagged against misconception Y" and get a non-empty, reviewed result. No hint reaches the learner without a KC tag in the log.

A definition, a misconception warning, a Socratic prompt, a worked example, and a rubric play different instructional roles. Untyped retrieval lets the model use a worked example where it should have used a prompt, which is one mechanism by which tutors substitute for productive struggle.

Type every corpus item at ingestion (definition, misconception, prompt, worked-example, rubric, transfer-item). The help policy decides which types are eligible at the current hint level and mode; retrieval filters on type before ranking.

Logs show the eligible type set and the selected type for each turn. A randomly sampled session can be replayed to confirm no answer-revealing types appeared before the policy permitted them.

Hallucinated instruction is worse than no instruction; it teaches incorrect content under the tutor's authority. Without a refusal path, the model will fluently answer items the curated corpus does not cover.

When retrieval confidence falls below a threshold or no KC matches, the tutor returns a scoped refusal ("this is outside what I've been prepared to teach") and routes to a human or to a "log for content authoring" queue.

Synthetic out-of-corpus probes in the regression suite consistently trigger refusal rather than confident answers. The content-gap queue is non-empty and reviewed.

Generative AI changes which behaviors predict learning. A learner can complete every item correctly while substituting the tutor for cognition. Modeling only knowledge and misconceptions misses the failure mode the PNAS 2025 result exposed.

Add at least four constructs to the learner model: knowledge state, misconception state, self-regulation state, and help-seeking state (productive vs. substitutive). Update help-seeking from per-turn signals: time to first attempt, hint level reached, explanation quality, and the assisted-vs-unassisted correctness gap.

Every session writes a help-seeking trajectory. Aggregate dashboards distinguish learners with high assisted correctness and low unassisted correctness from learners gaining on both.

Substitution leaves measurable traces — rapid hint escalation, "just tell me" patterns, copy-paste from tutor output to submission, skipped self-explanation prompts. If these are not logged, no dashboard can detect the harm.

Emit structured events for hint-level transitions, time-to-first-keystroke, answer-reveal requests, explanation-prompt skips, and clipboard-style paste events into submissions where the platform can observe them. Aggregate per learner and per KC.

A reviewer can pull the substitution dashboard for any cohort and see distributions, not just averages. At least one alarm threshold (e.g., session-level hint-ladder skip rate) is wired.

A help policy buried in a system prompt cannot be reviewed, tested, or governed. Different model versions will interpret the same prose differently, and instructional designers cannot revise policy without prompt-engineering access.

Maintain the help policy as code or structured config that the orchestrator reads — when to ask vs. hint vs. show a worked example vs. reveal an answer vs. request self-explanation vs. escalate. The system prompt enforces the policy's outputs but does not encode the policy.

The policy artifact has a version, an owner, and unit tests. A change to the policy produces a diff, not a prompt rewrite.

Without an attempt gate, the cheapest path through an item is to ask the tutor first. Bastani et al. (PNAS 2025) is the operational evidence that this trains a habit which survives the tutor's removal at exam time.

Practice mode requires the learner to commit to something — a prediction, an identified quantity, a chosen formula, partial code, or a stated reasoning step — before the tutor will provide more than an orienting question. Worked-example mode replaces the gate with a comparison or self-explanation requirement.

Logs show a non-empty learner attempt preceding any hint at level ≥ 2 in practice mode. Sessions where this invariant is violated are flagged as bugs.

Tutors that flatten hints into "best response" lose the gradient that protects effort. The level the learner reaches is also the most useful single feature for detecting substitution and over-help.

Define a discrete ladder — orienting question → relevant principle → local error cue → partial step → worked example → final answer (only when justified). The orchestrator chooses the next level based on policy and learner state, never skipping unless the policy permits it.

Every helpful turn logs hint_level. Per-KC distributions of hint level are inspectable. Learners who routinely reach the highest level surface in the substitution dashboard.

The same prompt should produce different behavior in practice, worked-example, and assessment contexts. Blurring the boundary is how assessment integrity erodes and how worked-example fading collapses into answer-giving.

The orchestrator maintains an explicit mode. Practice withholds answers; worked-example shows but requires comparison or fading; assessment disables AI help entirely. The UI surfaces the current mode to the learner. Homework/study contexts default to practice mode unless the instructor configures otherwise.

Mode is recorded on every turn. A simulated assessment session confirms the tutor refuses help. UX review confirms the learner can see the active mode without hunting for it.

Solving ability is not tutoring ability — see the solver-tutor gap. In math, code, and proof, an LLM that asserts correctness without a verifier will eventually authorize wrong answers fluently, and the tutor's authority makes that worse than a search result.

Route formal claims through a symbolic solver, code sandbox, or proof checker before delivering them. Where a verifier disagrees with the model, the tutor either rephrases as a hint or escalates rather than overriding the verifier.

Verifier-disagreement events appear in the log and trend toward zero with content fixes. No final-answer turn in a formal domain ships without a verifier pass.

MathTutorBench (Macina et al., EMNLP 2025) shows tutoring quality degrades over multi-turn dialogues. Without a coherence layer, the tutor forgets which hints it has given, which misconceptions remain, and which reasoning steps the learner has committed to.

Carry a structured dialogue state — attempts made, hints given by level, misconceptions still active, reasoning steps the learner has staked — separate from the chat transcript. The orchestrator reads from this state, not from raw history, when choosing the next move.

Replaying a session deterministically reproduces the orchestrator's decisions from the dialogue-state log. Long-dialogue regression tests do not show degradation in scaffolded-help selection.

Practice correctness with the tutor available is the metric most likely to mislead. Without scheduled unassisted items, the team cannot detect the PNAS-style harm before it ships.

Insert periodic checkpoint items the learner solves with AI help disabled. Include near-transfer and at least one far-transfer class per KC cluster. Report assisted-vs-unassisted gap by learner, KC, and cohort.

The cohort dashboard plots assisted and unassisted correctness side by side. Gap-widening is treated as a release-blocking signal, not a curiosity.

"AI tutor vs. nothing" is not a defensible comparison. Existing adaptive practice, active-learning lectures, and human tutoring have known effect sizes; a new tutor must be measured against one of them, and tutoring-quality benchmarks are how to catch regressions before learners do.

Use tutor-specific benchmarks (MathTutorBench, MRBench, TutorBench, BEA 2025) for response-quality regression. For learning evaluations, run RCT or quasi-experimental designs against an active comparator — not an unguarded LLM and not a no-tutor control alone.

A standing benchmark report exists per release. Outcome studies are pre-registered or at minimum design-reviewed before launch. Comparator is named in the study plan.

A silent provider model upgrade can change help-policy behavior, verifier disagreement rates, and substitution patterns overnight. Treating the model as a stable commodity ignores the only component most likely to change without notice.

Pin model versions per environment. Log the version on every session. Gate upgrades on the tutor-benchmark regression suite and on a smaller live-cohort revalidation of unassisted gains before broad rollout.

Session logs include model_version. A rollback path exists and has been exercised. No production upgrade has shipped without a passing regression report.

Some signals are not the tutor's to handle — repeated misconception after multiple scaffolds, affective distress, integrity risk, ambiguous correctness, low-confidence/low-performance combinations, and high-stakes assessment prep. The tutor's defaults will paper over these without an explicit route.

Encode escalation rules in the help policy. Surface them to the responsible facilitator (instructor, coach, support staff) with the conversation context the human needs. For deployments without a human in the loop, the escalation becomes a documented refusal plus a referral.

Escalation events are auditable. A facilitator can show which escalations they received last week and what they did about them. No escalation rule is dead code.

Time-on-task and engagement dashboards are precisely the metrics on which a substituting tutor will look successful. The facilitator needs the metrics that distinguish learning from completion.

The facilitator view surfaces top misconceptions, students needing review, KCs with high assisted but low unassisted performance, over-help patterns, and explanation-quality trends. Engagement metrics are present but not primary.

A facilitator can answer "which of my students has the largest assisted-vs-unassisted gap on KC X?" without opening a notebook. The dashboard's primary chart is not time on task.