Glitcher ASR Validation Bug and Stratified-Vocabulary Audit
Glitcher ASR Validation Bug and Stratified-Vocabulary Audit
Version: 1.1 (corrected) Original publication: 2026-02-27 Correction date: 2026-09-20 Author: Jeremy (Independent Security Researcher) Repository: https://github.com/binaryninja/glitcher License: MIT Companion to: Glitcher: A Research Toolkit for Detecting, Characterizing, and Steering Glitch Tokens in Large Language Models (v2.0, 2026-02-25)
Correction notice: Version 1.0 described a full-vocabulary census of 128,000 token IDs and reported 4,954 candidates, 1,131 confirmed tokens, and a ten-attempt ASR distribution. The repository does not contain the raw data or run metadata needed to substantiate those claims. The checked-in evidence is a 2,000-token stratified sample evaluated with three attempts per token. Version 1.1 withdraws the unsupported census counts and rewrites the results around the auditable artifacts.
Abstract
This follow-up audits Glitcher's multi-attempt anomaly score and compares an entropy-guided mining run with a stratified vocabulary sample for meta-llama/Llama-3.2-1B-Instruct.
The historical validator used greedy decoding for every attempt. For a fixed prompt and environment, those attempts repeated the same generation and forced the measured rate to an endpoint: 0% or 100%. A checked-in 18-token control audit confirms that all outputs were identical before the sampling patch and varied after it. All 18 control tokens had a measured anomaly rate of zero in both runs, so this audit demonstrates output variation rather than detector accuracy.
The checked-in vocabulary artifact records 283 detector-positive token IDs among 2,000 stratified samples at a threshold of 0.5. Each token had three attempts: 189 positives scored 3/3 and 94 scored 2/3. Of these 283 token IDs, 6 also appear in the 255-token output of the checked-in entropy-mining run; 277 do not. This establishes a coverage difference between those two runs. It does not establish vocabulary-wide prevalence, entropy mining recall, or a full census because the sample deliberately oversampled selected categories and the artifact discarded most negative records.
The audit also found that the sampling patch reseeded generation from wall-clock time, overriding the CLI's --seed promise. The tool now derives stable seeds from the user seed, token ID, attempt, and probe; records the decoding configuration; and leaves the caller's random-number stream intact when no seed is supplied.
Table of contents
- 1. Scope and evidence
- 2. Experimental artifacts
- 3. Confirmed ASR implementation bug
- 4. Corrected sample results
- 5. What the coverage comparison shows
- 6. What the results do not show
- 7. Tool corrections
- 8. Reproducibility
- 9. Recommendations
- 10. Limitations
1. Scope and evidence
This correction treats the repository as the source of record. It distinguishes three questions:
- Did deterministic decoding make repeated attempts redundant?
- Does the checked-in sample find detector-positive tokens outside one entropy-mining run?
- Can the artifacts support a vocabulary-wide count or operational severity taxonomy?
The answer to the first two questions is yes. The answer to the third is no.
The principal evidence is pinned to repository commit ef34b59, which added the audit code and data. Later commits changed production tooling but did not add a raw full-vocabulary run matching version 1.0 of this paper.
| Artifact | What it supports | Important limit |
|---|---|---|
data/asr_audit_unpatched.json | 18/18 control tokens produced identical outputs across greedy attempts | All 18 measured 0% anomaly rate; this is not a before/after positive-token ASR distribution |
data/asr_audit_patched.json | 0/18 control tokens had identical outputs after sampling | All 18 still measured 0%; variation alone does not validate the indicator |
data/vocab_scan_results.json | Summary and retained records for a 2,000-token stratified sample | Only 383 records remain: all 283 positives plus 100 sampled negatives |
data/mining_results.json | 255 unique token IDs from a 50-iteration entropy-mining run | It is one run, not the complete output of the mining method |
experiments/vocab_scanner.py | Sampling and detector implementation used for the checked-in scan | It used wall-clock reseeding and retained incomplete negative evidence |
2. Experimental artifacts
2.1 Recorded environment
The audit files identify the model as meta-llama/Llama-3.2-1B-Instruct. The accompanying findings document records an NVIDIA GeForce RTX 5090, bfloat16 precision, PyTorch 2.12.0.dev20260225+cu128, and Transformers 5.2.0. The result JSON does not embed a complete environment snapshot, so these values come from the audit report rather than the raw scan artifact.
2.2 Stratified sample
The scanner selected 2,000 token IDs using:
- equal-size samples from five embedding-L2-norm quintiles;
- additional sampling of whitespace, CJK, Cyrillic, and code-fragment categories;
- a final random trim to 2,000 unique IDs;
- sampling seed 42.
This is useful for finding examples outside low-norm neighborhoods. It is not a simple random sample of the vocabulary, and the raw inclusion probability for each token is not recorded. The observed positive fraction therefore must not be extrapolated directly to the full vocabulary.
2.3 Detector and score
For each token and each attempt, the scanner ran three prompts: repeat the token, explain its meaning, and repeat it with alternate phrasing. A probe failed when neither the token ID nor its decoded text appeared in the completion. An attempt counted as anomalous only when all three probes failed. The reported score was:
measured anomaly rate = anomalous attempts / total attempts
The code calls this value ASR. In this paper, it should be read as a detector-specific anomaly rate under fixed prompts and decoding settings, not as an intrinsic token property or a general attack-success probability.
The checked-in sample used three attempts, temperature=0.7, top_p=0.95, a maximum of 50 generated tokens, and a confirmation threshold of 0.5.
3. Confirmed ASR implementation bug
3.1 Root cause
Before commit 20c3609, all four model.generate() call sites in enhanced_glitch_verify used do_sample=False. Repeating greedy generation with the same prompt cannot measure sampling variability. It repeats the same decoding rule and, absent nondeterministic runtime effects, the same output.
The multi-attempt score consequently collapsed to 0/N or N/N. Increasing num_attempts consumed more compute without adding independent observations.
3.2 Direct audit
| Audit condition | Tokens | Attempts per token | Tokens with identical outputs | ASR endpoints |
|---|---|---|---|---|
| Greedy, historical behavior | 18 | 10 | 18 (100%) | 18 at 0%; 0 at 100% |
| Sampling, patched harness | 18 | 10 | 0 (0%) | 18 at 0%; 0 at 100% |
This is strong evidence for the decoding bug and its mechanical fix. It is not evidence that sampling changed any of these 18 classifications, because their classifications did not change.
3.3 Correct interpretation
Sampling makes repeated generations capable of varying. It does not make them statistically independent in a strict sense, validate the prompts, establish a ground-truth glitch label, or make a small number of attempts a precise probability estimate. Those properties require separate study.
4. Corrected sample results
The 2,000-token scan artifact reports 283 detector-positive tokens at ASR >= 0.5.
| Observed result | Count | Share of all 2,000 sampled IDs | Share of 283 positives |
|---|---|---|---|
| 3/3 anomalous attempts (ASR 1.0) | 189 | 9.45% | 66.8% |
| 2/3 anomalous attempts (ASR 0.667) | 94 | 4.70% | 33.2% |
| Confirmed at threshold >= 0.5 | 283 | 14.15% | 100% |
| Below threshold | 1,717 | 85.85% | — |
Three attempts permit only four possible estimates: 0, 1/3, 2/3, and 1. The artifact retains four below-threshold records at 1/3 and a 100-record subsample of negatives, but it discards the other negative per-token records. The paper therefore reports the positive distribution and aggregate negative count without inventing a more detailed histogram.
An observed 3/3 does not mean a token is deterministic, universally broken, or guaranteed to fail in production. It means three sampled attempts all satisfied this detector under this model, prompt set, and decoding configuration.
4.1 Embedding norms
Among the 277 sample positives absent from the entropy run's output, 143 (51.6%) had an embedding L2 norm within one standard deviation of the vocabulary mean recorded by the scanner. This supports a narrow conclusion: a low-norm-only candidate rule would not cover every token found by this detector in this sample.
The repository also reports a two-sample KS statistic of 0.895 with p=4.8e-6, comparing the 277 sample-only positives with the 6 overlapping positives. Because one group contains only six observations and the sample was stratified, this is exploratory evidence rather than a stable population estimate.
5. What the coverage comparison shows
The checked-in entropy run contains 255 unique token IDs. Comparing that set with the 283 detector-positive IDs from the stratified sample gives:
| Set relationship | Token IDs |
|---|---|
| Positive in stratified sample and present in entropy-run output | 6 |
| Positive in stratified sample and absent from entropy-run output | 277 |
| Total positives in stratified sample | 283 |
The direct result is that this stratified run surfaced 277 detector-positive token IDs that one 50-iteration entropy run did not return. That is useful evidence for broadening candidate generation beyond a single embedding-guided run.
It is not valid to call 6/283 the recall of entropy mining. The scan covered only part of the vocabulary, the sample was not uniform, the mining run had a finite search budget, and most negative sample records were removed. Likewise, the 249 entropy-run IDs absent from the retained scan file cannot be described as "not sampled": some may have been sampled, classified negative, and discarded during artifact trimming.
6. What the results do not show
Version 1.0 made claims beyond the checked-in evidence. The following are withdrawn:
- No full-vocabulary census: there is no auditable 128,000-token result artifact supporting 4,954 candidates.
- No 1,131-token confirmation set: the repository does not contain the claimed ten-attempt revalidation data.
- No 3.9% vocabulary hit rate: the auditable 14.15% is from a deliberately stratified sample and is not a population estimate.
- No continuous 0–100% distribution: three attempts yield a four-point grid. The positives occupy 2/3 and 1.
- No hard/soft/marginal severity taxonomy: three successes do not establish deterministic failure, and detector response has not been tied to downstream harm.
- No validated false-positive reduction: the artifacts do not include a labeled ground-truth set sufficient to estimate precision or false-positive rate.
- No production blocklist recommendation: repeatability under three research prompts is insufficient evidence for universal token blocking.
- No 2–3 hour full-scan runtime claim: no matching full-run log is checked in.
7. Tool corrections
7.1 Stochastic multi-attempt validation
Multi-attempt validation uses sampling; single-attempt validation remains greedy. The default multi-attempt settings are now:
do_sample = True
temperature = 0.7
top_p = 0.95
The temperature and top-p values are CLI options and are written into result metadata. Changing them changes the estimand, so results from different decoding configurations should not be pooled without qualification.
7.2 Reproducible seeding
The earlier patch used the wall clock to reseed each generation. That made outputs vary, but it overrode --seed and prevented exact reruns. The corrected validator accepts an optional base seed and derives each generation seed from:
(base seed, token ID, attempt index, probe index)
With a seed, the same token/probe/attempt schedule receives the same pseudorandom draws. Without a seed, the validator no longer overwrites the caller's random-number state with the current time.
This improves application-level reproducibility. Exact bitwise reproduction can still vary across GPU models, library versions, kernels, and nondeterministic CUDA operations.
7.3 API and caller fixes
The validator returns (is_glitch, asr). Two bundled callers treated that tuple as a single boolean, which is always truthy when nonempty. They now unpack and store both values. Parameter validation also rejects zero attempts, invalid thresholds, nonpositive sampling temperatures, and invalid top-p values.
7.4 Result metadata
CLI-managed test, comparison, entropy-mining, range, and stratified-scan outputs now record an environment/decoding block where the command controls the output file. The narrower wording matters: auxiliary scripts and manually written JSON files do not automatically receive metadata.
8. Reproducibility
The repository's APPENDIX-B/ directory contains environment collection, comparison, probe-template, control-token, and redaction utilities. The configuration template now matches the validator: one attempt records greedy decoding with null sampling parameters; multi-attempt snapshots record temperature=0.7 and top_p=0.95 unless overridden.
A reproducible validation command should specify at least:
glitcher test meta-llama/Llama-3.2-1B-Instruct \
--token-ids "..." \
--enhanced \
--num-attempts 10 \
--asr-threshold 0.5 \
--temperature 0.7 \
--top-p 0.95 \
--max-tokens 50 \
--seed 42 \
--output test_results.json
For a publishable experiment, retain all per-token records, the exact sampled ID list, raw or redacted per-attempt outputs, the code commit, model revision and tokenizer revision, decoding settings, hardware/software snapshot, and a cryptographic digest of each result artifact.
9. Recommendations
9.1 For detection research
- Use multiple candidate generators, including embedding-guided search and preregistered random or stratified samples.
- Define the target behavior before scanning. Failure to repeat a token is one behavior, not a universal definition of a glitch.
- Include labeled positive and negative controls to estimate detector precision and sensitivity.
- Preserve every sampled ID and outcome so sampling weights and overlap can be audited.
9.2 For rate estimation
- Use enough attempts to support the desired resolution and report uncertainty intervals.
- Treat ASR as conditional on model revision, prompt templates, sampling settings, and serving stack.
- Separate deterministic greedy tests from stochastic rate estimation and name each clearly.
- Do not convert sample incidence into a vocabulary total without known inclusion probabilities and a defensible estimator.
9.3 For mitigation
- Reproduce a token's effect in the actual downstream task before applying a mitigation.
- Compare blocklisting with contextual validation, tokenizer-aware normalization, output checks, and task-level retry policies.
- Measure collateral effects: whitespace, punctuation, multilingual text, and code fragments are common inputs, so broad token blocks can damage normal behavior.
10. Limitations
- Artifact audit rather than independent rerun. This correction recomputes counts from checked-in JSON and reviews source history. It does not claim a new model run.
- Incomplete negative records. The vocabulary artifact retained all 283 positives and only 100 negatives, preventing a full per-token audit of the 2,000 selections.
- No ground truth. Detector-positive means failure under the implemented probes, not a confirmed tokenizer defect or security vulnerability.
- Single model and precision. The artifacts cover one instruct model under bfloat16.
- Coarse rate estimates. Three attempts distinguish only 0, 1/3, 2/3, and 1.
- Sampling bias. The stratified design oversampled selected token categories and does not record final inclusion weights.
- Prompt dependence. A token can pass or fail because of prompt wording, chat templating, token position, or output length.
- Serving-stack dependence. Kernel, library, device, and generation implementation differences may change sampled outputs.
- Detector semantics. Requiring all three probes to omit the target is a design choice that has not been calibrated against human labels or downstream failures.
Changelog
| Date | Version | Change |
|---|---|---|
| 2026-02-27 | 1.0 | Initial publication |
| 2026-09-20 | 1.1 | Withdrew unsupported full-vocabulary counts; replaced them with the auditable 2,000-token stratified-sample results; corrected ASR interpretation, coverage claims, and tool reproducibility details |