Glitcher: Glitch-Token Discovery and ASR Validation

Glitcher is an open-source Python toolkit for finding under-trained or behaviorally anomalous tokens in modern LLM tokenizers and measuring their behavior under fixed probes. It is the runtime behind the Glitcher v2.0 research note and the corrected stratified-vocabulary audit, and it sits in the security pillar of this site as a research measurement tool.

What it does

Glitcher implements three families of glitch-token discovery and a validation harness on top of them:

• Embedding-norm thresholding — flags vocabulary entries whose embedding norms fall outside the expected distribution, a fast first pass that surfaces obvious under-trained candidates.
• Gradient-guided behavioral mining — uses gradient signal from a target prompt to walk the vocabulary toward tokens that induce specific failure modes.
• Range and stratified vocabulary scans — probes contiguous ranges or preregistered samples across embedding-norm and text categories.
• Multi-attempt ASR validation — scores each candidate across stochastic samples rather than repeated greedy decoding. The earlier determinism bug is documented in the methodology audit. The validator now supports reproducible seeds and records its sampling configuration.

The intended use cases are tokenizer audits before model release, red-team baselining of an open-weights model's vocabulary surface, and reproducing or extending the published sample-based audit.

Who it's for

ML engineers and platform owners who need to know what their tokenizer ships, security researchers studying input attack surface against LLMs, and red teamers building a baseline against a specific target. Glitcher is a research-grade measurement tool; it is not a jailbreak generator and it is not a managed service. If you are looking for a turnkey production scanner, this is not it.

How to use it

The published runs were validated against meta-llama/Llama-3.2-1B-Instruct on an RTX 5090 in bfloat16. The toolkit is model-agnostic in design but has only been exercised end-to-end against that target in the public audit. A typical first run is a vocabulary scan against your own tokenizer and target model, executed inside a research environment with no production traffic on the same host.

See the repository README for the full CLI surface, configuration options, and reproduction recipes for the published experiments. The README is the source of truth for flags and install steps; this page is the editorial overview.

Status and roadmap

Glitcher is active. The most recent work, in early 2026, was the methodology audit that corrected the ASR validator's determinism bug and added temperature-sampled multi-attempt probes; those patches have been prepared for the repository. Known limitations: only one model family has been validated end-to-end, and the gradient-guided strategy is sensitive to the choice of seed prompt. Roadmap items are tracked in the repository rather than here.

Source and license

The canonical source lives at the GitHub repository linked above. The repository's license file is the authoritative statement of redistribution terms; this page does not assert a license that has not been confirmed.

Responsible use. Glitcher demonstrates a class of LLM input weakness — under-trained tokens that confound filtering, logging, and extraction pipelines. It is a tool for auditing models you operate or are authorized to test, not for probing third-party endpoints. Use it against research deployments, not production systems or someone else's API.

Related research

• Glitcher v2.0 research note — the originating paper documenting the toolkit's design.
• Corrected stratified-vocabulary follow-on — the auditable sample results, withdrawn census claims, and ASR interpretation.
• ASR validation methodology audit — the determinism-bug correction and the case for temperature-sampled ASR.
• What are glitch tokens? — threat-model framing for readers who are new to the concept.

FAQ

Do I need a GPU to run Glitcher?

Small probe sets can run on CPU, but model-wide or large stratified scans are designed for a CUDA-capable GPU. Runtime depends on model size, vocabulary size, generated-token budget, and attempts per token; the published artifacts do not include a completed full-vocabulary runtime benchmark.

Is Glitcher safe to run against a production model?

Run it against a research or staging deployment. The probes are content-safe by design but they deliberately exercise the edge-case behavior of the tokenizer and model, which is exactly the surface you do not want hitting production traffic paths or shared logging pipelines. Treat the output as red-team data and keep it off live request paths.