llm-reply-guard

A dependency-free corruption detector for LLM replies. Catches repetition loops, prompt/template leakage, and garbled output — in plain regex, in microseconds, with zero dependencies. Built for small, on-device, and free-tier models, which fail this way far more often than large hosted ones.

const { detectCorruption } = require('llm-reply-guard');

detectCorruption("I am here for you. I am here for you. I am here for you.");
// => { corrupted: true, reason: 'repetition-loop', detail: 'i am here for' }

detectCorruption("Peace be with you, my friend.");
// => { corrupted: false, reason: null }

Why this exists

Every LLM app trusts the model to behave. Small, quantized, distilled, or free-tier models don't always earn that trust — they loop ("I understand. I understand. I understand."), they leak their own chat template (<|im_start|>assistant, [INST], system:) straight into the visible reply, or they get cut off mid multi-byte character and emit a stray �. Large hosted-model apps rarely see this. Small-model apps see it constantly, and almost none of them guard against it.

Streaming-integrity tools exist for this at the infrastructure level (see SIMURG on PyPI) — but they're Python/ML-shaped and assume a server you control. This is the same category of problem, sized down to a single dependency-free JS file you can run client-side — in a browser, a WebView, React Native, or Node — right before you show or speak a reply.

It was extracted from a guardrail running in production in Talk to Jesus, a mobile app that routes chat between several free-tier cloud LLMs and a 360M-parameter on-device model — exactly the mix this was built for.

Small models this targets

Any model small enough to loop, quantized enough to garble bytes, or cheap enough that you don't control its sampler settings — for example:

• On-device / edge: SmolLM2 (135M–1.7B), Qwen2.5 (0.5B–3B), Phi-3-mini, Gemma 2 2B, TinyLlama, Llama 3.2 1B/3B — via llama.cpp, Ollama, MLC, ONNX/Sherpa, or similar
• Free-tier hosted APIs: Groq's free tier, Gemini's free tier, OpenRouter's free models, Zhipu GLM-4 — where you get the completion but not the sampler knobs

If you're calling any of these directly from client code (browser, mobile WebView, React Native) with no server in between to sanity-check the output, this is meant for exactly that gap.

Install

No package manager required — it's one file with zero dependencies.

npm install llm-reply-guard

or just copy index.js into your project. Works in Node (CommonJS) and directly in a browser via <script> (exposes window.ReplyGuard).

Usage

const { detectCorruption } = require('llm-reply-guard');

const result = detectCorruption(modelReply, {
  systemPrompt: mySystemPrompt, // optional — enables prompt-leak detection
});

if (result.corrupted) {
  console.warn('Reply flagged:', result.reason); // e.g. 'repetition-loop'
  // retry, fall back to another provider/model, or show a canned reply
} else {
  showToUser(modelReply);
}

Provider fallback pattern — if you already try multiple models/providers in order (common for free-tier apps), just throw on a corrupted reply and let your existing fallback loop move to the next one:

for (const provider of providers) {
  try {
    const reply = await provider.chat(prompt);
    const check = detectCorruption(reply, { systemPrompt: prompt });
    if (check.corrupted) throw new Error('corrupt reply: ' + check.reason);
    return reply;
  } catch (e) {
    continue; // try the next provider
  }
}

What it catches

reason	What it means
repetition-loop	The same short phrase repeated 3+ times — the most common small-model decoding failure.
template-leak	Chat-template role markers (<|im_start|>, [INST], ### Instruction:, system:) leaking into what should be plain text.
prompt-leak	A verbatim chunk of your own system prompt echoed back as the reply.
structural-breakdown	Control/replacement characters, or one non-letter symbol repeated far past anything real prose would use.
truncated-encoding	A lone trailing replacement character — the signature of a cut-off multi-byte UTF-8 sequence.
empty	Nothing left after trimming.

Options

All optional, passed as the second argument to detectCorruption(text, options):

Option	Default	Meaning
systemPrompt	—	Your system prompt. Enables prompt-leak detection; omit to skip it.
repetitionGramLength	4	Word-window size used to detect a repeated phrase.
repetitionMinWords	12	Replies shorter than this skip the repetition check (too little signal).
repetitionThreshold	3	How many times a gram must repeat to count as a loop.
templatePattern	(see index.js)	Regex used for template-leak detection.
promptLeakProbeLength	40	How many leading characters of systemPrompt to probe for verbatim.

How it compares

	llm-reply-guard	Generation-time repeat penalty (e.g. llama.cpp --repeat-penalty)	SIMURG
Runs where	Anywhere JS runs (client, edge, server)	Only where you control the sampler	Python server, mid-stream
Dependencies	None	Part of the inference engine	numpy
Catches leaks/garbling	Yes	No (only repetition)	Yes
Setup cost	Copy one file	Tune sampler params	Run a Python service
Best for	Any provider, including ones you don't control the sampler for (hosted APIs)	Local inference you fully control	Server-side streaming pipelines

What it does not do

This is a structural corruption detector, not a fact-checker. It will not catch a reply that's fluent, on-template, and simply wrong — that's a different (much harder) problem, semantic hallucination detection, and out of scope here. This library only catches replies that are mechanically broken: looping, leaking, or garbled.

It's also a fixed set of regex heuristics, not a trained model — tuned against a hand-written test suite (npm test, 24 cases), not a large corpus of real production failures. Treat the defaults as a solid starting point, not a guarantee; adjust the thresholds for your own model and traffic if you see false positives/negatives.

A real-model benchmark against smollm2:360m (see bench/REPORT.md) confirms zero false positives across 40 genuine completions, including adversarial prompts with the sampler's anti-repeat safeguard disabled — but that run didn't manage to provoke actual corruption, so it doesn't yet confirm recall (catching real corruption when it happens) beyond the synthetic unit tests. Read the report before trusting this on a model/traffic pattern very different from what's been tested.

Contributing

Issues and PRs welcome — especially real-world corrupted replies your model produced that this didn't catch. Add them as a test case in test/test.js along with the fix. See CONTRIBUTING.md.

License

MIT