A consequence-aware retrieval scoring function that fixes a real failure mode in RAG-based storytelling. This is the research behind the Kalapatha interactive fiction engine.
Kalapatha is a full-stack AI storytelling system where player choices actually matter to what the LLM remembers. Standard RAG retrieves memories by semantic similarity alone, which means a betrayal from ten turns ago gets buried under irrelevant small talk about the weather. Kalapatha fixes this with a custom retrieval scoring function, Narrative Relevance Score (NRS), that weighs narrative importance alongside similarity. The approach is written up as a research paper (included in this repo) and implemented end to end as a working product.
Paper PDF · Backend · Frontend
In multi-turn AI storytelling, context windows are limited and memory has to be retrieved, not just appended. Every existing RAG pipeline ranks retrieved memories by cosine similarity to the current query. This works for factual QA. It breaks for stories.
Example: a player betrays their closest companion in beat 4. By beat 13, the story reaches its climax during a rainstorm. Cosine retrieval pulls up other rain-related beats because they're semantically closest to the query, so the betrayal, despite being the emotional core of the story, never makes it into context. The LLM generates a climax that has no idea what the player actually did.
We call this narrative importance displacement, and it's the central problem the paper addresses.
NRS re-ranks retrieved memories using three signals instead of one:
NRS(d, q) = α · similarity(d, q) + β · |consequence delta| + γ · recency decay
- Similarity: standard cosine similarity, same as baseline RAG
- Consequence weight: how morally/dramatically significant that memory was, scored by the LLM at generation time
- Recency decay: exponential decay so old beats don't dominate unless they're actually important
The weights are tuned so that a high-consequence beat from ten turns ago consistently outranks a low-consequence beat from one turn ago, which is the intended behavior for storytelling and the opposite of what plain cosine similarity does.
Tested across 200 auto-generated stories against a cosine-only baseline, using LLM-as-judge and automated coherence metrics:
| Metric | Baseline | Full NRS | Improvement |
|---|---|---|---|
| Character Consistency | 0.641 | 0.721 | +12.4% |
| Causal Chain Integrity | 0.618 | 0.678 | +9.7% |
| Consequence Fidelity | 0.720 | 0.779 | +8.2% |
| Narrative Drift (lower is better) | 0.381 | 0.327 | 14.2% lower |
Gains on character consistency and causal chain integrity are statistically significant (p < 0.01 and p < 0.05 respectively, paired t-test). An ablation study confirms the consequence-weighting term is the primary driver of the improvement, not just the recency decay.
Kalapatha is a production-shaped system, not a notebook experiment.
- Backend: FastAPI, orchestrating five concurrent AI subsystems
- Narrative engine: Llama 3.3 70B served via Groq's LPU inference, roughly 400 tokens/sec, sub-1.5s latency per story beat
- Memory: three-layer architecture (static world lore, per-session memory ranked by NRS, cross-session "world echoes" that persist across playthroughs)
- Retrieval store: ChromaDB with all-MiniLM-L6-v2 embeddings
- Voice: local Whisper for speech-to-text, ElevenLabs for text-to-speech
- Images: FLUX.1 for scene generation
- Frontend: React + Framer Motion, communicating over REST and server-sent events
Every player action triggers this sequence: embed input, retrieve candidates by cosine similarity, re-rank with NRS, assemble context from all three memory layers, then generate the next story beat. Full pipeline runs in under 1.5 seconds.
The core idea, augmenting semantic retrieval with a domain-specific importance signal, isn't specific to fiction. The same pattern applies anywhere similarity search alone misses what actually matters: legal retrieval (precedential value), medical history (clinical severity), tutoring systems (concept centrality or error recency). NRS is a general template for domain-aware re-ranking, and interactive narrative is just where it was validated.
- kalapatha-api: FastAPI backend with the NRS retrieval engine and three-layer memory architecture
- kalapatha-ui: React frontend
- The same consequence-weighted retrieval approach also powers Antaryami, a 3D Telugu mystery-thriller built in Godot 4, as a second applied implementation of this research.
Buvananand Vendotha, Vivek Chandra Telukuntla, Manohar Cherukuri, with Asst. Prof. B. Ranjith Kumar, Dept. of CSE, MVSR Engineering College, Hyderabad