RazorRecon

Built for the Razorpay AI Buildathon, AI Finance Controller track.

Deployed Link : https://razorrecon.streamlit.app/

First load can take ~30s: the free-tier host sleeps after a period of inactivity and wakes on the first visit, then generates and reconciles the 5,000-record benchmark. Reloading during that window is safe - the app detects a half-finished startup and rebuilds rather than showing partial results.

RazorRecon is a compact AI Finance Controller demo for Razorpay reconciliation. It keeps financial truth deterministic and uses a constrained investigator only to explain exceptions from read-only evidence.

Reconciliation is multi-source. Every payment is checked against four separate record sets: the captured payment, the settlement reconciliation records that map it, the settlement itself, and the bank entry behind that settlement. A payment only matches when the linkage and the arithmetic agree across all four; a disagreement at any one of them produces a typed exception rather than a silent pass.

Architecture

Full design rationale — the trust boundary, the approved read-only tool set, exposure classification, and the model fallback ladder — is in ARCHITECTURE.md.

Features

• Reproducible synthetic benchmark with 5,000 varied payments, a fixed seed, and separate development and held-out splits.
• Deterministic reconciliation for ID matching, settlement linkage, fee/tax adjustment, refunds, duplicates, timing windows, bank UTR checks, missing settlements, and amount discrepancies.
• Machine-readable reason codes for every reconciliation result.
• Exposure classification that separates money genuinely at risk from settlements that have simply not arrived yet.
• Read-only investigation tools with guardrails against fabricated evidence or direct financial mutation.
• Gemini investigation for exception explanations when LLM_API_KEY is configured, with deterministic fallback when it is not.
• Human review actions for resolve/escalate, written to an audit trail.
• Held-out evaluation metrics and exception taxonomy calculated from actual benchmark runs.
• Streamlit demo dashboard, exception queue, investigation view, evaluation page, and audit table.
• Razorpay Test Mode connector separated from synthetic benchmark data.

Verified Razorpay API Assumptions

Verified against official Razorpay documentation on 2026-08-27:

• Fetch payments: GET /v1/payments with from, to, count, and skip.
• Fetch settlements: GET /v1/settlements/ with from, to, count, and skip.
• Fetch settlement reconciliation: GET /v1/settlements/recon/combined with required year, month, optional day, count, and skip.
• Settlement recon rows can include payment, refund, transfer, and adjustment transaction types.

Additionally exercised against the live Razorpay API with Test Mode credentials on 2026-09-04. All three endpoints authenticated and returned HTTP 200 with a well-formed entity: "collection" envelope, called both directly and through core.RazorpayClient. POST /v1/orders also returns HTTP 200, so the credentials carry write scope. These are live-API results, not documentation readings.

Docs:

• https://razorpay.com/docs/api/payments/fetch-all-payments/
• https://razorpay.com/docs/api/settlements/fetch-all/
• https://razorpay.com/docs/api/settlements/fetch-recon/

Verified Gemini API Assumptions

Verified against official Google AI documentation on 2026-08-28:

• Model: gemini-3.5-flash-lite, selected as a stable low-cost Flash-Lite model with structured output support. gemini-3.5-flash is tried only when Flash-Lite is unavailable; see Failure Recovery.
• API: POST https://generativelanguage.googleapis.com/v1beta/interactions.
• Structured JSON output is requested with top-level response_format.
• max_output_tokens is 8192. This has to cover the slower model's thinking tokens, not just the JSON body.

Exercised against the live API on 2026-09-04: an investigation returned HTTP 200 with schema-valid structured output, root cause FEE_TAX_DISCREPANCY at confidence 0.95 with requires_human_review set.

Docs:

• https://ai.google.dev/gemini-api/docs/models
• https://ai.google.dev/gemini-api/docs/models/gemini-3.5-flash-lite
• https://ai.google.dev/gemini-api/docs/structured-output?lang=rest

Setup

pip install -r requirements.txt

Optional environment variables:

RAZORPAY_KEY_ID=
RAZORPAY_KEY_SECRET=
LLM_API_KEY=
DATABASE_URL=razorrecon.sqlite

Razorpay credentials must be Test Mode credentials. Live-money actions are not implemented.

Run The Demo

python -m streamlit run app.py

Then open the Streamlit URL, usually http://localhost:8501.

The app loads and reconciles the synthetic benchmark automatically when the local SQLite database is empty, so a deployed instance works without Razorpay account data.

On startup the app compares row counts rather than checking whether tables are merely non-empty. A page reload during the cold-start run kills that script run, and because the database connection is cached across runs the next run can see the abandoned transaction's partial rows. Treating that as a finished run would report a high match rate with an empty exception queue. When the counts disagree, the app discards the partial write and rebuilds.

Demo flow:

1. Review dashboard metrics from the automatically loaded synthetic benchmark.
2. Optionally click Load benchmark and Run reconciliation to reset/replay the demo.
3. Review records, matched count, match rate, exceptions, accuracy, precision, recall, throughput, and unresolved exposure by class.
4. Open the Exceptions tab for the taxonomy and the queue.
5. Open Investigation, filter by exception type and status, pick a case, and click Investigate.
6. Review investigation source, tool calls, evidence, confidence, recommendation, and audit history.
7. Click Resolve or Escalate to record a human decision.

Wide tables are paged to the first 500 rows so a free-tier host is not serialising 5,000 rows to the browser on every interaction. The complete figures are always available from the CLI benchmark.

Benchmark Methodology

The included benchmark is entirely synthetic. It is designed to make reconciliation behavior measurable and reproducible; it is not a claim about production accuracy.

• Default size: 5,000 payment records.
• Fixed random seed: 42.
• Split: 4,000 development-style records and 1,000 held-out records (80/20, stratified by scenario).
• Variation: payment amounts, fee rates, payment methods, timestamps, settlement timing, discrepancies, and identifiers are generated from the seeded random generator. Records are not duplicated copies of a smaller fixture.
• Ground truth: stored separately for every payment with its split, scenario, expected status, and expected exception type.
• Independence boundary: reconciliation queries only financial source columns. It does not receive the ground-truth label or expected result. Ground truth is joined only after deterministic results have been written, for evaluation and test compatibility.

This is not one cherry-picked match: every one of the 5,000 records is reconciled on each run, and every one of the 1,000 held-out records is scored against labels the reconciliation engine never saw.

Nor is it one lucky seed. The headline figures are reported from seed 42, so test_accuracy_holds_across_independent_seeds re-runs generation, reconciliation and held-out scoring on six unrelated seeds (7, 42, 101, 2024, 31337, 90210) and requires 100% accuracy, precision and recall with zero false positives and zero false negatives on every one. The same test also fingerprints the generated payment amounts and fails if any two seeds produce identical data, so the check cannot pass by scoring six copies of the same dataset.

The held-out set is scored independently after reconciliation. Reported metrics include total records, matches, exceptions, match rate, exact classification accuracy, exception precision and recall, false positives, false negatives, measured throughput, unresolved exception count/value, and exception counts by type.

Metric definitions:

• Accuracy: percentage whose matched/exception status is correct and, for exceptions, whose deterministic exception type exactly matches ground truth.
• Precision: true exception detections divided by all predicted exceptions.
• Recall: detected ground-truth exceptions divided by all ground-truth exceptions.
• Unresolved value: sum of the absolute reconciliation differences for open, human-review, or escalated exceptions. It is an exposure indicator, not a ledger balance, and it is reported split by exposure class rather than as one number.
• Throughput: always measured over the whole reconciliation run. A split is scored afterwards from stored results, so it has no separate timing of its own.

Throughput here is a full-pipeline figure, not an in-memory matching rate, so it is not comparable to benchmarks that time matching alone. The timed window covers clearing prior results, reading every payment, running roughly six SQLite queries per payment for related records, settlement, bank entry and expected-settlement arithmetic, and writing one reconciliation result row per payment plus one exception row per exception. On the default 5,000-record benchmark that is about 30,000 queries and 6,500 row inserts inside the measurement, all persisted to SQLite rather than held in memory.

Exposure Classes

A reconciliation difference does not mean the same thing for every exception type, so summing them into a single "unresolved value" overstates risk. Each type is classified:

• Amount at risk - the ledger disagrees about money that has already moved: AMOUNT_MISMATCH, BANK_UTR_AMOUNT_MISMATCH, REFUND_AMOUNT_MISMATCH, SETTLEMENT_AMOUNT_DISCREPANCY, DUPLICATE_RECONCILIATION_RECORD.
• Awaiting settlement - no settlement or reconciliation row exists yet, so the difference is the entire payment value. This is pipeline lag, not loss: MISSING_SETTLEMENT, MISSING_RECONCILIATION_RECORD.
• Structural - the amounts reconcile exactly but the linkage between records is wrong, so the difference is legitimately zero: WRONG_MAPPING, TIMING_WINDOW_EXCEEDED, PAYMENT_ID_MISMATCH.

On the default 5,000-record benchmark this separates roughly INR 1.86 lakh of genuine amount-at-risk from roughly INR 1.38 crore that is only awaiting settlement. An unrecognised exception type is classified as amount at risk rather than quietly discounted.

Exception taxonomy includes WRONG_MAPPING, MISSING_RECONCILIATION_RECORD, AMOUNT_MISMATCH, DUPLICATE_RECONCILIATION_RECORD, MISSING_SETTLEMENT, TIMING_WINDOW_EXCEEDED, REFUND_AMOUNT_MISMATCH, SETTLEMENT_AMOUNT_DISCREPANCY, and BANK_UTR_AMOUNT_MISMATCH. The engine also emits PAYMENT_ID_MISMATCH, which the synthetic generator does not produce; it is reachable on Razorpay connector data where a reconciliation row's entity does not match the captured payment. Counts, percentages, and unresolved values are calculated from each run; no example totals are hardcoded.

For WRONG_MAPPING, the generated evidence preserves the intended semantics: entity_id is the original payment ID and payment_id is the incorrect payment ID.

AI Safety Boundary

Financial truth comes only from deterministic Python reconciliation. Gemini runs only after a verified exception is selected, receives read-only evidence, and cannot mutate payments, settlements, reconciliation records, or bank entries. Any unresolved financial decision remains a human action recorded in the audit trail.

If LLM_API_KEY is absent, the API request fails, the response is malformed, or structured-output validation fails, RazorRecon continues with its evidence-grounded deterministic fallback. The UI always identifies the investigation source as either Gemini AI or Deterministic fallback.

Failure Recovery

Four real failures were found, reproduced, and fixed with a permanent regression test, not just handled in theory.

Gemini failure. If LLM_API_KEY is missing, the API call errors, the response is malformed, or the structured output fails schema validation, investigation falls back to the deterministic evidence-grounded classifier instead of crashing or guessing. The fallback reason is written to the audit trail (GEMINI_FALLBACK), never fed back into a prompt, and never shown as evidence. All three failure paths are covered by tests: test_malformed_gemini_response_falls_back, test_gemini_api_failure_uses_deterministic_fallback, test_gemini_structured_output_validation_rejects_invalid_fields.

Concurrent cold-start crash. @st.cache_resource hands every Streamlit session the same SQLite connection, and each session runs in its own thread. Two sessions hitting a cold start together both saw an empty benchmark table and both called load_benchmark on that shared connection at once, unsynchronized. Their insert loops interleaved and collided on the same deterministic payment_id sequence, crashing the app with sqlite3.IntegrityError: UNIQUE constraint failed. Reproduced deterministically with 8 threads sharing one connection. Fixed with a single process-wide lock (cached the same way as the connection itself) around the bootstrap section: one thread rebuilds, the rest wait and then see the result. Covered by test_concurrent_cold_starts_on_a_shared_connection_need_a_lock.

Colliding audit-trail ids. Audit ids were built as audit_{unix_second}_{randint(1000, 9999)}, which is only 9,000 distinct ids per second, and audit_events.id is a primary key. Audit rows are written in bursts inside a single second: investigate_exception alone writes EXCEPTION_INVESTIGATED and GEMINI_FALLBACK back to back, so a run writes four rows sharing one timestamp. Two rows drawing the same number raised sqlite3.IntegrityError: UNIQUE constraint failed: audit_events.id. This first appeared as a test that failed roughly one run in three and passed in isolation, which is the misleading part: the real cost was in the app, because the audit write happens on every investigation and every human decision, so a collision would crash the action a reviewer had just taken and lose the record of it. Reproduced deterministically by pinning the random draw, and measured on the old scheme at 2,000 writes in one second, where the first collision arrived at write 203. Fixed by replacing the random suffix with uuid4, removing the birthday problem rather than narrowing it. The same 9,000-value pattern was fixed in the Razorpay recon path, where unidentified rows shared an insert or replace key and silently overwrote each other instead of raising. Covered by test_audit_ids_do_not_collide_within_a_single_second.

Gemini capacity treated as an answer. Every investigation was falling back to the deterministic classifier. The audit trail held the reason: read timeouts at the old 20s ceiling, and HTTP 500 responses whose body read gemini-3.5-flash-lite is currently experiencing high demand. That is capacity pressure on one model at Google, not a bad key or a malformed request, and the code could not tell the difference between "the model is busy" and "the model has answered". Three things were wrong. The 20s timeout cut off healthy replies, since a measured good response took 26s. A single model meant one saturated model disabled the whole AI path. And max_output_tokens of 2048 truncated the larger model's reply mid-string, which surfaced as JSONDecodeError: Unterminated string and looked like a malformed response rather than a ceiling set too low. Fixed by trying gemini-3.5-flash-lite on a short 15s leash and falling through to gemini-3.5-flash at 40s, and by raising the ceiling to 8192. Transport faults and 5xx are retried on the next model; a 4xx or a schema-validation failure is not, because those fail identically everywhere and retrying them only delays the fallback the reviewer is waiting on. The deterministic classifier is still the last resort, so the safety property is unchanged. Verified end to end after the fix: the investigation returned Gemini AI, root cause FEE_TAX_DISCREPANCY, confidence 0.95. Covered by test_transient_gemini_timeout_is_retried_once_before_falling_back and test_non_transient_gemini_failure_is_not_retried.

Streamlit Community Cloud Deployment

Use app.py as the Streamlit entry point.

The deployed app defaults to synthetic benchmark mode. It does not require Razorpay credentials, Razorpay account data, a pre-existing SQLite database, or CLI setup by the judge before first use.

Required repository files for deployment:

• app.py
• core.py
• requirements.txt
• evaluation.py
• tests/test_reconciliation.py
• README.md
• assets/architecture.svg
• .streamlit/config.toml
• .env.example
• .gitignore

Do not commit .env, .streamlit/secrets.toml, local SQLite databases, logs, caches, or API keys.

Optional Streamlit Secrets:

RAZORPAY_KEY_ID = "rzp_test_..."
RAZORPAY_KEY_SECRET = "..."
LLM_API_KEY = "..."
DATABASE_URL = "razorrecon.sqlite"

All secrets are optional for the public demo. Without Razorpay keys, synthetic benchmark mode remains available. Without LLM_API_KEY, investigations use deterministic fallback.

Deployment steps:

1. Clone the repository.
2. Install requirements with pip install -r requirements.txt.
3. Optionally configure local environment variables in .env.
4. Run tests with python -m pytest.
5. Run the benchmark with python evaluation.py.
6. Run locally with python -m streamlit run app.py.
7. Push this repository to GitHub without .env, .streamlit/secrets.toml, or razorrecon.sqlite.
8. Go to https://share.streamlit.io.
9. Click Create app.
10. Select the GitHub repository and branch.
11. Set the main file path to app.py.
12. Open Advanced settings.
13. Paste any optional secrets in TOML format.
14. Deploy.

CLI Benchmark

python evaluation.py

The CLI uses an isolated in-memory SQLite database, generates the default seeded dataset, and prints both the complete benchmark and held-out report in a demo-ready format. Running it does not modify razorrecon.sqlite.

Results should be described as: "On the included synthetic held-out benchmark..." They must not be presented as production accuracy.

Tests

python -m pytest

The tests cover 5,000-record generation, fixed-seed reproducibility, multi-seed accuracy stability across six unrelated seeds, held-out independence and metrics, required exception classes, false-positive/false-negative safety, wrong mapping semantics, exposure-class partitioning, bank-entry referential integrity, rebuild of a partially written result set, approved investigation tools, Gemini structured validation and failure fallback, evidence recording, insufficient-evidence routing, mutation-tool blocking, audit-id uniqueness under same-second bursts, transient-versus-permanent Gemini failure handling, and Razorpay pagination.

File Map

• app.py - Streamlit UI and demo workflow.
• core.py - SQLite schema, benchmark generator, reconciliation engine, Razorpay connector, investigator tools, evaluation, and audit logic.
• evaluation.py - CLI benchmark runner.
• tests/test_reconciliation.py - focused tests for critical financial and guardrail behavior.
• .github/workflows/tests.yml - runs the test suite on every push and pull request.
• assets/architecture.svg - the pipeline diagram referenced above.
• .streamlit/config.toml - dashboard theme. Committed deliberately: without it the deployed app falls back to the default Streamlit palette, in which the exposure classes lose their colour coding.
• .env.example - required configuration shape.
• .gitignore - excludes secrets, local DBs, caches, and generated artifacts.

Known Limitations

• Gemini investigation requires LLM_API_KEY; without it, the deterministic evidence-grounded fallback remains fully usable.
• An investigation takes roughly 10 to 40 seconds depending on which model answers. When Flash-Lite is under load the request waits out a 15-second leash before the secondary model is tried, so a slow investigation is usually Google's capacity rather than a stall in this app.
• Razorpay Test Mode sync requires valid Test Mode keys and actual account data; the benchmark path remains separate and reliable for demos.
• Test Mode cannot demonstrate the full pipeline, and this is a property of Razorpay rather than of this connector. A Test Mode account holds no payments until a Checkout flow is completed by hand, and Razorpay does not run settlement cycles in Test Mode at all. So GET /v1/settlements/ and GET /v1/settlements/recon/combined return empty collections even after test payments exist. Reconciliation needs payments matched against settlements and recon rows, so the only way to exercise every rule is the synthetic benchmark. That is why the benchmark is the demo path and the connector is kept separate from it. Verified on 2026-09-04: all three endpoints return HTTP 200 with zero items on a Test Mode account.
• No autonomous refunds, payouts, or financial mutations exist.
• Synthetic benchmark behavior does not establish performance or accuracy on production Razorpay or bank data.
• The benchmark scores 100% accuracy, precision, and recall by construction: every scenario the generator produces is drawn from the failure modes the deterministic rules already cover. That result demonstrates rule coverage and guards against regressions. It is not evidence of accuracy on real data, which contains failure modes this generator does not create.
• Throughput is machine-dependent and is measured for the complete local benchmark run, not claimed as a service-level guarantee.

Matcher scope

Limits of the matching logic itself, separate from the data and accuracy limits above:

• Fixed timing window. The timing rule is a fixed seven-day deterministic window. Real settlement cycles vary by merchant, method, and payout schedule, and would need to be configurable.
• No fuzzy matching. Matching is by exact key and derived arithmetic. A reconciliation row whose identifier has been corrupted or reformatted surfaces as MISSING_RECONCILIATION_RECORD rather than being recovered.
• No batch decomposition. A payment is assumed to map to a settlement. Consolidated payouts, where many payments net into a single bank credit, are not decomposed into their components and would present as amount discrepancies.
• Single currency. Amounts are integer paise, single-currency. Multi-currency settlement and FX differences are not modelled.