Fraud Triage Demo

A polished, traceable suspicious-activity / fraud triage demo built with Next.js and TypeScript.

The project is optimized for live presentation:

• risk scoring stays rule-based and explainable
• guidance retrieval is local-first and file-based
• multi-agent execution is traceable in the API, terminal logs, and UI
• demo mode still works when OpenAI is unavailable
• HYBRID_OPENAI can make selected agents OpenAI-first while preserving local fallback and verification

Project Overview

This app simulates an analyst workstation that can:

• browse synthetic, generated, and imported cases
• normalize alerts into a richer CaseAlert domain model
• detect suspicious signals
• compute an explainable score and risk tier
• retrieve relevant local guidance passages with citations
• generate an investigator-facing summary narrative
• show a per-agent execution trace for debugging and demos

Demo Architecture

Browser workstation
  -> POST /api/triage
    -> orchestrator
      -> AlertIntakeAgent
      -> SignalDetectionAgent
      -> RiskAssessmentAgent
      -> GuidanceRetrievalAgent
      -> InvestigatorRecommendationAgent
      -> NarrativeSummaryAgent
    -> structured result + trace

Important directories:

• src/app/api/triage/route.ts: API entry point
• src/lib/orchestration/: multi-agent runtime and trace collection
• src/lib/agents/: local agent implementations
• src/lib/openai/: OpenAI-backed agent adapters
• src/lib/rag/: local guidance ingestion, chunking, and retrieval
• src/lib/data/: dataset loading, filtering, random selection, and comparison
• data/cases/: sample, generated, and imported case datasets
• data/knowledge/: raw and processed guidance documents
• tests/: lightweight reliability-focused tests

Agent Workflow

Execution order:

1. AlertIntakeAgent
2. SignalDetectionAgent
3. RiskAssessmentAgent
4. GuidanceRetrievalAgent
5. InvestigatorRecommendationAgent
6. NarrativeSummaryAgent

Responsibilities:

• AlertIntakeAgent: normalizes request payloads into the shared structured alert shape
• SignalDetectionAgent: detects suspicious signals and mitigating context
• RiskAssessmentAgent: calculates score, tier, and score breakdown
• GuidanceRetrievalAgent: retrieves local passages and, in hybrid mode, can use OpenAI to explain why they matter
• InvestigatorRecommendationAgent: turns risk output into an investigator-ready action recommendation
• NarrativeSummaryAgent: produces the final narrative summary

Execution Modes

LOCAL_RULES_ONLY

• every agent runs locally
• scoring is fully deterministic
• the summary uses the local template

HYBRID_OPENAI

• AlertIntakeAgent is OpenAI-first with local fallback/verification
• SignalDetectionAgent is OpenAI-first with local fallback/verification
• GuidanceRetrievalAgent keeps raw retrieval local, then uses OpenAI to interpret the passages
• InvestigatorRecommendationAgent is OpenAI-first with rule-based verification
• NarrativeSummaryAgent is OpenAI-first with local fallback
• RiskAssessmentAgent remains local and authoritative

Guidance / RAG Design

The retrieval layer is deliberately local-first and database-free.

• raw guidance lives in data/knowledge/raw
• the app chunks and processes documents into data/knowledge/processed/guidance-store.json
• retrieval is keyword-first with a lightweight similarity fallback
• results always return visible citation metadata
• in hybrid mode, retrieved passages can also receive an OpenAI-generated relevance explanation

This keeps the system easy to debug and safe for classroom or judge-panel demos.

More detail is in RAG_README.md.

Dataset Strategy

The demo uses a scalable local dataset workflow:

• data/cases/sample: curated sample cases
• data/cases/generated: synthetic cases generated by script
• data/cases/imported: CSV-imported cases normalized into the internal schema

The internal case model centers on CaseAlert, not on a flat alert row. A loader utility supports:

• browse all cases
• filter by source and expected risk tier
• load a random case
• compare two cases

More detail is in DATASET_README.md.

How To Run In Demo Mode

1. Install dependencies:

npm install

2. Copy .env.example to .env.local

3. Keep demo mode local and reliable:

TRIAGE_MODE=LOCAL_RULES_ONLY
OPENAI_API_KEY=$OPENAI_API_KEY
OPENAI_MODEL=gpt-4.1-mini
OPENAI_AGENT_MODEL=gpt-4.1-mini

4. Start the app:

npm run dev

5. Open http://localhost:3000

How To Enable OpenAI

Set:

TRIAGE_MODE=HYBRID_OPENAI
OPENAI_API_KEY=your_openai_key_here
OPENAI_MODEL=gpt-4.1-mini
OPENAI_AGENT_MODEL=gpt-4.1-mini

Behavior in this mode:

• OpenAI becomes the primary implementation for intake, signal detection, guidance interpretation, recommendation, and narrative summary
• local logic still verifies or replaces those outputs when needed
• RiskAssessmentAgent remains rule-based and final
• if OpenAI fails for any selected agent, that step falls back locally and the pipeline still completes

Tradeoffs:

• stronger demo storytelling and more natural agent behavior
• more API cost and dependency on network/API availability
• more trace complexity, which is why provider/model/fallback details are surfaced explicitly

Intermittent Fallback Diagnosis

Based on live logs and replay, intermittent fallback appears more likely than a permanent pipeline failure. The evidence is consistent with output variability on one or two schema-constrained steps, especially because the same case can later succeed without code changes.

This does not prove the root cause is only the model. Prompt/schema brittleness or other nondeterministic application factors could also contribute. The safest next step is to compare models under identical settings and add a single targeted retry on schema-validation failure before local fallback.

Recommended checks:

• verify the OpenAI call uses structured outputs such as response_format: { type: "json_schema", ... strict: true } or the equivalent Responses API path
• pin model versions for consistency-sensitive demos, since prompt behavior can vary across model snapshots
• run a small eval matrix such as 50-100 replays per case, per model, with the same settings
• log the exact validator failure type: parse error, missing key, enum drift, extra field, truncation, or timeout
• add one retry only for schema-validation failures before fallback

How To Inspect Traces

There are three trace surfaces:

1. Terminal logs
   • each agent step logs timing, provider, and metadata during local development
2. API response
   • /api/triage returns a structured trace object with run ID, mode, provider summary, timing, and per-agent summaries
3. UI trace panel
   • the workstation shows the step timeline, provider, model, fallback reason, timing, metadata, and expandable summaries

Trace format includes:

• runId
• caseId
• mode
• startedAt
• completedAt
• durationMs
• providerSummary
• events[]

Each event includes:

• agentName
• status
• startedAt
• completedAt
• durationMs
• inputSummary
• outputSummary
• inference (provider, model, attempted, verifiedWithRules, fallbackReason, requestId)
• optional details
• optional metadata

Useful Commands

npm run dev
npm run test
npm run test:watch
npm run generate:cases
npm run import:cases -- --input=data/cases/imported/example-import.csv --mapping=data/cases/imported/example-mapping.json

Limitations

• The scoring logic is intentionally simplified for explainability.
• Retrieval is local and lexical; it is not embedding-powered yet.
• Imported CSV support depends on a mapping file rather than automatic schema inference.
• OpenAI-primary behavior depends on strict schema validation and safe fallbacks; malformed model output is discarded in favor of local logic.
• This demo is not suitable for real fraud operations without stronger governance, controls, and data integrations.

Future Work

• add embedding-backed retrieval behind the existing RAG interfaces
• compare multiple models or providers behind the same agent interface
• expand imported dataset tooling with UI-assisted mapping
• add richer trace analytics and case history persistence