PharmaGuard

Intelligent Pharmacovigilance Signal Triage Orchestrator Grounded in Multi-Source Clinical Evidence

A Tool-Grounded, Tri-Source Evidence Fusion Agent for Postmarketing Adverse Event Triage
B.Tech 7th-Semester Capstone Project · Indian Institute of Information Technology, Allahabad

---

The Problem

Every year, millions of spontaneous adverse drug event reports are submitted to postmarketing safety databases such as the FDA Adverse Event Reporting System (FAERS). Clinical safety teams face an acute triage bottleneck: distinguishing true emergent pharmacological safety signals from background noise, uncorroborated reports, and confounded polypharmacy associations.

When generative foundation models (LLMs) are applied to clinical safety triage without strict tool grounding, they exhibit three fundamental failure modes:

1. Hallucinated Clinical Confidence: LLMs produce high, uncalibrated self-confidence scores without empirical statistical grounding.
2. Historical Regulatory Confusion: LLMs recall historical controversies that were investigated and formally dismissed by regulators (e.g. liraglutide + pancreatic cancer), confusing historical investigation with confirmed causation.
3. Parametric Epistemic Leakage: LLMs recall famous regulatory actions (e.g., FDA Boxed Warnings) directly from training memory, overriding biological mechanistic analysis with memorized clinical associations.

---

Our Solution

PharmaGuard is an automated pharmacovigilance triage orchestrator that evaluates drug–adverse event pairs by synthesizing evidence from three orthogonal, public biomedical data streams:

• 1. openFDA / FAERS: Computes postmarketing disproportionality statistics (2×2 contingency table, PRR, ROR, and Woolf 95% lower confidence bounds) with automatic down-weighting for wide confidence intervals.
• 2. ChEMBL Mechanism of Action: Evaluates target-level pharmacological mechanisms to determine biological plausibility (HIGH, MODERATE, LOW/UNKNOWN) via human-curated lookup with agent-derived fallback.
• 3. PubMed Literature Retrieval: Analyzes peer-reviewed abstracts using structured LLM grading against a versioned clinical rubric (Grade A for statistically significant odds ratios/CIs, Grade B for clinical observations, Grade C for unconfirmed/negative literature).

PharmaGuard synthesizes these signals via a deterministic composite confidence formula and applies a strict safety gate to output auditable decisions:

• ESCALATE — Statistically significant signal corroborated by high biological plausibility or Grade A literature.
• MONITOR — Genuine epidemiological signal with unconfirmed mechanism, or heavily confounded polypharmacy signal requiring clinical surveillance.
• DO_NOT_ESCALATE — No statistical postmarketing signal or dismissed non-causal association.

flowchart TD
    QP["Drug + Adverse Event Query Pair"] --> FAERS["openFDA / FAERS<br/>Disproportionality<br/>(PRR / ROR / Woolf 95% CI)"]
    QP --> CHEMBL["ChEMBL Target<br/>Biological Plausibility<br/>(HIGH / MOD / LOW)"]
    QP --> PUBMED["PubMed Evidence<br/>Literature Grade<br/>(Grade A / B / C)"]

    FAERS -->|"Weight: 0.40"| CONF["Deterministic Confidence Score [0, 1]<br/>Σ = 0.40·S_FAERS + 0.40·S_Lit + 0.20·S_Mech"]
    CHEMBL -->|"Weight: 0.20"| CONF
    PUBMED -->|"Weight: 0.40"| CONF

    CONF --> GATE{"FAERS == NO_SIGNAL ?"}
    GATE -->|"YES"| DNE_GATE["<b>DO_NOT_ESCALATE</b><br/>(Hard Safety Gate)"]
    GATE -->|"NO"| EVAL{"Evaluate Confidence"}

    EVAL -->|"Conf ≥ 0.70 & FAERS ≥ MODERATE"| ESC["<b>ESCALATE</b>"]
    EVAL -->|"Conf ≥ 0.35"| MON["<b>MONITOR</b>"]
    EVAL -->|"Conf < 0.35"| DNE["<b>DO_NOT_ESCALATE</b>"]

Loading

---

Core Architectural Pillars

1. Tri-Source Grounded Evidence Fusion

PharmaGuard eliminates LLM guesswork by querying live/cached biomedical APIs:

• FAERS Statistical Engine: Computes exact Proportional Reporting Ratios (PRR) and Reporting Odds Ratios (ROR) from openFDA records. Signals with lower 95% CI < 1.0 are automatically downgraded to prevent small-sample false alarms.
• ChEMBL Plausibility Layer: Routes through curated lookup (plausibility_ratings.json) with agent-derived biochemical fallback.
• PubMed Grading Pipeline: Extracts statistical markers (p < 0.05, odds ratios, 95% CIs) to grade supporting peer-reviewed literature.

2. Deterministic Scoring & Hard Safety Gating

$$\text{Confidence} = 0.40 \cdot S_{\text{FAERS}} + 0.40 \cdot S_{\text{PubMed}} + 0.20 \cdot S_{\text{Plausibility}}$$

• Hard Safety Gate: If FAERS == NO_SIGNAL, the pipeline immediately outputs DO_NOT_ESCALATE regardless of confidence score. This prevents theoretical literature or biological speculation from triggering false alerts on drugs with zero real-world patient reports (DECISIONS.md §5).
• Decision Boundaries:
  • Confidence >= 0.70 and FAERS >= MODERATE $\implies$ ESCALATE
  • Confidence >= 0.35 $\implies$ MONITOR
  • Otherwise $\implies$ DO_NOT_ESCALATE

3. Dual-Metric Benchmark Framework (Strict vs. Lenient)

Evaluating signal triage requires capturing both unhesitating escalation and safety-critical surveillance:

• Strict Metrics: Treats only ESCALATE as True Positive. Captures epistemic caution when biological mechanism is unconfirmed (e.g. montelukast::suicidal_ideation $\to$ MONITOR, strictly recorded as FN = 1).
• Lenient Metrics: Treats ESCALATE and MONITOR as True Positive. Confirms that no safety-critical signal is dropped (Recall = 1.000 on Core 15 pairs).

4. Anti-Leakage & Memorization Probe Discipline

Empirical probing revealed that unconstrained LLM plausibility derivation (force_agent mode) produced an artificial 1.000 Strict Recall by leaking regulatory memory (citing FDA Boxed Warnings) rather than performing biochemical reasoning (DECISIONS.md §19). PharmaGuard maintains a lookup_first configuration and treats agent-derived plausibility as grounded pharmacological knowledge retrieval and pathway synthesis, not de novo reasoning (DECISIONS.md §17).

5. High-Density Streamlit Dashboard (7 Dedicated Views)

A clinical review dashboard engineered in Streamlit and Plotly across 7 dedicated views:

1. Overview: Global executive summary, metric cards, Leave-One-Out (LOO) stability blocks, and multi-cohort switching.
2. Per-Pair Table: Detailed evidence drill-down with cross-source agreement badges (CONCORDANT / DISCORDANT) and confidence waterfall decompositions.
3. Disagreement Spotlight: Deep clinical case studies dissecting mechanistic caution and polypharmacy confounding.
4. Baseline Comparison: Direct side-by-side benchmarking against ungrounded single-shot LLMs.
5. Methodology Probes: Adversarial critic leakage audit, confounding self-probe, and temporal sensitivity checks.
6. OMOP Pilot Benchmark: External reference set evaluation across 32 pairs from OHDSI MethodEvaluation.
7. Live Signal Triage: Real-time interactive playground with 151 searchable benchmark presets, biomedical NLP auto-correction, and one-click clinical briefing generation.

6. Disease-Context Reasoning & Scoring-Inert Indication Concordance

To address confounding by indication—where a drug is prescribed for symptoms overlapping the suspected adverse event—PharmaGuard incorporates specialized clinical context modules:

• DiseaseContextTool & WHO ATC: Queries the ChEMBL API and local registries (atc_lookup.json) for WHO Anatomical Therapeutic Chemical (ATC) classification codes (Levels 1–4) and indication records to contextualize disease space.
• IndicationConcordanceTool: Evaluates semantic and pharmacological concordance between candidate adverse events and indicated pathologies using a 7-rule clinical heuristics cascade (IND-CONF-01 to IND-CONF-07).
• Scoring-Inert by Design: In production triage, indication concordance operates strictly as an informational surveillance flag without modifying composite numerical confidence or causing decision boundary crossings (DECISIONS.md §35).

7. Real-Time Biomedical NLP Canonicalization & Spelling Auto-Correction

Spontaneous reporting and clinical input fields are prone to typos, brand names, and colloquial lay terms. PharmaGuard embeds a two-stage hybrid normalizer (pharmaguard/utils/canonicalize.py):

• Stage 1 (Deterministic Exact & Alias): Maps high-confidence clinical synonyms, brand names to International Nonproprietary Names (e.g. Zestril $\to$ lisinopril, Singulair $\to$ montelukast), and lay terms to MedDRA Preferred Terms (e.g. heart attack $\to$ myocardial_infarction, kidney injury $\to$ acute_kidney_injury) with $S = 0.98$–$1.00$.
• Stage 2 (Bounded Fuzzy Sequence Matching):
  • $S \ge 0.85$: High-confidence automatic typo correction (e.g. lisinoprll $\to$ lisinopril, pancreatits $\to$ pancreatitis).
  • $0.65 \le S &lt; 0.85$: Interactive suggestion flag (Did you mean...?) alerting clinicians to near matches without silently mutating input terms.
  • $S &lt; 0.65$: Unmapped term retained with transparent audit notice.
• Orthographic Normalization: Automatically reconciles British English MedDRA PTs with American clinical records (e.g. hypoglycemia $\to$ hypoglycaemia, hemorrhage $\to$ haemorrhage).

8. Publication-Grade Clinical Safety Briefing & Triage Dossier

In the Live Signal Triage dashboard, clicking "Generate Clinical Briefing Dossier" compiles an exhaustive, regulatory-grade safety document (scripts/dashboard_modules/reports.py):

• Regulatory Compliance: Adheres strictly to ICH E2C(R2) (Periodic Benefit-Risk Evaluation Report) and CIOMS VIII (Signal Detection) structural standards.
• Emoji-Free Typography: Standardized on formal clinical serif/sans-serif headers, high-contrast badges, and clean markdown tables, eliminating informal emojis and raw unicode glyphs.
• Evans et al. 2001 Triad Audit: Complete disproportionality table detailing report counts ($a, b, c, d$), PRR, $\chi^2$ statistic ($p &lt; 0.001$), and Woolf 95% confidence intervals.
• KaTeX Mathematical Rigor: Step-by-step rendering of the composite confidence calculation and hard safety gate evaluation.
• Executive Sign-Off: Formally formatted for institutional review boards, Qualified Persons for Pharmacovigilance (QPPV), and medical safety officers.

---

Multi-Benchmark Performance Results

PharmaGuard has been evaluated across four formal benchmark cohorts comprising 165 curated drug–event pairs:

Benchmark Cohort	$N$	Strict Precision	Strict Recall	Strict Specificity	Strict $F_1$	Lenient Precision	Lenient Recall	Lenient Specificity	Lenient $F_1$
1. Core Ground Truth Benchmark	15	1.000 [0.610–1.000]	0.857 (6/7) [0.487–0.974]	1.000 [0.676–1.000]	0.923 [0.727–1.000]	0.875 [0.529–0.978]	1.000 (7/7) [0.646–1.000]	0.875 [0.529–0.978]	0.933 [0.769–1.000]
Single-Shot LLM Baseline (Core)	15	0.875 [0.529–0.978]	1.000 (7/7) [0.646–1.000]	0.875 [0.529–0.978]	0.933 [0.769–1.000]	0.700 [0.397–0.892]	1.000 (7/7) [0.646–1.000]	0.625 [0.306–0.863]	0.824 [0.615–0.941]
2. OMOP Pilot Reference Set	32	1.000 (1/1) [0.207–1.000]	0.063 (1/16) [0.011–0.270]	1.000 (16/16) [0.806–1.000]	0.118 [0.024–0.426]	0.846 (11/13) [0.578–0.957]	0.688 (11/16) [0.444–0.858]	0.813 (13/16) [0.570–0.934]	0.720 [0.540–0.850]
3. Top Prescribed Boxed Warnings	50	1.000 (10/10) [0.722–1.000]	0.400 (10/25) [0.232–0.593]	1.000 (25/25) [0.867–1.000]	0.571 [0.377–0.744]	0.958 (23/24) [0.798–0.993]	0.920 (23/25) [0.750–0.978]	0.960 (24/25) [0.805–0.993]	0.939 [0.848–0.978]
4. OMOP Expanded Standard	100	1.000 (1/1) [0.207–1.000]	0.020 (1/50) [0.004–0.105]	1.000 (50/50) [0.929–1.000]	0.039 [0.007–0.197]	0.933 (14/15) [0.702–0.988]	0.280 (14/50) [0.175–0.417]	0.980 (49/50) [0.895–0.997]	0.431 [0.283–0.589]

Key Findings: Across all 165 evaluated pairs, PharmaGuard achieved 100% Strict Specificity (91/91 negative controls correctly suppressed with zero false alarms), outperforming ungrounded models that over-escalate on historical debates. Under Lenient surveillance, it captured 92.0% Recall on top prescribed boxed warnings.

---

Documented Case Studies

1. montelukast + suicidal_ideation (Confirmed Positive → MONITOR)

• Evidence: FAERS MODERATE (PRR = 3.37, 1,259 reports), PubMed Grade A (ROR statistics with 95% CIs).
• Mechanism: CysLT1 receptors are primarily peripheral; no direct CNS pathway is pharmacologically confirmed (plausibility=LOW).
• Triage Result: Composite confidence drops to 0.664 (< 0.70), yielding MONITOR.
• Clinical Significance: Pharmacovigilance-correct outcome: signals real-world co-occurrence while flagging unresolved mechanistic uncertainty.

2. metformin + hypoglycaemia (Negative Control → MONITOR)

• Evidence: FAERS STRONG (PRR = 10.73, 9,344 reports) due to widespread polypharmacy with insulin/sulfonylureas.
• Mechanism: Metformin inhibits hepatic gluconeogenesis without stimulating insulin secretion (plausibility=LOW, PubMed Grade C).
• Triage Result: The 0.40 * S_FAERS term establishes a 0.400 confidence floor (>= 0.35), yielding MONITOR.
• Clinical Significance: Safety-first triage: discounts the confounded signal from ESCALATE down to MONITOR, preventing silent dropping when 9,000+ reports exist.

3. liraglutide + pancreatic_cancer (Negative Control → DO_NOT_ESCALATE)

• Baseline vs. PharmaGuard: The single-shot baseline confidently escalated based on recalled historical regulatory scrutiny. PharmaGuard checked live openFDA records, identified zero co-occurrence reports, and correctly applied the NO_SIGNAL Hard Safety Gate to return DO_NOT_ESCALATE (Confidence 0.300).

---

Tech Stack

• Core & Runtime: Python 3.13, Pandas, NumPy, Scipy
• Agent Orchestration: LangGraph, LangChain, ReAct Agent Loop
• Local & Cloud LLMs: Ollama (qwen2.5:7b), Google Gemini Flash (gemini-2.5-flash / gemini-1.5-flash)
• Biomedical APIs & Parsing: openFDA REST API, ChEMBL Web Resource Client, NCBI E-utilities (BioC / Entrez)
• Caching Layer: diskcache (persistent disk-backed cache with SHA-256 deterministic keying)
• Statistical Evaluation: Non-parametric Bootstrap Resampling ($B=1000$), Wilson Score Binomial Confidence Intervals
• Clinical Dashboard: Streamlit 1.61, Plotly Express & Graph Objects, Google Material Symbols
• CI/CD & DevOps: GitHub Actions matrix test runner (Python 3.11, 3.12, 3.13)

---

Repository Structure

PharmaGuard/
├── .agents/skills/                   # Antigravity agent skills & evaluation protocols
├── .github/workflows/
│   └── ci.yml                        # GitHub Actions automated CI/CD test matrix pipeline
├── assets/
│   ├── Logos/                        # Vector and raster brand identity assets
│   └── Screenshots/                  # High-resolution dashboard verification captures (Light & Dark)
├── configs/
│   └── config.yaml                   # Central pipeline, model, scoring, and cache configuration
├── docs/
│   ├── context/                      # Core architectural, historical, and engineering documentation
│   │   ├── UNDERSTAND.md             # Canonical plain-language project overview
│   │   ├── DECISIONS.md              # 38-section chronological record of architectural decisions
│   │   ├── PROGRESS.md               # Sprint changelog, verified metrics & reproduction steps
│   │   ├── ARCHITECTURE.md           # Exhaustive technical system & Pydantic schema specifications
│   │   └── CANONICALIZATION.md       # Biomedical NLP normalization & alias specification
│   ├── meetings/                     # Weekly stakeholder progress updates and briefing memos
│   ├── paper/                        # Complete 9-section conference paper manuscript (IEEE BIBM / ACM CHIL)
│   ├── presentation/                 # 20-slide 16:9 Capstone Defense presentation deck (.pptx) & notes
│   ├── proposals/                    # Formal capstone proposals & institutional briefs
│   └── test_cases.md                 # Standardized NLP auto-correction and triage test suite guide
├── outputs/
│   ├── core/                         # Frozen production TriageReport JSONs (15 pairs) + summary
│   ├── experiments/                  # Isolated experimental conditions (baseline, ablation, probes)
│   └── research/                     # Formal benchmarks (omop_pilot, top_prescribed, omop_expanded)
├── pharmaguard/
│   ├── agent/                        # Fixed Pipeline & ReAct LangGraph orchestrators, schemas
│   ├── data/                         # Benchmark pairs (165 pairs), plausibility ratings, ATC registries
│   ├── prompts/                      # Versioned system prompts & grading rubrics
│   ├── tools/                        # FAERS, ChEMBL, PubMed, Confounding, DiseaseContext tools & cache
│   └── utils/                        # NLP canonicalization, config loaders, metrics & normalizers
├── scripts/
│   ├── run.py                        # Standalone orchestrator runner
│   ├── dashboard.py                  # Streamlit evaluation dashboard driver (7 views)
│   ├── dashboard_modules/            # Modular dashboard package (views, components, reports, styles)
│   ├── run_eval.py                   # Production 15-pair benchmark evaluation runner
│   ├── evaluator.py                  # Strict & Lenient metric calculator with Bootstrap/Wilson CIs
│   ├── baseline.py                   # Single-shot LLM baseline evaluation runner
│   ├── dev/                          # Developer utilities & slide deck generator
│   └── research/                     # Research runners (OMOP pilot, top prescribed, stability, ablation)
├── tests/                            # 245 pytest unit & regression tests across 18 test files (all passing)
├── run.py                            # One-click multi-terminal launcher (Ollama + Streamlit dashboard)
├── requirements.txt                  # Pinned project dependencies
├── NOTICE.md                         # Third-party licenses (CC BY-SA 3.0, Apache 2.0) & citations
├── LICENSE                           # Project MIT License
└── README.md                         # Project entry point & overview

---

Quickstart & Reproduction

1. Installation

# Clone the repository
git clone https://github.com/Krishna200608/PharmaGuard.git
cd PharmaGuard

# Create and activate virtual environment
python -m venv .venv

# Windows (PowerShell)
.\.venv\Scripts\Activate.ps1
# Linux / macOS
# source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Configure environment keys (copy .env.example)
cp .env.example .env
# Edit .env and add GOOGLE_API_KEY and NCBI_API_KEY (optional for local Ollama)

2. One-Click Multi-Terminal Launcher (Recommended)

PharmaGuard features a unified launcher that automatically spins up the local Ollama LLM service and launches the Streamlit Clinical Dashboard in independent terminals with dynamic port discovery and collision avoidance:

python run.py

Spawns Ollama on http://localhost:11434 and opens the Dashboard at http://localhost:8501.

3. Run the Benchmark Evaluations

# Run 15-pair core evaluation against openFDA, ChEMBL, and PubMed
python scripts/run_eval.py

# Compute Strict and Lenient evaluation metrics with 95% CIs
python scripts/evaluator.py --outputs-dir outputs/core --title "PharmaGuard Final"

# Run single-shot baseline evaluation
python scripts/baseline.py

4. Run the Full Test Suite

pytest -v

(All 245 tests pass across 18 test files in ~50s with zero regressions).

---

Documentation Roadmap

Document	Purpose & Description
docs/context/UNDERSTAND.md	Start here. Plain-language guide covering system mechanics, data streams, and the dual-metric philosophy.
docs/context/DECISIONS.md	Complete 38-section chronological record of all architectural decisions, MedDRA PT audits, probe findings, and benchmark validations.
docs/context/PROGRESS.md	Sprint changelog, multi-benchmark results, exact Wilson/Bootstrap confidence interval tables, and reproduction verification.
docs/context/ARCHITECTURE.md	Exhaustive technical system architecture, data flows, Pydantic schemas, mathematical scoring equations, and directory tree.
docs/context/CANONICALIZATION.md	Two-stage biomedical NLP term canonicalization, alias mappings, and fuzzy typo correction specifications.
docs/test_cases.md	Standardized test case guide for live NLP auto-correction, suggestion flags, aliases, and regulatory gating.
docs/context/CONTRIBUTION.md	Grounded claims of project contributions, empirical findings, and architectural comparisons.
docs/paper/PharmaGuard_Conference_Paper.md	Complete 9-section academic manuscript prepared for conference submission.
docs/presentation/SLIDE_DECK_NOTES.md	20-slide 16:9 Capstone Defense presentation deck script and speaker notes.
docs/proposals/PharmaGuard_Proposal_2026-08-18.md	Formal Capstone Project Proposal submitted to Dr. Nikhilanand Arya.

---

Authors & Acknowledgments

• Team:
  • Krishna Sikheriya (IIT2023139) — Leader
  • Lokesh Bawariya (IIT2023138)
  • Naitik Jain (IIB2023036)
• Supervisor: Dr. Nikhilanand Arya — Assistant Professor, Department of Information Technology, IIIT Allahabad
• Institution: Indian Institute of Information Technology, Allahabad (IIIT-A)
• Academic Milestone: 7th-Semester B.Tech Capstone Project (2026–2027)

---

License & Third-Party Notices

• Software & Code: PharmaGuard's original software, agent orchestration, evaluation harnesses, and documentation are licensed under the MIT License.
• Third-Party Data & Lookups: Certain pharmacological registries and benchmark datasets distributed in pharmaguard/data/ are governed by open third-party licenses:
  • pharmaguard/data/chembl_lookup.json and pharmaguard/data/atc_lookup.json are derived from the EMBL-EBI ChEMBL database and licensed under CC BY-SA 3.0 (Creative Commons Attribution-ShareAlike 3.0 Unported). In accordance with the ShareAlike clause, these data files remain subject to CC BY-SA 3.0 and are not covered by the repo's MIT license.
  • pharmaguard/data/external/omopReferenceSet.rda and the derived benchmark files pharmaguard/data/ground_truth_omop_pilot.json and pharmaguard/data/ground_truth_omop_validation_holdout.json are distributed under the Apache License 2.0 (OHDSI MethodEvaluation).

See NOTICE.md for full third-party license texts, copyright notices, and required academic citations.