Activity-Aware VideoRAG Pipeline

Retrieval-Augmented Generation for Context-Aware Video Summarization

This repository implements a powerful Activity-Aware Graph Summarization pipeline. It extends basic Video Retrieval-Augmented Generation (RAG) by representing video-derived insights via an interconnected Knowledge Graph. Rather than executing flat, context-free queries across an entire vector database, the system dynamically clusters entities based on generated activities to build highly accurate, context-bound localized summaries.

Core Conceptual Flow

The pipeline operates in two major phases:

Phase 1: Video Extraction & Graph Construction (construct_graph.py)

1. Video Ingestion & Chunking: Audio is transcribed using the faster-distil-whisper-large-v3 model, and visual keyframes are captioned using the MiniCPM-V-2_6-int4 Vision Language Model. The outputs are binned into text chunks mapped tightly against the video timeline.
2. Extraction Engine (LLM): An LLM dynamically scans these chunks to identify distinct Entities, their inter-node Relationships, and their temporal Activities.
3. Graph Building: The extracted entities and relationships are stitched into a localized Graph Database. Critically, each entity node's metadata is permanently tagged with the precise dynamic activities found in its respective chunks (e.g., ["running", "speaking"]).

Phase 2: Activity-Aware Traversals & Summarization (ask_activity.py)

1. Clustering & Pruning: The system automatically pulls from the graph and groups entity nodes into localized subsets matching specific activities. Textually, it filters chunk references to structurally process data natively bound to a target_video_name.
2. Subgraph Traversal: For each local activity subset, the system selects the most structurally integral nodes using Degree Centrality. It then executes a Breadth-First-Search (BFS) expanding 2 hops outward to retrieve surrounding contextual peripheral nodes.
3. Hierarchical Summarization:
   • Local Level: It maps the filtered nodes exactly back to the source data transcripts and creates a localized summary describing that single activity event.
   • Global Level: It integrates and aggregates every structured localized mini-summary into one final, polished, macro summary describing the combined sequence of events without redundancy.

Repository Layout

ASTC-RAS/
├── README.md                         # This document
├── setup.sh                          # pip dependency install script (torch, faster-whisper, etc.)
├── .gitignore                        # Ignores weights, .env, and Python bytecode
├── LICENSE                           # Project license
│
├── construct_graph.py                # Phase 1: ingest videos -> transcript/captions -> build Knowledge Graph
├── ask_activity.py                   # Phase 2: Activity-Aware subgraph traversal + hierarchical summary
├── ask_graph.py                      # Auxiliary / testing direct graph query engine
│
├── videos/                           # Drop target .mp4 clips here (indexed by construct_graph.py)
│   ├── Amritsar Woman Stops Robbers.mp4
│   └── Human Activity Recognition I3D Demo 720P.mp4
│
└── VideoRAG_algorithm/               # Core algorithm library (forked VideoRAG, trimmed to essentials)
    └── videorag/
        ├── __init__.py               # Exposes VideoRAG, QueryParam
        ├── videorag.py               # Main pipeline: insert_video, query, generate_activity_summary
        ├── activity_summarization.py # Activity clustering + BFS subgraph traversal + summarization
        ├── _op.py                    # Graph/entity extraction + retrieval operator
        ├── _splitter.py              # Text chunk splitting
        ├── _llm.py                   # LLM + embedding configs (Gemini, OpenAI, Azure, Ollama)
        ├── base.py                   # Storage base interfaces
        ├── prompt.py                 # LLM prompts
        ├── _utils.py                 # Shared helpers / hashing
        ├── _videoutil/               # Video ingestion utilities
        │   ├── split.py              # Video -> clip splitting
        │   ├── feature.py            # Visual feature extraction
        │   ├── caption.py            # VLM captioning (MiniCPM-V)
        │   └── asr.py                # Audio transcription (faster-whisper)
        └── _storage/                 # Pluggable storage backends
            ├── kv_json.py            # JSON key-value store
            ├── gdb_networkx.py       # In-memory NetworkX graph
            ├── gdb_neo4j.py          # Optional Neo4j graph
            ├── vdb_hnswlib.py        # HNSW vector store
            └── vdb_nanovectordb.py   # NanoVectorDB vector store (+ video segment store)

Runtime note: the caption (MiniCPM-V-2_6-int4) and ASR (faster-distil-whisper-large-v3) model weights are downloaded locally and kept out of git via .gitignore (see the "Large local model weights" section). The videorag-workdir* directories generated at runtime are also gitignored.

Quick Start Guide

1. Environment Initialization: Ensure all necessary weights are downloaded to their folders natively and initialize your local conda virtual environment (conda activate videorag).
2. Setup API Credentials: Place a .env file at the root containing a valid GEMINI_API_KEY to grant the algorithm proper extraction capabilities.
3. Index your Videos: Place your desired .mp4 clip files directly into the videos/ directory. Target them uniformly from inside the array inside construct_graph.py and run python construct_graph.py to ingest the stream and build the network.
4. Acquire Summaries: Run python ask_activity.py to trigger the localized multi-video filtering querying (generate_activity_summary(target_video_name="...")).

This architecture allows numerous clips to be indexed into a single working directory while safely shielding extraction traversals to exact individual clip queries automatically!