DataGov QA

A web application that answers natural-language questions (Hebrew and English) by discovering and extracting datasets from Israel's open data portal — data.gov.il.

How It Works

1. You type a question in the search box (e.g. "כמות הגשמים שירדה באשקלון אתמול" or "gas stations in Jerusalem").
2. An OpenAI-powered agent interprets the question, searches data.gov.il's CKAN API for relevant datasets, selects the best resource, and extracts the data.
3. The app presents a direct answer, an evidence table of extracted records, full source provenance, and any warnings or assumptions.

Agent Tools

The LLM agent has access to 6 server-side functions via OpenAI function calling:

Tool	Purpose
search_datasets	Search data.gov.il for datasets by keyword
get_dataset	Get full metadata for a dataset
list_resources	List all resources in a dataset
preview_resource	Preview fields and sample rows from a DataStore resource
fetch_resource	Fetch DataStore data with filters, search, sort, pagination
download_and_sample	Download and parse CSV/JSON files (allowlisted domains only)

Project Structure

app/
  main.py                       # FastAPI routes, rate limiting, Jinja2 templates
  models.py                     # Pydantic models
  utils.py                      # Retry/backoff, caching, domain allowlist, sanitization
  smoke_test.py                 # CLI smoke test
  templates/
    index.html                  # Single-page RTL-aware UI
  services/
    openai_agent.py             # LLM agent loop with tool calling
    ckan_client.py              # CKAN Action API wrappers
    resource_resolver.py        # Resource ranking and selection
    filters.py                  # Date normalization, local filtering
    answer_formatter.py         # Structured response builder
    fetchers/
      datastore_fetcher.py      # DataStore-backed resource fetcher
      csv_fetcher.py            # CSV download and parse
      json_fetcher.py           # JSON download and parse
tests/                          # 31 unit tests
Dockerfile
render.yaml                     # Render deployment config
requirements.txt

Setup

Prerequisites

• Python 3.11+
• An OpenAI API key

Local Development

# Clone the repo
git clone https://github.com/YoniLabell/DataGovAI.git
cd DataGovAI

# Install dependencies
pip install -r requirements.txt

# Set your OpenAI API key
export OPENAI_API_KEY="sk-..."

# Run the app
uvicorn app.main:app --reload --port 8000

Open http://localhost:8000 in your browser.

Environment Variables

Variable	Required	Default	Description
OPENAI_API_KEY	Yes	—	OpenAI API key
OPENAI_MODEL	No	gpt-4.1-mini	OpenAI model to use
PORT	No	10000	Server port
LOG_LEVEL	No	INFO	Logging level
RATE_LIMIT_PER_MINUTE	No	15	Max requests per IP per minute

Running Tests

pytest tests/ -v

All 31 tests use mocked HTTP responses and require no API keys.

Smoke Test

Run a question end-to-end from the command line (requires OPENAI_API_KEY):

python -m app.smoke_test "כמות הגשמים שירדה באשקלון אתמול"
python -m app.smoke_test "gas stations in Jerusalem"

Deploy to Render

Option 1: Using render.yaml

1. Push this repo to GitHub.
2. Go to Render Dashboard and click New > Blueprint.
3. Connect the repo — Render will detect render.yaml automatically.
4. Set the OPENAI_API_KEY environment variable in the Render dashboard.
5. Deploy.

Option 2: Manual Setup

1. Create a new Web Service on Render.
2. Connect your GitHub repo.
3. Set Runtime to Docker.
4. Add environment variable OPENAI_API_KEY.
5. Set Health Check Path to /health.
6. Deploy.

The app runs on Render's free tier with no heavy dependencies.

Security

• Domain allowlist: Only fetches data from data.gov.il and its subdomains. Arbitrary URL fetching is blocked.
• Input sanitization: User input and CKAN data are sanitized before processing.
• Rate limiting: Per-IP rate limiting (15 requests/minute by default).
• Tool enforcement: The LLM proposes actions; the server validates and executes only allowed tool calls.

Tech Stack

• Backend: Python, FastAPI, httpx, Pydantic
• LLM: OpenAI API with function calling
• Data source: data.gov.il CKAN Action API
• Frontend: Server-rendered HTML with vanilla JavaScript
• Deployment: Docker, Render