Beeper

Beeper is an open-source agentic AI SRE platform that investigates production anomalies, correlates signals across observability layers, and generates root cause hypotheses with resolution recommendations.

Architecture

beeper/
├── operator/           # Rust K8s operator (Cargo.toml)
├── investigator/       # Python investigator agent (pyproject.toml)
├── ui/                 # Flask web UI (pyproject.toml)
├── openapi/            # OpenAPI specification
├── helm/               # Helm chart for deployment
├── scripts/            # Development scripts
└── docker-compose.yaml # Local development stack

Components

Component	Technology	Description
Operator	Rust + kube-rs	Kubernetes controller that watches for anomalies and spawns investigators
Investigator	Python	AI agent that correlates signals and generates RCA hypotheses
UI	Flask + HTMX	Web interface for viewing investigations and managing knowledge base

Getting Started

Prerequisites

Rust (stable)
Python 3.11+
Poetry 1.7+
Docker and Docker Compose
For K8s demo: kind (installed automatically by make demo-cluster), helm 3.x, ANTHROPIC_API_KEY env var (for investigations)
Kubernetes cluster (for production deployment)

Local Development

Clone the repository:

git clone https://github.com/your-org/beeper.git
cd beeper

Run the setup script:
```
./scripts/setup-dev.sh
```
Start the local development stack:
```
docker-compose up -d
```
Initialize Qdrant collections and seed sample data:
```
./scripts/seed-kb.sh
```
This creates the investigations and knowledge collections with sample runbooks and investigation entries.

Run each component:

# Operator
cd operator && cargo run

# Investigator
cd investigator && poetry run python -m beeper_investigator.main

# UI
cd ui && poetry run flask run

Demo Environment

Beeper ships with a demo that uses the OpenTelemetry Astronomy Shop — a real polyglot e-commerce application (16+ microservices) with built-in fault injection via feature flags.

Quick Start (K8s)

# Set your LLM API key (required for investigations to complete)
export ANTHROPIC_API_KEY=sk-ant-...

# One command: create kind cluster, build images, deploy Beeper + OTel demo
make demo-up

# Port-forward the UIs
make demo-ui
# → Beeper UI:     http://localhost:5050
# → OTel Shop:     http://localhost:8080
# → Feature Flags: http://localhost:8080/feature
# → Jaeger:        http://localhost:16686

# Tear everything down (deletes the kind cluster)
make demo-down

Quick Start (Local UI only)

./scripts/demo.sh              # Qdrant + seed KB + Flask UI
./scripts/demo.sh --k8s        # Also deploy OTel demo to K8s

Fault Injection

Inject real failures into the running application and watch Beeper detect and investigate them:

make demo-fault FAULT=payment-failure   # Payment service errors
make demo-fault FAULT=cart-failure      # Cart service failures
make demo-fault FAULT=kafka-problems    # Kafka queue overload
make demo-fault FAULT=slow-images       # Image loading delays
make demo-fault FAULT=high-cpu          # Ad service CPU spike

make demo-fault-status                  # Check which faults are active
make demo-recover                       # Clear all faults

See demo/README.md for full details on architecture, SLOs, and all available Makefile targets.

Development

Running Tests

# Rust operator
cd operator && cargo test

# Python investigator
cd investigator && poetry run pytest

# Python UI
cd ui && poetry run pytest

Linting

# Rust
cd operator && cargo fmt --check && cargo clippy

# Python
cd investigator && poetry run ruff check .
cd ui && poetry run ruff check .

Kubernetes Deployment

Prerequisites

Kubernetes cluster (1.26+)
Helm 3.x
kubectl configured for your cluster

Installing Beeper

Install the Helm chart:
```
helm install beeper ./helm/beeper
```

Create the LLM credentials secret (see LLM Configuration below):

kubectl create secret generic llm-credentials \
  --from-literal=api-key=YOUR_API_KEY

Verify the operator is running:

kubectl get pods -l app.kubernetes.io/component=operator
kubectl logs -l app.kubernetes.io/component=operator

Custom Resources

Beeper uses two Custom Resource Definitions (CRDs):

Source CRD

Configure data sources (Prometheus/Loki):

Prometheus Source:

apiVersion: beeper.dev/v1
kind: Source
metadata:
  name: prometheus-main
spec:
  source_type: prometheus
  endpoint: http://prometheus:9090
  credentials_secret: prometheus-creds  # Optional - for authenticated access

Loki Source:

apiVersion: beeper.dev/v1
kind: Source
metadata:
  name: loki-main
spec:
  source_type: loki
  endpoint: http://loki:3100
  credentials_secret: loki-creds  # Optional - for authenticated access

Credential Secret Format:

Both Prometheus and Loki sources use the same credential format. If authentication is required, create a Secret with username and password:

apiVersion: v1
kind: Secret
metadata:
  name: prometheus-creds  # or loki-creds
type: kubernetes.io/basic-auth
data:
  username: <base64-encoded-username>
  password: <base64-encoded-password>

Source Status:

The operator validates connectivity and updates the Source status:

kubectl get sources
NAME              TYPE         CONNECTED   AGE
prometheus-main   prometheus   true        5m

View detailed status:

kubectl describe source prometheus-main

Troubleshooting Connection Errors:

Error	Cause	Solution
"Connection refused"	Endpoint unreachable	Verify endpoint URL and network policies
"Authentication failed"	Invalid credentials	Check Secret username/password are correct
"Access forbidden"	Insufficient permissions	Ensure credentials have read access
"Connection timed out"	Slow network or overloaded server	Increase timeout or check server health
"Secret not found"	Missing credentials Secret	Create the Secret in the same namespace
"Prometheus API error"	Invalid PromQL query or API issue	Check Prometheus logs and query syntax
"Loki API error"	Invalid LogQL query or API issue	Check Loki logs and query syntax
"Invalid response"	Unexpected response format	Ensure source endpoint is correct type

Investigation CRD

Created automatically when anomalies are detected:

apiVersion: beeper.dev/v1
kind: Investigation
metadata:
  name: inv-abc123
spec:
  condition: "High error rate detected"
  service: payments
  severity: high
  triggered_at: "2026-02-09T12:00:00Z"

RBAC Permissions

The operator ServiceAccount requires:

Resource	Verbs
`sources.beeper.dev`	get, list, watch, create, update, patch, delete
`investigations.beeper.dev`	get, list, watch, create, update, patch, delete
`sources.beeper.dev/status`	get, update, patch
`investigations.beeper.dev/status`	get, update, patch
`jobs` (batch)	get, list, watch, create, update, patch, delete
`pods`	get, list, watch
`secrets`	get, list, watch
`configmaps`	get, list, watch
`events`	create, patch

Streaming Data Ingestion

The operator accepts pushed metrics and logs via streaming ingestion endpoints on port 9090:

Prometheus Remote Write

Configure Prometheus to push metrics to Beeper:

# prometheus.yml
remote_write:
  - url: http://beeper-operator:9090/api/v1/write

The endpoint accepts:

Method: POST
Path: /api/v1/write
Content-Type: application/x-protobuf
Content-Encoding: snappy (recommended)
Body: Snappy-compressed protobuf WriteRequest

Loki Push

Configure Loki to push logs to Beeper:

# loki.yaml (custom client or Promtail)
clients:
  - url: http://beeper-operator:9090/loki/api/v1/push

The endpoint accepts:

Method: POST
Path: /loki/api/v1/push
Content-Type: application/json
Content-Encoding: snappy (optional)
Body: JSON with streams array

Example JSON format:

{
  "streams": [
    {
      "stream": {"app": "myapp", "level": "error"},
      "values": [
        ["1676466135000000000", "log line content"]
      ]
    }
  ]
}

Backpressure Handling

The ingestion endpoints implement backpressure:

Response Code	Meaning	Action
200 OK (Prometheus)	Success	Continue sending
204 No Content (Loki)	Success	Continue sending
503 Service Unavailable	Buffer full	Retry with backoff
429 Too Many Requests	Rate limited	Retry with backoff
400 Bad Request	Invalid format	Fix request format

Configuration

Environment variables:

Variable	Default	Description
`BEEPER_INGESTION_PORT`	`9090`	Ingestion HTTP port
`BEEPER_INGESTION_BUFFER_SIZE`	`10000`	Max buffered samples

Health Endpoints

The operator exposes health endpoints on port 8080:

/healthz - Liveness probe (always returns 200 OK)
/readyz - Readiness probe (checks Kubernetes API connectivity)

API Endpoints

The operator exposes UI-facing API endpoints on port 8080:

Endpoint	Method	Description
`/api/v1/sources`	GET	List all configured sources with status
`/api/v1/health/components`	GET	Component health status
`/api/v1/ingestion/stats`	GET	Ingestion buffer statistics

LLM Configuration

Beeper uses LLM providers for AI-powered root cause analysis. Configuration is done via Helm values and Kubernetes Secrets.

Supported Providers

Provider	Model Examples	API Key Required
Anthropic	`claude-sonnet-4`, `claude-3-haiku`, `claude-opus-4`	Yes
OpenAI	`gpt-4o`, `gpt-4-turbo`	Yes
Azure OpenAI	`azure/<deployment-name>`	Yes (+ endpoint)
Ollama	`ollama/llama3`	No

Creating the LLM Secret

# For Anthropic (default)
kubectl create secret generic llm-credentials \
  --from-literal=api-key=YOUR_ANTHROPIC_API_KEY

# For OpenAI
kubectl create secret generic llm-credentials \
  --from-literal=api-key=YOUR_OPENAI_API_KEY

# For Azure OpenAI
kubectl create secret generic llm-credentials \
  --from-literal=api-key=YOUR_AZURE_API_KEY

Or use the example manifest:

# Edit helm/beeper/examples/llm-secret.yaml with your key
kubectl apply -f helm/beeper/examples/llm-secret.yaml

Helm Configuration

Configure the LLM provider in values.yaml:

llm:
  provider: anthropic          # anthropic, openai, azure, ollama
  model: claude-sonnet-4       # Model identifier
  apiKeySecret: llm-credentials  # Secret name
  apiKeySecretKey: api-key     # Key within secret
  # endpoint: ""               # Required for Azure, optional for Ollama

Health Monitoring

LLM connectivity status is visible in the /api/v1/health/components endpoint and the UI Health page. Status values:

Status	Meaning
`healthy`	LLM configured and credentials accessible
`unconfigured`	LLM provider not configured
`unhealthy`	Configuration error (missing secret, invalid model, etc.)

Local Development

For local investigator development, copy .env.example to .env:

cd investigator
cp .env.example .env
# Edit .env with your API key

Environment variables:

BEEPER_LLM_PROVIDER - Provider name
BEEPER_LLM_MODEL - Model identifier
BEEPER_LLM_API_KEY - API key
BEEPER_LLM_ENDPOINT - Custom endpoint (optional)

UI Development

The web UI is a Flask application with HTMX for dynamic updates.

Running the UI

cd ui
poetry install
poetry run flask run

The UI will be available at http://localhost:5000.

Configuration

Copy .env.example to .env and configure:

cp .env.example .env

Variable	Default	Description
`FLASK_ENV`	`development`	Flask environment
`BEEPER_OPERATOR_URL`	`http://localhost:8080`	Operator API URL
`BEEPER_OPERATOR_TIMEOUT`	`5.0`	API request timeout (seconds)
`BEEPER_UI_PORT`	`5000`	UI server port

UI Pages

Route	Description
`/`	Home page
`/sources/`	View configured data sources and their status
`/health/`	View operator component health
`/health/api`	UI health check endpoint

Testing

cd ui
poetry run pytest -v
poetry run ruff check .

License

Apache License 2.0 - see LICENSE for details.

Contributing

See CONTRIBUTING.md for contribution guidelines.

Name		Name	Last commit message	Last commit date
Latest commit History 267 Commits
.claude		.claude
.design-sync		.design-sync
.github		.github
.synthex		.synthex
Auto Run Docs		Auto Run Docs
_bmad-output		_bmad-output
_bmad		_bmad
demo		demo
docs		docs
helm/beeper		helm/beeper
investigator		investigator
openapi		openapi
operator		operator
scripts		scripts
ui		ui
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
VISION.md		VISION.md
docker-compose.yaml		docker-compose.yaml
kind-config.yaml		kind-config.yaml
whats-next.md		whats-next.md

Folders and files

Latest commit

History

Repository files navigation

Beeper

Architecture

Components

Getting Started

Prerequisites

Local Development

Demo Environment

Quick Start (K8s)

Quick Start (Local UI only)

Fault Injection

Development

Running Tests

Linting

Kubernetes Deployment

Prerequisites

Installing Beeper

Custom Resources

Source CRD

Investigation CRD

RBAC Permissions

Streaming Data Ingestion

Prometheus Remote Write

Loki Push

Backpressure Handling

Configuration

Health Endpoints

API Endpoints

LLM Configuration

Supported Providers

Creating the LLM Secret

Helm Configuration

Health Monitoring

Local Development

UI Development

Running the UI

Configuration

UI Pages

Testing

License

Contributing

About

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages