Skip to content
BranchesTags

Repository files navigation

Kubernetes Monitoring Agent

An LLM-powered proactive cluster monitoring tool that detects issues, misconfigurations, and potential problems before they cause incidents.

Overview

The Monitoring Agent takes a fundamentally different approach to Kubernetes observability. Instead of defining static alert rules or waiting for incidents to occur, it leverages large language models (LLMs) with massive context windows to analyze complete cluster state and identify issues through intelligent pattern recognition.

┌─────────────────────────────────────────────────────────────────────────┐
│                     MONITORING AGENT ARCHITECTURE                        │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│   ┌─────────────┐     ┌──────────────┐     ┌─────────────────────────┐ │
│   │  Kubernetes │────▶│ State        │────▶│ LLM Analysis            │ │
│   │  Cluster    │     │ Collector    │     │ (Gemini/Claude/GPT)     │ │
│   └─────────────┘     └──────────────┘     └───────────┬─────────────┘ │
│                                                        │               │
│   ┌─────────────┐     ┌──────────────┐     ┌──────────▼──────────────┐ │
│   │ Structured  │◀────│ Report       │◀────│ Issue Detection &       │ │
│   │ JSON Report │     │ Generator    │     │ Root Cause Analysis     │ │
│   └─────────────┘     └──────────────┘     └─────────────────────────┘ │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘

Key Features

1. Full Cluster Analysis

Collects and analyzes 23 different data types in a single pass:

  • Workloads: Pods, Deployments, ReplicaSets, StatefulSets, DaemonSets, Jobs, CronJobs
  • Networking: Services, Endpoints, Ingresses, NetworkPolicies
  • Storage: PersistentVolumeClaims, PersistentVolumes
  • Configuration: ConfigMaps, Secrets (metadata only), ResourceQuotas, LimitRanges
  • Cluster: Nodes, Namespaces, Events, HPAs
  • Metrics: Pod CPU/Memory usage, Node CPU/Memory usage (via metrics-server)

2. Multi-Model Analysis

Run analysis with multiple LLMs for comprehensive coverage:

# Single model
python monitoring_agent.py -m gemini-2.5-flash -o report.json

# Multiple runs for reliability
python monitoring_agent.py -m gemini-2.5-flash --runs 3 -o report.json

# Multi-model consensus
python monitoring_agent.py --models "gemini-2.5-flash:2,claude-sonnet-4-20250514:1"

3. Scheduled Monitoring

Continuous monitoring with automatic diff detection:

# Run every 30 minutes
python monitoring_agent.py --schedule 30 --output-dir output/

4. Report Comparison (Diff Mode)

Compare current state with previous reports to track changes:

python monitoring_agent.py --diff output/previous-report.json -o report.json

5. Trend Analysis

Analyze multiple historical reports to identify patterns:

python monitoring_agent.py --trend output/report1.json output/report2.json output/report3.json

6. Focused Investigation

Granular data collection for agentic investigation loops:

# Single resource
python monitoring_agent.py --focus-type deployment --focus-name yoozer --focus-namespace ciroos

# All resources in namespace
python monitoring_agent.py --dump-focused --focus-namespace ciroos-agent

# All deployments cluster-wide
python monitoring_agent.py --dump-focused --focus-type deployments

7. Metrics & Anomaly Detection

Real-time CPU/memory metrics from metrics-server for proactive anomaly detection:

# Requires metrics-server installed:
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

When metrics-server is available, pods and nodes are enriched with:

  • current_cpu_millicores, current_memory_mib - Real-time usage
  • cpu_utilization_percent, memory_utilization_percent - Usage vs limits
  • resource_anomalies - Detected issues

Anomaly Thresholds:

Level CPU Memory Risk
HIGH >90% of limit >90% of limit OOMKill / Throttling imminent
ELEVATED >80% of limit >80% of limit Approaching danger zone

Example enriched pod:

{
  "name": "my-app-xyz",
  "current_cpu_millicores": 450,
  "current_memory_mib": 480,
  "cpu_utilization_percent": 90.0,
  "memory_utilization_percent": 93.8,
  "resource_anomalies": [
    "HIGH_CPU: 90% of limit",
    "HIGH_MEMORY: 94% of limit (OOMKill risk)"
  ]
}

What it detects:

  • Pre-OOMKill conditions (memory >90% of limit)
  • CPU throttling candidates
  • Memory leaks (high memory + frequent restarts)
  • Node pressure before pod evictions
  • Pods without resource limits (NO_RESOURCE_LIMITS)

How It Differs from Other Tools

vs. Sleuth (Reactive RCA)

Aspect Sleuth Monitoring Agent
Trigger Incident/Alert occurs Scheduled or on-demand
Approach Reactive investigation Proactive discovery
Scope Focused on incident context Full cluster analysis
Goal Find root cause of known issue Detect unknown issues
Data Flow Incident → Investigation → RCA Scan → Analysis → Prevention 
SLEUTH (Reactive):
  Alert Fired ──▶ Start Investigation ──▶ Gather Evidence ──▶ Root Cause

MONITORING AGENT (Proactive):
  Schedule ──▶ Collect State ──▶ LLM Analysis ──▶ Discover Issues ──▶ Prevent Incidents

Sleuth is designed for reactive root cause analysis - it activates when an incident is reported and investigates using domain-specific tools (kubectl, AWS CLI, etc.) to find the cause.

Monitoring Agent is designed for proactive issue discovery - it continuously scans the cluster to find problems before they escalate into incidents.

vs. Prometheus + Alertmanager

Aspect Prometheus/Alertmanager Monitoring Agent
Rule Definition Manual PromQL rules LLM-inferred patterns
Thresholds Static, predefined Dynamic, contextual
Coverage What you explicitly monitor Comprehensive analysis
Maintenance Requires rule tuning Self-adapting
Novel Issues Only detects known patterns Discovers unknown issues

Prometheus requires you to define what to monitor. If you don't write a rule for it, you won't detect it.

Monitoring Agent analyzes everything and uses LLM reasoning to identify issues you might not have thought to monitor.

vs. Kubernetes Dashboard

Aspect K8s Dashboard Monitoring Agent
Analysis Visual inspection (human) Automated (LLM)
Scalability Limited by human attention Handles large clusters
Correlation Manual cross-referencing Automatic relationship analysis
Output Visual display Actionable reports

Dashboard shows you the data - you need to analyze it yourself.

Monitoring Agent analyzes the data and tells you what's wrong.

vs. Datadog/New Relic/Dynatrace

Aspect APM Platforms Monitoring Agent
Cost Per-host/container pricing API calls only
Data Residency Vendor cloud Your infrastructure
Customization Platform constraints Full control
Focus Metrics/APM/Logs Kubernetes state analysis

APM Platforms excel at metrics, traces, and logs but can be expensive and require agents.

Monitoring Agent focuses specifically on Kubernetes resource state analysis without requiring any cluster-side agents.

Architecture Deep Dive

Data Collection Pipeline

┌────────────────────────────────────────────────────────────────────────┐
│                        DATA COLLECTION                                  │
├────────────────────────────────────────────────────────────────────────┤
│                                                                        │
│  kubectl get pods -A -o json ─────┐                                    │
│  kubectl get deployments -A ──────┤                                    │
│  kubectl get services -A ─────────┼──▶ Slim & Normalize ──▶ ClusterState
│  kubectl get events -A ───────────┤                                    │
│  kubectl get nodes -o json ───────┤                                    │
│  ... (21 resource types) ─────────┘                                    │
│                                                                        │
└────────────────────────────────────────────────────────────────────────┘

Slimming Strategy: Raw Kubernetes resources contain verbose metadata. The agent extracts only operationally relevant fields:

# Pod slimming extracts:
- name, namespace, phase, nodeName
- container names, images, resources
- containerStatuses (ready, restartCount, state)
- conditions (type, status, reason)

# Deployment slimming extracts:
- name, namespace, replicas
- readyReplicas, availableReplicas, unavailableReplicas
- conditions (type, status, reason)

LLM Analysis Strategy

┌────────────────────────────────────────────────────────────────────────┐
│                        ANALYSIS PIPELINE                                │
├────────────────────────────────────────────────────────────────────────┤
│                                                                        │
│  Small Cluster (<900K tokens):                                         │
│    ClusterState ──▶ Single LLM Call ──▶ Results                        │
│                                                                        │
│  Large Cluster (>900K tokens):                                         │
│    ClusterState ──▶ Chunk by Namespace ──▶ Parallel LLM Calls          │
│                          │                        │                    │
│                          └────────────────────────┼──▶ Aggregate       │
│                                                   │                    │
│  Multi-Run Mode:                                                       │
│    ClusterState ──▶ N Parallel Calls ──▶ NLP Dedup ──▶ Consolidate    │
│                                                                        │
│  Multi-Model Mode:                                                     │
│    ClusterState ──▶ Model A (N runs) ─┐                                │
│                ──▶ Model B (M runs) ──┼──▶ Cross-Model Consolidate    │
│                ──▶ Model C (P runs) ──┘                                │
│                                                                        │
└────────────────────────────────────────────────────────────────────────┘

Issue Detection Categories

The LLM is prompted to detect issues across these categories:

Category Examples
Pod Health CrashLoopBackOff, Pending, OOMKilled, high restarts
Deployment State Unavailable replicas, stuck rollouts, ProgressDeadlineExceeded
Service Issues No endpoints (selector mismatch), wrong exposure type
Node Problems NotReady, memory/disk/PID pressure
Resource Config Missing limits/requests, over/under provisioning
Security Privileged containers, hostNetwork, sensitive exposure
Events Warning events indicating problems
Storage Pending PVCs, orphaned PVs
Missing Resources Secrets not found, ServiceAccounts missing

Report Schema

{
  "description": "Cluster monitoring scan results",
  "date": "2025-12-11",
  "timestamp": "2025-12-11T07:26:36.876751",
  "cluster": "microk8s",
  "scenarios": [
    {
      "id": 1,
      "name": "deployment_crashloopbackoff",
      "namespace": "ciroos",
      "severity": "critical|high|medium|low",
      "issue_summary": "One line description",
      "root_cause": "Detailed explanation",
      "affected_resources": ["pod/name", "deployment/name"],
      "symptoms": ["symptom 1", "symptom 2"],
      "recommendation": "How to fix"
    }
  ]
}

Focused Investigation for Agentic Loops

The focused investigation mode enables building agentic workflows:

┌─────────────────────────────────────────────────────────────────────────┐
│                     AGENTIC INVESTIGATION LOOP                          │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  1. Initial Scan                                                        │
│     └──▶ Full cluster analysis identifies issues                        │
│                                                                         │
│  2. Focus on Problem                                                    │
│     └──▶ collect_focused_state(deployment="yoozer", namespace="ciroos") │
│                                                                         │
│  3. Deep Investigation                                                  │
│     └──▶ Analyze focused state with targeted prompts                    │
│                                                                         │
│  4. Expand Investigation                                                │
│     └──▶ Follow relationships (owner refs, service selectors)           │
│                                                                         │
│  5. Generate Findings                                                   │
│     └──▶ Produce actionable recommendations                             │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘

Focus Modes

Mode Command Returns
Single Resource --focus-type pod --focus-name xyz --focus-namespace ns Resource + events + related
Resource Type --focus-type deployments --focus-namespace ns All deployments in namespace
Namespace --focus-namespace ns All resources in namespace
Cluster-wide Type --focus-type pods All pods cluster-wide

Related Resource Discovery

When investigating a specific resource, the agent automatically fetches related resources:

Resource Type Related Resources Fetched
Deployment Pods matching selector labels
Service Endpoints
Pod Owner references (ReplicaSet, Deployment, etc.)

Installation

Requirements

  • Python 3.12+
  • kubectl configured with cluster access
  • LLM API key (Gemini, Anthropic, or OpenAI)

Setup

# Clone repository
cd discovery-agent

# Install dependencies
pip install -r requirements.txt
# Or with uv:
uv pip install -r requirements.txt

# Set API key
export GEMINI_API_KEY="your-api-key"
# Or for other providers:
export ANTHROPIC_API_KEY="your-api-key"
export OPENAI_API_KEY="your-api-key"

Kubernetes Deployment

Deploy as a CronJob for continuous monitoring:

# Create namespace and secrets
kubectl create namespace monitoring
kubectl create secret generic monitoring-agent-secrets \
  --namespace monitoring \
  --from-literal=GEMINI_API_KEY="your-api-key"

# Build and push image
docker build -t your-registry/monitoring-agent:latest .
docker push your-registry/monitoring-agent:latest

# Deploy
kubectl apply -f k8s/cronjob.yaml

CLI Reference

usage: monitoring_agent.py [-h] [--provider {google,anthropic,openai}]
                           [--model MODEL] [--models MODELS]
                           [--namespaces NAMESPACES] [--output OUTPUT]
                           [--runs RUNS] [--sequential]
                           [--schedule MINUTES] [--output-dir OUTPUT_DIR]
                           [--diff PREVIOUS_REPORT] [--trend REPORT ...]
                           [--focus-type TYPE] [--focus-name NAME]
                           [--focus-namespace NS] [--no-events] [--no-related]
                           [--dump-state] [--dump-focused] [--no-save]

Options:
  --provider, -p        LLM provider (google, anthropic, openai)
  --model, -m           Model name
  --models              Multi-model specs: 'model1:runs,model2:runs'
  --namespaces, -n      Comma-separated namespaces to analyze
  --output, -o          Output JSON file path
  --runs, -r            Number of analysis runs (default: 1)
  --sequential, -s      Run LLM calls sequentially (default: parallel)

Scheduling:
  --schedule MINUTES    Run continuously every N minutes
  --output-dir          Output directory for scheduled runs

Comparison:
  --diff PREVIOUS       Compare with previous report
  --trend REPORTS       Analyze trends across multiple reports

Focused Investigation:
  --focus-type TYPE     Resource type (pod, deployment, service, etc.)
  --focus-name NAME     Specific resource name
  --focus-namespace NS  Namespace for focused investigation
  --no-events           Exclude events
  --no-related          Exclude related resources

Debug:
  --dump-state          Dump full cluster state JSON (no LLM)
  --dump-focused        Dump focused state JSON (no LLM)
  --no-save             Don't save to file, just print

Use Cases

1. Pre-deployment Health Check

# Before deploying new version
python monitoring_agent.py -o pre-deploy.json

# After deployment
python monitoring_agent.py --diff pre-deploy.json -o post-deploy.json

2. Continuous Monitoring

# Run as daemon, checking every 15 minutes
python monitoring_agent.py --schedule 15 --output-dir /reports/

3. Incident Investigation Support

# Quick focused investigation when alert fires
python monitoring_agent.py --focus-type pod --focus-name problematic-pod --focus-namespace production

4. Capacity Planning

# Analyze resource usage patterns over time
python monitoring_agent.py --trend reports/week1.json reports/week2.json reports/week3.json

5. Security Audit

# Full cluster scan looking for security issues
python monitoring_agent.py -m gemini-2.5-pro --runs 3 -o security-audit.json

Model Recommendations

Use Case Recommended Model Reasoning
Quick daily checks gemini-2.5-flash Fast, cost-effective
Thorough analysis gemini-2.5-pro Better reasoning
Multi-model consensus gemini-2.5-flash:2,claude-sonnet-4-20250514:1 Cross-validation
Large clusters gemini-2.5-flash 1M token context
Security-focused claude-sonnet-4-20250514 Detailed analysis

Context Window Limits

Model Context Window Estimated Cluster Size
Gemini 2.5/3 1,000,000 tokens ~500 pods
Claude Opus/Sonnet 200,000 tokens ~100 pods
GPT-4o 128,000 tokens ~60 pods

For clusters exceeding context limits, the agent automatically chunks by namespace and aggregates results.

Environment Variables

Variable Description
GOOGLE_API_KEY or GEMINI_API_KEY Google Gemini API key
ANTHROPIC_API_KEY Anthropic Claude API key
OPENAI_API_KEY OpenAI API key

Integration Ideas

With Sleuth

Monitoring Agent (proactive) ──▶ Detects degraded deployment
                                        │
                                        ▼
                               Trigger Sleuth RCA ──▶ Deep investigation

With PagerDuty/Slack

# In your integration script
report = json.load(open("report.json"))
critical_issues = [s for s in report["scenarios"] if s["severity"] == "critical"]
if critical_issues:
    send_to_pagerduty(critical_issues)

With CI/CD

# In your deployment pipeline
- name: Pre-deploy health check
  run: |
    python monitoring_agent.py -o pre-deploy.json
    CRITICAL=$(jq '[.scenarios[] | select(.severity=="critical")] | length' pre-deploy.json)
    if [ "$CRITICAL" -gt 0 ]; then
      echo "Critical issues detected, aborting deployment"
      exit 1
    fi

Contributing

See CONTRIBUTING.md for guidelines.

License

MIT License - See LICENSE for details.

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages