Goal: Run the Birtha / WrkHrs AI product (published OCI images; pin ${XLOTYL_IMAGE_PREFIX} + ${XLOTYL_VERSION} in config/xlotyl-images.env, typically ghcr.io/xlotyl; sources in XLOTYL/xlotyl) on hardened infrastructure: Proxmox VMs/LXCs, Docker Compose, reverse proxy (Caddy), queues (Redis), observability (MLflow, Prometheus, Grafana, Loki, Tempo), optional security (Fail2ban/CrowdSec), and DNS/homelab add-ons. A workstation with an RTX 4070 Ti hosts GPU inference workers, reachable from the server over LAN/mTLS.
- Server stack (Proxmox VM/LXC): Compose-driven platform services (observability, registry, Redis, reverse proxy), host MCP servers, and pull/run of the xlotyl AI stack via OCI images pinned in
config/xlotyl-images.env. AI application source lives only in the xlotyl product repo. - Workstation (RTX 4070 Ti): vLLM/Ollama LLM runners in Docker with
nvidia-container-toolkit, exposed only to the server via LAN/mTLS. - Dev flow: Remote dev via VS Code Dev Containers → CI in server (MCP + infra) and XLOTYL/xlotyl (full AI matrix) → Deploy pulls pinned images → Running containers use the published AI stack.
- This repo — Infrastructure and deployment: compose profiles, CI, MCP host packages, observability base, networking/media infra, and glue that runs the AI product from published images (not submodules).
- Legacy archive — Historical MBMH / harness material stays off the default tracked tree as a backup-style snapshot (sibling path or team storage). See
docs/legacy-archive.mdanddocs/repository-content-model.md.
Most of this repository is executable glue (compose, scripts, CI) plus documentation and links to upstreams so others can clone and reproduce how the stack is wired. We do not bulk-import large backups or runtime artifacts. Third-party components (e.g. Pi-hole) are referenced by official project links and pinned images in compose—see docs/runbooks/third-party-components.md. Custom packages we author live in separate repos (and/or under xlotyl) with CI calling this repo’s reusable workflows—see packages/README.md and docs/xlotyl-and-standalone-packages.md.
This repository is the primary web-based tracker for infrastructure layout, compose, CI, and image version pins (config/xlotyl-images.env) so changes are reviewable and reproducible. It does not replace on-machine operational truth.
GitHub is tertiary relative to local RAID (primary server) and backups on secondary machines: use git for history, diffs, and collaboration; use RAID and secondary copies for durability and recovery. See docs/infrastructure-and-git.md.
- AI product — XLOTYL/xlotyl (OpenClaw and related tooling are submodules inside that repo, not here). This server repo pulls pinned images only (
XLOTYL_IMAGE_PREFIX+XLOTYL_VERSIONinconfig/xlotyl-images.env); it does not vendor the product tree. - See
docs/external-repos.mdanddocs/external-orchestration-interfaces.md.
This project uses GitHub Projects for task tracking and project management.
- Project Board: https://github.com/users/mhold3n/projects/3
- Issue Templates: Available in
.github/ISSUE_TEMPLATE/ - Automation: Configured via
.github/workflows/project-automation.yml
Use the appropriate issue template:
- WrkHrs Features: Use
feature-wrkhrs.mdtemplate - MCP Servers: Use
mcp-server.mdtemplate - Observability: Use
observability.mdtemplate
- Backlog: All open issues with
wrkhrs-convergencelabel - In Progress: Assigned issues currently being worked on
- Done: Completed issues
- Blocked: Issues that are blocked or waiting for dependencies
- Proxmox host ready; create a VM/LXC for
agent-server. - Docker + Compose on server and workstation.
- (Workstation) NVIDIA driver +
nvidia-container-toolkit. uvfor the server-root Python workspace (host MCP tooling); clone XLOTYL/xlotyl beside this repo for AI Python/Node work.npmfor optional root scripts; usenpm ciinside the xlotyl clone for agent-platform builds.
uv sync --python 3.11
# AI product (optional sibling clone):
# git clone https://github.com/XLOTYL/xlotyl ../xlotyl && cd ../xlotyl && git submodule update --init --recursive && uv sync && npm ciFocused tool envs are bootstrapped explicitly and live under .cache/envs/:
scripts/bootstrap_tool_env.sh marker-pdf
scripts/bootstrap_tool_env.sh whisper-asr
scripts/bootstrap_tool_env.sh qwen-runtimeShared caches and reproducible local model state live under .cache/.
Copy .env.example to .env and fill values:
cp .env.example .env# Platform services (MLflow, observability)
make platform-up
# AI stack (WrkHrs services)
make ai-up
# Full server deployment (platform + AI + server)
make server-up
# addons (security/search/media/etc.; host/homelab-only)
make addons-up
# full AI dev stack (base + platform + AI + local overrides)
make up# Start GPU worker
make worker-up
# Or start with Ollama instead of vLLM
docker compose --project-directory "$(pwd)" -f docker/compose-profiles/docker-compose.worker.yml --profile ollama up -d# Health check all services
make health
# Smoke test with trace propagation
make smoke-test
# Test WrkHrs gateway
curl -s http://localhost:8080/health
# Test LLM inference via worker
curl -s https://worker.local:8443/v1/models
# Test MLflow UI
open http://localhost:5000- All devs SSH or VS Code Remote into the server (or work standalone in the XLOTYL/xlotyl clone for product code).
- API and Router (and the WrkHrs stack) run from published GHCR images; image host and tag are set in
config/xlotyl-images.env(XLOTYL_IMAGE_PREFIX,XLOTYL_VERSION). Compose wires ports and env. - WrkHrs / gateway stacks use the same published image set (built and released from the xlotyl repo).
- Internal services call the worker via OpenAI-compatible endpoints for LLM inference.
- MCP catalog (xlotyl-owned, baked into the mcp-registry image). Implementations under
mcp-servers/here are tracked for build/CI on primary hardware — seemcp-servers/README.md. - MLflow provides experiment tracking and model registry for all AI operations.
- WrkHrs Gateway: Main API gateway for AI requests with domain classification and request conditioning
- WrkHrs Orchestrator: Task orchestration and workflow management using LangChain/LangGraph
- WrkHrs RAG: Retrieval-augmented generation with Qdrant vector database
- WrkHrs ASR: Automatic speech recognition with Whisper integration
- WrkHrs Tool Registry: Tool discovery and registration for MCP integration
- WrkHrs MCP: Micro-capability platform service for tool and resource management
- Evidence Policy: Requires citations and evidence in AI responses
- Citation Policy: Validates citation quality and format
- Hedging Policy: Detects and flags hedging language
- Units Policy: Normalizes and validates SI units
- Policy Registry: Dynamic policy discovery and validation
- Policy Middleware: Automatic policy enforcement in chat completions with MLflow logging
- MLflow: Experiment tracking, model registry, and artifact storage
- OpenTelemetry: Distributed tracing with Tempo integration
- Prometheus: Metrics collection and monitoring
- Grafana: Dashboards and visualization
- Loki: Log aggregation and analysis
- Request Context Middleware: Automatic trace/run/policy header propagation
- Golden Trace: End-to-end trace validation from gateway to worker
- Default assumes models fit in 12 GB (e.g., 7–13B, FP16 or low-bit). Use vLLM paged attention & quantization for longer contexts.
- When models outgrow 12 GB: consider quantized variants or re-evaluate GPU topology.
- WrkHrs supports both vLLM and Ollama backends for LLM inference.
This system implements a hybrid MCP (Model Context Protocol) architecture:
| Dimension | Install globally on control plane | Install per-repo (in the project) |
|---|---|---|
| Scope & reuse | Cross-project tools (GitHub, Jira, search, vector DB, secrets broker, artifact registry) | Repo-unique tools (custom code indexer, domain schema, one-off integrations) |
| Versioning stability | You want one version for all teams; change managed by infra | The repo must pin exact versions compatible with its code/tooling |
| Security & secrets | Centralized secret custody, auditing, network ACLs | Least-privilege, repo-scoped tokens, sandboxes per project |
| Isolation / blast radius | Lower isolation (shared process) unless namespaced | High isolation; failures/updates affect only that repo |
| Reproducibility / CI | Good for common infra; less deterministic per-repo without careful tagging | Strong: repo contains its MCP stack → portable dev/CI |
| Performance / caching | Shared caches (code search, embeddings) benefit all repos | Tailored indexes/caches per repo; no cross-pollution |
| Ops overhead | Lower (one place to patch/observe) | Higher (N stacks), but automated with templates |
- GitHub MCP: Repository operations, issue tracking, pull requests, GitHub Projects integration
- Filesystem MCP: File operations, code analysis, dependency tracking
- Code Resources MCP: Indexed codebase datasets with embeddings for code search
- Document Resources MCP: PDF/textbook datasets with chunking for document search
- Secrets MCP: Secure secrets management with Vault integration
- Vector DB MCP: Embedding search, knowledge retrieval
- MCP Registry: Auto-registration and discovery of MCP servers with health monitoring
- Custom code indexers
- Domain-specific schemas
- Project-unique integrations
Test the complete observability stack with a golden trace:
# Start the full stack
make up-all
# Run smoke test with golden trace
make smoke-test
# Send a chat request with trace headers
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-H "x-trace-id: golden-trace-001" \
-H "x-run-id: golden-run-001" \
-H "x-policy-set: golden-policy" \
-d '{
"model": "mistralai/Mistral-7B-Instruct-v0.3",
"messages": [{"role": "user", "content": "Explain quantum computing with proper citations."}],
"temperature": 0.7,
"max_tokens": 200
}'The system automatically enforces policies on all chat completions:
- Hedging Detection: Flags uncertain language ("might", "could", "seems")
- Citation Requirements: Validates proper citations and references
- Evidence Standards: Ensures factual claims are supported
- SI Units: Normalizes measurements to standard units
Policy verdicts are logged to MLflow and included in response headers:
# Check policy verdict headers
curl -I -X POST http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "test", "messages": [{"role": "user", "content": "This might be correct."}]}'
# Response headers include:
# x-policy-verdict: False
# x-policy-score: 0.3MCP servers automatically register on startup:
# Register a new MCP server
curl -X POST http://localhost:8001/mcp/registry/register \
-H "Content-Type: application/json" \
-d '{
"name": "test-mcp",
"type": "tool",
"url": "http://test-mcp:7000",
"tools": [{"name": "test_tool", "description": "A test tool"}]
}'
# List registered servers
curl http://localhost:8001/mcp/registry- MLflow: Experiment tracking, model registry, and artifact storage for all AI operations
- Prometheus: Scrapes vLLM, WrkHrs services, and system metrics
- Grafana: Dashboards for service health, performance, and resource usage
- Loki: Log aggregation and analysis for all services
- Tempo: Distributed tracing for request flow analysis
- Jaeger: Trace visualization and analysis
- Comprehensive alerting for service health, performance, and resource usage
- LAN allowlist, mTLS (Caddy), token auth for internal APIs.
- Optional WireGuard/Tailscale for remote laptops into the server.
- Vault integration for secrets management.
- Local Development: Use VS Code Dev Containers for consistent environment
- Testing: Comprehensive test suite with pytest, mypy, ruff, black
- CI/CD: Automated testing, building, and deployment via GitHub Actions
- Deployment: SSH-based deployment to server and worker nodes
See deploy/server/provision_proxmox.md for VM/LXC creation, cloud-init, and networking (VLANs, bridges).
# Platform services (MLflow, observability)
make platform-up # Start platform services
make platform-down # Stop platform services
make logs-platform # View platform logs
# AI stack (WrkHrs services)
make ai-up # Start AI stack services
make ai-down # Stop AI stack services
make logs-ai # View AI stack logs
# Full server deployment (platform + AI + server)
make server-up # Start full server stack
make server-down # Stop full server stack
make logs-server-full # View full server logs
# Worker deployment
make worker-up # Start GPU worker
make worker-down # Stop GPU worker
make logs-worker # View worker logs
# Health and testing
make health # Health check all services
make smoke-orchestration # CLI smoke: orchestrator + gateway (no OpenClaw)
make bundle-orchestration-logs # Bundle logs + snapshots for debugging
make seed-corpora # Seed RAG corpora
make eval # Run evaluation harness
make mlflow-ui # Open MLflow UI
# Local development
make up # Start local stack
make down # Stop local stack
make logs # View logs
make test-chat # Test chat endpoint
# Addons
make up-addons # Start addon stack (profiles)
make logs-addons # View addon logs
make up-all # Start core + server + addons
# Testing
make test # Run all tests
make lint # Run linting
make type # Run type checking
make fix # Fix linting issues
# CI simulation
make ci # Run full CI pipeline- The reverse proxy uses subdomains with internal TLS. We now prefer the
.landomain to avoid mDNS conflicts with.local. Add DNS/hosts entries pointing these to the server IP (e.g., via Pi‑hole):- api.lan, router.lan
- ai.lan (WrkHrs AI stack)
- mlflow.lan, tempo.lan, loki.lan, jaeger.lan (observability)
- grafana.lan, prometheus.lan
- homarr.lan, pihole.lan
- meili.lan, searx.lan
- sso.lan (Authentik)
- nextcloud.lan, immich.lan, vikunja.lan, pelican.lan
- mcp-github.lan, mcp-files.lan, mcp-secrets.lan, mcp-vector.lan
- When accessing from a browser, use port 8443 (example: https://grafana.lan:8443).
- Upstream: Pi-hole Docker image and docs are not vendored here—see
docs/runbooks/third-party-components.mdanddocker/config/dns/pihole/README.md. - Pi-hole is available in the host/homelab server profile. AdGuard Home is scaffolded in its own add-on profile and is not started by default dev.
- Use only one DNS blocker at a time. To test AdGuard:
- Stop Pi-hole:
docker compose --project-directory "$(pwd)" -f docker-compose.yml -f docker/compose-profiles/docker-compose.platform.yml -f docker/compose-profiles/docker-compose.ai.yml -f docker/compose-profiles/docker-compose.server.yml stop pihole - Start AdGuard:
docker compose --project-directory "$(pwd)" -f docker-compose.yml -f docker/compose-profiles/docker-compose.addons.yml --profile adguard up -d - Visit https://adguard.local:8443 after adding DNS/hosts entries
- Stop Pi-hole:
Canonical Git remote: github.com/mhold3n/server. Use a single clone for day-to-day work (docs/runbooks/canonical-server-clone.md); optional legacy snapshots belong outside this tree (docs/legacy-archive.md). AI application sources live in XLOTYL/xlotyl; this repo pins registry + semver for those images via config/xlotyl-images.env (XLOTYL_IMAGE_PREFIX, XLOTYL_VERSION). Content model: docs/repository-content-model.md.
server/ # repository root (suggested clone folder name)
├── config/
│ └── xlotyl-images.env # Pin XLOTYL_IMAGE_PREFIX + XLOTYL_VERSION for AI stack pulls
├── services/ # Infra / platform services (mock-openai, queue, …)
├── mcp-servers/mcp/ # MCP servers
│ ├── servers/ # Global and per-repo MCP servers
│ │ ├── github-mcp/ # GitHub MCP with Projects integration
│ │ ├── code-resources-mcp/ # Code indexing and search
│ │ └── doc-resources-mcp/ # Document indexing and search
│ └── config/ # MCP server configuration
├── docker/
│ ├── compose-profiles/ # Canonical repo-level Compose profiles
│ └── config/ # Reverse proxy and observability config
├── worker/ # GPU worker configuration
│ ├── vllm/ # vLLM setup and docs
│ └── tgi/ # TGI alternative
├── deploy/ # Deployment scripts and guides
│ ├── server/ # Proxmox provisioning
│ └── ci/ # CI/CD scripts
├── scripts/ # Helper scripts
│ ├── health_check.sh # Unified health check
│ └── ingest/ # Corpus ingestion scripts
├── docs/ # Documentation (index: README.md; pointer-first: repository-content-model.md)
│ ├── runbooks/ # Operational runbooks
│ └── adr/ # Architecture Decision Records
├── packages/ # README: links to standalone domain package repos
└── dev/ # Development tools
├── containers/ # Dev container configuration
└── scripts/ # Helper scripts
- Install pre-commit hooks:
make install-pre-commit - Follow the coding standards (ruff, black, mypy --strict)
- Write tests for new functionality
- Update documentation as needed
- Submit pull requests with clear descriptions
MIT License - see LICENSE for details.