LLM Exerciser

LLM Exerciser runs one standardized expected-output suite against one or more OpenAI-compatible models, then optionally repeats the workload for hours while checking correctness, request failures, client/server RSS, host memory, swap, and arbitrary Prometheus metrics. The runner keeps only bounded reservoir samples and failure previews in memory, so the test harness should not grow in proportion to soak duration.

The tool is deliberately small and dependency-free for its built-in mode. It can also supervise GuideLLM as an external high-rate load generator while continuing to send assertion-checked canary requests. Prometheus probes allow telemetry from AMD Device Metrics Exporter, node_exporter, or an inference runtime's own /metrics endpoint to participate in the same pass/fail report.

What it proves

Each run produces evidence for:

• the same deterministic and structured-output tests across every configured target;
• request, HTTP, timeout, malformed-stream, and assertion failures;
• pre-soak and post-soak correctness, which catches state corruption that only appears after prolonged use;
• request counts, pass rate, TTFT, latency percentiles, and token totals;
• client RSS and optionally one local server PID per target;
• system available-memory and swap limits on Linux;
• post-warmup memory growth and least-squares slope gates;
• configured Prometheus minimum, maximum, growth, and slope gates;
• clean interruption with a final summary and non-zero exit status.

A passing soak is bounded evidence for the exact duration and workload that ran. It is not proof that a leak can never occur. An 8-hour production gate should actually run for eight hours with realistic context sizes and concurrency.

Install

Python 3.11 or newer is required.

cd llm-exerciser
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -e .

Built-in mode has no runtime dependencies. GuideLLM is optional and currently supports Python 3.10 through 3.13, so install it in a compatible environment or point [driver].executable at its executable:

python -m pip install 'guidellm[recommended]'

Configure

Copy examples/config.toml and examples/standard-tests.json. Add one [[targets]] block per model or endpoint. Base URLs are server roots and API paths include /v1:

tests_file = "standard-tests.json"
output_dir = "results"

[[targets]]
name = "qwen-rocm"
base_url = "http://127.0.0.1:8000"
model = "Qwen/Qwen3.5-35B-A3B"
endpoint = "/v1/chat/completions"
models_endpoint = "/v1/models"
stream = true
concurrency = 2
monitor_pid_file = "/run/user/1000/qwen.pid"

[[targets]]
name = "deepseek-vulkan"
base_url = "http://127.0.0.1:8001"
model = "deepseek-coder-v2"
stream = true
concurrency = 1
monitor_pid = 12345

API secrets are read only from an environment variable and are never copied to the run manifest:

api_key_env = "LOCAL_LLM_API_KEY"

Target request_defaults are merged first, then each test's request object. This permits model-specific defaults while keeping test inputs standardized.

Define expected output

The JSON test file is an array. Each test supplies chat messages, request overrides, and one or more assertions:

{
  "id": "json_contract",
  "messages": [
    {"role": "system", "content": "Return only JSON."},
    {"role": "user", "content": "Return {\"status\":\"ok\"}."}
  ],
  "request": {"temperature": 0, "max_tokens": 32},
  "assertions": [
    {"type": "json_valid"},
    {"type": "json_path", "path": "$.status", "equals": "ok"}
  ]
}

Supported assertions are:

Type	Required fields	Meaning
exact	value	Exact text after optional trimming
contains, not_contains	value	Substring present or absent
regex, not_regex	pattern	Python regular-expression match
json_valid	none	Entire output parses as JSON
json_path	path, optional equals/one_of	Simple $.field[0] lookup
min_chars, max_chars	value	Output-size boundary
finish_reason	value or one_of	OpenAI finish reason
latency_ms	value	Maximum end-to-end latency

Text assertions accept case_sensitive and strip. A test can use targets to limit it to named targets and repeats for functional-run repetition.

Run

Validate without contacting any endpoint:

llm-exerciser validate -c config.toml

Run the functional suite once, or override every test's repeat count:

llm-exerciser run -c config.toml
llm-exerciser run -c config.toml --repeats 10

Run the configured soak or a shorter qualification stage:

llm-exerciser soak -c config.toml
llm-exerciser soak -c config.toml --duration 30m
llm-exerciser soak -c config.toml --duration 8h

SIGINT and SIGTERM stop new work, finish artifact generation, and return a failing/interrupted summary. Request timeouts bound how long in-flight HTTP calls can delay shutdown.

GuideLLM integration

Set the driver to guidellm to run synthetic load against each target in parallel. LLM Exerciser supervises the processes, bounds GuideLLM's retained request samples, captures its JSON/CSV/log files, monitors memory, and sends one continuous assertion-checked canary stream per target.

[driver]
kind = "guidellm"
profile = "concurrent" # concurrent, constant, poisson, or throughput
prompt_tokens = 2048
output_tokens = 128
extra_args = ["--disable-console-interactive"]

For concurrent, target concurrency becomes GuideLLM's stream count. For constant or poisson, it becomes requests per second. For throughput, it becomes maximum concurrency. The generated command and GuideLLM output are preserved under guidellm/<target>/.

Prometheus and AMD telemetry

Add one or more [[prometheus_probes]] blocks. Every metric can filter exact labels, aggregate matching series, apply a scale, and define gates:

[[prometheus_probes]]
name = "amd-gpu"
url = "http://127.0.0.1:5000/metrics"
required = true

[[prometheus_probes.metrics]]
name = "replace_with_exported_vram_metric"
aggregate = "sum"
scale = 0.00000095367431640625 # bytes to MiB
max_growth = 1024
max_slope_per_hour = 256

[[prometheus_probes.metrics]]
name = "replace_with_exported_temperature_metric"
aggregate = "max"
max_value = 90

Inspect the exporter's /metrics output and use its actual metric names. A required metric that is absent or has too few post-warmup samples fails the soak rather than silently weakening the evidence.

Artifacts and exit codes

Each timestamped result directory contains:

• manifest.json: hashes of the exact config and test suite plus non-secret target/driver settings;
• events.jsonl: one bounded request record per attempt, flushed immediately;
• telemetry.jsonl: raw time-series samples and scrape errors;
• summary.json: aggregate metrics, trend calculations, failure previews, and the explicit gate table;
• report.md: human-readable gate, request, and telemetry summary;
• junit.xml: CI-friendly aggregated functional results;
• guidellm/<target>/: optional external-driver logs and reports.

Exit code 0 means every evaluated gate passed, 1 means a request/assertion, driver, resource, or leak gate failed, 2 means configuration was invalid, and 130 means the run was interrupted.

The process uses constant-memory online statistics plus a reservoir of at most 4,096 latency samples and the 100 most recent failures. JSONL artifact size still grows with request count by design; rotate or compress completed result directories according to local retention policy.

Development verification

python -m unittest discover -s tests -v

See Open-source landscape for the build-versus- integrate decision and the role of Promptfoo, GuideLLM, AIPerf, vllm-bench, MLPerf LoadGen, and Prometheus exporters.