ttft

Zero-config local LLM optimization for Ollama, LM Studio, and Apple Silicon MLX. Reduces TTFT by 40%, wall time for local agents by 46%, and RAM usage by 3x.

LLM inference benchmarking toolkit. Measure TTFT, inter-token latency, throughput, and P50–P99 across concurrency levels.

The only voice agent context manager with a TTFT feedback loop

Mesure les métriques d'inférence LLM (TTFT, TPOT, débit, coût, VRAM) sur n'importe quelle API OpenAI-compatible. Inclut infer-serve pour héberger un GGUF via llama.cpp en une commande.

An asynchronous, neuro-symbolic VLA (Vision-Language-Action) orchestration stack for edge autonomy. Fuses probabilistic Qwen2-VL visual reasoning and faster-whisper ASR with deterministic PX4/MAVSDK flight-control loops and HSV color guardrails.

LLM inference observability sidecar for vLLM in Python: FastAPI middleware exports TTFT/TBT/E2E, KV-cache and queue-depth metrics to Prometheus + Grafana; Docker Compose stack; 32/32 pytest passing. Demo: TTFT p50 84ms / p99 244ms, TBT p50 17ms, E2E p50 384ms.

Benchmark for prefill/decode interference on a single GPU, measuring TTFT inflation when new requests arrive while decode is already running.

Load generator and performance analyzer for LLM inference serving. Measures TTFT, TPOT, and throughput against vLLM/TGI/Ollama or a built-in mock, and flags serving anti-patterns with tuning advice.

Measure LLM provider mechanics — TTFT, tokens/sec, and cost at API list rates — one JSONL schema any benchmark can consume. A math vs vibes project.

Event-driven benchmark of dynamic batch size controllers for LLM serving, measuring the throughput vs TTFT SLO trade-off under variable load across five workload profiles.

Benchmark study: NVIDIA L4 + Llama 3.1 8B FP8 on Red Hat OpenShift AI. Concurrent load benchmarks, quality evaluation, and deployment advisor grounded in real measured data.

Benchmark comparing three LLM serving strategies on identical hardware: naive (no cache, no batching), KV-cache only, and continuous batching. Batched server achieves 21.8x throughput gain and constant 7-9ms TTFT under concurrent load.

Evidence-first AI serving capacity tests for HTTP/SSE/WebSocket, measured as SLO-qualified goodput.

Reproducible ML inference benchmarking across heterogeneous hardware: one YAML config drives llama.cpp and ONNX Runtime sweeps on laptops, Raspberry Pi (SSH) and GPUs, with TTFT, throughput, per-rail power, thermal and utilization telemetry in self-describing records, plus publication-quality tables and figures.

Benchmark of token streaming flush policies for LLM serving, measuring user-perceived interactivity, chunk readability, and infrastructure overhead through TTI, TTRC, SPIS, and SPIS-R metrics.

A tiny, stdlib-only Python script that benchmarks local models served by Ollama

Thin benchmark harness for local LLM endpoints — TTFT/prefill/decode with distribution, correlated with macOS memory pressure and thermal throttling. Same code on M1 (LM Studio) and M5 (mlx-lm), just swap the config.

Event-driven benchmark of speculative prefill during user reading time in multi-turn LLM conversations, measuring net TTFT benefit, contention penalty, and the conditions under which speculation becomes net-negative.