InferLoad
A client-side performance-testing and capacity-planning laboratory for LLM inference endpoints (vLLM, Ollama, OpenAI-compatible) designed around the split mechanics of generative prefill and decode.
Project overview
InferLoad is an empirical performance-testing and capacity-analysis tool designed specifically for LLM inference endpoints (vLLM, Ollama, and OpenAI-compatible APIs). Sitting outside the model server, it generates controlled workloads, decodes streaming responses in real time, and produces comprehensive statistical and SLO reports.
The problem: Why standard HTTP load testing fails on LLMs
Standard load testing tools such as wrk, k6, or Locust treat endpoints as static request-response systems. They measure end-to-end round-trip latency, which completely obscures the fundamental execution mechanics of generative inference:
- Prefill vs. Decode Split: The compute-bound prompt ingestion phase (prefill) operates under entirely different hardware constraints than the memory-bandwidth-bound auto-regressive generation phase (decode).
- Time to First Token (TTFT): In interactive user applications, TTFT determines perceived responsiveness, while round-trip time depends heavily on requested generation length.
- KV-Cache Starvation: Under concurrent load, continuous batching engines run out of GPU KV-cache slots, causing request queueing, chunk preemption, and extreme p99 tail latency spikes that average metrics conceal.
Architecture & execution flow
Define target endpoint, model name, prompt distribution, token budgets, and concurrency or arrival rate models.
Supports closed-loop fixed concurrency (client sends next request immediately upon receipt) and open-loop Poisson arrival streams.
Decodes streaming chunks from Ollama, vLLM, or OpenAI-compatible endpoints with high-resolution monotonic timestamps.
Aggregates latency distributions, detects outliers, and evaluates service level objectives (SLO) against predefined target budgets.
What InferLoad measures
Time to First Token (TTFT)
Elapsed duration from request dispatch to arrival and decoding of the first streamed token chunk, isolating prefill computation.
First Byte Latency
Time from request dispatch to receipt of initial HTTP response headers, isolating network RTT from model computation.
Inter-Token Latency (ITL)
Time deltas between consecutive token arrivals, capturing token generation fluidity and memory-bandwidth stalls.
Generation Throughput
Tokens generated per second per individual request, alongside aggregate token production across the entire server.
Statistical Distributions
p50, p95, and p99 percentiles computed with linear interpolation on unrounded measurements to expose tail latencies.
SLO Compliance & Saturation
Inflection point analysis detecting the exact concurrency threshold where queueing triggers KV-cache preemption.
Saturation analysis & capacity planning
InferLoad helps infrastructure engineers identify the exact operating ceiling of an LLM cluster before latency degrades. By scaling concurrency in stepping intervals, InferLoad plots the inflection curve where GPU memory reaches capacity and batch prefill queue times start scaling linearly.
Operational Guarantee
InferLoad does not invent synthetic token counts or drop failed stream chunks. Every metric is computed directly from client-received chunk deltas, preserving statistical truth for capacity planning decisions.Technical architecture & tools
- Core Runtime: Python 3.12, AsyncIO, HTTPX streaming client
- Web Console: FastAPI backend, Render cloud deployment
- Target Compatibility: Ollama, vLLM, DeepSeek, OpenAI-compatible APIs
- Distribution Engine: Exact unrounded percentiles (p50, p95, p99), histogram interpolation
- Configuration: Strict YAML schemas for repeatable experiment definitions