FairLane
An experimental token-fair scheduling and admission-control benchmark comparing FIFO, Round-Robin, and Virtual Token Counter (VTC) policies under continuous batching constraints for multi-tenant LLM serving.
Project overview
FairLane is an empirical systems research project that evaluates token-aware scheduling and admission control policies at the queue layer for multi-tenant LLM inference servers. By modeling continuous batching dynamics, FairLane compares how different scheduling policies affect fairness, queue wait times, and tail latency when co-located tenants have asymmetric workload characteristics.
The problem: Monopolization under continuous batching
In multi-tenant LLM serving environments, clients share GPU compute and KV-cache memory. When standard FIFO or naive request-count scheduling is used, a single “flooder” tenant submitting high-token requests (e.g., long-context RAG prompts of 1,024 to 4,000 tokens) will saturate available batch slots.
A co-located “victim” tenant submitting lightweight, latency-sensitive requests (e.g., 128 prompt / 64 decode tokens) is forced to wait behind the flooder's massive queue. In measured experiments, this results in the victim experiencing an astounding 160x slowdown and mean queue wait times surpassing 52 seconds.
Implemented scheduling policies
Processes requests strictly in arrival order. Fails catastrophically under heterogeneous workloads when high-volume tenants saturate the queue.
Alternates turns across active tenants by request count. Protects against simple request volume flooders but remains vulnerable to token-length disparities.
Tracks cumulative token allocations per tenant and dispatches from the tenant with the lowest accumulated virtual token count, enforcing genuine resource fairness.
Measured experimental benchmark results (N = 30 Seeds)
Experimental setup: Flooder tenant (240 requests, 1,024 prompt / 256 decode, 83.3 req/s Poisson arrival) co-located with a Victim tenant (60 requests, 128 prompt / 64 decode, 20 req/s Poisson). Server KV capacity: 4,096 tokens, max batch size: 16. Evaluated with 95% non-parametric bootstrap confidence intervals.
| Policy | Victim Slowdown (95% CI) | Victim TTFT Mean | Queue Wait Mean |
|---|---|---|---|
| FIFO (break mode) | 160.37x [151.46, 169.23]* | 52,461 ms | 52,460.7 ms |
| FIFO (skip mode) | 159.20x [150.31, 168.10]* | 52,148 ms | 52,147.4 ms |
| Round-Robin (break) | 2.19x [1.96, 2.45] | 398 ms | 397.4 ms |
| Round-Robin (skip) | 1.83x [1.63, 2.01] | 311 ms | 310.7 ms |
| Virtual Token Counter (VTC default) | 2.00x [1.82, 2.21] | 336 ms | 336.1 ms |
*Note on FIFO Slowdown: The measured 160.37x ratio is horizon-dependent because the flooder arrival rate exceeds the single-node service rate, compounding queue backlog over time.
Scientific transparency & falsification
FairLane is presented strictly as a research project with empirical methodology. Key methodology disclosures include:
Pre-Registration & Falsification Notes
- Initial Hypothesis Falsification: The original finite-trace hypothesis was falsified due to the tokens conservation invariant (total served tokens must identically equal submitted tokens over finite drain horizons).
- Pilot-Informed Steady-State: Steady-state Poisson evaluations were refined based on pilot simulation runs before freezing experiment configurations.
- Scope Limitation: Evaluates single-node queue scheduling algorithms rather than distributed cluster orchestration across multiple GPU nodes.
Technical architecture & tools
- Simulation Kernel: Python & Go discrete-event simulation harness
- Batching Model: Continuous batching with dynamic KV-cache slot allocation
- Statistical Analysis: Non-parametric bootstrap (1,000 resamples, 95% CI), Jain's Fairness Index
- Workload Distributions: Synthetic Poisson arrival with heterogeneous prompt/decode token distributions