Legacy Benchmarks (Python FastAPI)
These benchmark results were collected using the original Python FastAPI implementation (now on the
legacy-fastapibranch). They are preserved for reference but may not reflect current performance of the Go/Gin rewrite.
Here we collect results running on latest supported inference engines.
The benchmark tool used is the Hugging Face inference benchmarker. The dataset used for benchmark captures a wide variety of common prompts with various lengths and complexity.
The collected performance metrics are:
- Queries per second (QPS)
- Inter-token latency (ITL)
- Time to first token (TTFT)
- End to end (E2E) latency
- Throughput measured in tokens/s for all queries running in parallel
The inference engines are launched using the scripts provided in src/core/scripts.
The benchmark runs on a set of fixed rates, increasing by 50% for the next rate amount.
vLLM on LUMI
Reference benchmark without network and services overhead. The benchmarker software and vLLM both run on the same compute node.
HPC: LUMI supercomputer, queue: dev-g, gpu count: 1
GPU: AMD MI250x
Container: laifs-lumi-multi-20250728_103251.sif (custom LUMI image), running vLLM v0.10 with V0 engine
Model: Qwen/Qwen2.5-Coder-7B-Instruct
Run command:
./inference-benchmarker --tokenizer-name "Qwen/Qwen2.5-Coder-7B-Instruct" \
--max-vus 800 \
--duration 120s \
--url "http://localhost:8111" \
--warmup 60s \
--benchmark-kind rate --rates 0.5 --rates 0.75 --rates 1.12 --rates 1.68 --rates 2.53 --rates 3.79 --rates 5.69 --rates 8.54 --rates 12.81 \
--prompt-options "num_tokens=200,max_tokens=220,min_tokens=180,variance=10" \
--decode-options "num_tokens=200,max_tokens=220,min_tokens=180,variance=10"
| QPS | ITL (P90) (ms) | TTFT (P90) (ms) | E2E (P90) (ms) | Throughput (tokens/s) |
|---|---|---|---|---|
| 0.50 | 15.96 | 68.36 | 3334.70 | 91.00 |
| 0.75 | 17.36 | 65.67 | 3607.30 | 139.14 |
| 1.12 | 19.43 | 67.68 | 4049.61 | 202.34 |
| 1.68 | 23.14 | 73.65 | 4958.47 | 311.66 |
| 2.53 | 25.40 | 73.26 | 5409.09 | 471.48 |
| 3.79 | 29.24 | 76.80 | 6224.82 | 704.91 |
| 5.69 | 38.00 | 81.98 | 7909.88 | 993.93 |
| 8.54 | 89.16 | 106.94 | 18221.90 | 1385.79 |
| 12.81 | 158.55 | 18786.22 | 49194.62 | 1340.24 |
vLLM + HEAppE on LUMI
This benchmark is running against live service at chat.exa4mind.eu. Note that these results are limited by communication with HEAppE virtual machine, as well as extra overhead on SSH tunnel communication. In comparison with vLLM on LUMI without HEAppE, there is an additional delay on Time-to-first-token. Another thing to note is that when approaching the server limit at ~8 requests per second, the inference engine starts to queue the incoming requests, which results in significant delay for TTFT.
HPC: LUMI supercomputer, gpu count: 1
HEAppE virtual machine: HEAppE 6.1.1 on poutavm.fi
GPU: AMD MI250x
Container: laifs-lumi-multi-20250728_103251.sif (custom LUMI AI image), running vLLM v0.10 with V0 engine
Model: Qwen/Qwen2.5-Coder-7B-Instruct
Run command:
./inference-benchmarker --tokenizer-name "Qwen/Qwen2.5-Coder-7B-Instruct" \
--max-vus 800 --duration 120s --url https://chat.exa4mind.eu/lumi/ \
-a <api-key> \
--warmup 60s \
--benchmark-kind rate --rates 0.5 --rates 0.75 --rates 1.12 --rates 1.68 --rates 2.53 --rates 3.79 --rates 5.69 --rates 8.54 --rates 12.81 \
--prompt-options "num_tokens=200,max_tokens=220,min_tokens=180,variance=10" \
--decode-options "num_tokens=200,max_tokens=220,min_tokens=180,variance=10"
| QPS | ITL (P90) (ms) | TTFT (P90) (ms) | E2E (P90) (ms) | Throughput (tokens/s) |
|---|---|---|---|---|
| 0.50 | 11.84 | 1365.65 | 3758.64 | 87.21 |
| 0.75 | 13.23 | 1503.20 | 4166.15 | 129.12 |
| 1.12 | 15.87 | 1793.84 | 4981.15 | 178.01 |
| 1.68 | 17.69 | 1951.60 | 5707.67 | 233.18 |
| 2.53 | 19.85 | 2674.75 | 6588.92 | 401.44 |
| 3.79 | 23.62 | 3432.89 | 7588.16 | 628.73 |
| 5.69 | 28.42 | 3393.31 | 9036.41 | 880.58 |
| 8.54 | 37.00 | 31152.10 | 36995.62 | 1046.19 |
| 12.81 | 40.09 | 40909.62 | 47162.50 | 1084.86 |
vLLM on Karolina
This is a baremetal run of the benchmarker and vLLM running on the same HPC node. The benchmark values are average out of 5 independent runs.
Date: 2026-03-19
HPC: IT4I Karolina, queue: qgpu, gpu count: 1
GPU: NVIDIA A100 40GB
Container: vllm-openai-nightly-ef96fa3f.sif, running vLLM v0.14.0 with V1 engine
Model: Qwen/Qwen3-4B-Instruct-2507
Run command:
./inference-benchmarker --tokenizer-name "Qwen/Qwen3-4B-Instruct-2507" \
--max-vus 800 \
--duration 120s \
--url "http://localhost:8111" \
--warmup 60s \
--benchmark-kind rate --rates 2.53 --rates 3.79 --rates 5.69 --rates 8.54 --rates 12.81 --rates 19.22 \
--prompt-options "num_tokens=200,max_tokens=220,min_tokens=180,variance=10" \
--decode-options "num_tokens=200,max_tokens=220,min_tokens=180,variance=10"
| Benchmark | QPS (req/s) | E2E Latency (avg-sec) | TTFT (avg-ms) | ITL (avg-ms) | Throughput (tokens/sec) | Error Rate | Successful Requests |
|---|---|---|---|---|---|---|---|
| @2,53req/s | 2.514 | 1.584 | 31.406 | 8.584 | 454.656 | 0% | 300/300 |
| @3,79req/s | 3.756 | 1.63 | 32.188 | 8.766 | 685.222 | 0% | 450/450 |
| @5,69req/s | 5.634 | 1.666 | 33.434 | 9.114 | 1005.498 | 0% | 671/671 |
| @8,54req/s | 8.466 | 1.776 | 35.134 | 9.58 | 1532.756 | 0% | 1008/1008 |
| @12,81req/s | 12.67 | 1.9 | 37.744 | 10.214 | 2302.122 | 0% | 1516/1516 |
| @19,22req/s | 18.92 | 2.27 | 44.10 | 12.23 | 3418.70 | 0% | 2260/2260 |
vLLM + HEAppE on Karolina
Using HEAppE and the service itself, we can compare the results to the bare-metal run. There is an additional increase in the TTFT. The benchmark values are average out of 5 independent runs.
Model: Qwen/Qwen3-4B-Instruct-2507
Date: 2026-03-19
HPC: IT4I Karolina, queue: qgpu, gpu count: 1
GPU: NVIDIA A100 40GB
Container: vllm-openai-nightly-ef96fa3f.sif, running vLLM v0.14.0 with V1 engine
Run command:
./inference-benchmarker --tokenizer-name "Qwen/Qwen3-4B-Instruct-2507" \
--duration 120s \
--url "https://chat.exa4mind.eu/fta-26-13-dev/" \
-a <api-key> \
--warmup 60s \
--benchmark-kind rate --rates 2.53 --rates 3.79 --rates 5.69 --rates 8.54 --rates 12.81 --rates 19.22 \
--prompt-options "num_tokens=200,max_tokens=220,min_tokens=180,variance=10" \
--decode-options "num_tokens=200,max_tokens=220,min_tokens=180,variance=10"
| Benchmark | QPS (req/s) | E2E Latency (avg-sec) | TTFT (avg-ms) | ITL (avg-ms) | Throughput (tokens/sec) | Error Rate | Successful Requests |
|---|---|---|---|---|---|---|---|
| @2.53req/s | 2.50 | 1.83 | 720.13 | 5.68 | 465.04 | 0% | 298/298 |
| @3.79req/s | 3.79 | 1.72 | 624.24 | 5.57 | 694.58 | 0% | 448/448 |
| @5.69req/s | 5.62 | 1.83 | 737.47 | 5.63 | 991.39 | 0% | 672/672 |
| @8.54req/s | 8.47 | 1.87 | 685.79 | 6.06 | 1517.43 | 0% | 1009/1009 |
| @12.81req/s | 12.62 | 2.14 | 838.79 | 6.65 | 2284.74 | 0% | 1510/1510 |
| @19.22req/s | 18.90 | 2.89 | 1334.99 | 7.98 | 3355.15 | 0% | 2256/2256 |
Model: Qwen/Qwen3-Coder-30B-A3B-Instruct
Running a larger model that requires multi-GPU serve using tensor parallelism.
Date: 2026-04-24
HPC: IT4I Karolina, queue: qgpu, gpu count: 4
GPU: NVIDIA A100 40GB
Container: vllm-openai-nightly-ef96fa3f.sif, running vLLM v0.14.0 with V1 engine
Run command:
./inference-benchmarker --tokenizer-name "Qwen/Qwen3-Coder-30B-A3B-Instruct" \
--duration 120s \
--url "https://chat.exa4mind.eu/fta-26-13-dev/" \
-a <api-key> \
--warmup 60s \
--benchmark-kind rate --rates 2.53 --rates 3.79 --rates 5.69 --rates 8.54 --rates 12.81 --rates 19.22 \
--prompt-options "num_tokens=200,max_tokens=220,min_tokens=180,variance=10" \
--decode-options "num_tokens=200,max_tokens=220,min_tokens=180,variance=10"
| Benchmark | QPS (req/s) | E2E Latency (avg-sec) | TTFT (avg-ms) | ITL (avg-ms) | Throughput (tokens/sec) | Error Rate | Successful Requests |
|---|---|---|---|---|---|---|---|
| @2.53req/s | 2.49 | 2.17 | 782.77 | 7.11 | 468.48 | 0% | 297/297 |
| @3.79req/s | 3.75 | 2.18 | 825.86 | 7.02 | 627.26 | 0% | 444/444 |
| @5.69req/s | 5.56 | 2.63 | 959.85 | 8.55 | 1009.42 | 0% | 659/659 |
| @8.54req/s | 8.35 | 2.82 | 1083.43 | 8.95 | 1448.30 | 0% | 1009/1009 |
| @12.81req/s | 12.50 | 3.83 | 1722.48 | 10.83 | 2238.89 | 0% | 1490/1490 |
| @19.22req/s | 18.90 | 6.49 | 3513.13 | 15.34 | 3004.82 | 0% | 2040/2040 |
During the peak benchmark run (19.22req/s), there were 78 response streams concurrently.