ML systems
Measure the queue, not just the tokens.
I served an open-weight language model on a 6 GB laptop GPU and measured the tradeoffs between precision, throughput, latency and answer quality.
My role
I built the load harness, evaluation runner and observability stack, and recorded the experiments.
Where it stands
Reproducible personal benchmark on one laptop. The quantized weights come from the model publisher; I did not quantize them myself.
What changed.
A repeatable comparison of FP16, AWQ and GPTQ serving, with raw results and hardware scope attached.
The setup
I compared three precision variants of Qwen2.5-1.5B-Instruct using synthetic load prompts and a separate quality set. Model revisions were pinned, and the repository records the tested hardware and measurement conditions.
Two different load questions
Closed-loop workers show how throughput changes with concurrency. Open-loop arrivals expose what happens when requests arrive faster than the service can handle them. A server can produce more tokens and still make a person wait longer for the first one.
The tradeoff
The lower-precision arms used less weight memory and delivered more output tokens per second in this setup. Quality changed too, so I report it alongside latency instead of describing compression as a free improvement.
The stack
- Python
- vLLM
- Docker
- Prometheus
- Grafana