News
RTX 3060 12 GB under sustained load: 60–68 tok/s to the last token
We measured speed inside every long answer, token by token, instead of averaging: 60–68 tok/s with reasoning across an 8192-token answer at 16K context, 79–96 tok/s without reasoning, and only about 8% lost from a 1K to a 30K prompt. All 145 raw files are available to download.
On 30 October 2026 we ran a long series of measurements on B2003, our experiment computer with an RTX 3060 12 GB. The result: with reasoning on (the default), Qwen3.8-Flash-Next IQ3_S on Strata holds 60–68 tok/s from the first token to the 8192nd at a 16K prompt, with no drops. With reasoning off it runs at 79–96 tok/s. A nearly full context costs only about 8%: from a 1K to a 30K prompt the median goes from 88 to 84 tok/s.
Our earlier article reported up to 68 tok/s decoding on this card. The two questions we are asked most on X are whether that speed holds through a long answer or is only a warm-up peak, and whether performance holds in the second half of a long context. To answer them, we timed every streamed token instead of averaging over the whole answer.
We checked speed in intervals of the same answer, context from 1K to 30K tokens, a cold and a warm prompt cache, fast and precise expert routing, three content types (JSON, C code, Russian technical prose), GPU power, clocks and temperature, and whether the output itself stayed correct.
Setup
- Machine B2003: ASUS Pro Q870I-C, Intel Core Ultra 7 265, 2×24 GiB DDR5-5600.
- GPU: RTX 3060 12 GB (GA106), driver 615.71.10, GPU unlock on, power limit 170 W.
- Storage: Lexar NM1090 PRO 2 TB NVMe (PCIe 5.0 ×4).
- System: Ubuntu 24.04.5, kernel 7.0.0-38.
- Engine: Strata 0.1.52 (build f-027-rel).
- Model: Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S (78 GiB, 2 shards).
--max-context 32768 --prefill 4096 --pool-workers 12 --spec 4 --spec-min-p 0.5
--kv int8 --kv-resident 32768 --conversation-cache-mib 1536
--resident-experts --resident-budget-gib 24
STRATA_UNBUFFERED_LOAD=1 STRATA_RESIDENT_HEADROOM_GIB=10Methodology
- The server does not expose /tokenize, so tokens were counted from the timestamps of streamed SSE chunks (1 chunk ≈ 1 token; check: 5478 chunks against 5479 tokens in usage.completion_tokens) and anchored to the exact final token count. Words and lines were not used.
- Prompts were calibrated with probes using max_tokens: 1 (20.2 tokens per filler sentence, 145 tokens of fixed overhead); the actual size was checked against usage.prompt_tokens.
- Answer length was 32768 minus the prompt minus a margin, up to 8192 tokens; at 30K this is 2048 tokens.
- Each key scenario was repeated at least 3 times; we report the median, minimum and maximum. Decode noise is ±20% and prefill noise ±5%; smaller differences were not counted as results.
- Hardware telemetry was recorded every 3 seconds (clocks, power, temperature, utilisation, VRAM, free RAM, swap, NVMe read counters). Prefill, MTP draft acceptance, expert-cache hits and file-tier reads came from the engine log.
- Two expert routing modes: fast (default routing, prior 1.5) and precise (STRATA_ROUTE_PRIOR=0). Modes were switched by restarting the engine.
- 4 rows in the precise-mode file were produced by the fast engine because the block script was rewritten during the run. They are excluded from the tables and kept in the archive as they are.
- In total: 52 measured runs in 13 JSONL files (14 in fast mode, 13 in precise mode and 25 variants: reasoning, warm cache, code, prose, context limit, calibration), plus the 4 excluded rows.
Results: speed inside the answer
Values in tok/s, median (min–max). Context is the prompt size in tokens; the answer is up to 8192 tokens (2048 at 30K).
Fast mode, reasoning off
| Context | Runs | 0–0.5K | 0.5K–1K | 1K–2K | 2K–4K | 4K–6K | 6K–8K | Tail | Whole answer |
|---|---|---|---|---|---|---|---|---|---|
| 983–1516 | 3 | 86 (56–86) | 87 (87–88) | 86 (85–88) | 88 (88–90) | — | — | 92 | 88 (55–89) |
| 7865 | 2 | 84 (81–87) | 96 (93–99) | 94 | 95 (94–96) | — | — | 94 | 93 (93–94) |
| 15816 | 3 | 79 (77–88) | 86 (86–96) | 90 (90–96) | 92 (90–98) | 93 (93–96) | 93 (92–96) | 95 | 91 (90–96) |
| 23917 | 3 | 81 (77–85) | 86 (83–91) | 83 (81–92) | 90 (80–90) | 96 | — | 83 | 86 (80–92) |
| 30153 | 3 | 78 (77–78) | 86 (86–87) | 86 (85–88) | — | — | — | 86 | 84 (84–85) |
Fast mode, reasoning on (default behaviour)
| Context | Runs | 0–0.5K | 0.5K–1K | 1K–2K | 2K–4K | 4K–6K | 6K–8K | Tail | Whole answer |
|---|---|---|---|---|---|---|---|---|---|
| 15854 | 1 | 61 | 68 | 61 | 59 | 58 | 60 | 85 | 60 |
| 30191 | 1 | 66 | 56 | 65 | — | — | — | 96 | 63 |
Precise mode, reasoning off
| Context | Runs | 0–0.5K | 0.5K–1K | 1K–2K | 2K–4K | 4K–6K | 6K–8K | Tail | Whole answer |
|---|---|---|---|---|---|---|---|---|---|
| 1011 | 2 | 27 (26–29) | 28 (26–29) | 27 (26–28) | 25 (24–26) | 25 (24–25) | 23 | 26 | 25 (25–26) |
| 7893 | 2 | 29 (29–30) | 30 (29–31) | 28 (24–31) | 27 (22–31) | 26 (21–31) | 27 | 24 | 26 (23–30) |
| 15814 | 3 | 29 (26–32) | 27 (25–32) | 26 (22–28) | 23 (20–24) | 23 (20–23) | 22 | 21 | 24 (21–24) |
| 23915 | 3 | 31 (30–32) | 32 (31–34) | 29 (26–31) | 25 (25–28) | 24 | — | 24 | 26 (26–29) |
| 30151 | 3 | 32 (28–32) | 32 (30–33) | 29 (29–30) | — | — | — | 29 | 30 |
In fast mode the speed is flat or rises towards the end of the answer (16K: 79 → 93, tail 95). A slowdown towards the end appears only in precise mode: 29 → 22 over 8000 tokens (−24%). Precise mode with reasoning on at 16K gives 25 tok/s for the whole answer. Fast routing is 3.8× faster than precise routing.
Content type (fast mode, 16K context)
| Task | Answer, tokens | Whole answer, tok/s | MTP drafts accepted |
|---|---|---|---|
| JSON array of records | 8192 | 91 (90–96) | 92% (90–98%) |
| C code | 7332 | 59 | 74% |
| Russian technical prose | 4730 | 40 (40–43) | 38% (1327/3485) |
The differences are several times larger than decode noise: predictable structure speeds generation up 2–2.3×, free prose slows it down. This is a property of the content, not a slowdown over time.
Prefill, time to first token and prompt cache
| Case | Prompt, tokens | Cold prefill, tok/s | Cold TTFT | Warm TTFT | Reused tokens |
|---|---|---|---|---|---|
| 1K | 983–1516 | 293 | 3.4–3.8 s | — | 0 |
| 8K | 7865 | 865 | 9.1–9.2 s | — | 0 |
| 16K | 15816 | 908 | 17.3–17.6 s | 0.2 s | 15 803 |
| 24K | 23917 | 914 | 25.4–26.3 s | — | 0 |
| 30K | 30153 | 892 | 33.6–34.5 s | 0.4 s | 30 141 |
A warm prompt cache removes almost all prefill (17.4 s → 0.2 s at 16K, 33.6 s → 0.4 s at 30K) and does not change decode speed. Expert-cache hits were 98.7–98.8% in fast mode; NVMe reads were 1.1–3.3 GB/s while experts were loading and almost zero during steady JSON decoding.
At a 30,151-token prompt with a 32,768 limit, the engine trims the answer budget itself and ends with stop or length, without an error. Only a prompt that does not fit on its own returns error 400.
Hardware during the runs
| Block | Power | Temperature | SM clock | VRAM | Swap | NVMe read |
|---|---|---|---|---|---|---|
| Fast mode | 162–169 W (max 170.2) | 74–76 °C | 1815–1875 MHz | 11 409–11 423 MiB | 0 | 0.2–4.7 GB/s (6.6) |
| Precise mode | 88–105 W (max 167) | 62–71 °C | 1965–1972 MHz | 11 425 MiB | 0 | 1.6–2.3 GB/s (6.1) |
Ranges across the measured blocks; peak values in brackets. Fast mode is limited by GPU power: at the 170 W cap the clocks drop from 2160 MHz to 1815–1875 MHz at 75 °C, with no thermal throttling, 11.4 of 12.3 GiB VRAM in use and no swap. In precise mode the GPU draws about half the power at maximum clocks: it is waiting for experts rather than working, which is why speed drops.
Output quality
- The JSON format does not break: in all 12 runs cut off by the limit every record stayed valid; only the array itself was left unclosed.
- An instruction from the very beginning of the prompt is still followed at 30K (the tag rule held in the tail, 50/50 and 22/22).
- A marker from the middle of the document is recalled at 16K and 24K but lost at 30K, in both modes.
- The main weakness of fast mode is repeating entities in long lists (26–28 unique codes out of about 190) and one loop of 122 duplicate lines at 30K. Precise mode at 16K kept 168/168 and 193/193 unique, but at 24K and 30K it can also collapse to 26–33 unique codes in a run.
- C code in fast mode produced 255 repeated lines out of 681, so code tasks in fast mode are not reliable.
Limits of this experiment
One machine and one build: the conclusions about the GPU power limit and expert supply apply to this card with a 170 W limit and 12 GiB of VRAM. Modes were switched by restarting the engine, so runs within one mode were consecutive and the order was not fully randomised. Precise mode with a warm cache was run once, and precise mode at 1K and 8K had 2 repeats instead of 3.
What this means for a Box customer
- An affordable 12 GB card keeps 60–68 tok/s through long answers, not only at the start.
- Long documents in context cost little speed, and a repeated context is answered almost instantly thanks to the prompt cache.
- The mode is chosen per task: fast for structured data, precise when long lists must stay unique; generated code needs review.
Raw data
All 145 files of this experiment are available to download: 13 JSONL result files, 102 answer and reasoning texts, hardware telemetry, engine logs, scripts, the system state, the original report in Russian and the post draft in English. Access keys, internal addresses, user names and internal paths were removed before publication.