Our experimental configuration: an ~83.6 GB model running locally on a budget RTX 3060

This is incredible!

Qwen3.8-Flash-Next IQ3_S (~83.6 GB) running locally on a budget RTX 3060 12 GB with just 48 GB of system RAM!

Up to 68 tokens/sec decoding, ~916 tokens/sec prompt processing, and a 30K-token prompt processed in just 34 seconds.

Powered by Strata by coldniko. No cloud inference. Just affordable consumer hardware!

Full benchmarks, hardware specs, optimizations, and trade-offs below.

You can jump straight to the key takeaways in the final section.

The hardware behind these results

No datacenter GPU, no expensive multi-GPU server, and no massive RAM configuration.

The entire experiment was performed on a fairly ordinary consumer/workstation PC.

CPU

• Intel Core Ultra 7 265 (Arrow Lake)
• 20 cores / 20 threads
• Up to 5.3 GHz
• 30 MB L3 cache
• AVX-VNNI / BF16 support; no AVX-512

Motherboard

• ASUS Pro Q870I-C
• Intel Q870 chipset, LGA1851
• Mini-ITX form factor

Memory

• 48 GB DDR5-5600 (2 × 24 GB Micron)
• Approximately 46 GiB usable

GPU

• NVIDIA GeForce RTX 3060 12 GB
• GA106, compute capability 8.6
• PCIe 4.0 ×16
• 170 W configured power limit

Storage

• Lexar NM1090 PRO 2 TB
• PCIe 5.0 ×4 NVMe

OS

• Ubuntu 24.04.5 LTS
• Linux kernel 7.0.0-38
• NVIDIA driver 615.71.10

Network

• Intel 1 GbE Ethernet

The crucial detail: the model is much larger than the available VRAM and RAM combined.

That’s exactly what makes this experiment interesting.

Model, engine, and memory architecture

Model

Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S

Two GGUF shards:

• 54.82 GB
• 28.80 GB
• Total: 83.62 GB (77.87 GiB)

Additional components

• Packed dense weights: ~1.5 GiB
• MTP speculative-decoding drafter
• BF16 vision adapter: ~866 MB
• CPU-based vision encoder (12 threads)

Inference engine

• Strata engine 0.1.52
• CUDA 13.0 backend
• CUDA architecture sm_86

Actual memory allocation

• GPU VRAM: 11.14 GiB used
• Resident experts in system RAM: 24 GiB
• Remaining expert data streamed from NVMe
• No swap activity during normal inference

Important configuration settings

--expert-cache auto
--prefill 4096
--spec 4
--spec-min-p 0.5
--mtp mtp/rt
--max-context 32768
--kv int8
--kv-grow
--kv-resident 32768
--vision
--vram-reserve-mib 960
--pool-workers 12
--conversation-cache-mib 1536
--prompt-cache-tail
--prompt-cache-spread
--resident-experts
--resident-budget-gib 24

This is not a conventional “load the whole model into GPU memory” setup.

Strata uses GPU memory, system RAM, expert caching, and NVMe storage together.

The important optimization is minimizing expensive expert transfers while keeping the GPU productive.

Real benchmark results

Same model, same hardware, two routing configurations.

Mode A: fast, residency-biased expert routing (prior = 1.5)

Cold time to first token:

• 1K: 3.7 s
• 4K: 4.9 s
• 8K: 9.7 s
• 16K: 17.3 s
• 30K: 33.9 s

For comparison, Mode B disables residency-biased routing (prior = 0):

500-token generation run:

• Fast: 62 tok/s, 7.1 s total
• Precise: 21 tok/s, 20.8 s total

Speed measurements were cross-checked against client-side streaming timings.

Main observation:

Disabling cache-biased routing can reduce decoding speed by approximately 2–2.4×.

But there’s an important reason not to leave aggressive routing enabled for every workload. More on that in the next section.

Speed vs model quality: the hidden trade-off

On this Strata engine version, GPUs with ≤13 GB VRAM automatically use residency-biased routing:

STRATA_ROUTE_PRIOR=1.5

This favors experts already available in fast memory instead of always selecting the experts that the original router would prefer.

The benefit is dramatic:

• Expert cache hit rate: 97.2% in fast mode
• Without the bias: 65.3%
• Storage reads during the compared workload: 5.5 GiB vs 46 GiB
• Decoding speed: approximately 2× faster

But this is NOT a free optimization.

Changing expert selection can change model behavior, increase divergence from the original model, and affect long reasoning or agent workflows.

According to the developer’s evaluation on 10 agent tasks:

• Routing prior 1.5: 4.5/10
• Routing prior 0: 5.6/10

The baseline results themselves varied between runs, so this is a directional result, not a definitive quality measurement.

My own six short agent checks passed in both modes.

However, a separate uniqueness stress run exposed repetition:

• Fast mode: 205 lines, 96 duplicates
• Precise mode: 209 lines, 79 duplicates

The conclusion is straightforward:

• Fast routing for chat, interactive use, and short tasks.
• Original routing for longer, quality-sensitive agent workflows.

And benchmark speed alone should never be confused with model quality.

Prompt caching changes everything

One of the most impressive results wasn’t decoding speed. It was the response time for repeated requests using the same prompt.

With a fully cached prompt:

That’s roughly 45–90× less time to the first token in these runs.

The prompt cache makes a huge difference when repeatedly querying the same long document or maintaining a conversation with a stable context.

The engine is also configured for conversation caching:

• 1,536 MiB cache budget
• Minimum 4,096 tokens for parking
• Minimum free memory threshold: 1,536 MiB

There’s a catch: these results are for identical prompts with a fully reusable cache.

They don’t mean arbitrary new 30K-token prompts will process in half a second.

But for interactive local AI, a warm cache is a game changer.

The SSD mystery: PCIe 5.0 isn’t the bottleneck

At first, the NVMe drive appeared suspiciously slow. We investigated the PCIe link, storage topology, and actual I/O behavior.

The drive: Lexar NM1090 PRO 2 TB, PCIe 5.0 ×4.

Verified PCIe connection:

• Current link: 32.0 GT/s ×4
• Maximum link: 32.0 GT/s ×4
• CPU root port: 32.0 GT/s ×4

So the drive is running at its full PCIe 5.0 link speed.

Direct uncached read benchmarks:

• Single stream, 1–4 MiB blocks: 2.9–3.4 GB/s
• Four parallel streams: 10.94 GB/s
• Four streams, 64 KiB blocks: 2.25 GB/s
• Four streams, 4 KiB blocks: 0.26 GB/s (~63K IOPS)

That 10.94 GB/s parallel result is particularly important: it exceeds the practical bandwidth of PCIe 4.0 ×4.

Why is single-stream throughput much lower?

Because PCIe bandwidth isn’t the same thing as application-level throughput.

Queue depth, I/O size, request parallelism, and latency all matter.

Strata’s expert-loading pattern is much closer to the lower-concurrency case than to a synthetic maximum-throughput benchmark.

Observed engine storage throughput:

• Fast routing: ~0.3 GB/s
• Original routing: ~2.4 GB/s

The difference comes from how many experts must be fetched, not from a PCIe downgrade.

One more useful detail: the RTX 3060 itself is a PCIe 4.0 device. Seeing its link drop to Gen1 while idle is normal power management behavior.

Always check which device you’re measuring before diagnosing a PCIe problem.

An important lesson: don’t let Linux cache an 83.6 GB model on a 48 GB machine

We hit a serious problem during initial startup.

The first configuration allowed buffered loading of the model files.

The model is ~83.6 GB, while the system has only 48 GB of physical RAM.

Under that workload, Linux entered severe memory reclaim pressure. The machine became almost unresponsive, including SSH connectivity, and required a forced restart.

The solution was to change the loading strategy and introduce memory safeguards.

Current settings:

• STRATA_UNBUFFERED_LOAD=1
• STRATA_RESIDENT_HEADROOM_GIB=10
• 24 GiB resident expert budget
• Memory availability checked before launch
• 12 GiB additional safety margin in the launch policy

After these changes:

• No swap activity during normal inference
• Approximately 34.6 GiB RAM in use at the observed point
• Approximately 12.7 GiB available
• Stable generation under load

The lesson:

Running a model larger than your physical memory is possible, but memory accounting is absolutely critical.

Page-cache pressure, pinned model data, resident experts, and KV cache must be considered together.

Don’t blindly copy maximum-performance flags from machines with much larger RAM configurations.

Power consumption, thermals, and inference cost

This is another pleasant surprise.

Measured component power:

Idle with model loaded:

• CPU: ~5.8 W
• GPU: 15–18 W

During generation:

• CPU: ~62.8 W
• GPU: 152–167 W
• Brief GPU peaks around 174 W

Estimated total system power at the wall (not measured using a wall meter):

• Idle: ~54 W
• Generation: ~280 W

Observed temperatures:

• GPU under load: 62–68°C
• GPU at idle: ~40°C
• NVMe: 38–47°C

No observed thermal throttling.

At an assumed electricity price of €0.25/kWh:

• 24 hours idle: ~€0.32
• 30 days continuously idle: ~€9.72
• One hour of active generation: ~€0.07

This makes it entirely realistic to operate a capable, privacy-preserving local AI endpoint at home.

Of course, those are estimated electricity costs, not total cost of ownership. Hardware depreciation, cooling, and maintenance are excluded.

Where the RTX 3060 setup stops making sense

This configuration is excellent for short interactive requests, but it has limits.

Most importantly:

• Maximum context: 32,768 tokens
• A prompt near 32K leaves insufficient space for generation
• The practical benchmark ceiling was ~30,165 input tokens
• Thinking/reasoning can consume hundreds of additional generated tokens
• Long-running agent sessions quickly become constrained

For comparison, I ran the same model on a more powerful local system with a 24 GB GPU.

Measured performance on that system:

• 4K prompt prefill: 1,339 tok/s
• 4K decode: 94 tok/s
• 16K prompt prefill: 2,956 tok/s
• 16K decode: 125 tok/s
• Maximum context: 262,144 tokens
• 15 worker threads

The RTX 3060 system delivers much lower prefill throughput and roughly half the decoding throughput in comparable cases, with one-eighth of the available context.

However, the 12 GB card is far more accessible and remains impressively useful.

My current approach:

• More powerful hardware for complex, long-running agents
• Affordable 12 GB GPUs for chat, short workflows, experiments, and backup inference

The point isn’t to pretend a cheap GPU replaces a high-end machine.

The point is how much useful local AI can already run on inexpensive hardware.

What could still be optimized?

The system is already performing well, but there are a few interesting avenues for further experiments.

Concurrent inference

The --slots 2 option could improve total throughput with two simultaneous conversations.

An estimated +10–12% aggregate throughput is possible for this workload, but it needs validation.

This does not mean a single conversation becomes 12% faster.

Larger prefill batches

Current: --prefill 4096

Candidate: --prefill auto:16384

Larger prefill windows might improve long-prompt processing, but available VRAM is already tight.

With roughly 11.14 GiB of the GPU’s 12 GiB occupied, memory behavior must be measured carefully.

More system RAM

Using --ple-io ram could bring more model data into system memory and reduce storage reads.

The developer’s documentation suggests:

• +16–31% prefill performance
• +3–11% decoding performance

But this configuration would require substantially more RAM, approximately 72+ GiB for the measured layout.

It is not practical on the current 48 GB configuration.

Speculative decoding

MTP speculative decoding is already enabled.

Observed acceptance: ~33–41 accepted tokens out of 50–60 drafted, or approximately 65–70%.

Longer context

The current 32K limit is an operational configuration limit, not evidence that the model itself supports only 32K.

Increasing it requires re-checking memory capacity, KV cache behavior, latency, and stability.

For this workload, reducing expensive expert reads and improving access patterns may matter more than raising the drive’s maximum sequential bandwidth.

As always: benchmark the real workload, not just hardware marketing numbers.

Final thoughts

Here’s what this experiment demonstrates.

An ~83.6 GB quantized MoE model can run locally on:

• RTX 3060 12 GB
• 48 GB DDR5 RAM
• Consumer Intel CPU
• PCIe 5.0 NVMe SSD
• Ubuntu + Strata

With measured results up to:

• 68 tok/s warm decoding
• 916 tok/s prompt processing
• 30K-token cold prompt in ~34 seconds
• Fully cached 30K prompt: ~0.44 s to first token
• No swap under normal operating conditions

But there are important caveats.

• Maximum configured context is 32K
• Fast expert routing changes model behavior
• The highest speed is not necessarily the highest quality
• Storage performance depends on access patterns
• Memory management requires careful tuning
• Long-running agents benefit from more powerful hardware

I am particularly interested in the practical economics of local AI.

Not just the maximum benchmark on a €10,000 system, but the useful performance attainable on consumer machines and affordable configurations starting around €1,000.

Privacy, independence, controlled operating costs, and real-world usefulness matter more than chasing impressive benchmark numbers alone.

This is still an evolving experiment, and I’ll keep sharing the measurements, failures, configuration changes, and improvements.

The future of local AI is already becoming surprisingly affordable.

Later additions

Oh, I forgot to mention that the GPU is running with NVIDIA Unlock, which was released just a few days ago, last week.

I’ve now asked my AI agent to run the same benchmarks without the unlock, so we’ll be able to compare the results in about 15–20 minutes.

In the next few days, though not today, I’ll install this GPU in a system costing less than $1,000 and see how it performs there.

Original thread on X · All news