256K context on an RTX 3050 8GB in a DDR3 workstation

On 11 November 2026, results were published from workstation P3001: an Intel Xeon E5-2697 v2 (12 cores / 24 threads, AVX1 only), 64GB of DDR3 ECC RAM installed (the system sees about 62GB) at 1333 MT/s, an NVIDIA RTX 3050 8GB, an NVMe SSD, and Ubuntu 24.04. The software was Strata 0.1.55 (an AVX-compatible build) running Qwen3.8-Flash-Next IQ3_S, a compressed model.

Earlier 16K configuration: 59 tok/s on real text

Earlier the same day, a 16K-context configuration with an 8-bit KV cache was reported on the same machine. KV cache is the model’s working memory for tokens already processed. The figures published with that post:

The same post reports 59 tok/s real-text decode, and 120–132 tok/s prefill. A later thread states that this 59 tok/s figure came from speculative decoding: Strata predicts several upcoming tokens and verifies them together. Real text produced high prediction acceptance; artificial benchmark filler did not (about 70% acceptance on synthetic filler, according to the 16K post). Prefill was limited by DDR3-1333 and AVX1; in that 16K configuration, 8GB of VRAM limited context to 16K.

256K configuration

The stated goal was a 256K-token context window — capacity for approximately 262,144 tokens in a single request. A first attempt simply raised the limit to 256K with an 8-bit KV cache. According to the post, Strata then tried to keep too much of that working memory in the GPU’s 8GB of VRAM, the GPU expert cache dropped to zero slots, and a cuBLAS error followed. The server was still responding, but GPU acceleration was not working properly.

The published solution: switch KV cache from int8 to q4_0 (4-bit compression), with memory per token falling from roughly 71KB to 29.5KB; set –kv-resident 20480, so only 20,480 KV positions stay resident in GPU memory and additional KV data is streamed from system RAM; keep 46.47GB of model experts in page-locked DDR3 RAM; preserve 498 expert-cache slots in GPU memory; and limit prompt caching to one active checkpoint. Final settings listed: –max-context 262144, –kv q4_0, –kv-resident 20480, –prompt-cache 1, –prompt-cache-every 0. The post states that Strata then started successfully and reported max_context: 262144, loaded: true.

Measured speed

The following figures are measurements from the inference engine itself, one run per row — preliminary, not averages from repeated benchmarks. Prefill is the processing of the input before an answer is generated; decode is token-by-token generation; TTFT is the wait until the first answer token.

The post notes that raising the input from roughly 6K to 160K tokens only reduced prefill from 188 to 173 tok/s, while decode stayed around 23–26 tok/s. A repeated 24K-token run with cached context dropped TTFT from 128.3 seconds to 0.49 seconds. A 160K-token cold input took about 15.5 minutes before the first answer token — attributed to the old Xeon and DDR3 memory bottleneck. An agent report of a 200K-token run is mentioned, but the full numerical row for that run is not in the table.

An important distinction in the thread: 256K is the configured and reported context limit. The detailed measurements shown reach 160K, not the full 256K. The full 262K window had not been benchmarked at the time of the posts. A reply estimates about 26 minutes of cold prefill for the full 262K, and states that this still needs to be measured. 59 tok/s has not been established for 256K-context workloads. Prefill in the new configuration was higher than in the previous 16K setup; the thread treats this as an observed difference between configurations, not proof of a single cause, because the KV format and memory layout also changed.

Limits and an energy estimate

Limits listed in the thread: GPU memory almost full, about 7.5–7.7GB used out of 8GB, with some runs warning that only 193–331MB remained free; system RAM around 53GB out of 62GB, with about 46.5GB occupied by model experts locked in memory; only one warm prompt-cache checkpoint, so switching between very large conversations may require processing inputs again (a 200K-token checkpoint alone could consume roughly 4.7GB); and long cold prompts are slow.

A rough power estimate for the 160K prefill is given as 300W × 15.5 minutes ≈ 0.078 kWh. At €0.25/kWh, that is about €0.02 of electricity. The post marks this as an estimate, not a measured energy benchmark.

The thread’s practical conclusion is that inexpensive local AI is not meant to compete with cloud models on long, demanding sessions, where cloud services are often faster and more economical. The main advantage of such old, inexpensive computers is privacy: processing sensitive documents locally. For many such tasks, a small context window is enough. The published achievement is that an RTX 3050 8GB can run a 256K-configured local model without moving all of its working memory into VRAM. Further measurements — including the full context range, repeatability and task quality — are still needed.

Benchmark results and logs are published in a public repository.

Project repository on GitHub · Post on X · 16K results · All news