Have 8 GB of VRAM and 32 GB of RAM? You can run Qwen 3.8 Flash Next at up to 900 t/s PP and 30 t/s TG!
First, this isn’t clickbait - these are real results. I managed to run Qwen 3.8 Flash Next Q2 and got better results in benchmarks and real-world tests than I did with any Qwen 3.6 35B A3B fine-tune or Qwen 3.8 27B model I tested. It is also faster than my current Qwen 3.6 35B A3B setup. Here are the details. About a week ago, I learned about Strata and what it can do from this subreddit. Its ability to distribute workloads across VRAM, RAM, and SSD storage, run a 125B-parameter model, and deliver incredible prompt-processing and token-generation speeds sounded almost too good to be true - but only if you had 12 GB of VRAM. At first, I thought I probably wouldn’t be able to achieve similar results on my machine, but I decided to try it myself. The results surprised me, so I wanted to share them with you and provide some guidance for getting the model running on your own hardware. First, here are my system specs: GPU: RTX 5060 Laptop, 8 GB VRAM CPU: AMD Ryzen 9 8945HX, up to 5.4 GHz, 16 cores / 32 threads RAM: 32 GB (16 + 16 GB) DDR5-5200 SSD: 1 TB, 5,200 MiB/s read, 4,200 MiB/s write PCIe: Gen 4 x8 OS: Linux You can find my Strata configurations here: Chat (Pastebin) | Chat (GDrive) | Agentic (Pastebin) | Agentic (GDrive) . Remember to update the paths to your model and expert files. Use the Chat configuration for a single session with the model, such as a regular chat. Use Agentic when working with multiple sessions and delegation, for example in agent-based systems. I’ll explain the difference in more detail below. Key parameters to tune --mmap-experts - places the experts that do not fit in memory on the SSD. --kv-resident 20480 - distributes the KV cache between VRAM and RAM. A lower value leaves more VRAM available. 20480 is the minimum value. --vram-reserve-mib 500 - leaves some VRAM available for other tasks. In my setup, 500 is the lowest workable value, but I recommend using 900 . --pcie-frac 0.35 - controls the fraction of per-layer VRAM misses that the GPU reads over PCIe. You can tune this parameter using the calibration command: ./setup.sh --calibrate . I found a value that works best for my setup. You can also calibrate it yourself if you are not satisfied with your prompt-processing speed. STRATA_RESIDENT_HEADROOM_GIB=4 - reserves RAM, in GiB, for other tasks. Set this higher if you are using Windows. The biggest performance gains in my setup came from adjusting --vram-reserve-mib and STRATA_RESIDENT_HEADROOM_GIB . I used the Gyro S model because I consider it the best quant for my hardware, and it delivered the strongest results among the models I tested. I had previously tried GSQ-RCO Coder, GSQ-RCO IQ2_XS, and other quants, but they either got stuck in repetitive loops or struggled with languages other than English. Gyro S offers a good balance between quality and speed. I’m not using MTP. It can improve TG, but it also consumes VRAM and reduces PP, which is likely to be a worse trade-off for this setup. You can enable it if you have 10 GB of VRAM or more. I run Vision on the CPU. Image processing takes around 1-2 seconds per image, but this saves a substantial amount of VRAM. Strata runs Vision in a separate process, so it can be moved to swap. This does not significantly affect model performance, while freeing up more RAM. If you use local models for parallel agent calls, there is one important limitation: with a context size of 131K tokens or more, you cannot safely use --parallel because it consumes a large amount of both VRAM and RAM. This can cause a severe drop in speed or an out-of-memory (OOM) error. A safer alternative is --conversation-cache . It keeps a limited number of sessions parked so they can be resumed sequentially. This does not provide true parallelism, but it avoids having to re-read long contexts when switching between sessions. Here are the relevant parameters: --conversation-cache-mib 4096 - the size of the parked-session cache. A larger value reduces the chance that a session’s cache will be evicted and its context will need to be read again. --conversation-cache-slots 2 - the number of parked-session slots. Two slots allow two sessions to be stored at once. --conversation-cache-min-tokens 8000 - do not cache a session if its token count is at or below this threshold. The right value depends on your PP speed and how long you are willing to wait for that many tokens to be processed. --conversation-cache-min-free-mib 1024 - the minimum amount of free RAM to preserve while caching. If caching a new entry would leave around 1,024 MiB of free RAM, the old cache will be cleared to reduce the risk of OOM. Be careful when tuning these parameters yourself: incorrect settings can affect system stability or push the model into heavy swapping. Do not exceed the total memory available on your system. All tests were run on Linux, which gives me more flexibility with RAM allocation. If you use Windows, try allocating less memory to the model. This will reduce throughput significantly, but it also lowers the risk of running out of memory. On Linux, I created a dedicated 32 GB swap partition for other applications. I also pinned the server process in memory so it could not be swapped out, using a systemd service with this setting: [Service] MemorySwapMax=0 This helps ensure that the server’s memory does not get swapped out, preserving consistent read and write performance even with a large context and parked-session cache. With the Chat configuration, I get: 262K context: PP 800-970 t/s and TG 30-40 t/s 131K context: PP 930-1,040 t/s and TG 30-45 t/s With the Agentic configuration, I get: 262K context: PP 520-740 t/s and TG 30-40 t/s 131K context: PP 670-890 t/s and TG 30-45 t/s In my view, these are excellent results-even compared with Qwen 3.6 35B A3B running in llama.cpp. With Qwen 3.6 35B A3B, using a VRAM-constrained expert layout and MTP/Vision disabled, I get PP 630-680 t/s and TG 27-35 t/s. Do you think Qwen 3.6 35B A3B is still worth using for agentic tasks today? Strata does not affect response quality; it helps distribute the model across available resources and keeps frequently used experts resident. That is only one of its major optimizations, alongside several others. The model’s quantization is what affects output quality. If you have more than 8 GB of VRAM, you could try Gyro M/L or other quants. Use my configurations as starting points and adapt them to your hardware and workload. Your component combination and use case may be different from mine. Real-world quality tests Now that I’ve covered how to fit the model onto this hardware, let’s look at the quality of its outputs. I’ve included several representative tests. I did not run an extensive benchmark suite or standardized evaluations; instead, I spent the past week testing the model on practical, everyday, and demanding tasks. I’ll leave broader benchmark testing to the community. I’ll compare it with my current high-performing setup, Cyber-Tiel 35B A3B , based on Ornith 1.5 Heretic. Tiel 35B A3B often gave me repetitive, looping responses, so I switched to Cyber-Tiel. The Flint model was released recently, but I won’t include it in this comparison. It falls behind Qwen 3.6 in my view; I may compare the two later and add those tests to this post. Test 1: Pagoda This test was run in Pi Agent with a minimal set of tools: LSP and Playwright MCP. For Qwen Flash Next Gyro S, I set reasoning_effort to high . https://reddit.com/link/1xft2ae/video/vycdc57ifsxh1/player Test 2: Flappy Bird clone The conditions were identical to those in the previous test. https://reddit.com/link/1xft2ae/video/97i98026fsxh1/player Test 3: Pelican For this test, I compared not only two different models but also different reasoning_effort settings for Qwen Flash Next Gyro S. The agent environment was the same: Pi Agent. https://preview.redd.it/ifhdl3y7fsxh1.png?width=2067&format=png&auto=webp&s=e9eb5d9ad5046a2a58d339406ab83f8f4429f1c4 I ran many more tests, but I cannot share them for confidentiality reasons. What I can say is that I’ve replaced all my coding agents with this model. That change has accelerated my development and improved code quality by a factor of five. As you can see from these examples, the results are better than those of any other model I’ve tested that fits within 8 GB of VRAM. Qwen 3.8 Flash Next is impressive, and the Gyro S quant makes it possible to run the model even on hardware like mine for local coding and agentic tasks. My goal with this post is to show people who still use Qwen 3.6 35B A3B or its fine-tunes locally that there is now a model that, in my experience, outperforms them by a wide margin. I hope the examples make that case. I’m always open to a different perspective, so feel free to share criticism in the comments. If you have suggestions or improvements for the configuration, please share them as well-other users may benefit from them too. That’s all from me. Use good tools, and happy coding! --- EDIT 1 : I tested the Gyro M on my device. You can see the results in this comment . EDIT 2: I published the Pagoda Benchmark results for the Gyro M here .