2026-11-01

Betelgeuse: Running Qwen 3.8 Flash Next at 22 tok/s sustained (spiking to 48 tok/s) on an RTX 3060 12GB + 16GB DDR4 RAM 100% bit-exact!

About two months ago I posted here about trying to predict which MoE experts get used on the next token to speed up CPU/GPU offload on Qwen 3.6 35B. Quick confession on that older post before anything else: I ended up throwing that prototype away. To get the big numbers I was seeing back then, I was basically cheating with gate pruning (dropping cold experts when their router weights were low). It made the tok/s counter go brrr, but on an already low-bit quant it quietly degraded output quality and reasoning. I didn't want to ship something that lobotomizes the model just to look fast in a screenshot, so I scrapped it. When Qwen 3.8 Flash Next dropped, I wanted to see if we could actually run the 68GB GSQ-RCO-IQ2_XS build at real usable speeds on a normal budget gaming PC— without dropping a single expert . For context, here is the exact box I tested this on (I actually pulled my extra RAM sticks out specifically to test what a regular 16GB PC gets): GPU: RTX 3060 12GB RAM: 16GB DDR4, single-channel (~19 GB/s) OS: CachyOS / Arch Linux (tested inside a strict MemoryMax=12G cgroup) Storage: Consumer PCIe NVMe SSD (~2.1 GB/s) If you try to load a 68GB MoE on 12GB VRAM + 16GB RAM in stock llama.cpp , you already know what happens: mmap thrashes the OS page cache to death, you get ~1,500 major page faults per token, and generation crawls at 1.47 tok/s . Then I tested Strata . Strata is super cool and hits 40+ tok/s if you have 48GB–64GB of RAM to mlock 24GB+ of experts in system memory. But on a 16GB RAM setup, it refuses to load unless you force --low-ram mmap , where it drops down to 8.11 tok/s because it's bound by cold SSD reads. So over the last week I built Betelgeuse ( bel-cli / bel-server , a fork of llama.cpp , and I'm also opening a clean upstream PR for llama.cpp even if it probably won't get merged lol). What it actually does under the hood: Zero gate pruning / 100% bit-exact: Every single one of the 10 routed experts per layer across all 48 layers is computed at full precision. We verified 28,080 router records and 576 generated tokens byte-for-byte against golden stock output ( 0.000e+00 diff). Arrival-Slack VRAM Slot Pool (~5,000 slots in VRAM): All 49 dense layers + GDN + QSA + LM head stay in ~3.2 GB of VRAM. The remaining VRAM holds ~5,000 expert slots distributed non-uniformly (240 slots/layer on early layers 0–4 where there is 0ms of compute slack, 92 on middle layers, 60 on tail layers). In-Stream GPU Doorbell + 1-Split CUDA Graph Replay: Instead of ggml_backend_sched splitting the graph 48 times per token to ask the CPU which experts missed (which burns ~360ms/token in cudaStreamSynchronize ), a custom in-stream CUDA kernel resolves hits in ~2 µs on-device and only rings a mapped host doorbell when an expert is missing. Per-layer token embeddings ( per_layer_token_embd ) are also staged to CUDA0 so the entire decode pass runs as 1 single CUDA Graph replay . RAM Victim Cache + io_uring Pager: When an expert gets evicted from the VRAM pool, it moves into a dedicated host RAM victim cache instead of getting thrown away.VRAM and RAM stay 100% exclusive instead of duplicating the same hot experts in the OS page cache. Physical NVMe reads drop from ~71 MB/token down to 12.43 MB/token . No duplicate 35GB expert bloat on disk: Instead of keeping 35GB of dead expert tensors inside Shard 1 .gguf AND another 35GB in experts.bin , we stripped the expert payloads out of Shard 1 so the main GGUF is only 3.2 GB ( Dense.gguf ) + 35.3 GB ( experts.bin , 4KB-aligned single-read blobs) + **26.8 GB (** PLE.gguf n-gram shard) . Measured Results (RTX 3060 12GB + 16GB DDR4, MemoryMax=12G) Engine Cold Decode (Run 1) Warm Decode (Run 2) Peak Bursts Prompt Processing (PP) Physical SSD Reads Output Parity Stock llama.cpp (mmap) 1.47 tok/s 1.63 tok/s ~2 tok/s 14–20 tok/s 208–312 MiB/tok Baseline Strata ( --low-ram mmap ) 8.11 tok/s 8.11 tok/s ~9 tok/s 34.6 tok/s ~105 ms pool wait ~0.022 KL Betelgeuse ( bel-server ) 16.80 tok/s 22.03 tok/s Up to 48 tok/s 16.2 – 19.3 tok/s 12.43 MiB/tok 100% Byte-Exact On warm runs it holds 21.8–22.03 tok/s sustained across long coding prompts, and when generation hits stretches where all 10 routed experts are resident in the VRAM/victim pools, it spikes up to 48 tok/s . Links & What's Next GitHub (Betelgeuse Engine — bel-cli & bel-server ): [https://github.com/Betelgeuse-AI/Betelgeuse] HuggingFace (Pre-packed Alpha-Orionis-XS 3.2GB Dense + Experts bundle): https://huggingface.co/Betelgeuse-AI/Alpha-Orionis-XS Upstream llama.cpp PR: [https://github.com/ggml-org/llama.cpp/pull/31469] Even on the massive Q8_K_XL quant (~190GB total), bel-server runs at 8 tok/s cold and 12 tok/s warm (with bursts up to 21 tok/s) on the exact same 12GB 3060 + 16GB DDR4 box, where stock llama.cpp immediately OOMs or crawls at Zero experts are dropped or thresholded (0.000e+00 diff across 28,080 router records and 576 tokens vs unmodified stock). The speedup comes from eliminating the 48 per-layer CPU/GPU scheduler splits (KICK/WAIT doorbell around the shared expert), keeping the RAM victim cache strictly exclusive of VRAM (MOE_DROP_MMAP), and pre-staging cold misses via the upgraded oracle + 4KB-aligned io_uring pack. What I'm working on next: We're currently bringing this exact streaming + victim-cache architecture over to DeepSeek V4.1 Flash (1017) —early testing under development is already sitting around ~25 tok/s on the same hardware class. Happy to answer any questions or have people poke holes in the benchmarks! Leave us a Star :)

HackerTube: YouTube built from videos recommended by HN

YouTube-style UI over 6,454 videos sourced from 39,834 HN comments since 2007. https://hackertube.org/

Serbia just elected the world's brainiest parliament