Speculative-Decoding

Flash-Next is Qwen’s sparse-attention flagship and its official FP8 recipe assumes datacenter GB300. Adapting it to 2x96 GB workstation Blackwell took one legit trick (offloading the 51B-entry PLE n-gram table to pinned host RAM, freeing ~25 GiB per GPU for KV), one config archaeology exercise (YaRN via –hf-overrides because the build has no –rope-scaling flag), and discipline about what not to retry: fp8 KV hard-requires BF16 on QSA and fails differently on vLLM and SGLang, and 786K+ contexts OOM during compile no matter the KV math. Final stack: 512K context, 1.42M-token pool, 146.5 tok/s decode with MTP ns=3, 9-11.2K tok/s prefill, vision and tools validated. The SGLang NVFP4 day-0 lane runs too (after three SM120 gates), but serves one request at 262K slower than the vLLM lane serves two at 512K, so vLLM keeps the port.

My 27B coding brain was hard-capped at ~107K context by fp8 KV and fat weights. A stranger’s gist promised a 451K-token KV pool on the same GPU via NVFP4 KV cache and a patched vLLM. I ported it, pinned it, validated it with planted-password recall tests at 235K tokens — and then watched it crash spectacularly the first time a real agent used it. The final stack: full 262,144-token context, ~125 tok/s decode, vision, MTP speculative decoding, all on one card.