Qwen has announced Qwen3.8-Flash-Next, a multimodal mixture-of-experts (MoE) model that serves as an early preview of the architecture planned for Qwen4. It follows Qwen3-Next, whose hybrid Gated DeltaNet (GDN) and Gated Attention design was later used across the Qwen3.5, Qwen3.6, Qwen3.7 and Qwen3.8 series.
Qwen3.8-Flash-Next
The new model introduces changes to attention, residual connections, embedding and optimization. It has a 125-billion-parameter main model, an additional 51 billion N-gram Embedding parameters, and 6 billion parameters activated per token. Compared with Qwen3.7-Plus, Qwen says training requires about one-ninth the compute, with lower training and inference costs and better results in coding and office tasks.
GDN and QSA attention
Qwen3.8-Flash-Next uses a Gated DeltaNet (GDN) + Qwen Sparse Attention (QSA) hybrid architecture. Three out of every four layers use GDN to compress historical information into a fixed-size state, while the remaining layer uses global Attention for retrieval across the full context.
QSA reduces the cost of global Attention by using a lightweight indexer to divide sequences into micro-blocks, estimate their importance and select relevant regions. Unlike token-level indexing approaches such as DSA, QSA performs sequence compression independently within each layer.
At a 1-million-token context, Qwen reports:
- 7.6× speedup for Prefill
- 4.9× speedup for Decode
- 8.6× higher Prefill throughput than Qwen3.7-Plus with a 90% Prefix Cache hit rate
Gated Residual and N-gram Embedding
Gated Residual (GR) expands the residual stream from one branch to four parallel branches. It combines the multi-branch design of Hyper-Connection with element-wise dynamic gating from GatedNorm. The gate controls information read from and written to each branch, while one branch was found to form a long-range pathway connecting the first Attention layer with most middle and later layers.
GR removes additional branch mixing used by Hyper-Connection, reducing memory-access overhead and sources of instability. The gate also suppresses activation outliers, and the residual state supports FP8 storage.
The model adds 51 billion N-gram Embedding parameters, inspired by Per-Layer Embedding in Gemma 3n and DeepSeek Engram. Instead of using only a single token for lookup, N-gram Embedding uses the current token and preceding tokens to represent local patterns.
The embedding parameters can be stored in host memory and asynchronously prefetched during model computation. The model uses a single N-gram Embedding layer near the beginning of the network.
Muon optimizer and other optimizations
Qwen3.8-Flash-Next is trained using the Muon optimizer, with changes to orthogonalization accuracy, parameter assignment between Muon and AdamW, and fused parameter splitting.
- Muon: Attention, GDN and MoE Expert main weights
- AdamW: Embeddings, MoE Router and low-rank GR parameters
Fused QKV, SwiGLU and GDN projections are split into their independent transformations before orthogonalization. Qwen also refitted the scaling law and found that larger learning rates and batch sizes can be used while maintaining stable training.
The training recipe does not use Batch Size Warmup. Qwen says warmup did not improve the final result and required 18.8% more optimizer steps.
The model also uses an ultra-sparse MoE with a large expert pool, a small number of routed experts per token, one shared expert and global load balancing. Its Multi-Token Prediction (MTP) module uses multiple training steps and replaces its full-Attention layers with QSA.
Training-stability measures include zero-centered RMSNorm with weight decay on norm weights, attention output gating and normalized MoE Router initialization.
Performance and context
With 6 billion activated parameters, Qwen3.8-Flash-Next-Base achieved the best results on 8 of 14 benchmarks:
- MMLU-Pro
- SuperGPQA
- BBH
- GSM8K
- EvalPlus
- SWEBench-Pretrain
- MGSM
- MMMLU
It remained close to Qwen3.7-Plus-Base on MMLU, MMLU-Redux, GPQA, MATH and MultiPL-E.
The model supports 262,144 tokens natively and can be extended to 1 million tokens using YaRN. The 51 billion N-gram Embedding parameters are deterministically addressed and do not count toward the per-token matrix-multiplication budget.
API, availability and pricing
The production model is served on QwenCloud as qwen3.8-flash. It supports OpenAI-compatible Chat Completions and Responses APIs, an Anthropic-compatible interface, and reasoning levels of low, medium and xhigh.
Qwen lists support for Claude Code, Codex, Qoder CLI, Qwen Code and OpenClaw, while QwenWork integrates Qwen3.8-Flash-Next into its Standard mode. The Codex configuration supports a 1-million-token context window, 95% effective context-window usage, parallel tool calls, text and image inputs, the three reasoning levels and up to 65,536 output tokens.
Availability and pricing:
- Open-weight model: Hugging Face and ModelScope
- QwenCloud model: Qwen3.8-Flash
- Context on QwenCloud: 1 million tokens by default
- Built-in tools: Supported on QwenCloud
- Input pricing: $0.16 per million tokens
- Output pricing: $0.47 per million tokens