Xiaohongshu dots3 note Open Source Deep Dive: 280B MoE Multimodal Model — A Content Platform Enters the Open-Source LLM Arena

1. Introduction: A Strategic Pivot for a Content Platform

On August 14, 2026, dots studio, the AI research lab at Xiaohongshu (also known as “Little Red Book”), officially released dots3-note preview — the first open-source model in the dots3 family and the first time a Chinese content platform has publicly entered the open-source LLM arena. This move carries significance far beyond “yet another open-source model”; it signals Xiaohongshu’s transition from an AI application-layer player to a foundational model capability builder.

The dots3-note preview adopts a Mixture of Experts (MoE) architecture with 280B total parameters and only 16B activated parameters per token, supports a 512K ultra-long context window, and possesses full multimodal understanding across text, vision, and audio. Even more notably, a branch version of the same model series achieved a perfect 42/42 gold medal score at the International Mathematical Olympiad (IMO) 2026 — an official AI first.

This article provides a comprehensive technical deep dive into dots3-note preview, covering architecture design, MoE routing, attention mechanisms, multimodal processing, training methodology, deployment practices, benchmark performance, and industry impact.

2. Overall Architecture Overview

The dots3-note preview architecture consists of three core subsystems: the Language Backbone, the Vision Encoder, and the Audio Encoder. These work together to support unified input understanding across text, images, video, and audio.

2.1 Architecture Parameter Summary

2.2 Overall Architecture Diagram

┌─────────────────────────────────────────────────────────────┐
│                  dots3-note preview Architecture             │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌──────────────────┐  ┌──────────────────┐                │
│  │  Vision Encoder  │  │  Audio Encoder   │                │
│  │  MoE ViT         │  │  Dense Encoder   │                │
│  │  7B total/1.2B   │  │  800M params     │                │
│  │  image+video     │  │  16kHz audio     │                │
│  └────────┬─────────┘  └────────┬─────────┘                │
│           │                     │                          │
│           ▼                     ▼                          │
│  ┌──────────────────────────────────────────────────────┐  │
│  │          Multimodal Projection Alignment Layer        │  │
│  │     (Visual/Audio Projector → LLM Embedding Space)   │  │
│  └──────────────────────┬───────────────────────────────┘  │
│                         │                                   │
│                         ▼                                   │
│  ┌──────────────────────────────────────────────────────┐  │
│  │              Language Backbone (46 layers)            │  │
│  │                                                       │  │
│  │  ┌───────┐  ┌───────┐  ┌───────┐      ┌───────┐     │  │
│  │  │ Dense │  │ MoE 1 │  │ MoE 2 │ ...  │ MoE 45│     │  │
│  │  │ Layer │  │ Layer │  │ Layer │      │ Layer │     │  │
│  │  └───┬───┘  └───┬───┘  └───┬───┘      └───┬───┘     │  │
│  │      │          │          │              │         │  │
│  │      ▼          ▼          ▼              ▼         │  │
│  │  ┌──────────────────────────────────────────────┐   │  │
│  │  │  MTP Head (1.13B shared multi-token pred)    │   │  │
│  │  └──────────────────────────────────────────────┘   │  │
│  └──────────────────────────────────────────────────────┘  │
│                         │                                   │
│                         ▼                                   │
│  ┌──────────────────────────────────────────────────────┐  │
│  │              Output (Text Generation)                 │  │
│  └──────────────────────────────────────────────────────┘  │
│                                                             │
└─────────────────────────────────────────────────────────────┘

3. Deep Dive into MoE Architecture

3.1 Expert Routing Mechanism

The dots3-note MoE layer employs 256 routed experts plus 1 shared expert. Each token activates the top-8 most relevant experts through a learned gating network.

import torch
import torch.nn.functional as F

class Dots3MoELayer(torch.nn.Module):
    """
    Simplified implementation of the dots3-note MoE layer.
    
    Key design choices:
    - 256 routed experts + 1 shared expert
    - Top-8 routing per token
    - Shared expert always active for stable base signal
    """
    
    def __init__(self, hidden_dim=5120, n_experts=256, n_shared_experts=1, 
                 top_k=8, expert_dim=1536):
        super().__init__()
        self.hidden_dim = hidden_dim
        self.n_experts = n_experts
        self.top_k = top_k
        
        # Gating network
        self.gate = torch.nn.Linear(hidden_dim, n_experts, bias=False)
        
        # Shared expert (always active)
        self.shared_expert = torch.nn.Sequential(
            torch.nn.Linear(hidden_dim, expert_dim * 4),
            torch.nn.SiLU(),
            torch.nn.Linear(expert_dim * 4, hidden_dim),
        )
        
        # Routed experts (illustrative; actual impl uses grouped GEMM)
        self.experts = torch.nn.ModuleList([
            torch.nn.Sequential(
                torch.nn.Linear(hidden_dim, expert_dim * 4),
                torch.nn.SiLU(),
                torch.nn.Linear(expert_dim * 4, hidden_dim),
            )
            for _ in range(n_experts)
        ])
    
    def forward(self, x):
        # x: [batch_size, seq_len, hidden_dim]
        batch_size, seq_len, _ = x.shape
        
        # 1. Compute routing scores
        gate_logits = self.gate(x)  # [B, S, 256]
        gate_scores = F.softmax(gate_logits, dim=-1, dtype=torch.float32)
        
        # 2. Top-8 selection
        topk_scores, topk_indices = torch.topk(gate_scores, self.top_k, dim=-1)
        topk_scores = topk_scores / topk_scores.sum(dim=-1, keepdim=True)
        
        # 3. Shared expert output (always active)
        shared_out = self.shared_expert(x)
        
        # 4. Routed expert output (only for selected experts)
        # In production: dispatch/combine fused operators (FUSED_MC2)
        routed_output = torch.zeros_like(x)
        for i in range(self.n_experts):
            mask = (topk_indices == i).any(dim=-1)
            if mask.any():
                expert_out = self.experts[i](x[mask])
                expert_weight = topk_scores[mask][topk_indices[mask] == i]
                routed_output[mask] += expert_out * expert_weight.unsqueeze(-1)
        
        return shared_out + routed_output

3.2 The Economics of Activated Parameters

The 280B/16B parameter split is the most consequential design decision in dots3-note. The economics are clear:

• Knowledge Capacity: 280B total weights provide ample representational capacity across diverse domains
• Inference Cost: Only 16B parameters activated per token, keeping compute and memory close to a 16B dense model
• Deployment Friendly: FP8 quantization fits on a single 8×H100 80GB node

Comparison with contemporary open-source models:

3.3 MoE Routing and Attention Hybrid Architecture

┌──────────────────────────────────────────────────────────────┐
│              dots3-note Single Transformer Layer              │
├──────────────────────────────────────────────────────────────┤
│                                                              │
│                     Input x (hidden_dim=5120)                 │
│                           │                                  │
│                           ▼                                  │
│  ┌──────────────────────────────────────────────────────┐   │
│  │                  RMSNorm                             │   │
│  └──────────────────────┬───────────────────────────────┘   │
│                         │                                    │
│                         ▼                                    │
│  ┌──────────────────────────────────────────────────────┐   │
│  │              Attention Layer (one per layer)          │   │
│  │                                                       │   │
│  │  ┌─────────────────┐      ┌──────────────────────┐   │   │
│  │  │  DSA (13 layers) │  or  │  SWA (33 layers)     │   │   │
│  │  │  Top-2048 sparse │      │  Sliding window=513  │   │   │
│  │  │  MLA + Indexer   │      │  MLA + window mask  │   │   │
│  │  └─────────────────┘      └──────────────────────┘   │   │
│  └──────────────────────┬───────────────────────────────┘   │
│                         │                                    │
│                         ▼                                    │
│  ┌──────────────────────────────────────────────────────┐   │
│  │              Residual + RMSNorm                       │   │
│  └──────────────────────┬───────────────────────────────┘   │
│                         │                                    │
│                         ▼                                    │
│  ┌──────────────────────────────────────────────────────┐   │
│  │                  MoE FFN Layer                        │   │
│  │                                                       │   │
│  │   ┌──────────────────────────────────────────────┐    │   │
│  │   │      Gate (Linear 5120→256)                  │    │   │
│  │   │              │  Top-8 routing                 │    │   │
│  │   │              ▼                               │    │   │
│  │   │  ┌─────┐ ┌─────┐ ┌─────┐     ┌─────┐       │    │   │
│  │   │  │Exp 1│ │Exp 2│ │Exp 3│ ... │Exp 8│       │    │   │
│  │   │  └──┬──┘ └──┬──┘ └──┬──┘     └──┬──┘       │    │   │
│  │   │     │        │        │          │          │    │   │
│  │   │     └────────┴────────┴──────────┘          │    │   │
│  │   │              │  Weighted sum                 │    │   │
│  │   │              ▼                               │    │   │
│  │   │    ┌────────────────────────┐               │    │   │
│  │   │    │  Shared Expert (always)│               │    │   │
│  │   │    └───────────┬────────────┘               │    │   │
│  │   │                │  add                        │    │   │
│  │   └────────────────┼────────────────────────────┘    │   │
│  └──────────────────────┬───────────────────────────────┘   │
│                         │                                    │
│                         ▼                                    │
│  ┌──────────────────────────────────────────────────────┐   │
│  │              Residual + Output                        │   │
│  └──────────────────────────────────────────────────────┘   │
│                                                              │
└──────────────────────────────────────────────────────────────┘

4. DSA Sparse Attention and the 512K Context Window

4.1 Why Sparse Attention?

Standard full attention has O(N²) complexity. When scaling from 4K to 512K context, the computation grows not by 128× but by approximately 16,000×. Without sparsification, a single 512K attention matrix alone would consume ~1TB of GPU memory.

dots3-note employs a hybrid attention architecture: 13 DSA (Dots Sparse Attention) layers + 33 Sliding Window Attention (SWA) layers, at a ratio of approximately 1:3.

4.2 How DSA Works

DSA’s core idea is “select first, compute later” — a lightweight Indexer quickly filters the most relevant tokens, and only those selected tokens undergo full attention computation.

import torch
import torch.nn.functional as F

class DSAttention(torch.nn.Module):
    """
    Dots Sparse Attention (DSA) Implementation
    
    Core pipeline:
    1. Lightweight Indexer scores all tokens
    2. Select Top-2048 most relevant KV entries
    3. Compute full attention only on selected entries
    """
    
    def __init__(self, hidden_dim=5120, n_heads=64, head_dim=128,
                 index_n_heads=64, index_head_dim=128, index_topk=2048):
        super().__init__()
        self.hidden_dim = hidden_dim
        self.n_heads = n_heads
        self.head_dim = head_dim
        self.index_topk = index_topk
        
        # MLA: Low-rank QKV projection
        self.q_proj = torch.nn.Linear(hidden_dim, n_heads * head_dim, bias=False)
        self.kv_proj = torch.nn.Linear(hidden_dim, 2 * head_dim, bias=False)
        self.o_proj = torch.nn.Linear(n_heads * head_dim, hidden_dim, bias=False)
        
        # DSA Indexer: lightweight retriever
        self.indexer = DSAIndexer(
            hidden_dim=hidden_dim,
            n_heads=index_n_heads,
            head_dim=index_head_dim
        )
    
    def forward(self, x, attention_mask=None):
        batch_size, seq_len, _ = x.shape
        
        # 1. Compute Q and shared KV (MLA style)
        q = self.q_proj(x).view(batch_size, seq_len, self.n_heads, self.head_dim)
        kv = self.kv_proj(x)
        k, v = kv.chunk(2, dim=-1)
        k = k.unsqueeze(2)  # [B, S, 1, head_dim]
        v = v.unsqueeze(2)  # [B, S, 1, head_dim]
        
        # 2. Use Indexer to select top-K positions
        selected_indices = self.indexer(x, self.index_topk)
        
        # 3. Gather selected KV entries
        expand_k = k.unsqueeze(1).expand(-1, seq_len, -1, -1, -1)
        expand_v = v.unsqueeze(1).expand(-1, seq_len, -1, -1, -1)
        
        selected_k = torch.gather(
            expand_k, 2,
            selected_indices.unsqueeze(-1).unsqueeze(-1)
            .expand(-1, -1, -1, 1, self.head_dim)
        )
        selected_v = torch.gather(
            expand_v, 2,
            selected_indices.unsqueeze(-1).unsqueeze(-1)
            .expand(-1, -1, -1, 1, self.head_dim)
        )
        
        # 4. Attention computation (top-k only)
        attn_scores = torch.einsum(
            "bqhd,bqkhd->bqhk",
            q.unsqueeze(2),
            selected_k
        ) / (self.head_dim ** 0.5)
        
        if attention_mask is not None:
            attn_scores = attn_scores + attention_mask
        
        attn_weights = F.softmax(attn_scores, dim=-1)
        attn_output = torch.einsum("bqhk,bqkhd->bqhd", attn_weights, selected_v)
        
        attn_output = attn_output.reshape(batch_size, seq_len, -1)
        return self.o_proj(attn_output)


class DSAIndexer(torch.nn.Module):
    """
    DSA Indexer — The "smart radar" for token selection
    
    Uses independent low-dimensional projections to quickly encode
    token positions and semantics, computing index scores for retrieval.
    """
    
    def __init__(self, hidden_dim, n_heads=64, head_dim=128):
        super().__init__()
        self.n_heads = n_heads
        self.head_dim = head_dim
        
        self.wq = torch.nn.Linear(hidden_dim, n_heads * head_dim, bias=False)
        self.wk = torch.nn.Linear(hidden_dim, n_heads * head_dim, bias=False)
        self.weights_proj = torch.nn.Linear(hidden_dim, n_heads, bias=False)
        self.k_norm = torch.nn.LayerNorm(head_dim)
    
    def forward(self, x, topk):
        batch_size, seq_len, _ = x.shape
        
        idx_q = self.wq(x).view(batch_size, seq_len, self.n_heads, self.head_dim)
        idx_k = self.k_norm(
            self.wk(x).view(batch_size, seq_len, self.n_heads, self.head_dim)
        )
        
        # Index score: I(t,s) = sum_j(w_j * ReLU(q_j · k_j))
        idx_scores = torch.einsum("bqhd,bkhd->bqkh", idx_q, idx_k)
        idx_scores = F.relu(idx_scores)
        
        idx_weights = self.weights_proj(x)  # [B, S, n_heads]
        idx_scores = torch.einsum("bqkh,bqh->bqk", idx_scores, idx_weights)
        
        _, topk_indices = torch.topk(idx_scores, topk, dim=-1)
        return topk_indices

4.3 DSA + SWA Hybrid Attention Layout

┌──────────────────────────────────────────────────────────────────┐
│              dots3-note 46-Layer Attention Layout                │
├──────────────────────────────────────────────────────────────────┤
│                                                                  │
│  Layer   Type        Description          Capacity               │
│  ──────  ──────────  ────────────────     ──────────────────     │
│   0      Dense       Initial dense layer  Full attention         │
│   1      DSA         Top-2048 sparse      Long-range modeling    │
│   2      SWA         Sliding window 513   Local context          │
│   3      DSA         Top-2048 sparse      Long-range modeling    │
│   4      SWA         Sliding window 513   Local context          │
│   5      SWA         Sliding window 513   Local context          │
│   ...    ...         Interleaved pattern                         │
│   45     MoE+SWA     Last MoE layer       Local context          │
│                                                                  │
│  Count: DSA × 13  |  SWA × 33  |  Ratio ≈ 1:3                  │
│                                                                  │
│  Design Philosophy:                                               │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │  Most layers (SWA) handle nearby information cheaply     │   │
│  │  Few layers (DSA) provide global coverage sparsely       │   │
│  │  Local and global each serve their role, balancing cost  │   │
│  │  and effectiveness                                       │   │
│  └──────────────────────────────────────────────────────────┘   │
│                                                                  │
│  DSA Layer Internal Flow:                                        │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │  Step 1: Indexer → Score all tokens for relevance        │   │
│  │  Step 2: Selection → Pick Top-2048 candidates            │   │
│  │  Step 3: Attention → Full attention on selected only     │   │
│  │  Step 4: Output → Project back to hidden dim             │   │
│  │  Complexity: O(N × topk) ≈ O(N) instead of O(N²)        │   │
│  └──────────────────────────────────────────────────────────┘   │
│                                                                  │
└──────────────────────────────────────────────────────────────────┘

5. Multimodal Processing Pipeline

5.1 Vision Encoder: MoE ViT

The vision encoder in dots3-note is itself an MoE architecture with 7B total parameters and 1.2B activated parameters — a remarkably aggressive scale for a vision encoder in a large language model.

import torch
import torchvision.transforms as T
from PIL import Image

class Dots3MultimodalProcessor:
    """
    dots3-note multimodal processing pipeline
    Supports unified input handling for text, images, video, and audio
    """
    
    def __init__(self, model, processor):
        self.model = model
        self.processor = processor
        self.device = next(model.parameters()).device
        
        self.image_transform = T.Compose([
            T.Resize((384, 384)),
            T.ToTensor(),
            T.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]),
        ])
    
    def process_text_only(self, text, enable_thinking=False):
        """Pure text inference"""
        messages = [{"role": "user", "content": text}]
        inputs = self.processor.tokenizer.apply_chat_template(
            messages, add_generation_prompt=True,
            return_tensors="pt", return_dict=True,
            enable_thinking=enable_thinking,
        ).to(self.device)
        
        outputs = self.model.generate(
            **inputs, max_new_tokens=2048,
            temperature=0.7, top_p=0.95,
        )
        return self.processor.decode(
            outputs[0, inputs.input_ids.shape[1]:],
            skip_special_tokens=True
        )
    
    def process_image_text(self, image_path, text):
        """Image + text understanding"""
        image = Image.open(image_path).convert("RGB")
        image_tensor = self.image_transform(image).unsqueeze(0).to(self.device)
        
        messages = [{
            "role": "user",
            "content": [
                {"type": "image", "image": image_tensor},
                {"type": "text", "text": text},
            ]
        }]
        
        inputs = self.processor.tokenizer.apply_chat_template(
            messages, add_generation_prompt=True,
            return_tensors="pt", return_dict=True,
        ).to(self.device)
        
        outputs = self.model.generate(
            **inputs, max_new_tokens=2048, temperature=0.7,
        )
        return self.processor.decode(
            outputs[0, inputs.input_ids.shape[1]:],
            skip_special_tokens=True
        )

# OpenAI-compatible API usage
def run_multimodal_example():
    from openai import OpenAI
    
    client = OpenAI(
        base_url="http://127.0.0.1:8000/v1",
        api_key="EMPTY"
    )
    
    # Image understanding
    response = client.chat.completions.create(
        model="dots3-note-prev",
        messages=[{
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {
                    "url": "https://example.com/photo.jpg"
                }},
                {"type": "text", "text": "Describe this image in detail"},
            ]
        }],
        temperature=0.7, max_tokens=1024,
    )
    print(response.choices[0].message.content)
    
    # Reasoning mode
    response = client.chat.completions.create(
        model="dots3-note-prev",
        messages=[{"role": "user", "content": "Prove that sqrt(2) is irrational"}],
        temperature=1.0, max_tokens=4096,
        extra_body={"chat_template_kwargs": {"enable_thinking": True}},
    )
    print(response.choices[0].message.content)

5.2 Multimodal Processing Flow

┌──────────────────────────────────────────────────────────────────┐
│                  dots3-note Multimodal Processing Flow           │
├──────────────────────────────────────────────────────────────────┤
│                                                                  │
│  Input Types                                                     │
│  ┌──────┐  ┌──────┐  ┌──────┐  ┌──────┐                        │
│  │ Text │  │ Image│  │ Video│  │ Audio│                        │
│  └──┬───┘  └──┬───┘  └──┬───┘  └──┬───┘                        │
│     │         │         │         │                             │
│     ▼         ▼         ▼         ▼                             │
│  ┌──────┐  ┌──────────────┐  ┌──────────────┐                  │
│  │Tokenizer│  │  MoE ViT    │  │  Dense Audio  │                  │
│  │152K vocab│  │  7B/1.2B    │  │  Encoder      │                  │
│  │         │  │  384×384    │  │  800M         │                  │
│  │         │  │  [frames]   │  │  16kHz        │                  │
│  └────────┘  └──────────────┘  └──────────────┘                  │
│       │              │                  │                        │
│       │              │    ┌─────────────┘                        │
│       │              │    │  Video: frame sampling + audio       │
│       ▼              ▼    ▼  extraction + timestamp interleaving │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │         Multimodal Token Embedding + Position Encoding     │  │
│  │  [text] [image×N] [audio×M] [timestamps] ...              │  │
│  └──────────────────────────┬─────────────────────────────────┘  │
│                             │                                    │
│                             ▼                                    │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │           dots3-note Language Backbone (46 layers)          │  │
│  │                                                             │  │
│  │  ┌───────┐ ┌───────┐ ┌───────┐       ┌───────┐           │  │
│  │  │ Dense │ │MoE+SWA│ │MoE+DSA│  ...  │MoE+SWA│           │  │
│  │  │ Attn  │ │  FFN  │ │  FFN  │       │  FFN  │           │  │
│  │  └───┬───┘ └───┬───┘ └───┬───┘       └───┬───┘           │  │
│  │      └─────────┴─────────┴───────────────┘                │  │
│  └──────────────────────────┬─────────────────────────────────┘  │
│                             │                                    │
│                             ▼                                    │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │  MTP Head: parallel next-3-token prediction + verification  │  │
│  └────────────────────────────────────────────────────────────┘  │
│                             │                                    │
│                             ▼                                    │
│                       Output: Text Generation                     │
│                                                                  │
└──────────────────────────────────────────────────────────────────┘

6. TEMPO Reinforcement Learning: A Training Innovation for Long-Horizon Agent Tasks

6.1 The Core Challenge

In agent scenarios, task execution can span hours or even tens of hours. Traditional value-free RL methods (e.g., GRPO) face two fundamental difficulties:

1. Intolerable training efficiency: A single exploration trajectory can take 10+ hours
2. Sparse reward credit assignment: When final success/failure comes after hundreds of steps, it’s extremely difficult to attribute outcomes to specific decisions

6.2 The TEMPO Method

TEMPO (Test-time scaled Value Estimation with Macro-step Policy Optimization) decomposes long-horizon tasks into macro-steps. At the end of each macro-step, the same agent switches from actor to critic role, using test-time scaling inference to estimate the expected remaining return from the current state.

import torch
from typing import List, Dict, Any

class TEMPOTrainer:
    """
    TEMPO: Test-time scaled Value Estimation with Macro-step Policy Optimization
    
    Core insight: Decompose long-horizon tasks into macro-steps.
    At each macro-step boundary, the agent switches to critic mode,
    using test-time compute scaling to estimate value.
    """
    
    def __init__(self, model, tokenizer, 
                 macro_step_size: int = 10,
                 n_critic_rollouts: int = 8):
        self.model = model
        self.tokenizer = tokenizer
        self.macro_step_size = macro_step_size
        self.n_critic_rollouts = n_critic_rollouts
    
    def rollout_with_macro_steps(self, env) -> List[Dict]:
        """Execute rollout with macro-step boundaries"""
        trajectory = []
        state = env.reset()
        done = False
        
        while not done:
            # Actor phase: execute one macro-step
            macro_actions = []
            for _ in range(self.macro_step_size):
                if done:
                    break
                action = self.act(state)
                next_state, reward, done, info = env.step(action)
                macro_actions.append({
                    "state": state, "action": action,
                    "reward": reward, "next_state": next_state,
                })
                state = next_state
            
            # Critic phase: estimate value via test-time scaling
            value_estimate = 0.0 if done else self.estimate_value(
                state, trajectory, env.get_privileged_info()
            )
            
            trajectory.append({
                "macro_actions": macro_actions,
                "value_estimate": value_estimate,
                "final_reward": reward if done else None,
            })
        
        return trajectory
    
    def act(self, state) -> str:
        """Actor: generate action"""
        prompt = self.build_actor_prompt(state)
        inputs = self.tokenizer(prompt, return_tensors="pt").to(self.model.device)
        outputs = self.model.generate(
            **inputs, max_new_tokens=256,
            temperature=1.0, top_p=0.95,
        )
        return self.tokenizer.decode(
            outputs[0, inputs.input_ids.shape[1]:],
            skip_special_tokens=True
        )
    
    def estimate_value(self, state, history, priv_info) -> float:
        """
        Critic: estimate value via test-time scaling inference.
        
        Key finding: "evaluation is easier than generation" holds for
        long-horizon tasks. Even when the agent cannot solve the problem,
        it can still accurately estimate value as a critic.
        """
        prompt = self.build_critic_prompt(state, history, priv_info)
        
        estimates = []
        for _ in range(self.n_critic_rollouts):
            inputs = self.tokenizer(prompt, return_tensors="pt").to(self.model.device)
            outputs = self.model.generate(
                **inputs, max_new_tokens=512,
                temperature=0.7,
                chat_template_kwargs={"enable_thinking": True},
            )
            analysis = self.tokenizer.decode(
                outputs[0, inputs.input_ids.shape[1]:],
                skip_special_tokens=True
            )
            estimates.append(self.extract_value_from_analysis(analysis))
        
        return torch.mean(torch.tensor(estimates)).item()

6.3 TEMPO Training Results

According to official benchmarks, models trained with TEMPO on ARC-AGI-3 achieved:

• 31.5% average score improvement over baseline checkpoint
• 20.6% improvement over GRPO-trained models
• Fewer steps to reach the same level with higher scores

7. MTP Multi-Token Prediction and Speculative Decoding

7.1 MTP Acceleration

dots3-note includes a built-in 1.13B parameter shared multi-token prediction (MTP) layer. While generating the current token, it can predict up to 3 subsequent tokens in parallel. This acts as a built-in draft model, enabling speculative decoding without deploying a separate model.

import torch
import torch.nn.functional as F

class MTPDecoding:
    """
    MTP (Multi-Token Prediction) Speculative Decoding
    
    The built-in MTP layer generates multiple candidate tokens in a
    single forward pass, then verifies them, drastically reducing
    the number of decoding steps.
    """
    
    def __init__(self, model, mtp_head, n_speculative_tokens=3):
        self.model = model
        self.mtp_head = mtp_head
        self.n_speculative_tokens = n_speculative_tokens
    
    @torch.no_grad()
    def generate_with_mtp(self, input_ids, max_new_tokens=1024,
                          temperature=0.7, top_p=0.95):
        """
        Generate with MTP speculative decoding.
        
        Flow:
        1. Main model forward → get current token logits
        2. MTP Head → predict next n tokens in parallel
        3. Verify → accept correct predictions, reject wrong ones
        4. Repeat until completion
        """
        for _ in range(max_new_tokens // (self.n_speculative_tokens + 1) + 1):
            # Step 1: Main model forward
            main_outputs = self.model(input_ids, use_cache=True)
            main_logits = main_outputs.logits[:, -1, :]
            next_token = self.sample(main_logits, temperature, top_p)
            
            # Step 2: MTP Head → draft tokens
            draft_tokens = []
            hidden_state = main_outputs.last_hidden_state[:, -1:, :]
            
            for _ in range(self.n_speculative_tokens):
                mtp_logits = self.mtp_head(hidden_state)
                draft_token = self.sample(mtp_logits[:, -1, :], temperature, top_p)
                draft_tokens.append(draft_token)
                draft_embed = self.model.get_input_embeddings()(draft_token)
                hidden_state = self.mtp_head.forward_embed(draft_embed)
            
            # Step 3: Verify draft tokens
            draft_ids = torch.cat([next_token] + draft_tokens, dim=-1)
            verify_outputs = self.model(
                torch.cat([input_ids, draft_ids[:, :-1]], dim=-1),
                use_cache=False
            )
            verify_logits = verify_outputs.logits[:, -self.n_speculative_tokens:, :]
            
            accepted_tokens = [next_token]
            for i, draft_token in enumerate(draft_tokens):
                target_logits = main_logits if i == 0 else verify_logits[:, i-1, :]
                target_prob = F.softmax(target_logits, dim=-1)
                draft_prob = F.softmax(verify_logits[:, i, :], dim=-1)
                
                p_accept = torch.min(
                    torch.ones_like(target_prob[0, draft_token[0]]),
                    draft_prob[0, draft_token[0]] / target_prob[0, draft_token[0]]
                )
                
                if torch.rand(1).item() < p_accept.item():
                    accepted_tokens.append(draft_token)
                else:
                    corrected = self.sample(target_logits, temperature, top_p)
                    accepted_tokens.append(corrected)
                    break
            
            new_tokens = torch.cat(accepted_tokens, dim=-1)
            input_ids = torch.cat([input_ids, new_tokens], dim=-1)
            
            if (new_tokens == self.model.config.eos_token_id).any():
                break
        
        return input_ids
    
    def sample(self, logits, temperature, top_p):
        """Temperature and top-p sampling"""
        logits = logits / temperature
        
        sorted_logits, sorted_indices = torch.sort(logits, descending=True)
        cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)
        
        sorted_indices_to_remove = cumulative_probs > top_p
        sorted_indices_to_remove[..., 1:] = sorted_indices_to_remove[..., :-1].clone()
        sorted_indices_to_remove[..., 0] = 0
        
        indices_to_remove = sorted_indices_to_remove.scatter(
            -1, sorted_indices, sorted_indices_to_remove
        )
        logits[indices_to_remove] = float('-inf')
        
        probs = F.softmax(logits, dim=-1)
        return torch.multinomial(probs, num_samples=1)

7.2 MTP Performance Impact

With MTP speculative decoding enabled on vLLM Ascend:

• TPOT (time per output token) reduced by over 50%
• Throughput significantly improved for high-concurrency online scenarios
• No separate draft model needed — the built-in MTP layer handles everything

8. Ascend NPU Adaptation and Domestic Computing Deployment

8.1 Day-0 Ascend Adaptation

Huawei’s Ascend team completed Day-0 adaptation on the same day dots3-note was released — a milestone for domestic model + domestic hardware integration. The adaptation covers Atlas 800 A3 and Atlas 900 A3 SuperPoD clusters.

8.2 Ascend Adaptation Architecture

┌──────────────────────────────────────────────────────────────────┐
│              dots3-note × Ascend Adaptation Architecture         │
├──────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │                  Atlas 900 A3 SuperPoD                    │   │
│  │  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐ │   │
│  │  │ Atlas    │  │ Atlas    │  │ Atlas    │  │ Atlas    │ │   │
│  │  │ 800 A3  │  │ 800 A3  │  │ 800 A3  │  │ 800 A3  │ │   │
│  │  │ Node 1  │  │ Node 2  │  │ Node 3  │  │ Node N  │ │   │
│  │  └────┬────┘  └────┬────┘  └────┬────┘  └────┬────┘ │   │
│  │       └────────────┴────────────┴────────────┘       │   │
│  │                    │ SuperPoD interconnect            │   │
│  └────────────────────┼──────────────────────────────────┘   │
│                       │                                       │
│                       ▼                                       │
│  ┌──────────────────────────────────────────────────────────┐ │
│  │              vLLM Ascend Inference Framework              │ │
│  ├──────────────────────────────────────────────────────────┤ │
│  │                                                          │ │
│  │  1. Full-Modality End-to-End Adaptation                  │ │
│  │     ┌──────────────┐ ┌──────────────┐ ┌──────────────┐  │ │
│  │     │ Vision Enc.  │ │ Audio Enc.   │ │ LLM Backbone │  │ │
│  │     │ (MoE ViT)    │ │ (Dense)      │ │ (46 MoE)     │  │ │
│  │     └──────────────┘ └──────────────┘ └──────────────┘  │ │
│  │                                                          │ │
│  │  2. FlashComm Optimization                               │ │
│  │     AllReduce → ReduceScatter + AllGather split          │ │
│  │     Column-independent ops moved between 2-phase comm   │ │
│  │     → Eliminates redundant multi-card computation        │ │
│  │                                                          │ │
│  │  3. FUSED_MC2 Compute-Communication Fusion               │ │
│  │     dispatch → gmm1 → swiglu → gmm2 → combine           │ │
│  │     Merged into single large kernel                      │ │
│  │     → Fewer launches, deep pipelining, no intermediate   │ │
│  │     → Significant MoE throughput boost                   │ │
│  │                                                          │ │
│  │  4. MTP Speculative Decoding Native Support              │ │
│  │     Parallel multi-token generation → verification       │ │
│  │     → TPOT reduced by 50%+                               │ │
│  │                                                          │ │
│  └──────────────────────────────────────────────────────────┘ │
│                                                                  │
└──────────────────────────────────────────────────────────────────┘

8.3 Ascend Deployment Command

# Deploy dots3-note on Ascend Atlas 800 A3 using vLLM Ascend
vllm serve dots-studio/dots3-note-prev-fp8 \
  --served-model-name dots3-note-prev \
  --host 0.0.0.0 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --moe-backend deep_gemm \
  --max-model-len 262144 \
  --enable-auto-tool-choice \
  --tool-call-parser dots \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

9. Benchmark Performance and Competitive Analysis

9.1 Reasoning and Agent Tasks

Key Observations:

• ARC-AGI-2 (abstract visual reasoning): dots3-note scores 81.4, dramatically outperforming other same-scale models and trailing only GPT-5.5
• Agent tasks (ClawEval, WildClawBench): With only 16B activated params, dots3-note achieves scores comparable to models with several times its activation budget
• IMOAnswerBench: The IMO gold medal lineage is validated — near-perfect reasoning capability

9.2 Multimodal Vision Tasks

9.3 Model Capability Matrix

┌──────────────────────────────────────────────────────────────────────┐
│             dots3-note vs Peer Open-Source Models — Radar            │
├──────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  Capability       dots3 note   DS-v4-flash   Hy3       GLM-5.2      │
│  ──────────       ──────────   ──────────   ──────    ──────────    │
│  Reasoning         ████████    ████████     ████████  █████████     │
│                    ════════    ════════     ════════  ═════════     │
│  Coding            ███████     ████████     ████████  █████████     │
│                    ════════    ════════     ════════  ═════════     │
│  Agent             ████████    ██████       ██████    █████████     │
│                    ════════    ════════     ════════  ═════════     │
│  Long Context      █████████   ██████       ██████    ████████      │
│  (512K)            ════════    ════════     ════════  ═════════     │
│  Multimodal Vision ████████    ██           ██        ██████        │
│                    ════════    ════════     ════════  ═════════     │
│  Multimodal Audio  ████████    ██           ██        ██            │
│                    ════════    ════════     ════════  ═════════     │
│  Ascend Support    █████████   ██████       ██████    ██████        │
│                    ════════    ════════     ════════  ═════════     │
│  Activation Eff.   █████████   █████████    ████████  ██████        │
│  (16B/13B/21B/39B) ════════    ════════     ════════  ═════════     │
│                                                                      │
│  ████████ = Score range (each block ≈ 12.5 points)                   │
│                                                                      │
│  Note: dots3-note with 16B activated params establishes             │
│  differentiated advantages in Agent, long context, multimodal,       │
│  and Ascend adaptation — particularly outstanding in "activation     │
│  efficiency"                                                         │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘

10. Deployment Practice and Getting Started

10.1 Hardware Requirements

10.2 Complete Deployment Workflow

# 1. Deploy with SGLang (recommended, full MTP support)
docker run --gpus all --ipc=host -p 8000:8000 \
  lmsysorg/sglang:dev-dots3-note \
  sglang serve \
  --model-path dots-studio/dots3-note-prev-fp8 \
  --served-model-name dots3-note-prev \
  --host 0.0.0.0 \
  --port 8000 \
  --context-length 524288 \
  --enable-dp-attention \
  --dp-size 8 \
  --tp-size 8 \
  --ep-size 8 \
  --moe-dense-tp-size 1 \
  --page-size 64 \
  --trust-remote-code \
  --attention-backend fa3 \
  --moe-a2a-backend deepep \
  --enable-multimodal \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --speculative-draft-model-path dots-studio/dots3-note-prev-fp8

# 2. Deploy with vLLM (mature and stable)
vllm serve dots-studio/dots3-note-prev-fp8 \
  --served-model-name dots3-note-prev \
  --host 0.0.0.0 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --moe-backend deep_gemm \
  --max-model-len 262144 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --enable-auto-tool-choice --tool-call-parser dots

10.3 Client API Usage

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="EMPTY"
)

# Text inference with reasoning
response = client.chat.completions.create(
    model="dots3-note-prev",
    messages=[{"role": "user", "content": "Explain how MoE architecture works"}],
    temperature=1.0, top_p=0.95, max_tokens=2048,
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(response.choices[0].message.content)

# Multimodal: image understanding
response = client.chat.completions.create(
    model="dots3-note-prev",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}},
            {"type": "text", "text": "Analyze the trend in this chart"},
        ]
    }],
    max_tokens=1024,
)
print(response.choices[0].message.content)

# Tool calling (Agent mode)
response = client.chat.completions.create(
    model="dots3-note-prev",
    messages=[{"role": "user", "content": "Check weather in Beijing and plan my itinerary"}],
    tools=[{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get weather for a city",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string", "description": "City name"}
                },
                "required": ["city"]
            }
        }
    }],
    tool_choice="auto",
)
print(response.choices[0].message)

11. The dots3 Series Roadmap and Industry Impact

11.1 dots3 Model Hierarchy

The dots3 family will include three tiers of models, each targeting different scenarios:

┌──────────────────────────────────────────────────────────────────┐
│                    dots3 Series Roadmap                          │
├──────────────────────────────────────────────────────────────────┤
│                                                                  │
│  Tier      Model      Positioning         Target Scenarios       │
│  ──────   ──────────  ────────────────   ────────────────────    │
│                                                                  │
│  Light     note       Most lightweight    Long-horizon Agent     │
│            (open)     280B/16B            Multimodal daily tasks │
│                       512K context        Personal assistant     │
│                                                                  │
│  Mid       jazz       Balance of speed    Complex reasoning      │
│            (coming)   and capability      Professional domains   │
│                       Larger activation   Enterprise workflows   │
│                                                                  │
│  Flagship  aria       Maximum capability  Frontier research      │
│            (coming)   Largest params      Scientific discovery   │
│                       Full context        IMO-level tasks        │
│                                                                  │
│  ──────────────────────────────────────────────────────────────  │
│                                                                  │
│  The official dots3-note release (non-preview) will also be      │
│  open-sourced soon, with a complete technical report,            │
│  training details, and more stable model weights.                │
│                                                                  │
└──────────────────────────────────────────────────────────────────┘

11.2 Why Would a Content Platform Build Its Own Model?

Xiaohongshu’s move has sparked deep industry reflection on why content platforms are building their own AI models:

1. Cost Control: With 300M+ MAU, API call costs for search and recommendation at scale explode. Self-built models are the long-term cost-optimal solution
2. Data Flywheel: The platform possesses massive image-text and consumer decision data. Self-built models can turn this data advantage into a model capability advantage
3. Domain Depth: Xiaohongshu has unique data in life decision scenarios (travel, weddings, home renovation). General models struggle with these long-tail but high-value needs
4. Ecosystem Control: Once model capability becomes part of the core product pipeline, relying entirely on external providers introduces risk and uncertainty

The Dual Significance of Open-Source:

• Direct benefit: Access to community testing, feedback, ecosystem support, and hardware adaptation
• Strategic signal: A public demonstration to the technical community that Xiaohongshu is not just using AI at the application layer but also building foundational capabilities

12. Conclusion and Outlook

The release of dots3-note preview marks a watershed moment — the first time a content platform has entered the open-source LLM arena. With its 280B total / 16B activated MoE architecture, 512K context window, full multimodal understanding, and TEMPO reinforcement learning innovation, it delivers competitive performance across reasoning, agent, and multimodal benchmarks.

Key Takeaways:

1. MoE Extreme Efficiency: 16B activated params matching or exceeding models with several times the parameters validates the efficiency advantage of MoE architecture
2. DSA + SWA Hybrid Attention: Provides a practical technical path for 512K context — sparse without missing key information
3. TEMPO Training Method: Opens new avenues for RL training of long-horizon agent tasks; self-critiquing ability may be key to general-purpose agents
4. Ascend Day-0 Adaptation: Domestic model + domestic hardware full-stack solution holds special significance for government and enterprise deployments
5. Apache 2.0 Full Open-Source: Lowers the barrier for developers and accelerates ecosystem building

The preview label is honest about current limitations: RL training is still incomplete, and the model has room for improvement in hallucination control, text-vision capability balance, and stability. The formal release, with a complete technical report and more stable weights, is coming soon.

In essence, dots3-note preview is not just a technical release — it’s a declaration of a content platform redefining its position in the AI era. When platforms shift from “using AI” to “building AI,” the competitive landscape of large language models is quietly changing.

References:

• dots3-note preview Official Tech Blog (https://studio.dots.ai/dots/dots3-zh.html)
• GitHub Repository (https://github.com/studio-dots-ai/dots3-note-prev)
• Hugging Face Model Card (https://huggingface.co/dots-studio/dots3-note-prev)
• Huawei Ascend Adaptation Announcement (IT之家)
• DeepSeek-V3.2: DSA Sparse Attention Technical Report

• ←Previous
• Next→