Tencent Hunyuan Hy3 Deep Dive: 295B MoE Fast-Slow Thinking Fusion, Apache 2.0 Open Source, 90% Agent Task Success Rate

Abstract: On July 6, 2026, Tencent officially released Hunyuan Hy3. This article systematically dissects the full technical details of this 295B-parameter MoE fast-slow thinking fusion model across architecture design, training strategy, inference optimization, agent capabilities, hallucination management, and multi-product synergy, with complete Go/Python code implementations.

1. Model Overview: A Six-Month Sprint from Infrastructure Rebuild to Product Feedback

In late January 2026, the Tencent Hunyuan team initiated an infrastructure rebuild. On April 23, Hy3 preview was released. On July 6, the official Hy3 launched. In less than six months, Tencent completed the full R&D cycle from infrastructure rebuild to product feedback.

Core Specifications:

Hy3 demonstrated comprehensive improvements across 12 benchmarks, with SkillsBench surging from 29.1 to 55.3 (+90%), MathArena Apex from 12.8 to 38.7 (+202%), agent and code capabilities improving 20%-30%, and hallucination rate halved.

2. MoE Architecture: 295B Total, 21B Active Sparse Expert Design

2.1 Core Architecture

Hy3 adopts a standard Sparse MoE (Mixture of Experts) architecture. Unlike Dense models, MoE models have far more total parameters than the parameters actually activated per inference, achieving greater model capacity while maintaining inference efficiency.

┌────────────────────────────────────────────────────────────┐
│              Hy3 MoE Overall Architecture                   │
├────────────────────────────────────────────────────────────┤
│  ┌──────────┐  ┌──────────┐  ┌──────────┐                 │
│  │ Tokenizer│→ │ Embedding│→ │  Router  │                 │
│  └──────────┘  └──────────┘  └────┬─────┘                 │
│                                    │                        │
│           ┌────────────────────────┼──────────┐            │
│           │         Top-2 Routing  │          │            │
│           ▼                        ▼          │            │
│  ┌─────────────────────────────────────────┐  │            │
│  │  ┌──────┐  ┌──────┐  ┌──────┐  ┌──────┐│  │            │
│  │  │Expert│  │Expert│  │Expert│  │Expert││  │            │
│  │  │  1   │  │  2   │  │  ...  │  │  N   ││  │            │
│  │  └──────┘  └──────┘  └──────┘  └──────┘│  │            │
│  │          N=32 Experts, Top-2 Active     │  │            │
│  └─────────────────────────────────────────┘  │            │
│                    │                           │            │
│  ┌─────────────────┼─────────────────────┐   │            │
│  │   3.8B Multi-Token Prediction (MTP)   │   │            │
│  │   ┌────────┐  ┌────────┐  ┌────────┐  │   │            │
│  │   │Head 1  │  │Head 2  │  │Head 3  │  │   │            │
│  │   └───┬────┘  └───┬────┘  └───┬────┘  │   │            │
│  │       └───────────┼───────────┘       │   │            │
│  │                   ▼                    │   │            │
│  │            Next 3 Tokens              │   │            │
│  └─────────────────────────────────────────┘   │            │
│                                                  │            │
│  ┌──────────────────────────────────────────┐   │            │
│  │       Fast-Slow Thinking Fusion Module   │   │            │
│  │  ┌─────────────┐  ┌─────────────────┐   │   │            │
│  │  │ Fast Path   │  │  Slow Path      │   │   │            │
│  │  │ (System 1)  │  │  (System 2)     │   │   │            │
│  │  │ 1-2 step    │  │  Chain-of-Thought│   │   │            │
│  │  └─────────────┘  └─────────────────┘   │   │            │
│  └──────────────────────────────────────────┘   │            │
└────────────────────────────────────────────────────────────┘

2.2 MoE Router Implementation

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import List, Tuple, Optional

class SparseMoERouter(nn.Module):
    """
    Hy3 Sparse MoE Router with Top-2 routing and load balancing.
    
    Design:
    1. For each input token, compute relevance scores with all N experts
    2. Select Top-2 highest-scoring experts for activation
    3. Use auxiliary loss to ensure load balancing, avoiding "dead experts"
    """
    def __init__(
        self,
        hidden_dim: int = 7168,        # Hy3 hidden dimension
        num_experts: int = 32,          # Total experts
        top_k: int = 2,                 # Experts activated per token
        capacity_factor: float = 1.25,
    ):
        super().__init__()
        self.num_experts = num_experts
        self.top_k = top_k
        self.capacity_factor = capacity_factor
        self.gate = nn.Linear(hidden_dim, num_experts, bias=False)
        self.register_buffer("expert_counts", torch.zeros(num_experts))
        self.register_buffer("total_tokens", torch.tensor(0.0))
        
    def forward(self, x: torch.Tensor, training: bool = True):
        batch, seq_len, _ = x.shape
        x_flat = x.view(-1, x.size(-1))
        n_tokens = x_flat.size(0)
        
        logits = self.gate(x_flat)
        routing_scores = F.softmax(logits, dim=-1)
        top_k_scores, top_k_indices = torch.topk(
            routing_scores, k=self.top_k, dim=-1
        )
        routing_weights = top_k_scores / (top_k_scores.sum(dim=-1, keepdim=True) + 1e-8)
        aux_loss = self._compute_load_balancing_loss(routing_scores, top_k_indices, training)
        
        return routing_weights.view(batch, seq_len, self.top_k), \
               top_k_indices.view(batch, seq_len, self.top_k), aux_loss

2.3 3.8B Multi-Token Prediction (MTP) Layer

Traditional autoregressive models predict only the next token at each step. Hy3’s 3.8B MTP layer simultaneously predicts multiple future tokens, improving throughput, long-range dependency modeling, and decoding speed.

package mtp

type MTPHead struct {
	Heads           [3]*LinearTransform
	Norm            *LayerNorm
	SharedEmbedding *EmbeddingMatrix
}

type MTPOutput struct {
	MainLogits      []float32
	FutureLogits    [3][]float32
	ConfidenceScores [3]float32
}

func (m *MTPHead) Forward(hiddenState []float32) *MTPOutput {
	output := &MTPOutput{}
	normalized := m.Norm.Forward(hiddenState)
	output.MainLogits = m.Heads[0].Forward(normalized)
	
	for i := 0; i < 3; i++ {
		output.FutureLogits[i] = m.Heads[i].Forward(normalized)
		output.ConfidenceScores[i] = computeConfidence(output.FutureLogits[i])
	}
	return output
}

3. Fast-Slow Thinking Fusion: System 1 & System 2 Synergy

Hy3’s biggest differentiator is its “fast-slow thinking fusion” — it supports both rapid intuitive reasoning (System 1) and deep chain-of-thought reasoning (System 2), automatically switching based on problem complexity.

3.1 Thinking Router Implementation

class ThinkingRouter:
    def __init__(self, fast_threshold: float = 0.3, slow_threshold: float = 0.7):
        self.fast_threshold = fast_threshold
        self.slow_threshold = slow_threshold
    
    def select_mode(self, features: ComplexityFeatures) -> ThinkingMode:
        score = features.complexity_score
        if score < self.fast_threshold:
            return ThinkingMode.FAST
        elif score > self.slow_threshold:
            return ThinkingMode.SLOW
        return ThinkingMode.HYBRID

4. Hallucination Mitigation: 12.5% → 5.4%

Hy3 achieved significant hallucination reduction through three systematic engineering approaches: fine-grained data cleaning, hallucination-aware training constraints, and inference-time fact verification.

5. Agent Capabilities: 72% → 90% Task Success Rate

Agent task success rate in WorkBuddy surged from 72% to 90%, with average completion time reduced by 34%. This improvement stems from enhanced tool call stability, multi-turn reasoning consistency, and long-context memory management.

6. Multi-Product Integration Results

7. Pricing & Open Source

• Input: ¥1/1M tokens (cache hit ¥0.25)
• Output: ¥4/1M tokens
• License: Apache 2.0 (commercial-friendly)
• Platforms: HuggingFace, ModelScope, OpenRouter, Hermes, Kilo, Cline, OpenClaw

Sources:

• Tencent Hunyuan official 21世纪经济报道
• CSDN AI Daily CSDN

• ←Previous
• Next→