Live Web Viewer Multi-Language Real-Time Subtitle Synchronization Solution — R&D Analysis Report

Version: v1.0
Date: 2026-08-13
Status: Draft

1. Requirements Analysis

1.1 Background

Our live streaming system already has the following capabilities:

• The publisher (host client) has completed audio capture;
• The server/publisher side has integrated an ASR (Automatic Speech Recognition) engine that can generate real-time subtitle text with PTS (Presentation Time Stamp);
• Multi-language translation capabilities are already implemented, supporting Chinese, English, Japanese, Korean, and other languages;
• The host client can already display subtitles.

The missing key link is:

• Distributing multi-language subtitle data to web viewers in real time;
• Web viewers can receive subtitles and synchronize them precisely with the audio/video stream;
• Web viewers can freely switch subtitle languages (or turn subtitles off).

1.2 Objectives

1. Establish an independent real‑time subtitle distribution channel decoupled from the video stream;
2. Achieve low latency (< 800 ms) and high synchronization accuracy (error < 50 ms) for subtitle display on the web viewer side;
3. Support viewers to switch subtitle languages in real time without interrupting video playback or requiring a re‑pull of the stream;
4. The solution must support high concurrency (tens of thousands of concurrent viewers);
5. Guarantee reliability and graceful degradation under poor network conditions.

1.3 Non‑Functional Requirements

2. Architecture Design

2.1 Overall Architecture Diagram

┌────────────────────────────────────────────────────────────────────────────────┐
│                              Overall Architecture                              │
├────────────────────────────────────────────────────────────────────────────────┤
│                                                                                │
│  ┌─────────────┐    ┌─────────────┐    ┌───────────────────────────────────┐ │
│  │ Publisher   │    │  ASR /      │    │      New: Subtitle Distribution   │ │
│  │ (Existing)  │───▶│ Translation │───▶│           Gateway                 │ │
│  │ Audio       │    │  Services   │    │  ┌─────────────────────────────┐ │ │
│  │ Capture     │    │ (Existing)  │    │  │ • Ingest subtitle data      │ │ │
│  └─────────────┘    │ Generate    │    │  │ • Cache & distribute by lang │ │ │
│                     │ multi‑lang  │    │  │ • WebSocket connection pool │ │ │
│                     │ subtitles   │    │  │ • Heartbeat / retransmission │ │ │
│                     │ with PTS    │    │  └──────────────┬───────────────┘ │ │
│                     └─────────────┘    │                 │ WebSocket        │ │
│                                        └─────────────────┼─────────────────┘ │
│                                                          ▼                     │
│                     ┌────────────────────────────────────────────────────────┐ │
│                     │             Web Viewer (New Modules)                  │ │
│                     │  ┌──────────────────────────────────────────────────┐ │ │
│                     │  │ ① WebSocket Client (connect/auth/heartbeat/reconnect)│ │
│                     │  └─────────────────┬───────────────────────────────┘ │ │
│                     │                    ▼                                 │ │
│                     │  ┌──────────────────────────────────────────────────┐ │ │
│                     │  │ ② Subtitle Buffer Manager (PTS sort/dedup/expiry)│ │ │
│                     │  └─────────────────┬───────────────────────────────┘ │ │
│                     │                    ▼                                 │ │
│                     │  ┌──────────────────────────────────────────────────┐ │ │
│                     │  │ ③ Sync Rendering Engine (based on video.currentTime)│ │
│                     │  └─────────────────┬───────────────────────────────┘ │ │
│                     │                    ▼                                 │ │
│                     │  ┌──────────────────────────────────────────────────┐ │ │
│                     │  │ ④ Subtitle Overlay (CSS/Canvas rendering)       │ │ │
│                     │  └──────────────────────────────────────────────────┘ │ │
│                     │  ┌──────────────────────────────────────────────────┐ │ │
│                     │  │ ⑤ Language Switcher UI (sends switch signal)    │ │ │
│                     │  └──────────────────────────────────────────────────┘ │ │
│                     └──────────────────────┬───────────────────────────────┘ │
│                                            │                                 │
│                     ┌──────────────────────┴───────────────────────────────┐ │
│                     │ Video Player (Existing, hls.js/WebRTC)              │ │
│                     │ Provides currentTime as the sync reference          │ │
│                     └──────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────────────────┘

2.2 Data Flow Description

1. The publisher captures audio → sends to the ASR/Translation service (existing);
2. The ASR/Translation service generates multi‑language subtitle data with PTS (existing);
3. The new “Subtitle Distribution Gateway” subscribes to this data stream and pushes it via WebSocket to each web viewer;
4. The web viewer receives subtitles → stores them in a buffer → matches and renders based on the video’s currentTime;
5. When a viewer switches language, the web client sends a signal, and the gateway switches the pushed language accordingly.

2.3 Key Design Principles

• Separation of transport: subtitles and A/V streams are transmitted independently, without interfering with each other;
• Player‑side alignment: all synchronisation uses the video player’s currentTime as the sole reference;
• On‑demand push: the server pushes only the language currently selected by the viewer, saving bandwidth;
• Fault‑tolerant design: reconnect, retransmission, and graceful degradation ensure usability under poor networks.

3. Detailed R&D Plan

3.1 Server Side – Subtitle Distribution Gateway

3.1.1 Technology Stack

• Language/Framework: Go + gorilla/websocket (high concurrency) or Node.js + ws (faster development)
• Protocol: WebSocket (RFC 6455)
• Deployment: Kubernetes cluster, supporting horizontal auto‑scaling

3.1.2 Core Functional Modules

3.1.3 Message Protocol (Server → Client)

{
  "type": "subtitle",
  "version": "1.0",
  "data": {
    "id": "sub_20260813_001",
    "language": "zh-CN",
    "text": "Hello, welcome to the live stream",
    "pts": 12345678,          // milliseconds
    "start_time": 12345678,
    "end_time": 12345900,
    "duration": 222,
    "is_final": true,
    "confidence": 0.96
  }
}

3.1.4 Client → Server Signaling

• Authentication:

{ "type": "auth", "data": { "token": "xxx", "room_id": "live_001", "language": "zh-CN" } }

• Switch language:

{ "type": "switch_language", "data": { "language": "en-US" } }

• Pong response (heartbeat):

{ "type": "pong", "data": { "timestamp": 1723536000000 } }

3.2 Web Side – Subtitle Consumption & Sync Modules

3.2.1 Module Breakdown

1. WebSocket Client: handles connection, authentication, heartbeat, automatic reconnection (exponential backoff), and message reception.
2. Subtitle Buffer:
   • Stores subtitles in ascending PTS order;
   • Deduplicates by id;
   • Supports filtering by language;
   • Automatically cleans expired subtitles (data older than 10s beyond current playback time).
3. Sync Rendering Engine:
   • Driven by requestAnimationFrame (syncs with screen refresh);
   • Reads video.currentTime (in ms) each frame;
   • Finds a subtitle in the buffer with start_time ≤ currentTime ≤ end_time and matching language;
   • Updates the DOM only when the text changes, to minimise reflows.
4. Subtitle Overlay:
   • Absolutely positioned over the video;
   • Supports customisable styling (font, size, colour, background, position);
   • Supports multi‑line display.
5. Language Switcher UI:
   • Dropdown or button listing available languages;
   • On click, calls ws.switchLanguage(lang) and updates local language preference;
   • Provides an option to turn subtitles off.

3.2.2 Core Code Examples (Simplified)

WebSocket Connection Management

class SubtitleWS {
  constructor(roomId, token, onMessage) {
    this.ws = new WebSocket(`wss://api.example.com/subtitle?room=${roomId}&token=${token}`);
    this.ws.onopen = () => this.sendAuth(roomId, token);
    this.ws.onmessage = (e) => onMessage(JSON.parse(e.data));
    this.ws.onclose = () => this.reconnect();
  }
  sendAuth(roomId, token) {
    this.ws.send(JSON.stringify({ type: 'auth', data: { room_id: roomId, token, language: 'zh-CN' } }));
  }
  switchLanguage(lang) {
    this.ws.send(JSON.stringify({ type: 'switch_language', data: { language: lang } }));
  }
  reconnect() { /* exponential backoff */ }
}

Buffer & Sync

class SubtitleSync {
  constructor(video) {
    this.video = video;
    this.buffer = [];
    this.lang = 'zh-CN';
    this.currentText = '';
    this.renderLoop();
  }
  push(sub) {
    if (this.buffer.some(s => s.id === sub.id)) return;
    this.buffer.push(sub);
    this.buffer.sort((a, b) => a.pts - b.pts);
    this.cleanup();
  }
  getActive() {
    const now = this.video.currentTime * 1000;
    return this.buffer.find(s => s.language === this.lang && s.start_time <= now && s.end_time >= now);
  }
  renderLoop() {
    const active = this.getActive();
    const text = active ? active.text : '';
    if (text !== this.currentText) {
      this.currentText = text;
      document.getElementById('subtitle-overlay').textContent = text;
      document.getElementById('subtitle-overlay').style.display = text ? 'block' : 'none';
    }
    requestAnimationFrame(() => this.renderLoop());
  }
  cleanup() {
    const now = Date.now();
    this.buffer = this.buffer.filter(s => (now - s.pts) < 10000);
  }
  switchLanguage(lang) { this.lang = lang; }
}

3.3 Data Flow Integration (Critical)

The existing ASR/Translation service must add a new output destination to send subtitle data to the “Subtitle Distribution Gateway”. Options:

• If the ASR service already uses a message queue (e.g., Kafka), let the gateway consume the same topic;
• If the ASR service outputs via gRPC stream, add a gRPC client subscription;
• If subtitles are currently generated only on the publisher side, the publisher must report them to the server via a signaling channel, then the gateway distributes them.

Recommendation: Establish a unified subtitle data bus on the server side (e.g., Redis Streams). The ASR/Translation service writes data to it, and the gateway consumes and broadcasts.

3.4 Deployment & Scaling

• Stateless gateway – can be scaled horizontally. Use consistent hashing by Room ID or message queue partitions to optionally route connections of the same room to the same gateway instance.
• Monitoring metrics: connection count, message throughput, latency distribution (P99), error rate, reconnection rate.
• Alert rules: sudden drop in connections > 20%, latency P99 > 1s, error rate > 1%.

4. Development Considerations

4.1 Timestamp Alignment

• The subtitle PTS generated by ASR must use the same clock source as the video stream PTS (preferably using the live stream’s absolute time).
• On the web, video.currentTime returns seconds – convert to milliseconds for comparison with subtitle PTS.
• If A/V sync is an issue (common with HLS), ensure the subtitle PTS aligns with the video track PTS, not the audio track.

4.2 Poor Network & Packet Loss Handling

• WebSocket reconnection should use exponential backoff (1s, 2s, 4s…) to avoid reconnect storms.
• The server should retain the most recent 20–50 subtitles for retransmission after reconnection.
• Consider using WebSocket extensions (e.g., permessage‑deflate) to compress data and save bandwidth.

4.3 Performance Optimisation

• Use requestAnimationFrame for subtitle rendering – avoid setInterval which can cause stutter.
• Update the DOM only when the subtitle text changes, to minimise reflows.
• Limit the buffer to a maximum number of entries (e.g., 200) and clean up expired data periodically to prevent memory leaks.
• When many viewers switch language simultaneously, the gateway should handle the load smoothly to avoid traffic spikes.

4.4 UI/UX Design

• The subtitle overlay should have a semi‑transparent background, clear font, appropriate size, and be adapted for mobile and PC.
• The language switcher should be simple, with common languages placed prominently, and include a “turn off” option.
• Under poor network conditions or disconnection, provide clear feedback (e.g., “Subtitle connection lost, reconnecting…”).

4.5 Security

• All WebSocket connections must be authenticated via JWT, with the token containing room_id to prevent cross‑room subscription.
• Subtitle content should be filtered for sensitive words (if required).
• Limit the number of connections per IP or per token to prevent malicious attacks.

4.6 Compatibility

• WebSocket is well supported in mainstream browsers (Chrome/Firefox/Safari/Edge) and WebViews.
• Ensure requestAnimationFrame and video.currentTime behave consistently across mobile browsers.

5. Test & Verification Plan

5.1 Functional Testing

5.2 Performance Testing

5.3 Weak Network Testing

Use Chrome DevTools or Network Link Conditioner to simulate:

5.4 Compatibility Testing

5.5 Security Testing

• Bypass authentication: attempt to connect without a token or with a forged token – must be rejected.
• Privilege escalation: use a token from room A to access room B – must be rejected.
• Injection attacks: subtitle content containing scripts – must be filtered or escaped.

5.6 Monitoring & Observability

Post‑launch, continuously monitor:

• Business metrics: subtitle delivery rate, average latency, number of language switches, reconnection count.
• System metrics: gateway CPU/memory, WebSocket connection count, message queue backlog.
• Alert rules: sudden drop in connections > 20%, latency P99 > 1s, error rate > 1%.

6. R&D Schedule & Milestones

Total: 5.5 weeks (with some tasks parallelisable)

7. Risks & Mitigation

8. Summary & Recommendations

Based on the existing ASR and translation capabilities, this solution adds a “Subtitle Distribution Gateway” and web subtitle consumption modules to achieve real‑time multi‑language subtitle distribution and precise synchronisation via an independent WebSocket channel. The solution offers:

• Decoupling: subtitle transport is separate from the video stream, with no interference;
• Low latency: end‑to‑end < 800 ms, sync error < 50 ms;
• Flexible switching: language changes without stream re‑pull, smooth user experience;
• Scalability: stateless gateway supports large concurrency;
• Resilience: reconnect, retransmission, and graceful degradation are built in.

It is recommended to implement the core functionality first (display and switching), then enhance monitoring and fault tolerance. Close collaboration with the ASR/Translation team on data format and ingestion is essential to ensure seamless integration.

This report serves as the technical baseline for subsequent R&D implementation. Should any questions or requirement changes arise, the design can be adjusted accordingly.