Today for AI
HOT RADAR
computeHEAT 0.0°

AI CURATED HOT STORY · 10/10/2026

Tsinghua's TokenRouter Boosts LLM Serving Throughput by 64x via Per-Token Model Routing

1 reports archived0 independent sourcesupdated 10/10/2026, 11:32:00
Synthesis & Latest Updates

Tsinghua University introduced TokenRouter, a serving system that dynamically routes between small and large models at the token level to eliminate blocking delays in traditional frameworks. By enabling independent model deployment with shared KV caches and request buffering for optimized batching, it achieves throughput improvements ranging from 2.01x to 64.15x across five routing strategies.

LATEST/Tsinghua University introduced TokenRouter, a serving system that dynamically routes between small and large models at the token level to eliminate blocking delays in traditional frameworks. By enabling independent model deployment with shared KV caches and request buffering for optimized batching, it achieves throughput improvements ranging from 2.01x to 64.15x across five routing strategies.

Coverage timeline

Total 1 reports · Latest first
  1. Rohan Paul (@rohanpaul_ai)T2·78 pts

    Tsinghua's TokenRouter Boosts LLM Serving Throughput by 64x via Per-Token Model Routing

    • TokenRouter enables dynamic model routing at the token level, breaking the single-model-per-request limitation.
    • Decoupling model servers while sharing KV caches eliminates synchronization overhead where fast models wait for slow ones.
    • Experiments show up to 64.15x throughput improvement over existing state-of-the-art configurations across various routing strategies.