Taobao MNN Team Open-Sources 0.3B Mobile Model, Achieving 350 Tokens/s Offline Decoding on iPhone 18 Pro

Taobao's MNN team has trained from scratch and open-sourced MNN-Pocket-0.3B, a ~300M-parameter on-device language model that runs fully offline on iPhone 18 Pro via the MNN engine, achieving an average decoding speed of 350 tokens/s. The INT4 deployment package is only 173MB, with peak memory below 300MB. This breakthrough indicates that on-device language model inference can now match cloud-level speed, paving the way for high-frequency edge AI applications.

Rather than simply compressing an existing large model, the team co-optimized model architecture, vocabulary, training data, post-training, and the quantized inference engine to maximize end-to-end efficiency. Official targets include power consumption of no more than 1% of battery per 100 inferences, and a DFlash2 speculative decoding draft model that boosts throughput up to 2.06x on specific tasks. On iPad Pro M5, peak Metal resident memory increment is about 228MB, demonstrating good cross-device adaptability.

This open-source achievement has substantial impact on the edge AI ecosystem: developers can deploy high-performance language capabilities in native mobile scenarios with an ultra-lightweight model, reducing cloud dependency, latency, and privacy risks. Meanwhile, the goal of matching 8B-level model performance in business scenarios, if widely validated, will shift on-device model design from 'compression' to 'native optimization', accelerating the migration of AI applications to the edge.