EdgeRock-4B Open-Sourced: Cuts KV Cache by 99%, Boosts Long Inference Throughput 6x

By Yunzhong, reporting from Ao Fei Si QbitAI | Official Account: QbitAI

A paper published by Google this year was enough to cause a stir in the stock prices of major memory chip manufacturers.

The paper claimed that KV Cache could be compressed by at least 6x.

As we know, large model inference is increasingly memory-intensive, and KV Cache represents a significant portion of that overhead.

When Google suddenly announced such substantial compression, the stock prices of storage giants like Micron dropped accordingly.

Yet just months later, someone has taken an even more aggressive cut to KV Cache.

This time, mere compression wasn't enough.

Context length increased from 4K all the way to 256K, yet KV Cache usage remained unchanged.

At a 256K context length, it stays fixed at 72 MiB. Compared to the source model, KV Cache usage is reduced by 99.1%.

Even more impressive, a single 80GB GPU can simultaneously run 512 concurrent long-inference streams, achieving an output throughput of 12,124 tokens/s—6.42 times that of the source model.

Furthermore, reasoning capabilities are largely preserved. The average pass@1 rate on AIME 2024–2026 reached 82.9%, whereas bounded-state models among the competitors, even those scaled to 14B parameters, achieved a best-case score of only 18.8%.

This is EdgeRock-4B, recently released and open-sourced by Rockchip Matrix (under Rockchip Technology), in collaboration with HKUST, CMU, and other universities.

It brings a fundamental shift to long-context inference:

No matter how long the inference, memory usage does not increase.

Model weights and inference code are currently open-source, supporting both text and image reasoning.

Why Does Longer Inference Cost More?

Modern models often require tens of thousands of tokens for "thinking," while Agents can operate continuously over dozens of rounds, constantly stuffing search results, code, and tool-call logs into the context.

As the model works harder, VRAM runs out first.

The culprit is KV Cache.

Most current models store inference history in KV Cache, adding more data with every generated token.

Take Qwen3.5-4B as an example: each new token adds 32 KiB to the KV Cache. At 64K inference, a single sequence consumes 2 GiB; at 256K, it hits 8 GiB.

This issue is particularly critical in the era of reasoning models.

Models are increasingly prone to "deep thinking," often outputting tens of thousands of tokens. Agents running dozens of tool-call rounds cause contexts to balloon rapidly.

The longer the inference, the more VRAM it occupies, reducing the number of requests a single card can serve concurrently.

On edge devices, KV Cache must also compete with perception and control modules for already limited memory.

Industry solutions generally fall into two categories.

One approach is to "slim down" each token:

Multi-head Latent Attention (MLA) compresses KV, quantization methods like KIVI reduce precision, and Google's TurboQuant follows this path.

However, these only lower the cost per token; the cache still grows linearly with inference length.

The other category consists of bounded-state models:

Recurrent models, state-space models, and sliding-window models have inherently fixed memory footprints.

The problem is that they often fail to retain distant details required for subsequent reasoning, significantly degrading reasoning capability.

Thus, the true challenge becomes:

Can we retain the critical details needed for reasoning within a fixed memory footprint?

EdgeRock's answer is: Yes.

One Card, 512 Concurrent Long-Inference Streams

Let's look at the most intuitive metric: throughput.

The test setup involved 32K input and 32K output, with all models running on a single Hopper 80GB GPU, using their respective fastest measured batch sizes.

The result: EdgeRock-4B opened up to 512 concurrent streams, achieving an output throughput of 12,124 tokens/s.

The source model, Qwen3.5-4B, peaked at a batch size of 64 with only 1,889 tokens/s. Gemma 4 E4B, MiniCPM5-2B, and LFM2.5-2.6B also hovered between 1,700 and 2,200 tokens/s.

Why is the gap so large? The math is straightforward.

After completing 32K input plus 32K output, Qwen3.5-4B requires 2 GiB of KV Cache per sequence.

A single 80GB GPU fills up quickly with just a few dozen concurrent sequences, forcing additional requests to queue.

In contrast, when EdgeRock-4B runs in batches, the total historical state saved per sequence is fixed at 97.76 MiB, regardless of length.

To put it in perspective: if Qwen3.5-4B were to run these same 512 sequences concurrently, the KV Cache alone at generation completion would require 1 TiB—equivalent to the VRAM of roughly thirteen 80GB GPUs.

EdgeRock fits everything on one card, peaking at 61.26 GiB, including model weights.

Now consider scalability.

Across 16 workload combinations ranging from 4K to 32K inputs and outputs, as batch size increased from 32 to 512, weighted decoding throughput rose from 4,128 to 12,497 tokens/s, while peak VRAM usage never hit the single-card limit.

Single-request latency is also competitive.

Models where KV Cache grows with inference share a common flaw: they get slower as they write more.

This is because generating each new token requires reviewing the ever-expanding KV Cache.

EdgeRock's computational cost per token does not scale with context length.

After processing 128K input and generating another 128K tokens, its speed remains steady at 233.7 tokens/s throughout.

Qwen3.5-4B, however, drops from around 169 to 151.3 tokens/s.

The Longer the Context, The Cheaper It Gets

Multiplying throughput several times over essentially means dividing costs by the same factor.

On the same card, under the same load, generating the same number of tokens:

• With 4K input, EdgeRock requires only 24.9% of the GPU time compared to Qwen3.5-4B;
• With 32K input, it requires only 15.6%.

If billing by GPU-hour, the compute cost per token decreases proportionally.

Averaged across four input lengths, EdgeRock’s throughput is 4.94 times that of Qwen3.5-4B and 7.18 times that of LFM2.5-2.6B.

More critically, the trend shows that the longer the context, the greater the performance gap.

As context length increases, Qwen3.5-4B’s KV Cache grows significantly, whereas EdgeRock’s remains constant.

Beyond memory savings, fixed memory allocation offers an often-overlooked benefit: determinism.

The memory requirement for each request can be calculated precisely before inference begins.

• Inference services can plan capacity based on request count, eliminating the need to guess how long each request will take;
• On-device deployments allocate memory once upfront, ensuring that even lengthy inferences do not exhaust available resources;
• Agent history memory no longer scales with the number of tool calls;
• Specialized chips can store all historical state directly in on-chip memory.

In an era where memory is increasingly precious, this efficiency becomes more valuable by the day.

Memory Savings Without Sacrificing Intelligence

At this point, industry experts might ask: RNNs and Mamba have already achieved constant memory usage—what makes this special?

The distinction lies in maintaining reasoning capabilities while reducing memory consumption.

The team benchmarked EdgeRock-4B against 16 bounded-state models, covering architectures such as GDN, RWKV, and Mamba, including three models at the 14B scale.

On AIME 2024–2026, EdgeRock-4B achieved an average pass@1 rate of 82.9%. In comparison, the strongest baseline model, RWKV-7 G1j 13.3B, scored only 18.8%, while AHN-GDN Qwen2.5 14B reached 11.4%.

Across 29 text benchmarks, EdgeRock-4B ranked first among bounded-state models in 24 categories. Its equal-weighted average score across all 29 benchmarks exceeded that of the strongest baseline, AHN-GDN Qwen2.5 14B, by 23.8 percentage points.

Compared to its source model, EdgeRock-4B largely retains its capabilities:

• On AIME, it maintains 96.3% of Qwen3.5-4B’s average pass@1 rate; on MATH-500, it achieves 97.5%;
• Among the 29 text benchmarks, it outperforms Qwen3.5-4B in 8 categories, including MMLU-Pro, SuperGPQA, HumanEval, and MBPP;
• Across six vision benchmarks, it scores an average of 84.9%, retaining 95.3% of Qwen3.5-4B’s performance.

These results place EdgeRock-4B on par with leading small-scale models whose memory usage scales with inference length.

The Secret: Anchoring Memory with "Rock"

So, how does EdgeRock achieve this?

The core innovation is a new memory mechanism called Rock Memory Anchors (RMA).

Inspired by rock anchors used in climbing, the design aims to fixate critical information required for reasoning.

Just as climbers secure themselves with reliable anchor points on a rock face, RMA "pins" essential details encountered during reasoning into a fixed-capacity anchor library, allowing direct retrieval when needed.

Specifically, EdgeRock divides the history of each sequence into three components, each serving a distinct purpose:

• Recurrent states summarize the evolving context, capturing the general gist;
• Local windows cover the most recent 1024 tokens, handling immediate context;
• Anchor libraries preserve the most valuable details from earlier history, recording key facts.

The process for selecting anchors works as follows:

A writer generates a candidate record for each token, which is then scored by a rating network.

Every time 256 candidates accumulate, they are compared against existing records in the anchor library. Only the top 1024 highest-scoring records are retained; the rest are discarded.

This process occurs independently for each of the four KV groups.

Retained anchors remain independent and immutable. Consequently, even if their original context has slid out of the local window, subsequent reasoning steps can still access them directly.

In essence, the model learns not to "remember everything," but to "remember only what matters."

By combining local windows with anchor libraries, the attention KV cache for each sequence is fixed at 72 MiB. This is the source of the constant memory usage.

Architecturally, EdgeRock-4B is derived from Qwen3.5-4B: it retains 24 Gated DeltaNet layers, replaces 8 full attention layers with sliding window attention (window size 1024), and adds an independent RMA module.

The entire model was converted from Qwen3.5-4B through staged training, eliminating the need for pre-training from scratch.

Do the anchors actually work?

The team conducted an experiment: during inference, all 8 RMA modules were disabled, while recurrent layers and local attention continued to operate normally.

Using 256 segments of long-context inputs ranging from 16K to 130K tokens, with four questions per segment covering retrieval, state updates, cross-document information synthesis, and variable binding:

• With RMA enabled, the model answered 964 out of 1024 questions correctly, achieving a 94.1% accuracy rate;
• With RMA disabled, it answered only 4 questions correctly, resulting in a 0.4% accuracy rate.

Long-range recall effectively dropped to zero. This demonstrates that EdgeRock relies on anchors to retrieve distant information for answering queries.

Furthermore, each anchor records its source position, allowing developers to inspect exactly what the model chose to remember.

Will Memory Costs Continue to Drop?

EdgeRock was developed by Rockchip Matrix, a subsidiary of Rockchip Technology, in collaboration with universities including HKUST and CMU. The model is currently open-source and available for commercial use.

The team revealed that EdgeRock variants with different parameter sizes are in development.

For robots requiring long-running operations, this mechanism holds significant potential.

After all, robots simultaneously perceive environments and plan actions. Tasks may run for hours or longer, making unbounded growth of historical caches impractical.

If memory usage during inference can be determined in advance, edge deployment becomes considerably simpler.

Whether memory costs will continue to fall is left to market forces.

But one thing has certainly changed: no matter how long a model thinks, its memory footprint will no longer expand.

Paper: https://rockmatrix.ai/documents/edgerock-report-zh.pdf GitHub: https://github.com/rockmatrix-ai/EdgeRock Hugging Face: https://huggingface.co/rockmatrix-ai/EdgeRock