MemT 1.0 Released: First Memory-Native Model at ~300B Scale

By Luyu, reporting from Ao Fei Si QbitAI | Official Account: QbitAI

The AI community has recently collectively turned its attention to three letters: RSI.

Hinton’s RSI paper features 22 signatories, including Bengio, OpenAI Chief Scientist Pachocki, and Anthropic co-founder Jack Clark… essentially half of the AI industry is represented here.

As everyone shifts their gaze toward AI self-improvement, a more fundamental question worth asking emerges:

How does one improvement become the starting point for the next?

Effective methods need to be retained, and missteps need to be identified. The mechanism responsible for this is actually memory.

The conventional approach is simple: attach an external experience library directly to the AI, allowing operational methods and historical lessons to be saved and retrieved.

Memory Tensor, however, offers a completely different answer in this area of exploration—memory-native.

What does "memory-native" mean? It means turning the retention, updating, and invocation of experience into capabilities that the model can learn.

When memory takes root within the model, it constitutes a memory-native model.

Its latest open-source release, the MemT 1.0 series, provides a rather concrete path forward:

It encodes experience into internal dynamic states, learns how to write, update, and read these states through training, and finally allows these states to participate in subsequent computations via a memory branch.

Compared to external memory, not only are its capabilities undiminished, but its future expansion ceiling is also higher.

This also marks the first time a memory-native model has achieved engineering validation at a scale of nearly 300B parameters.

(Paper link: https://arxiv.org/abs/2610.26825)

Wow, memory is invisible and intangible, yet it has solidly become the foundation for model growth.

From early requirements for models to simply remember information, to effectively utilizing experience, demands on model memory have continuously escalated, pushing memory step-by-step from behind the scenes to center stage.

Putting Memory Inside the Model

MemT 1.0 directly released two models this time:

MemT-1.0-Nano has a total of 36B parameters with 3B active, based on the "Source God" Qwen3.6-35B-A3B; MemT-1.0-Flash breaks upward, reaching up to 288B parameters with 17B active, based on DeepSeek-V4-Flash which has undergone further post-training.

Their common change is the addition of a parallel memory computation pathway alongside the original attention mechanism within selected Transformer modules.

Let’s illustrate with a simple example.

For instance, when facing a new task, the external memory approach involves stuffing raw memory text, retrieval evidence, and even abstract methods back into the context, allowing the model to reuse experience.

Learning mechanisms can also be introduced into the retrieval and organization process.

However, memory-native models like MemT 1.0 choose to delve deeper inside the model:

Experience is encoded into updatable matrix states; when the model generates answers or actions, it reads these states through a dedicated memory branch. The read results are then fused with the original attention output, influencing subsequent generation.

The difference lies in how experience is represented and through what computational process it exerts its effect.

But the benefits are obvious: involving reading and writing in training leads to smoother method reuse. However, there is a cost: if precise deletion, explanation, and recovery are required, additional mechanisms must be added.

External text is certainly more convenient for inspection and tracing. Currently, both approaches have their respective pros and cons.

To truly see the difference, one must observe how the model performs after using experience.

First, let’s look at a scenario familiar to coders: fixing code.

The workflow typically goes: receive bug → locate code → make modifications. But if the problem isn’t reproduced initially or boundary conditions aren’t confirmed properly, failed fixes lead to cascading vulnerabilities, passed down from one version to the next.

Take SWE-bench Verified as an example. One task involves Django’s parse_http_date function, requiring a fix for parsing two-digit years.

The research team compared execution trajectories under three settings: no experience, textual experience, and memory-native.

Without experience, the Agent found the bug and quickly started modifying the parsing logic, only introducing targeted reproduction and verification in later steps. Both groups with experience first constructed reproduction tests before modifying the code.

Notably, MemT-1.0-Nano re-checked a boundary condition after the initial modification, corrected the implementation, and performed additional verification.

It is clear that experience changed the order of actions and participated in checks and corrections within the task.

Comparing on a larger sample, across 500 SWE-bench Verified tasks, each run independently three times, MemT-1.0-Nano achieved an average success rate of 69.73% at critical breakpoints, compared to 66.07% for the baseline without experience and 69.13% for the textual experience group.

The proportion of tasks succeeded in all three runs increased from 54.00% (without experience) to 62.80%, while the textual experience group was at 61.80%.

Looking at SWE-bench Multilingual tasks, under the same trigger settings, the memory-native group had an average success rate of 63.56%, compared to 60.00% for the no-experience group and 63.00% for the textual experience group.

In other words, these procedural experiences can be retained through memory-native mechanisms, performing comparably to explicit textual experience, with advantages primarily reflected in average success rates and reliability during repeated executions.

To put it more intuitively, let’s look at a pixel-art style maze exploration demo.

Same maze, same movement speed. On the left is an Agent without memory-native capabilities; on the right is MemT 1.0 with memory-native capabilities.

Round 1: Both sides need to explore unknown paths, and both will hit dead ends.

The difference appears after this round concludes: successful routes and forks to avoid are organized, archived, and loaded into MemT 1.0’s memory, providing reference for the next round of exploration.

In the second round, starting again from the origin, the left side still repeats trial-and-error at the fork, while the right side leverages experience from the previous round to avoid dead ends already explored and reach the destination via an effective path.

Video link: https://mp.weixin.qq.com/s/DTdaGUk8o0dwkx1wxnnbYw

The path established in the previous round directly influences how the next round is navigated. This is a concrete manifestation of why memory-native approaches are worth pursuing.

Let’s look at another scenario closer to our daily office work: information updates.

An exhibition was originally scheduled for Saturday at 3 PM, then officially changed to Sunday at 4 PM. Later, someone suggested moving it to Monday, but this decision has not yet been finalized.

This kind of incremental information with repeated changes is actually quite troublesome for AI.

It is not enough to simply remember the last date mentioned; the model must also distinguish between formal decisions and casual suggestions, while retaining unchanged content.

After multiple rounds of dialogue, MemT-1.0-Nano still successfully grasped the situation, correctly answering the original time, the officially revised time, and confirming that other arrangements remained unchanged.

In the comprehensive evaluation across 11 memory benchmarks, Flash and Nano also demonstrated outstanding performance.

With average scores of 50.6 and 47.0 respectively, they ranked as the top two among the parametric memory methods evaluated, leading other models by a significant margin.

However, to be frank, text-based memory systems still hold an overall advantage, particularly in recalling details from long histories.

What MemT 1.0 delivered today is merely an initial computational implementation of the new memory-native paradigm. Yet its actual performance already rivals that of text-based memory, suggesting infinite potential as development continues.

Where exactly does "memory-native" reside?

So the question arises: if experience is not retained in the task context, where does the model retrieve and utilize it?

This is precisely the focus of MemT 1.0’s architecture.

The native path provided by MemT 1.0 first encodes relevant experiences into internal dynamic states. When generating subsequent actions, the current query reads this state via a dedicated memory branch, and the result is fused with the original attention output.

In specific parametric implementations, MemT 1.0 preserves the base model’s original computational flow while adding a native memory branch within selected Transformer modules. Broken down, this includes three layers:

• Backbone weights: Carry the model’s existing general capabilities and remain frozen during memory training.
• Hyper-memory: Trainable parameters such as read/write projections and gating mechanisms determine specifically how memory is encoded, updated, and read.
• Local memory: Dynamic matrix states store newly added information from interactions.

By making this distinction, the model can incorporate new experiences without needing to fine-tune the entire backbone every time; only the dynamic states corresponding to the task trajectory change.

The read/write mechanism of this system is also highly sophisticated.

During writing, hidden states undergo projection to generate memory keys and values; erase, write, and forget gates jointly regulate associations within the existing matrix. During reading, the current input is mapped to a memory query to retrieve relevant information, which then passes through normalization and output gating to participate in subsequent generation.

The update mechanism employs token-by-token generation based on Gated DeltaNet-2, with stabilization measures applied.

(Insider info: The author of Gated DeltaNet-2 also participated in related discussions.)

It triggers after each round ends, using gating to adjust existing memory associations, while within a round, states are adjusted sequentially by token.

Ablation studies confirmed this point: replacing token-by-token updates with whole-round aggregation caused Nano’s average score on memory benchmarks to drop from approximately 47.0 to 41.0.

This six-point gap sufficiently demonstrates that continuous token-level updates better handle memory decay and timely correct intra-round associations.

But what happens when the volume of memory content expands continuously and exceeds the model’s capacity?

At this point, MemT 1.0 relies on MoE (Mixture of Experts) to propose the MoS (Mixture-of-Storage) solution.

MoS expands a single memory-native module into multiple memory experts, learning storage and usage routing separately. It adopts dense fusion, where all memory experts participate in computation, and routing weights determine their respective contributions.

This differs significantly from the sparse expert routing used in the base MoE architecture.

However, structure is just the starting point.

To enable the model to truly learn how to use this pathway, the team constructed a 34 billion token memory training corpus, comprising 20 billion tokens for continued pre-training and 14 billion tokens for post-training.

The format for continued pre-training is uniform: long texts are first split into continuous segments written into memory, and then the model uses the memory state to answer queries.

After processing each segment, the model updates the state for use in the next segment.

Post-training further aligns task outcomes. The data includes three categories: memory-native language modeling data, memory storage and utilization data (such as summarization, continuation, compression, etc.), and task data specifically designed to strengthen long-term memory.

One point here deserves special mention: the previously frozen backbone acts as the teacher model, seeing the complete historical information, while MemT 1.0 serves as the student, seeing only its own memory state. Off-policy distillation is then used to align the student’s distribution with the teacher’s.

Synthesizing these elements, the core of memory-native design becomes clear: encoding history into states is only the first step; how to retain, update, and read them constitutes the entirety of a memory-native model.

The value of this approach needs to be judged comprehensively by considering experience reuse effectiveness, resource overhead, and long-running conditions.

From Continual Learning to RSI: Memory Opens Space for Capability Growth

So, some friends might ask: Why use memory-native approaches?

Existing external memory solutions are already mature and have good practical cases. In comparison, memory-native approaches are just getting started, and their effectiveness is roughly similar.

Not quite! Memory-native architecture has its own unique advantage:

It makes the representation, reading, and writing of experience itself a target for expanding model capabilities. This is what the MemT team refers to as "Memory Scaling"—finding a third curve for performance growth beyond inference scaling.

Image generated by AI

Previously, the utility of past experience was judged solely by whether retrieval improved task outcomes. With MemT 1.0, we can go further and ask: Can the same experience be utilized more effectively by changing its internal representation, update rules, or read mechanisms?

This introduces a new set of variables that can be optimized through training and architectural design to enhance memory effectiveness.

Admittedly, this does not yet prove that the long-term ceiling of native memory exceeds that of external memory. However, it demonstrates that training and capacity expansion around internal states can yield additional capability gains.

The next step is to verify whether these gains persist beyond the current training distribution, maintaining their value in longer interactions and novel tasks.

Returning to the concept of Recursive Self-Improvement (RSI) mentioned earlier.

Once memory-native architectures are fully realized, they will significantly benefit breakthroughs in continual learning and RSI.

As is well known, RSI consists of three key components: memory, feedback, and validation/update mechanisms. Memory ensures that effective results are retained, while the latter two help determine which changes are worth incorporating into the next iteration.

What MemT 1.0 is currently exploring is precisely the step of inheriting and reusing experience. It validates that experience can guide actions via native memory, filling the first—and arguably most critical—gap at the model level.

Image generated by AI

If we push further, experiences accumulated in local memory could potentially be distilled into super-memory parameters through subsequent training, which in turn would improve how the model manages its own memory.

Once this loop is established, the system might achieve an intelligence explosion across tasks by leveraging recorded results and methods—the ultimate goal of RSI.

These are still theoretical benefits. On the application side, memory-native architectures also facilitate enterprise experience accumulation.

Currently, when discussing enterprise AI, the industry focuses primarily on the early deployment stage, asking questions like: Is the current model usable? What can it achieve?

While this perspective is valid, enterprises realize after adoption that experience accumulation is equally important.

Long-term enterprise use involves project rules, operational sequences, and validation methods. If every new task requires re-stating similar constraints and repeating similar failures, previous investments struggle to translate into subsequent returns.

Therefore, for such repetitive workflows, how many times validated experience can be reused should enter enterprise AI selection criteria alongside single-instance model performance.

Memory-native architectures provide a model-side mechanism to support this.

With the same 128K long history, Nano’s end-to-end inference speed is approximately 1.86x that of full-context solutions. For the same history length, the persistent state footprint per trajectory is one-sixty-sixth that of the full-context KV Cache.

In other words, this delivers immediate cost control benefits.

State budgets for long sessions become easier to estimate, and multiple queries can reuse pre-built states, amortizing preparation overhead.

However, to be fair, memory-native approaches do not necessarily guarantee higher task quality than external memory. Yet, in scenarios with comparable quality, they are more likely to steadily reduce rework through underlying experience reuse—a tangible benefit for enterprise operations.

In essence, memory-native and external memory are not mutually exclusive adversaries. Collaborative division of labor between the two may generate greater synergistic benefits.

For queries requiring exact original text, sources, or versions, external records remain valuable. For invoking experience for subsequent execution, the role of native mechanisms can continue to be tested.

Regardless, once effective progress can be retained and repeatedly validated in new tasks, continuous self-improvement finally has a foundation for accumulation.

Through comprehensive hierarchical memory management and scheduling, global read/write efficiency optimization becomes achievable. Today, the release of memory-native models has made this engineering deployment and implementation possible.

The value of memory in driving continuous self-improvement will gradually emerge through such long-term validation.

GitHub Link: https://github.com/MemTensor/MemT-1.0 HuggingFace Link: https://huggingface.co/collections/MemTensor/memt-1