Growing Memory into the Model: MemT 1.0, a Near-300B Memory-Native Model, Makes Note-Taking a Trainable Parametric Capability

Growing Memory into the Model: MemT 1.0 and the Memory-Native Architecture

Part One: When a Model Can Only Pass Knowledge “in a Letter to the Next Generation”

Model capability has surged for two years, but one gap has never truly closed — memory.

Rules, methods, and lessons learned can sit in external notes or a vector store today, and the model can “see” them every time; but the moment a fresh inference run begins, the weights are unchanged and the model starts over like it is clocking in for the first time. One developer put it sharply: today’s models pass on experience “by writing letters to the next generation” — however detailed the letter, the reader itself has not grown stronger.

On October 20, 2026, the Shanghai-based MemTensor team released and open-sourced MemT 1.0, the first memory-native foundation model, which for the first time encodes history into a persistent, per-token-updatable internal memory state that continuously participates in generation. It ships in two sizes:

• MemT-1.0-Flash: 288B total / 17B activated, 336M dynamic memory, based on a further post-trained DeepSeek-V4-Flash;
• MemT-1.0-Nano: 36B total / 3B activated, 10M dynamic memory, based on Qwen3.6-35B-A3B.

It is the largest open parametric-memory model to date, and the first near-300B-scale engineering validation of the memory-native route. Weights are on Hugging Face and ModelScope; the paper is arXiv 2610.26825; code is at MemTensor/MemT-1.0.

Against the backdrop of the RSI (recursive self-improvement) discussion sparked by Hinton’s 22-author paper (Bengio, OpenAI chief scientist Pachocki, Anthropic co-founder Jack Clark among them), MemT attacks a prerequisite question: how does one improvement become the starting point of the next? The underlying capability is memory. Without efficient retention, updating, and retrieval of experience, recursive self-improvement has nothing to recurse on.

This article dissects: how does memory-native differ fundamentally from the bolt-on memory we know? What “memory branch” does MemT add inside the Transformer? What do the Gated DeltaNet-2 per-token update, the four gates, and Mixture-of-Storage each solve? And what do its results on 11 memory benchmarks and SWE-bench agent tasks reveal about its limits?

Part Two: Three Old Memory Types and the Memory-Native Shift

To grasp MemT’s trade-offs, survey the three existing memory types and their read/write economics:

Mainstream long-term-memory engineering over the past two years has mostly been bolt-on type three: stuff raw memories, retrieved evidence, even abstracted methods back into the context for reuse. RAG and MemGPT-style systems live here. They solve real problems but never change the model — the weights stay fixed and the notes are re-read every time.

Memory-native takes a different path: history is no longer re-injected as text but encoded into a persistent, fixed-size, updatable internal state matrix the model reads and writes directly during every token’s generation. Bolt-on memory is “carry an endless notebook and read it before each problem”; memory-native is “the knowledge actually grew into the brain.”

The conceptual shift is key: turn memory from a data problem into an optimization problem.

• If write/read are only forward computation (compress text into state, read it out), the model learns at most “how to compress text”;
• MemT lets the error gradient from answer prediction flow back along the memory readout and then along the earlier storage operations. The mistakes from one exam literally determine what notes you should keep and how.

As the paper puts it, the “ownership” of memory lands on the model side for the first time. At deployment, a memory state is bound to a user/task trajectory and serialized, saved, and restored across sessions; external storage merely hosts the states and raw materials, while forming, updating, and using memory all happen through the model’s internal mechanisms.

A full-architecture picture:

                    Current token hidden state h
                                  |
              +-------------------+-------------------+
              |                                       |
              v                                       v
   +---------------------+              +-----------------------------+
   |  Standard attention  |              |   Memory attention (new)    |
   |      path           |              |  memory query -> read matrix|
   +---------------------+              |  -> normalize -> output gate |
              |                         +-----------------------------+
              |                                       |
              +-------------------+-------------------+
                                  v
                     Fuse (sum) -> next layers
                                  ^
                                  |  write at end of turn
        +-------------------------+--------------------------+
        | Dynamic local memory (starts at 0 per trajectory)    |
        |  project to key/value; erase/write/forget gates,     |
        |  updated per token                                    |
        +-----------------------------------------------------+
        | Hyper memory (projection + gate params; frozen online)|
        +-----------------------------------------------------+

Part Three: A Memory Branch in Parallel with Attention, and Two Memories

In selected Transformer blocks, MemT adds a memory branch in parallel with the existing attention. The current input flows through the standard attention path while another path issues a “memory query” to read historical state; the two outputs are summed and fused for downstream computation. Information from prior turns then shapes the current token without re-feeding the raw history.

Memory has two components:

1. Dynamic local memory: a state matrix compressing the concrete experiences of this conversation/task trajectory. It initializes at a zero matrix for each new trajectory and updates through interaction.
2. Hyper memory: the trainable parameters providing projection matrices and gates — deciding how to write, read, and forget. It learns during training and stays frozen during online interaction. What changes is the experience; what stays fixed is the note-taking method.

The division of labor is elegant: taking on new experience only updates the trajectory’s dynamic state, with no need to fine-tune the backbone each time. “Updating knowledge” is decoupled from “updating capability” — continuous accumulation without paying a full fine-tune per new experience.

3.1 Writing: Projection + Three Gates

At the end of a turn, input and response hidden states are projected into memory keys and values, then three gates shape the state token by token:

• Erase gate: softens old associations, deciding what gets overwritten;
• Write gate: injects new information with a learned strength;
• Forget gate: controls retention, handling natural decay.

On read, the current input maps to a memory query, retrieves from the matrix, and after normalization and an output gate, the result is added to the standard attention output. The output gate governs how strongly memory should influence the current answer — keeping irrelevant old experience from hijacking it.

Simplified Python conveys the math intuition of the erase/write/forget per-token update (teaching sketch, not official code):

import numpy as np

def sigmoid(x):
    return 1.0 / (1.0 + np.exp(-x))

class ToyMemoryState:
    """
    Teaching version: M is a fixed-size memory matrix.
    Each token updates M via erase / write / forget gates,
    in the spirit of a Gated DeltaNet associative-memory write.
    """
    def __init__(self, d_k=32, d_v=32):
        self.M = np.zeros((d_v, d_k))          # dynamic local memory, starts at 0

    def update(self, k, v, W_erase, W_write, W_forget, x):
        e = sigmoid(x @ W_erase)               # erase gate
        w = sigmoid(x @ W_write)               # write gate
        f = sigmoid(x @ W_forget)              # forget gate

        self.M = self.M * f                              # 1) forget: decay old memory
        self.M = self.M * (1 - np.outer(np.ones_like(v), e))  # 2) erase (simplified)
        self.M = self.M + w * np.outer(v, k)             # 3) write key->value
        return self.M

    def read(self, q):
        return q @ self.M.T                    # query the matrix for related value

# Simulate a trajectory: write one fact, then many distractors, then recall
rng = np.random.default_rng(1)
d = 32
mem = ToyMemoryState(d, d)
W_e, W_w, W_f = [rng.normal(size=(d, d)) * 0.1 for _ in range(3)]

k1, v1, x1 = rng.normal(size=(3, d))           # turn 1: "meeting moved to Wednesday"
mem.update(k1, v1, W_e, W_w, W_f, x1)
for _ in range(20):                            # 20 turns of irrelevant chatter
    k, v, x = rng.normal(size=(3, d))
    mem.update(k, v, W_e, W_w, W_f, x)

out = mem.read(k1)
cos = float(out @ v1 / (np.linalg.norm(out) * np.linalg.norm(v1) + 1e-9))
print(f"cosine of recall with early fact after 20 distractors: {cos:.3f}")

The key point: the matrix M stays fixed in size and does not grow with the conversation, yet gated writes keep an early association queryable after heavy interference — the core difference between bounded, updatable state and unbounded full context.

Part Four: Gated DeltaNet-2 Per-Token Updates — Why the 6-Point Gap Matters

The update builds on Gated DeltaNet-2 (GDN-2), with a stabilization fix, replacing the token-selection scheme of the team’s earlier Metis system.

A central design choice is update granularity:

• Within a turn: state adjusts in token order, per token;
• At turn end: gated updates fuse the turn’s information into existing memory.

Why per-token rather than one aggregation per turn? The ablation on MemT-1.0-Nano is decisive:

• Full model (per-token stable update): memory average 47.0;
• Remove stabilization, revert to plain GDN-2: 46.4 (close — stabilization is a small gain);
• Swap per-token for whole-turn aggregation: drops to 41.0.

A 6-point gap says continuous token-level updates handle memory decay and correct within-turn associations better. Waiting until a turn ends to write loses the transient associative signals in the middle.

The stabilization fix is worth noting: MemT makes the erase operation symmetric by using square-root-scaled keys and erase gates. In the restricted “erase-only, identity decay, no write” case, the map contracts in one direction and does not expand in orthogonal directions, keeping the state numerically stable under repeated updates — an engineering insurance policy required to push a research associative-memory mechanism to 288B scale.

Part Five: When Memory Runs Out of Capacity — Mixture-of-Storage

A fixed-size matrix eventually has a capacity ceiling. Borrowing from MoE, MemT introduces MoS (Mixture-of-Storage).

It expands one memory-native module into several “memory experts,” each learning storage and usage routing. But unlike the backbone MoE’s sparse expert routing, MoS uses dense fusion: every memory expert participates, with routing weights determining contributions, rather than activating only a few.

     Backbone MoE (sparse)                 MoS memory (dense)
  ------------------------------        --------------------------------
  Top-k experts participate            All memory experts participate
  Suited for compute division          Suited for "memory must not miss"
  Experts specialize, disjoint          Experts share addresses, dense blend

Why dense for memory? Intuitively, the cost of forgetting one experience may exceed the cost of computing a few extra experts. With sparse routing, if a fact is not routed to a responsible expert, it may be absent when needed; dense fusion lets every expert’s memory vote on the current answer, and capacity scales independently by adding experts without touching the backbone. In a small controlled experiment, four memory experts beat two at matched later checkpoints — the paper honestly flags this as limited evidence, not a scaling law.

To make the model actually learn to use the new branch, the team built 34B tokens of memory training data (20B continued pre-training + 14B post-training) in two stages: continued pre-training first, then post-training where the memory parameters learn to approximate a frozen, full-context backbone teacher — the student receives history only through its memory state yet must match the teacher’s answers. It is a distillation of “read everything” into “remember what matters.”

Part Six: Empirical Results — Memory Benchmarks, Agentic Coding, and Long-History Savings

Three result sets matter.

First, 11 memory benchmarks. Flash averages 50.6, Nano 47.0, ranking first and second among evaluated parametric-memory methods; versus the strongest parametric baseline Temp-LoRA (38.1), they lead by 12.5 and 8.9 points — a steep margin — while largely preserving general capability.

Second, SWE-bench agent coding. An execution framework triggers experience writes at key breakpoints, storing procedural experience in native memory (the same guidance in textual context is the control). On SWE-bench Verified’s 500 tasks, each run three times:

• Nano average success at key breakpoints: no-experience baseline 66.07% -> 69.73%;
• All-three-runs success (pass³, measuring consistency): 54.00% -> 62.80%;
• SWE-bench Multilingual (300 tasks, 9 languages): 60.00% -> 62.89%/63.56%; pass³ 45.33% -> 53.67%/55.33%.

The gains are not earth-shaking, but the ~9-point pass³ jump shows something more valuable: memory-native improves not just the mean but repeat-execution consistency — the model, leaning on internalized procedure, has fewer “got it this time, failed next time” flukes.

Third, long-history efficiency. On dual GPU, single concurrency, fixed 128-token generation, at 128K history Nano runs about 1.86x the end-to-end speed of full-context, and 3.99x at 256K; the persistent memory state is about 1/66 the size of the full-context KV cache. That is bounded fixed state beating unbounded KV — the longer the history, the bigger the advantage.

Part Seven: Limits — Memory Is Not a Transcript, and Text Still Wins Somewhere

Cool the enthusiasm. The paper itself and third-party reviews (e.g., alphaXiv) draw clear boundaries:

1. Explicit textual memory still wins on some benchmarks. On LoCoMo, Flash scores 65.1 and Nano 57.5, while the textual memory system MemOS reaches 85.7/83.4. Where verbatim recall and citable provenance matter, “reading the notes” beats “remembering.”
2. Precise deletion, explanation, and recovery are harder. Once experience is fused into the matrix, auditing “what exactly did it remember about me” or surgically erasing an item for compliance needs extra machinery; external text memory is naturally inspectable and traceable. This is the same trust problem as the “outputs cannot distinguish sources” issue in last week’s ART-enzyme dispute.
3. The stability proof has scope. The contraction guarantee covers the restricted erase-only case, not the entire memory system under every configuration; the MoS expert-count gain is a small experiment, not an established law.

The accurate framing is not “memory-native replaces RAG/text memory” but a division of labor: parametric native memory excels at implicit method reuse, cross-task synthesis, and long-history storage/speed savings; explicit text memory excels at precise facts, auditability, and deletion. Production will likely be hybrid — what should grow into the brain grows in; what must stay in the ledger stays there.

Part Eight: A Necessary (Not Sufficient) Condition for RSI

Return to RSI. Recursive self-improvement requires one improvement to sediment into the next, which needs at least three abilities: forming effective methods, retaining them, and invoking them on new tasks. Bolt-on text memory showed retention and retrieval can be done with data engineering, but the model does not grow on that path; MemT shows memory can exist as a trainable, differentiable, forward-computation-participating intrinsic capability, turning “learning from experience” from a prompt trick into part of the architecture.

It is necessary, not sufficient: with an accumulable memory base, a model can begin to integrate experience across tasks and acquire experience autonomously; but genuine RSI also needs more reliable self-evaluation, an improvement-verification loop, and the memory interpretability/deletion machinery noted above.

For deployment, the team points to clear use cases: customer service (per-customer history and preferences), software R&D automation (reusing debugging lessons across tickets, validated by SWE-bench), and personalization — plus a subtler commercial implication: a deployer can let an open model quickly grow unique memory that “knows only your business and keeps absolute secrets,” with memory ownership and data never leaving its own trajectory. Next steps: longer-term memory, larger capacity, lower update cost, and autonomous experience acquisition with cross-task integration.

If reasoning sets the ceiling of single-shot intelligence, memory may set the floor of every inference. At near-300B scale, MemT 1.0 shows a model need not start from zero every time — it can remember and grow. How steep this “Memory Scaling” curve climbs may become the next growth curve for foundation models after pre-training scaling and inference-time scaling.

Sources: MemTensor MemT-1.0 technical report (arXiv:2610.26825); GitHub MemTensor/MemT-1.0; HuggingFace/ModelScope collections; 36Kr, Eastday/Xinmin Evening News, Huanqiu, alphaXiv, and related reporting (2026-10-19/20). Code is illustrative, not official.

• ←Previous
• Next→