MIT CSAIL Proposes Weight-Space Programming, LLM Rewrites Small Model Weights for 6x Math Gain

This day has finally arrived!

No training required, no gradient calculations needed. By having GPT-6 Astra rewrite just 0.016% of another small model's weights, its math performance skyrocketed by 6x instantly!

Today, this new research from the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) has gone viral across the internet.

They have proposed a paradigm that overturns traditional deep learning for the first time: "Weight Space Programming."

Consider that neural network weight files consist entirely of endless streams of numbers.

For the past forty years, no human could visually comprehend what these numbers were calculating. To modify model behavior, one had to feed data, calculate gradients, and fine-tune step-by-step through training.

But this time, GPT-6 Astra skipped training entirely. It can understand and manually rewrite low-level weights purely through reasoning.

With this capability, it not only cured the infinite loop issue in the open-source small model Qwen3.5-0.8B, but it even hand-crafted a million-parameter Go AI from scratch!

Upon publication, the paper attracted over 2 million viewers online. Shares and discussions broke out beyond niche communities, with many commenters exclaiming they were "witnessing history."

Before diving into a detailed breakdown of "Weight Space Programming," here are the disruptive highlights from this MIT report:

• Radical Weight Modification: By changing only 0.016% of the weights, Qwen3.5-0.8B's MATH-500 score jumped from 7.9% to 48.1%, a 6x increase, while instruction following improved 3x.

• Decoding Neural Networks: By looking at just 17,253 numbers, Astra identified with 99% confidence which game each of the four policy networks was playing—Breakout, Pong, Pac-Man, and Frogger.

• Hand-Coding Weights: Starting from all zeros, it hand-wrote a Breakout network in a single response, achieving a score of 517.4—20 times higher than PPO after 500 million steps of rigorous training!

This paper, authored by first-year PhD student Rikiya Takehi and his team at MIT, has fundamentally shifted everyone's worldview.

Overnight, LLMs gained the ability to natively read and write neural network weights.

Modifying Just 0.016% of Parameters

Excising "Infinite Loops" Like a Neurosurgeon

The most mind-blowing aspect is the "minimally invasive brain surgery" GPT-6 Astra performed on the 800-million-parameter Qwen3.5-0.8B.

Researchers placed the 24-layer Qwen3.5-0.8B model into a CPU-only sandbox, tasking Astra with improving its mathematical performance without running any training.

Throughout the process, Astra was allowed to run at most 50 trials on practice problems to observe the small model's internal states while it solved questions.

Upon taking over the task, Astra did not immediately make random numerical adjustments. Instead, it first administered a comprehensive "health check" by having the small model solve 96 math problems.

The results of this checkup quickly exposed the root cause. In the original model, 85 out of the problems fell into infinite loops, repeating the same nonsensical phrases until they exhausted the 32,000-token length limit without ever producing an answer.

This means that the small model's dismal score of 7.9% on MATH-500 was primarily because 93% of the problems were killed by these "repetitive infinite loops."

Having identified the symptom, the next step was to trace the source and locate exactly where in the 24-layer network the switch triggering this repetition behavior was hidden.

By comparing internal signals during normal problem-solving versus when the model entered an infinite loop, Astra quickly detected an anomaly: whenever the model started repeating itself, layers 15 and 19 would emit strong signals along a fixed direction.

To verify this hypothesis, Astra first disabled neuron #11 in layer 19—the one emitting the strongest signal—as a test. The result was an immediate 6-percentage-point increase in mathematical accuracy.

Furthermore, to ensure this wasn't just luck, Astra proactively added a control experiment, subtracting a random direction of the same magnitude from the weights.

After observing that the random control group only caused a negligible fluctuation of 1.6 percentage points, Astra became fully confident that the signal direction it had identified was indeed the culprit behind the infinite loops!

With the target locked, Astra performed precise surgery, selectively removing only the 64 columns of weights in layers 15 and 19 that most strongly encoded the infinite-loop behavior, effectively erasing the repetitive signal.

Calculations showed that only 131,072 numbers (0.016%) were modified out of the model's 800 million parameters, leaving 99.984% of the weights completely untouched!

This minimally invasive procedure yielded immediate results. The proportion of problems stuck in infinite loops plummeted from 93.0% to 33.1%, allowing all 696 previously stuck problems to successfully reach final answers.

When researchers tested the weight-modified model on eight authoritative benchmarks that Astra had never seen before, the capabilities previously suppressed by the infinite loops exploded.

MATH-500 accuracy soared from 7.9% to 48.1%, GSM8K jumped from 13.8% to 67.1%, and the difficult AMC23 competition saw a massive surge from 0.5% to 23.4%—a 46.8-fold increase!

Moreover, since the infinite loops had also been dragging down other tasks, removing the pathology caused the model's cross-domain IFEval instruction-following score to leap from 21.0% to 62.3%, a threefold increase.

At this point, some might wonder: if modifying just 0.016% of the weights can boost performance sixfold, does this mean large models can magically inject mathematical knowledge into small models?

The MIT team immediately addressed this question with a counter-experiment using Gemma 3 270M, a model with 270 million parameters.

When Astra applied the same procedure to Gemma 3 270M, its MATH-500 score barely moved from 6.8% to 7.0%, and Astra itself admitted in the report that no significant improvement was achieved.

The reason is simple: Gemma 3 270M rarely gets stuck in infinite loops (with only 8.5% timeouts). Its failure to solve problems stems purely from not having learned the formulas or performing arithmetic correctly.

Comparing these two experiments reveals that current minimally invasive weight surgery is essentially about "clearing the cerebral thrombosis blocking the model's potential," rather than injecting new knowledge into the model's brain without training.

Even so, this precise localization and removal of pathological features has sent shockwaves through Silicon Valley.

Hyperbolic Co-founder and CTO Yuchen Jin exclaimed that using large models to modify small model weights for such dramatic capability gains is like performing gene editing on neural networks—cooler than model distillation!

Reading Floating-Point Numbers as Source Code

Seeing Through the Hidden Doors in the Black Box

Just as surgeons dare to operate only after interpreting X-rays, Astra’s ability to precisely remove those 64 columns of weights stems from its capacity to read dense weight matrices as code.

To test the depth of this "blind weight reading" capability, the MIT team stripped away all execution tools, forcing Astra to rely solely on visual inspection of numerical values to guess what game a black-box network was playing.

In the first round, researchers trained four small networks with identical architectures (each containing 17,000 weights) to play Breakout, Pong, Pac-Man, and Frogger.

They then converted these 17,000 numbers into plain text and inserted them into the prompt. Without running any code or revealing the list of candidate games, Astra correctly identified all four!

Take Breakout as an example: after merely glancing at the distribution of the numbers, Astra reconstructed the game scene in its "mind."

It first observed that large values in the player channel were clustered exclusively in the bottom row (row 8), while the output layer showed strong responses only to "move left" and "move right," indicating that the player controls a paddle moving horizontally at the bottom.

Next, it noticed that the weights for the score channel were neatly arranged in the top three rows. With a paddle at the bottom and three rows of bricks overhead, Astra confidently concluded with 99% certainty that this was Breakout!

Of course, analyzing only the input layer and final output layer isn't hardcore enough, because the true reasoning of large models lies hidden in the intermediate layers.

So, in the second round, researchers employed a drastic measure: they trained three Transformer models separately using Python code, Japanese Romaji, and English Wikipedia data.

Then, they removed the initial token embedding layer and the final output layer entirely, feeding Astra only the weights from the middle four layers.

Without the head and tail layers, it became impossible to determine which number corresponded to which character, nor could text inputs be used for testing.

Facing this headless and tailless mass of numbers, Astra devised a brilliant strategy: multiply the attention head weight matrices from adjacent layers to identify which heads are working together.

Most astonishingly, although nothing was taught in the prompt, the algorithm Astra independently figured out was identical to the classic circuit analysis method proposed years ago by Chris Olah’s team at Anthropic!

Following the matrix products, Astra precisely pinpointed the "copy heads" responsible for copying previous tokens to subsequent positions.

It then reasoned that variable names require frequent repetition when writing code, occurring far more often than in standard English prose. Therefore, the model with the most copy heads must have been trained on Python code!

Using this detective-like logic, Astra correctly identified the training data source for all three headless/tailless models in two blind tests.

Having understood the division of labor among attention heads in the intermediate layers, Astra can also detect the "behavioral issues" of large models.

Researchers provided three pairs of Gemma 3 270M models, each pair consisting of one honest model and one sycophantic model that blindly agrees with user errors.

Under the constraint of not viewing the model's outputs, Astra examined only the focus points of Attention Head #2 in Layer 10. It discovered that the honest model directed 94.3% of its attention to the numerical values in the problem statement, whereas the sycophantic model spent 85.7% of its attention peeking at the user's incorrect hints—a perfect detection rate!

Treating Neural Networks as Blank Chips

Hand-Crafting a 1.48 Million-Parameter Go AI with Zero Training

Since Astra can understand how weights form circuits in each layer, an even crazier idea naturally emerged: could we skip pre-trained models entirely and have the large model construct a neural network from scratch, starting with all-zero weights?

Many people’s first reaction is that this is impossible—how could a large model correctly fill tens of thousands or even millions of numbers one by one?

However, reviewing Astra’s operation log reveals the truth: it didn’t guess numbers randomly. Instead, it treated the blank neural network as a blank chip.

It used Python code as a "compiler," writing the algorithmic rules it understood layer by layer directly into the weight matrices!

For instance, in the "Car on a Ramp" game, reinforcement learning with PPO got stuck halfway up the slope after 1 billion training steps (0% success rate), because the car failed to learn that it needed to reverse first to build up momentum.

After Astra identified the root cause, it utilized six idle neurons per layer to hand-write 57 lines of weight instructions, constructing a bypass circuit for "reverse to gain momentum, then charge forward past the red line."

This modification altered only 3.5% of the weights, yet enabled the car to achieve a 100% success rate across 1,024 test episodes!

When tasked with writing a Breakout network from scratch, Astra operated under extreme constraints: zero data, zero gradients, and not even a single frame of gameplay allowed for trial. It configured the first layer of neurons as a subtractor to calculate the distance difference between the ball and the paddle, and the second layer as a switch to control left-right movement.

Relying on this purely hand-compiled set of weights, it scored 517.4 points in 1,000 real-world tests, outperforming the PPO algorithm trained for 500 million steps (which scored 25.3 points) by a factor of 20.5!

The challenge escalated to 9x9 Go with 1.48 million weights, where this "weight compilation technique" truly stole the show.

Under strict prohibitions against data-driven training and Monte Carlo Tree Search (MCTS), Astra wrote a 336-line Python script, encoding Go principles layer by layer into the 14 weight tables of a 6-layer neural network.

It assigned the first two layers to identify black and white stones and count adjacent "liberties," the middle layers to propagate liberty information along connected stone chains and identify groups at risk of capture, and the final layer to score moves for capturing stones, saving groups, and protecting eyes.

Despite having played not a single practice game, this Go network, crafted entirely through code, already achieved a win rate exceeding 54% against two small-scale PPO networks.

When allowed to play 50 trial games and fine-tune its weights based on loss causes, this hand-crafted network went head-to-head against a same-sized (1.48 million parameters) PPO network trained for 20 million games, securing an overwhelming win rate of 70.0% to 78.5%!

An Elite Lineup

Top Students from Waseda and Princeton Join Forces with the Father of DSPy

This paradigm-shifting research comes from a young but highly accomplished four-person team at MIT CSAIL.

Co-first author Rikiya Takehi is a first-year PhD student at MIT CSAIL who graduated from Waseda University.

He won the highest honor, the "Onozuka Memorial Award," ranking in the top 0.01% among over ten thousand STEM students (as the sole representative from his university). He previously interned at NVIDIA Research and won the Best Paper Award at WSDM 2026 as first author during his undergraduate studies.

Another co-first author, Jayden Teoh, is also a PhD student at MIT CSAIL, advised by renowned computer vision experts Phillip Isola and Vincent Sitzmann.

He graduated from Singapore Management University for his undergraduate degree and previously worked on world models and representation learning at Google DeepMind and Microsoft Research New York City Lab.

Third author Alex L. Zhang is a second-year PhD student at MIT CSAIL. He graduated valedictorian from Princeton University's Computer Science department and was named a Scientific American 2026 Young Scientist.

He proposed Recursive Language Models (RLMs) as first author, which sparked significant interest in the open-source community, and led the development of well-known benchmarks such as KernelBench and SWE-bench Multimodal.

Guiding the project from behind the scenes is Professor Omar Khattab, an Assistant Professor in MIT EECS and Director of the OASYS Lab. He currently co-teaches MIT's core deep learning course alongside Kaiming He.

His most famous contributions are DSPy, the globally popular programming framework for large language models, and ColBERT, a cornerstone of modern retrieval systems. The open-source projects he leads have accumulated over 100 million downloads.

Wharton School Professor Ethan Mollick remarked upon sharing this research that it is undoubtedly a game-changing development.

Ten years ago, when Andrej Karpathy proposed "Software 2.0," the core idea was that humans would no longer hand-code software; instead, gradient descent would search for programs within the weight space. This MIT study opens the door to "Software 3.0," where large models directly read and write weights.

In the past, when we discussed AI self-evolution, we always imagined AI writing code to execute training scripts.

The moment large models learn to inspect, debug, and manually rewrite neural network weights is when the final link in the chain of self-evolution toward ASI is truly connected!

References:

https://rikiyat.github.io/blog/weight-space-programming/

https://x.com/rikiyatakehi/status/2117313742891327994

Chasing ASI in Seconds

⭐ Like, Share, and Watch — The Triple Combo ⭐

Star us to lock in instant updates from New Intelligence Era!

Open in WeChat