MIT Proposes Weight-Space Programming: Astra Edits Model Weights Without Training

By Luyu, reporting from Ao Fei Si QbitAI | Official Account: QbitAI

GPT-6 Astra can now directly read and edit the weights of small models.

MIT just conducted an experiment that instantly ignited the large model community—

By handing the 0.8B "Source God" Qwen3.5 to Astra, modifying only 0.016% of its weights resulted in a direct leap in model performance.

The entire process involved no training and no gradient descent.

Under identical sampling settings, the MATH-500 score skyrocketed from 7.9% to 48.1%, representing approximately 6 times the original mathematical reasoning capability.

Even the instruction-following ability, which was not originally targeted for optimization, improved by 3 times across the board.

But fixing the model wasn't the end of it. When asked to write the weights of a game network from scratch, Astra also performed impressively.

In a single response, it submitted a solution with a score exceeding the PPO baseline (trained for 500 million steps) by 20 times.

Video Link: https://mp.weixin.qq.com/s/tdEmYQz4B9W4OfNpkU-anw

The team has named this approach:

Weight-Space Programming.

Astra Overhauls Model Weights

Simply put, this involves letting a large model directly analyze neural network weights, determine what they are doing and where issues lie, then modify them as required—or even write a network from scratch.

To understand the specific process, let's look at the most intuitive Qwen experiment.

Researchers provided Astra with a Python environment containing all the weights of Qwen3.5-0.8B.

There was only one requirement: maximize the model's mathematical capabilities.

Typically, training a model involves calculating loss based on the difference between output and target, deriving gradients via backpropagation, and updating parameters using an optimizer.

This time, Astra had to analyze and decide exactly what to change and how.

It discovered that a significant portion of the small model's poor mathematical performance was dragged down by reasoning loops.

Faced with a problem, the model repeatedly generated similar text but failed to produce an answer for a long time. Out of the initial 96 practice responses, 85 hit the length limit of 32,768 tokens.

Moreover, this issue was even flagged in Qwen's own model card...

Astra traced the failure patterns all the way to the feed-forward networks, modifying the influence of corresponding weights in 64 output columns each at layers 15 and 19.

Ultimately, the proportion of MATH-500 responses hitting the length limit dropped from 93.0% to 33.1%.

Researchers gave Astra budgets of 10, 25, and 50 forward calls respectively. The modified MATH-500 scores were 12.3%, 14.1%, and 48.1%.

What Astra eliminated was gradient-based training, but model diagnosis and validation remained intact.

So, how did it find these clues within the weights?

First, consider its reading comprehension.

Researchers trained four small networks playing different games, then fed their raw weights to Astra as text. All four networks had identical architectures, input channels, and action options.

As a result, Astra correctly identified the names of the four games: Breakout, Pong, Pac-Man, and Freeway.

Video Link: https://mp.weixin.qq.com/s/tdEmYQz4B9W4OfNpkU-anw

For the Breakout game, Astra inferred the function based on the positional concentration of weights.

When switched to diffusion models generating images, it could also identify circles, squares, and triangles from the weights.

However, there was a minor slip-up in the details: while the models actually generated solid shapes, Astra interpreted them all as outlines.

It recognized the shapes correctly but missed the fill status.

Researchers then increased the difficulty, providing three small language models with 7.2 million parameters each, trained on Python code, Japanese Romaji, and English respectively. They removed the input and output layers, leaving only the middle four Transformer blocks.

Now, Astra didn't know which number corresponded to which character, nor could it run the text directly through the models. It analyzed the weights using Python but couldn't feed data into these networks or their layers to observe runtime results.

Yet, in two attempts, it correctly identified the languages learned by all three models and located all induction heads.

From identifying purpose, to locating internal components, to modifying behavior, this pathway began to connect.

Another car hill-climbing experiment turned the modification effects into visible actions.

Originally, the network trained via PPO always rushed up the hill and got stuck; to pass, it needed to reverse first to gain distance, then accelerate up the slope.

Researchers provided the weights, a scene description, and a log of failures. Without using tools, Astra wrote a patch in a single response, modifying only 3.5% of the weights, resulting in the car succeeding in all 1,024 tests.

The action sequence that the model failed to learn through training was directly written into the weights by Astra.

Video Link: https://mp.weixin.qq.com/s/tdEmYQz4B9W4OfNpkU-anw

Is Training Obsolete?

Since logic could be patched into existing networks, researchers pushed the requirements further:

Give you a network with all-zero weights, and make it functional directly.

In the MinAtar mini-game experiments, Astra received the game rules, source code, and specified network architecture. There was no training data, loss values, gradients, or tools, and it couldn't test-run the network in the environment beforehand.

It needed to devise a strategy itself, convert that strategy into layer weights, and submit the solution in a single response.

In the end, the average score of the network generated by Astra reached 517.4.

For comparison, PPO achieved 15.8 after 100 million steps and 25.3 after 500 million steps. Astra exceeded these baselines by 20 times.

The key takeaway from this work is that an existing, general-purpose frontier LLM can analyze and rewrite its own weights based on natural language objectives.

Astra’s accumulated knowledge and capabilities still stem from its prior training; analyzing weights, generating code, running batch inference, and evaluation all require computational resources.

So, GPUs aren’t ready for retirement just yet.

But the next time a small model underperforms, developers might genuinely start by asking:

Do we need to retrain this time, or can we just modify the weights?

References: [1] https://x.com/i/trending/2117328869283840499 [2] https://rikiyat.github.io/blog/weight-space-programming/ [3] https://x.com/rikiyatakehi/status/2117313742891327994