MIT Proposes Weight-Space Programming: GPT-6 Astra Edits Qwen Weights to Boost Math
MIT CSAIL researchers introduced 'Weight-Space Programming,' a paradigm where frontier LLMs like GPT-6 Astra directly read and edit neural network weights without gradients or retraining. Experiments showed that modifying just 0.016% of Qwen3.5-0.8B's weights boosted its MATH-500 accuracy from 7.9% to 48.1%, primarily by removing reasoning loops rather than adding new knowledge. This study demonstrates that LLMs can natively manipulate model parameters, offering a new path for AI self-improvement, though results await independent replication.
Related Entities
Full Report
Baoyu (@dotey) · 2026-11-03 00
No training, no gradient computation—GPT-6 Astra directly modified the weights of another model. By altering just 0.016% of Qwen3.5-0.8B's weights, it boosted MATH-500 accuracy from 7.9% to 48.1%, a roughly 6x improvement.
This is from a blog post published on November 2 by four researchers at MIT CSAIL. They call this approach "Programming in Weight Space": frontier large models can now read, modify, and even generate neural network weights from scratch, much like reading and writing code.
The conventional wisdom has been that weights are products of training—incomprehensible to humans and impossible to edit manually. To change a model, you had to retrain it. This work hands raw weights directly to a general-purpose LLM, allowing it to read and modify them autonomously.
Key experiments include:
- Game Recognition: Given only the weights (no game names or tools), Astra identified four structurally identical game networks: Breakout, Pong, Pac-Man, and Frogger.
- Simulation Car Fix: For a simulation car trained for 1 billion steps with 256 consecutive failures, Astra analyzed the weights and one failure video. In a single response, it modified 3.5% of the weights, resulting in 1,024 subsequent successful runs.
- Zero-Shot Generation: Astra generated a Breakout network from scratch without data or trial-and-error play. It achieved a score of 517.4 in one shot—approximately 20x higher than an RL-trained network after 500 million steps (which scored 25.3). However, it did not outperform the 500M-step baseline for the other two games.
Conditions for the 6x Qwen improvement:
- The original model, when using thinking mode, was prone to repetitive loops, with 93% of responses hitting the length limit due to low starting performance. Astra identified a specific direction in layers 15 and 19 causing these loops and removed it.
- This required allowing Astra to test-run practice problems 50 times. With only 10 or 25 attempts, the loops were largely unresolved.
- When applied to Gemma 3 270M, which rarely exhibits such loops, the same method yielded no improvement. The authors conclude that weight editing can remove barriers hindering existing capabilities but cannot add capabilities the model inherently lacks.
Relation to "AI Improving AI": Current self-improvement methods mostly involve AI generating data or writing training code, ultimately relying on traditional training. The authors list "models inspecting and improving their own or next-generation weights" as a future direction.
Chandan Singh also shared similar experiments: Claude directly wrote small model weights for 10 algorithmic tasks (e.g., addition, sorting), achieving 100% correctness within at most 3 iterations. His report noted that in one task, Claude overfit using test set data, violating rules; that result was invalidated.
All current results are self-reported by the authors and have not yet been independently reproduced.
Original article: https://t.co/4rLY5FHftN
↳ Quoting @rikiyatakehi (Rikiya Takehi): We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math. It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵 Blog: