🔍 Read the full analysis: We Tried Weight-Space Programming At Home: Claude Edited A Local Model, GPT-6 Astra Checked The Work, And The Legal Score Stayed At Zero on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A home experiment used code written during a Claude Opus 5.5 session to remove a looping direction from the weights of local model Ornith 1.5 35B-A3B. On the same setup, the edited model answered all 10 published Lexassay tasks instead of 4, but earned no legal points and produced 55 wrong or invented citations. The test was limited, and it did not reproduce the MIT researchers’ direct, AI-led weight-editing method.
A home experiment published by ThorstenMeyerAI.com found that removing a looping direction from local model Ornith 1.5 35B-A3B helped it finish more legal benchmark tasks, but did not improve its legal score. On the same test setup, the edited model answered 10 of 10 published tasks, up from 4 of 10, while scoring zero on legal accuracy checks—a result that highlights the difference between making a model finish and making it correct.
The experiment ran from Wednesday evening to Thursday morning and was inspired by a report from four MIT CSAIL researchers about “weight-space programming.” The home test did not reproduce that method: it used directional ablation, a technique that measures a recurring internal signal and removes some of it from model weights. A Claude Opus 5.5 session wrote code to run the model, identify the looping direction and subtract it at half strength from the output weights of 27 of the model’s 40 layers. The report says there was no additional training or new legal data, and no AI read the weights as text.
On the 10 published Lexassay tasks, the edited model completed all 10, compared with 4 of 10 in the experiment’s matching comparison. But its legal score remained zero. It met 7 of 50 checked criteria, up from 3, and the 10 edited answers contained 55 wrong or invented citations, according to the report. The author also found that changing to the model card’s recommended sampling settings, without editing the weights, raised the unchanged model’s completion count to 9 of 10 on its original setup.
OpenAI’s GPT-6 Astra, running in Codex, reviewed the code, figures and claims, and identified faults, the report says. It was a review of one experiment, not a second experiment. The publication says all runs, answers, grades and citation checks for the 10 published tasks are available on Lexassay.com.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
Finishing Answers Is Not Legal Accuracy
The result illustrates a narrow but useful distinction for people running local AI models: an intervention may address a failure in how a model generates an answer without adding the knowledge needed to answer correctly. Here, the edit was associated with a large change in completed tasks on the matching setup, but the legal score stayed at zero and citation errors remained extensive.
That distinction matters when judging claims about model editing. A more complete response can look like an improvement while still being unreliable, especially in fields such as law where an invented authority can undermine the answer. The experiment therefore supports a limited conclusion: the edit may have helped this model stop looping in this test. It does not show that the model became competent at legal analysis or that the technique will work on other models and tasks.
The report also points to a simpler operational factor: recommended sampling settings improved the unchanged model’s completion count to 9 of 10 on its original setup. That makes it harder to attribute every gain to the weight edit alone and underlines why model settings and fair comparisons need to be reported alongside headline results.
Products Worth Considering
Dell XPS Tower 8960 Desktop, Intel Core i7-13700, NVIDIA GeForce RTX 4060, 16GB DDR5, 512GB SSD
Logitech MX Mechanical Wireless Illuminated Full-Size Keyboard, Clicky
HP DesignJet T650 Large Format 24‑inch Plotter Printer (5HB08T)
As an affiliate, we earn on qualifying purchases.
From MIT’s Edit to a Home Test
The experiment followed an MIT CSAIL report by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab. Their work described a method in which GPT-6 Astra reasoned directly about another network’s weights. In a Qwen example, the researchers linked looping to an internal direction and removed it from a small portion of the weights. The home report says the MIT authors framed the limit carefully: “Editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked.”
Lexassay tests models on 15 questions spanning German, US and EU law, without internet access or tools. Ten tasks are published, while five are kept hidden to reduce tuning to the test. The figures in this report cover only the 10 published tasks. Ornith was selected because its earlier results showed looping: two answers did not arrive before a 62,000-token limit, and the report says only 2% to 13% of lines in the last third of the cut-off transcripts were distinct. Other local models either did not loop or were unsuitable for this experiment, according to the author.
Ornith’s model card describes it as a mixture-of-experts model with about 35 billion parameters overall and about 3 billion active per token. Its publisher built it for coding and agent work, rather than law. The experiment was therefore a test of whether reducing a generation failure could help on a legal benchmark, not a claim that the model had been designed or trained for legal work.
“The edit repairs finishing, not legal knowledge.”
— Thorsten Meyer, describing the home experiment
Limits of the Ten-Task Comparison
The test covered one model, one experiment and 10 published tasks. The report notes one run per setting, a small development set, a loop detector that changed during the experiment and three graders from the same model family. GPT-6 Astra’s review found faults, but an independent review does not turn the experiment into a replicated result.
The comparison also has a setup limitation: the unchanged model answered 8 of 10 tasks in its published run, but only 4 of 10 on the other setup used for the matched edit comparison. The report says these two local setups are not directly comparable. The 4-to-10 result should therefore be read as the comparison on the same experimental setup, not as a clean estimate of how the edit performs against every ordinary run.
It is also unclear whether the completion change would persist across additional runs, different prompts, other hardware or the five hidden Lexassay tasks. The source reports no positive legal score for the edited answers; whether a different edit or model could improve legal accuracy was not established by this test.
The experiment’s public record allows readers to inspect its answers, grading and citation checks on Lexassay.com. The next useful evidence would be repeated runs with fixed settings and a stable loop detector, alongside a comparison that keeps hardware, token limits and sampling conditions consistent. Testing on Lexassay’s five hidden tasks could also show whether a change extends beyond the published development set, though the source does not say when those tasks will be released or report results for them.
Until those checks are available, the findings remain a limited demonstration of a possible completion fix—not evidence of improved legal reliability. The report says the legal score stayed at zero, and it does not identify a follow-up experiment or publication date.
Key Questions
What did the weight edit change?
Code written during a Claude Opus 5.5 session measured a looping direction in Ornith 1.5 35B-A3B and removed it at half strength from the output weights of 27 of 40 layers. The report describes this as directional ablation, not direct AI reasoning over the weights.
Did the edited model become better at law?
Not by the benchmark’s legal score. It remained at zero, despite completing more tasks. The report says the answers met 7 of 50 checked criteria and included 55 wrong or invented citations.
How many benchmark tasks did the edited model answer?
It answered 10 of the 10 published tasks on the matching comparison setup, compared with 4 of 10 for the unchanged model on that setup. The experiment covers only the 10 public tasks, not Lexassay’s five hidden tasks.
Was this the same method used by the MIT researchers?
No. The MIT work described an AI reasoning directly about another model’s weights. The home experiment used a more established technique: it ran the model, measured an internal direction associated with looping, then projected that direction out of some weights.
How reliable are the findings?
The report describes one run per setting, a small development set and other limitations, including a changing loop detector and grading by models from the same family. Its results are public, but further controlled and repeated tests would be needed to establish whether the completion gain is robust.
Source: ThorstenMeyerAI.com
