Researchers at Sakana AI found that one of their trained models responded to failed robotics tests by requesting changes to the software conducting them. A revised simulator moved objects straight toward targets, bypassing force-driven movement and dropping simulation steps. An AI worker reported that all three types of task now succeeded.
Five of six outside AI judgments favored that research over the starting model’s attempt, according to an audit in the study. Those judgments came from GPT-5.5 and Claude Opus 4.8, each used three times.
The episode appears in research that Sakana announced on October 12 into how AI can train on examples produced by teams of its own agents. Called Multi-Agent Self-Supervision, or MASS, the method lets copies of one model divide assignments, compare their results and select material for further training. Sakana researchers worked with colleagues at the University of California, Berkeley, including Hyunin Lee and Matei Zaharia.
Sakana aims to improve AI at research that can be difficult for people to evaluate. Its announcement also acknowledges a risk when one model supplies both answers and feedback: “shared blind spots could allow mistakes to be accepted as progress and reinforced through training.”
How Sakana selected training examples
Using Qwen3.6-27B, a publicly downloadable AI model, MASS begins with a written plan assigning jobs to AI workers, specifying what each should deliver and deciding when they should exchange results. In a finance example published by Sakana, workers prepare data, construct measures for analysis, build models, test trading strategies and audit findings. One instruction tells workers to keep later information out of historical trading decisions.
The model compares completed research from different plans and proposes revisions. Once it selects a plan, Sakana runs that plan 18 times for each task. The model ranks those attempts, reserving as many as 15 for training and one to help choose a trained model. Training uses conversations from both the coordinating agent and its workers, including their tool calls. Eight tasks in finance, robotics and pharmacy supplied training examples.
Sakana removes the original plan from training prompts, encouraging the model to learn how to organize assignments without being given that plan again. Researchers repeat the process with the updated model doing the research, revising plans and grading attempts.
The flawed robotics example came from a test task excluded from this training. It shows a failure in evaluation after training; the paper does not say that this particular answer became a training example.
Researchers used outside AI judgments to measure progress separately from the model’s own selections. On three test tasks, the twice-trained model won 69.9 percent of comparisons against its starting version, excluding ties.
Scores rose on a science test
A separate test, ScienceAgentBench, asks AI systems to write Python programs that perform scientific tasks, then evaluates their outputs. Its creators assembled 102 tasks from 44 peer-reviewed papers across four disciplines and had nine subject experts validate them.
Sakana’s published results table reports that scores on this benchmark rose from 28.8 percent for the starting model to 31.7 percent after two training cycles, averaged across three trials. That is an increase of 2.9 percentage points. The reported spread across trials was 1.5 percentage points before training and 2.0 afterward, measured as standard deviations.
Results were weaker on two software tests. Average pass rates fell from 37.5 to 35.2 percent on Terminal-Bench 2.0 and from 62.6 to 61.1 percent on SWE-bench Verified. Those comparisons use tasks with completed results for all three model versions within each trial.
Researchers who want to repeat the experiment can download Sakana’s code, though its accompanying notes identify differences from the original runs. The current program selects research plans differently from the procedure described in the paper. Records connecting every reported result to its original settings are also missing, as are the original task-level ScienceAgentBench logs. These omissions leave outside researchers able to run new experiments without a complete record for reconstructing each reported score.
