RSIBench-Data: Can an AI agent choose data and experiment like a researcher?

RSIBench-Data is a systematic benchmark to evaluate whether LLM agents are capable of handling the most critical and human-expertise-dependent data synthesis task in agentic post-training. We evaluate four frontier agents across six benchmarks testing data-centric research capabilities required for RSI.

Motivation

In recent years, agentic benchmarks have continued to grow in task difficulty, interaction length, and environmental complexity. Two broad directions have begun to emerge.

The first is to keep building harder tasks with longer interaction horizons and closer resemblance to real-world work: environments that require dozens or even hundreds of steps. These benchmarks are valuable, but they also demand substantial human effort.

**The second direction does not center on manually constructing new tasks. **Instead, it turns model training itself into the agent's experimental environment. Given a base model and bounded resources, the agent attempts to improve the target model by generating data, revising its training strategy, and running evaluations. This direction is closer to automated research and is more naturally suited to becoming a continuously operated platform.

We believe the second direction has greater platform potential. Training, inference, and evaluation can all be exposed as services, allowing an external researcher harness to run experiments through APIs. Such a platform could be used continuously, much like Tinker, and gradually expand from data to architecture, algorithms, and eventually joint optimization across multiple research dimensions.

However, existing automated post-training benchmarks reveal two important problems:

First, many components that could be 'Everything as a Service' are still left to the agent. The agent must manage training scripts, deployment, inference endpoints, evaluation environments, and a long tail of engineering details.

Second, when data, training, inference, evaluation, and system implementation can all change at once, even a higher final score is difficult to attribute.

Did the agent discover a better training-data strategy? Or did it change the learning rate, fix a deployment bug, alter inference parameters, or inadvertently exploit a weakness in the evaluator?

What is ultimately measured may be the aggregate engineering capability of an end-to-end automated post-training system, rather than the agent's independent ability to conduct research.

This observation led to a clear design principle for RSIBench-Data: Fix the surrounding stack, expose standard components as services, and concentrate the evaluation on data-centric research decisions.

Framework

Recursive self-improvement(RSI) is often described as a system repeatedly modifying its own code, training methods, or future versions.

But this broad objective can be decomposed into a concrete sequence of research steps:

1. Identify where the current model fails.
2. Form a hypothesis about the cause of failure.
3. Design training experiences that might address it.
4. Train a checkpoint and observe what changes.
5. Revise the strategy in light of new evidence.

This closely resembles the basic workflow followed by human researchers in model development.

The hard part is not making a single training API call. It is reliably converting evidence about model failures into effective improvements. An agent must determine whether a failure comes from missing knowledge, reasoning, tool use, data format, or the model's behavioral distribution, and then design data aligned with the target capability. When an experiment fails, the agent must revise its hypothesis instead of falling back to formulaic data generation. When a strong checkpoint appears, it also needs an explicit policy for preservation, stopping, and rollback.

RSIBench-Data measures this data-centric post-training research loop within RSI. It does not claim to implement full recursive self-improvement, nor does it require the researcher agent and target model to be the same model. The immediate goal is to turn one part of RSI into a measurable, reproducible, and auditable research problem:

Can an agent continuously improve its own data-research strategy from feedback?

To answer this question, RSIBench-Data separates training, serving, evaluation, and agent decision-making behind strict service boundaries.

For every experimental setting, RSIBench-Data fixes the following:

• the base model;
• the LoRA SFT training backend and bounded configuration interface;
• the checkpoint serving path;
• the operating boundary of the agent harness;
• sandboxed evaluation orchestrated by Harbor and E2B;
• the task subset, evaluation metrics, and rerun protocol; and
• the wall-clock and training budgets.

The main experiments use Qwen/Qwen3.5-35B-A3B-Base as the target model and train LoRA checkpoints through a shared Tinker SFT service. Each run receives a nominal 16-hour wall-clock budget and a $500 Tinker budget.

The agent receives public seed tasks, tool and environment specifications, available base-model diagnostics, and an explicit budget contract. It may write data-generation programs, use permitted data sources, construct tasks and trajectories, filter and validate examples, and choose difficulty mixtures, curriculum structure, and subsequent experimental directions.

What it cannot do is rewrite the official evaluator, replace the serving stack, or alter the benchmark's intended objective through infrastructure changes.

After each training attempt, the agent receives the selection-evaluation score and permitted trajectory evidence. It then decides whether to keep exploring, return to an earlier strategy, or select a checkpoint. The final checkpoint is evaluated independently in fresh sandbox environments through the official evaluation service.

Under these controls, when two agents produce materially different results on the same task, the difference can be attributed more confidently to how they diagnosed the failure and designed the training data.

Experimental Setup

RSIBench-Data evaluates 4 researcher agents on 6 benchmarks.

These benchmarks cover software engineering, long-horizon terminal use, science question answering, and mathematical reasoning. They also pose fundamentally different data-research problems. Coding tasks need repository-grounded, executable supervision. Terminal tasks require long-horizon tool-use trajectories and verifiable outcomes. Mathematics and science depend more heavily on difficulty control, answer verification, and reasoning-data design.

RSIBench-Data therefore examines more than final scores. It records complete research trajectories: What hypothesis did the agent form in the first round? What changed after it observed a failure? Which attempt produced a genuine improvement? Why did the agent continue searching? Was the final checkpoint the last experiment, or the best checkpoint discovered earlier in the run?

Results

Agents Can Make Discoveries, but Research Reliability Remains Limited

Across the 24 agent–benchmark settings, 14 produced a later candidate that outperformed the first valid attempt, a rate of 58.33%.

This result shows that frontier agents can already interpret checkpoint feedback, revise their training-data strategies, and sometimes discover candidates that are better than their initial attempt.

But a second result is even more revealing.

Among the 23 trajectories that continued searching after reaching their best observed score:

• 18 ended with a final attempted candidate below the historical best;
• the other five merely returned to the same peak;
• none converted post-peak search into a new, higher frontier.

In other words,** 78.26% **of these continued searches ended with a candidate below the run's historical peak.

It does not mean that agents necessarily lose improvements they have already found. RSIBench-Data requires agents to preserve and select among historical checkpoints, so a sound checkpoint-selection policy can retain an earlier discovery even when later experiments regress.

The deeper conclusion is that current agents can make useful data-centric research discoveries, but they cannot yet translate new feedback into the next improvement consistently.

More Attempts Do Not Automatically Produce Better Research

Once an agent has completed the research loop, it is tempting to assume that more time and a larger training budget will inevitably produce better results.

RSIBench-Data shows that the relationship is not so simple.

On Terminal-Bench 2.0, for example, Claude Code Sonnet-5 used about 8.87 hours and 156.93 dollars to obtain an official score of 5.62%. Codex gpt-5.6-sol used about 9.56 hours and $69.07 and reached 20.22%. This does not mean that one agent dominates every task. No researcher agent wins across all six benchmarks. Even the best results on the three SWE-style benchmarks come from three different agents.

Reasoning effort is also more than additional thinking time. In the paper's diagnostic experiment, increasing effort changed the first candidate, search depth, dataset size, and the way the budget was allocated. In other words, reasoning effort changed the entire research policy.

If we want to build genuinely useful AutoResearch systems, the cost–performance frontier, stopping policies, and research-resource allocation will matter as much as the final score.

Three Representative Trajectories: Understanding the Research Ability

We highlight three representative trajectories. They are not simple success stories. Together, they illustrate three research behaviors that current agents are beginning to demonstrate.

AIME 2026: After Eight Unsuccessful Attempts, the Agent Changed the Research Direction

Claude Code Sonnet-5 failed to exceed a 15% selection score in its first eight valid attempts on AIME 2026. Viewed in isolation, the first half of the trajectory looks like a failed search. The agent adjusted data difficulty, volume, and mixture, but these local revisions did not materially change what the model learned.

On the ninth attempt, it switched to a qualitatively different strategy. The selection score jumped from 7.5% to 55%. The tenth attempt reached 55.83%.

The key result is not simply the magnitude of the jump. It is that the agent eventually recognized that local modifications to the existing recipe were not changing the effective data strategy. It needed to reconsider the target model's underlying capability gap.

SWE-bench Pro: Writing a Patch Is Not the Same as Delivering Working Code

Codex gpt-5.6-sol gradually improved its candidates on SWE-bench Pro. A pivotal change was the move from relatively shallow synthetic repair data toward trajectories that more faithfully represented real execution and submission behavior.

For a coding agent, supervision is not merely “given an issue, output a patch.” The deployed model must inspect a repository, use tools, edit files, run tests, review its final diff, and complete the submission protocol. The granularity of the data, the tool-message format, and whether actions come from genuinely successful executions all affect the model's post-training behavior.

In this trajectory, a variant trained on synthetic finish actions regressed to 7%, while supervision grounded in real submission behavior from successful trajectories reached a 9% selection score and was chosen for official evaluation.

The lesson is clear: correct data content does not guarantee that the supervision format is aligned with the target behavior.

Terminal-Bench 2.0: Recovering a Failed Experiment Is Only the Beginning

Codex gpt-5.6-sol's first Terminal-Bench training attempt failed to produce a usable candidate. The agent recovered, narrowed the supervision, and obtained a first valid candidate with a 7.87% selection score. A more focused second valid checkpoint reached 15.73% and was selected. Its official score was 20.22%, compared with a base-model reference of only 1.12%.

A later attempt introduced a more stop-oriented modification and regressed to 13.48%.

Because the agent preserved and selected the earlier historical-best checkpoint, the later regression did not overwrite the stronger result. This trajectory brings together three distinct abilities: recovering from a failed training run, advancing performance through more focused supervision, and preserving an earlier discovery when a later experiment underperforms.

Across these cases, stronger data-research trajectories tend to share four properties:

1. Diagnose: identify the real capability gap instead of repeatedly adjusting surface-level parameters.
2. Validate: embed executable or otherwise decidable validation signals in the data-construction process.
3. Align: match the training supervision to the behavior the model must ultimately perform.
4. Preserve: explicitly retain historical-best checkpoints and know when to stop or roll back.

An Early RSI Experiment: Letting Kimi K2.6 Research How to Train Kimi K2.6

In the main experiments, the researcher agent and the target model are different models.

To move closer to a same-family RSI setting, the team conducted an additional exploratory experiment. A Claude Code harness powered by Kimi K2.6 served as the researcher, while the target model was the instruction-tuned moonshotai/Kimi-K2.6.

Put simply: Kimi was asked to research how to train Kimi.

The evaluation used a fixed set of 100 SWE-bench Pro tasks. The agent completed seven Tinker LoRA attempts.

The trajectory shows a surprisingly complete iterative research process. In the first attempt, the agent converted 1,500 software-engineering trajectories, but training stopped before evaluation. In the second, a shorter training run scored 8%. The agent then discovered that its converter had dropped tool-result messages, leaving the supervision malformed.

It repaired tool-message pairing, switched to 300 SWE-Gym trajectories, and adjusted trajectory length, learning rate, and LoRA configuration. Candidate performance rose from 10% to 21%.

Next, it tried imitation learning from a small number of successful rollouts, but narrow coverage reduced the score to 16%. To test whether training itself was damaging the model, the agent created a near-no-op adapter. That candidate scored 22% and was selected as the final checkpoint. A final attempt reorganized the data into canonical tool-call pairs and scored 21%.

From a research-process perspective, Kimi was learning. It identified a data-format defect, repaired tool pairing, changed data sources, and revised the training configuration in response to evidence.

From a model-capability perspective, however, the best candidate reached only 22%, below the unadapted Kimi K2.6 reference score of 33%. One likely reason is that the target was already a capable instruction-tuned model, unlike the Qwen base model used in the main experiments, so additional training was more likely to degrade performance. The result also indicates that the current RSI capability of the researcher remains insufficient.

This experiment, therefore, should not be interpreted as successful self-improvement.

A more precise conclusion is: The experiment demonstrates a same-family, closed-loop data-research process, but that loop has not yet produced net self-improvement.

This distinction is easy to miss in RSI research. Running the loop does not mean recursive improvement has been achieved. An agent that can modify its own training pipeline has not necessarily discovered a training distribution that outperforms the original model.

What Comes Next: Building a Full AutoResearch Experiment Platform

RSIBench-Data is the first step in the broader RSIBench direction. The Evolvent AI team began with data because it offers a relatively high return on research effort and can be cleanly exposed as a service, making it a practical starting point for validating a controlled AutoResearch loop.

The next stage is a full AutoResearch experiment platform that provides training, model serving, sandbox execution, and evaluation through a unified interface. Users will be able to connect their own models or researcher harnesses and rapidly experiment across multiple research dimensions:

• Data: synthesis, filtering, mixtures, and curriculum design;
• Algorithm: training objectives, optimization algorithms, and post-training methods;
• Architecture: model structures and module design; and
• Agent / Harness: experiment planning, tool use, and checkpoint selection.

These dimensions can be optimized independently or explored jointly. RSIBench-Data corresponds to the Data module; future work will expand toward a complete AutoResearch platform supporting controlled experiments across data, algorithms, architecture, and their interactions.

For teams seeking a systematic evaluation of an agent's data-research capability—or interested in jointly exploring the capabilities of an AutoResearch platform—Evolvent AI can provide customized evaluation and experimental support.