FrontierRefactor: Benchmarking Multi-Scale, Performance-Oriented Codebase Refactoring

The first performance-oriented codebase refactoring benchmark, spanning four codebase scales from individual computational routines to entire repositories.

Codebase Refactoring is a routine activity in software engineering: it changes a program’s internal implementation without altering its externally observable behavior, typically for one of two purposes—improving readability and maintainability, or improving runtime performance. The former can often be achieved simply by restructuring code or even changing programming languages; its quality depends on subjective human judgment and is difficult to measure directly. The latter requires accurately identifying the sources of program cost and eliminating them while preserving external behavior. Its effects appear directly in service response time and resource consumption and can therefore be measured. Performance-oriented codebase refactoring is consequently more challenging from an engineering perspective and more meaningful in practice.

We therefore introduce FrontierRefactor, a performance-oriented codebase-refactoring benchmark. It organizes tasks into four scales according to the size of the codebase being refactored, ranging from an individual computational routine to an entire repository.

This blog presents the design of FrontierRefactor: how tasks are organized, how functional and performance tests are conducted, and how both tests are protected against circumvention. On this basis, we release a public subset, FrontierRefactor Preview-30, comprising 30 tasks, and report evaluation results on this set.

Why Evaluate Performance-Oriented Codebase Refactoring

Code execution efficiency directly affects service latency and operating cost, and also determines how long a build or test cycle takes. A substantial portion of experienced engineers’ time is therefore spent optimizing existing implementations: using profiling to locate hot spots, replacing operations whose cost grows quadratically with data size, reducing memory allocation and copying, batching I/O, or changing the execution backend. The key to such optimization is identifying where the cost actually comes from: distinguishing overhead caused by the algorithm itself from overhead caused by a particular implementation, and eliminating only the latter. This judgment depends on understanding the program’s execution process and cannot be reduced to copying a known pattern.

This class of work has already attracted the attention of frontier AI Labs. When OpenAI released GPT-5.1-Codex-Max, it identified project-level refactoring as one of the complex tasks that requires sustained work over an extended period. Existing coding benchmarks, however, do not match real-world codebase refactoring needs: prior work typically covers only one codebase scale. KernelBench evaluates rewriting individual GPU kernels, PIE evaluates acceleration of contest programs, and SWE-Perf and GSO evaluate modifying code in existing repositories to improve performance. Engineering practice, by contrast, spans multiple scales—from local optimization of a single function to wholesale rewriting at repository scale. No existing work covers this full range, and repository-level rewrites in particular remain under-evaluated.

FrontierRefactor therefore defines four scales according to the size of the refactored code: Kernel (a single computational routine), Module, Package, and Repository. Determining whether a refactoring is acceptable requires answering three questions:

1. How can we construct sufficiently comprehensive functional tests? Tests must cover every function promised by the refactored code’s external interface; any omission could allow behaviorally inconsistent implementations to pass.
2. How can we measure credible speedups? Whether a program is faster must be determined by measuring runtime, yet runtime is affected by machine state. An observed difference may reflect environmental variation rather than an actual improvement in the code.
3. How can we prevent either test from being bypassed? A model under evaluation might avoid optimizing the code and instead attempt to invalidate performance tests or speed measurements.

The following sections first describe the task format and then explain how we address these three questions.

The Design of FrontierRefactor

Four Codebase Refactoring Scales

The feasible optimization strategies differ with the scale of the codebase being refactored. We consequently divide tasks into four scales:

• Kernel: The refactored code is a single computational routine. The computation to be performed is specified by the task; only data layout and instruction ordering may be changed.
• Module: The refactored code is a function or module. The function signature and observable behavior must remain unchanged, while the algorithm and data structures may be replaced.
• Package: The refactored code comprises all public interfaces of a package. Calling conventions and return values must remain unchanged, but the internal implementation may be redesigned wholesale.
• Repository: The refactored code is an entire repository. The externally visible command behavior must remain unchanged, while the internal structure and implementation language may be freely selected.

As Table 1 shows, as codebase scale increases, effective optimization shifts from local modification toward wholesale rewriting, and the choice of programming language correspondingly broadens. Local changes must build on the existing code and therefore remain in its original language; a complete rewrite reimplements all externally visible behavior, so the choice of language is no longer constrained.

Task Format

Each task is packaged in Harbor format and provides the Coding Agent under evaluation with a task description and an initial workspace. The workspace contains one correctly functioning implementation, called the reference implementation and denoted by S. The Agent modifies it; the version left in the workspace at the end is its submission, called the candidate implementation and denoted by S′. The inputs used for functional testing and timing are held by the evaluator and do not appear in the workspace, so the Agent cannot hard-code special handling for particular inputs.

Once a candidate implementation is received, the evaluator applies a fixed sequence of checks: it first verifies that functionality has been preserved, then measures the performance improvement. Submissions that fail functional testing do not proceed to timing. Both checks compare only the program’s external behavior and do not inspect the code itself. There is therefore no reference answer that an implementation must match; the implementation strategy does not affect the scoring criteria.

Functional Testing: Covering All Behavior

The credibility of functional testing depends on complete test coverage. Tests are prepared in three stages by an Agent on the task-authoring side, and the output of each stage is reviewed by an engineer familiar with the relevant domain.

• Enumerating the functionality. If the code being refactored already includes a complete test suite, we use it directly. Otherwise, the Agent reads the code in full and lists each externally promised function. The granularity depends on scale: Package- and Repository-level tasks decompose public interfaces into a set of operations, while Kernel- and Module-level tasks list properties that must remain unchanged, including return values, data types, element order, exception behavior, and whether inputs are modified in place.
• Generating tests from the checklist. Each functionality item must have corresponding test inputs. Expected outputs are taken from the reference implementation itself: each input is run five times on the reference implementation, and only results consistent across all five runs are retained as comparison targets, excluding content that naturally varies between runs. Timestamps, temporary files, timing information, and other irreproducible outputs are excluded from scoring; the Coding Agent under evaluation is therefore not required to reproduce inherently nondeterministic output.
• Auditing coverage. After test preparation, the Agent checks every item in the checklist and confirms that it has corresponding test coverage. The audit result and any uncovered items are recorded in the task metadata for human review; omissions trigger a return to the previous step for additional test generation.

The resulting tests are grounded in the reference implementation’s actual behavior rather than the task author’s experience or intuition. During evaluation, candidate and reference outputs are compared item by item. In a Repository-level task, one test case is a sequence of commands; each command’s exit code, standard output, standard error, and filesystem state are checked separately. Functional testing provides no partial credit: a submission passes only when every test case matches exactly.

Performance Testing: Measuring Credible Speedups

The speedup ratio is defined as r = c(S) / c(S′), where c(·) is the execution cost measured by the evaluator rather than a value reported by the candidate program. Most tasks measure cost using wall-clock time; a small number of tasks run in closed simulators and instead use simulated instruction-cycle counts, which are identical across runs and unaffected by environmental variation.

The principal difficulty in timing is noise: the same code can take different amounts of time in two runs, so a single measurement is insufficient evidence of a genuine performance improvement. The evaluator therefore uses three safeguards:

• Paired interleaving. The reference and candidate implementations run in pairs on the same batch of inputs, with their order alternated from one pair to the next. Perturbations such as CPU frequency scaling and competition for resources between adjacent processes consequently affect both sides equally. The measured speedup thus reflects the difference between the two implementations rather than fluctuations in the environment.
• Warm-up before timing. Several untimed runs are performed before formal measurement so that caches and compilation reach a stable state.
• Repeated calls for short functions. For functions whose execution time is too short to measure reliably, each measurement invokes the function repeatedly, making the total duration exceed the timer’s precision limit.

If the runtime difference between the original and refactored implementations falls within the variation observed across repeated measurements, it is treated as no speedup. The required speedup threshold is specified by each task and must hold consistently across repeated measurements.

Compliance and Anti-Circumvention

Both functional testing and timing assume that the candidate implementation participates in evaluation according to the rules. To enforce that assumption, we add two measures beyond the two tests themselves:

• Source inspection. This occurs before functional testing. Each task provides a checklist specifying permitted imports and prohibited function calls. The evaluator checks every import and function call appearing in the submission against the checklist. If a prohibited item is found, the submission receives zero credit and neither functional testing nor timing is performed.
• Execution isolation. This takes effect during timing. The reference and candidate implementations run under separate restricted accounts, each with a limit on the number of processes it may create, preventing computation from being delegated to another process to evade timing. While one side is being timed, the other side’s processes are suspended; neither implementation can prolong the other’s runtime by consuming processor resources.

Neither measure involves model judgment: the rules are fixed by the task description, and the same submission receives the same determination on every run.

In total, a submission must pass three criteria: functionality, performance, and compliance. A submission that passes all three is called a strict success in this blog. In reporting results, functionality and performance are listed separately rather than combined into a single score: a behaviorally correct submission that does not become faster and a faster submission that contains errors are different failure modes, and only separate reporting can distinguish them.

FrontierRefactor Preview-30

The design above has been implemented across a set of tasks. We first release 30 of them as FrontierRefactor Preview-30: 3 Kernel-level, 17 Module-level, 8 Package-level, and 2 Repository-level tasks.

Experimental Results

Experimental Setup

The model under evaluation was GPT-5.5, with reasoning effort set to medium and driven by Codex CLI 0.145.0-alpha.27; task packaging and execution were handled by Harbor 0.18. Each of the 30 tasks was attempted independently three times, for a total of 90 attempts. Every attempt started from the task’s initial workspace; no process or result from an earlier attempt was carried into the next. The three results are therefore independent and can be used to assess stability on the same task.

Per-Task Results

Tables 3–6 report the results of three attempts for each task, separately for the Kernel, Module, Package, and Repository scales. The columns have the following meanings.

Functionality: whether functional testing was passed; ✓ denotes a pass and ✗ a failure.

Speedup:r = c(S) / c(S′), measured by the evaluator. An attempt that fails functional testing is not timed and is recorded as N/A; if all three attempts fail functional testing, the entire cell is recorded as —.

Strict success: whether all three criteria—functionality, performance, and compliance—were passed.

Steps: the number of interactions between the Agent and the environment during the attempt. Each interaction consists of one model output and the tool calls it triggers; the step count therefore reflects how many times the Agent repeatedly modified the task.

Results Analysis

Most attempts passed functional testing, but only a few met the speedup requirement

Refactoring tasks require two conditions to be satisfied simultaneously: preserving existing functionality and improving runtime. To compare their difficulty, we applied three successive checks to the 90 submissions: whether they passed functional testing, whether runtime genuinely decreased, and whether the decrease reached the task-specific threshold. As Figure 2 shows, 53 of the 90 submissions passed functional testing, 23 genuinely reduced runtime, and 17 ultimately reached the required magnitude. The first check eliminated 37 submissions, while the latter two eliminated a further 36.

Among the submissions eliminated by the latter two checks, a substantial fraction did not merely remain at the original level: of the 53 submissions that passed functionality, 14 were slower than the reference implementation, while the runtime change for another 16 was too small to distinguish from measurement noise. async-write-buffer produced 0.94×, 0.95×, and 0.95× across its three attempts; multi-level-exact-lookup produced 0.82×, 0.78×, and 0.44×. Their behavior was fully preserved, but they became slower—a class of result that only timing can reveal.

The 17 submissions that passed all three checks were also concentrated in a small number of tasks. The far-right panel of Figure 2 shows how many tasks remain successful as the criteria become progressively stricter: 23 of the 30 tasks were solved correctly at least once, the number fell to 10 when strict success was required, and only idna achieved strict success on all three attempts. iterable-shard-skipping achieved a measured 299× speedup, the largest in the entire task set, but its other two attempts failed to reproduce the full iteration semantics. Finding an effective refactoring strategy once and finding one reliably on every attempt represent different levels of capability.

As codebase scale increases, functional pass rates decline while attainable speedups rise

Figure 3 shows the results broken down by codebase scale. From Kernel to Repository, the scope of the refactored code increases step by step. We report three statistics for each scale: the functional pass rate, the number of steps used to complete an attempt, and the measured speedup. Functional pass rate does not decline smoothly; instead, it drops sharply at one transition: Kernel and Module achieve 67% and 69%, respectively; Package falls to 50%; and all six Repository-level attempts fail. The drop occurs when the code scope expands from a single function or routine to a complete set of public interfaces.

Although all six Repository-level attempts fail, none is wholly incorrect; each retains only a small number of discrepancies. As described above, a Repository-level test case checks multiple results individually. The two tasks contain 5,588 and 160 comparison items, respectively: across three attempts, rtk passes 4,191, 4,191, and 4,318 items, while uv-pip-compile passes 120, 120, and 110.

The second statistic is the number of steps used to complete an attempt, which also rises with codebase scale and indicates that larger tasks require more rounds of interaction. The median step count across all 90 attempts is 14; Kernel and Module have medians of 9 and 10, Package rises to 26.5, and Repository reaches 40. Step count alone, however, is related only to task scale and does not predict whether an individual attempt will succeed. At Module scale, tree-enumeration has the highest mean step count—19.7 steps (19, 16, and 24)—yet only one of its three attempts passes functional testing and none meets the speedup requirement, whereas the largest speedup, 299×, takes only 11 steps. Too few steps reveal another problem: the first two gif-frame-counting attempts take only 1 and 0 steps, respectively, and both submissions have a speedup of 0.03×, making them more than thirty times slower than the reference. The Agent did not time its own refactoring process and therefore could not tell whether a change improved or degraded performance.

Although Package-level tasks are harder to pass functionally, their measured speedups are substantially larger once they do pass. Of the 23 attempts that reduce runtime, 11 are Package-level, with a median speedup of 9.67×; 9 are Module-level, with a median of 4.34×. All three idna attempts remain above 30×, while cssselect and netaddr reach 26.67× and 16.10×, respectively. Although Package-level tasks permit any implementation language, none of the 24 submissions switches languages; all speedups come from rewriting the complete interface in Python. In netaddr, for example, the reference implementation represents addresses and networks with a collection of classes, whereas the rewritten version represents addresses directly as integers and replaces layered object construction with standard-library parsing functions. Package-level tasks therefore test not merely local optimization techniques, but the ability to preserve correctness and performance simultaneously during a wholesale rewrite.

Failed functional attempts differ substantially in the proportion of test cases they pass

Functional testing produces only a binary pass/fail result, yet being one test case short and being half of the test cases short imply very different next steps. Across Package- and Repository-level tasks, 18 of 30 attempts failed; as Figure 4 shows, 7 had pass rates below 70%, 7 fell between 70% and 99%, and another 4 exceeded 99%.

The closest of these 18 attempts to passing came from netaddr: its second and third attempts each passed 1,564 of 1,567 test cases, missing only three, so further correction of the existing implementation might have sufficed. At the other extreme was sqlglot, whose three attempts all had pass rates below 53%; the implementation strategy was misguided from the outset and required redesign. Recording the comparison result for every test case is therefore essential for judging the distance between failure and success and deciding whether to continue correcting the current implementation or start over.

The same implementation can exhibit different speedups on different inputs

Tables 3–6 list only one speedup ratio for each attempt, but a task typically includes multiple timing inputs. These may differ in the amount of data processed by the same operation or in the operation itself. We time and record each input separately rather than merging the results. As Figure 5 shows, 12 Package-level submissions passed functional testing and proceeded to timing. The second commonmark attempt passed all 1,305 functional tests, achieving a 2.97× speedup when rendering one input set but only 1.47× when parsing another, below that task’s 1.5× requirement. We score against the worst input set, so this submission is not counted as a strict success. If the two inputs were averaged, the result would be 2.2× and would appear to pass; the strength on one input would conceal the weakness on the other, and the resulting number would correspond to no actual invocation mode.

Future Work

We will continue this work in three directions.

Expanding the task set. We will broaden both coverage and difficulty by adding more programming languages, application domains, and larger-scale refactoring scenarios. We will also increase the optimization difficulty of individual tasks so that they more closely resemble real engineering situations requiring repeated trade-offs.

Tracking frontier models. Coding capabilities continue to evolve rapidly. We will keep incorporating newly released models into the evaluation and run them under the same standards described here, enabling direct comparison across time.

Publishing a public leaderboard. We will compile each model’s performance on FrontierRefactor into a public leaderboard and update it continuously as new models and tasks are added.