VibeLifeBench: Teaching Agents to Work Over the Long Term in a living World
A 32-year-old product manager is planning a 20-day trip to Japan with her 62-year-old mother and 65-year-old father. It sounds like an ordinary family vacation. Yet the background materials contain a chain of details that could derail the entire trip—each waiting for the AI assistant to identify and handle at the right moment.
- The mother’s passport expires on November 8, 2026, while the return flight is scheduled for May 16. By then, the passport will have only five months and 22 days of validity remaining—exactly eight days short of the six-month requirement. The problem must be surfaced before flights are booked. If it is discovered too late, the entire itinerary may need to be rebuilt.
- The father has insulin-dependent diabetes. He must carry a 20-day supply of insulin, plus a backup supply, in his carry-on luggage. This requires a bilingual doctor’s letter and the appropriate customs declaration. The itinerary must also accommodate his health constraints: no more than four kilometers of walking per day, and no delayed meals.
- The total budget is CNY 60,000, and it is an unbreakable hard cap. The task requires the AI assistant to manage the budget proactively—calculating every expense accurately, reconciling the running total, and identifying deviations before they become problems.
- During the trip, the risks continue to evolve. The airline quietly swaps the aircraft from a B787-9 to a B737-800, invalidating the original seat assignments without sending any notification.
- A phishing email disguised as a “visa expedite fee” appears in the inbox.
- A typhoon forecast is gradually upgraded from low confidence to high confidence, with landfall expected in Kansai.
- The return flight is then delayed by four hours and ten minutes.
We reviewed the complete task trajectories of seven frontier models. The result was striking: not a single model surfaced the mother’s passport-validity problem before flights were chosen, and not one re-selected the seats after the aircraft swap.
Why VibeLifeBench?
The Life Domain: Agents Have Not Yet Truly Entered Everyday Life
In recent years, agent evaluation has focused overwhelmingly on two areas: coding and office work. Benchmarks now cover software engineering, cross-file refactoring, frontend generation, PowerPoint, Excel, and other professional tasks. The priorities of the field are clearly reflected in the benchmark landscape.
The VibeLifeBench paper compares representative agent benchmarks in a single table. Coding, office work, web tasks, and general tool use occupy most of it, while everyday life remains almost entirely absent.
Yet everyday life is where agents are closest to ordinary users—and where users can least afford an assistant that is only “usually correct.”
Life tasks are not simply coding tasks with a different toolset. They introduce at least four additional layers of difficulty:
- Ambiguity: Users often have not fully worked out what they need when they first ask. Requirements emerge gradually during the process.
- Personalization: The correct answer to “help me book a hotel” changes completely depending on the person, family structure, preferences, and constraints.
- Long horizons: A single task may continue for weeks, months, or even across calendar years.
- Difficult verification: There is no equivalent of a test suite that can definitively determine whether an entire family trip has been planned correctly.
Dynamic Tasks: The World Does Not Stand Still
Most previous agent benchmarks follow a one-shot task paradigm. The user provides a single instruction containing the goal, constraints, and background information, after which the agent completes the task in a relatively static environment.
The real world does not behave this way.
Flights are delayed. Listings disappear. New emails arrive. Calendars change. Weather forecasts shift. Prices and inventory fluctuate. An agent cannot simply execute its initial plan; it must continually perceive external changes, interpret new information, and update its subsequent actions.
The central question in dynamic-task evaluation is therefore not whether an agent can complete a fixed instruction. It is whether the agent can update its judgment after the environment changes, take the right action at the right time, and keep the final outcome consistent with the current state of the world.
Long-Horizon Tasks: Progress Must Be Sustained
Real tasks are rarely completed in a single interaction. Preparing a family trip, handling a rental dispute, following an exam schedule, or coordinating a renovation may take weeks or months.
The user cannot provide every detail in the opening message. The task unfolds across multiple stages, conversations, decisions, and tool calls. The agent must preserve long-term context, remember what has already been completed, identify the next required action, and continue moving the task forward at critical moments.
VibeLifeBench defines each task as a living world with a clock.
Each task is a scripted timeline with explicit stages and timestamps. The median simulated horizon is 29 days. The benchmark contains:
- 200 tasks across ten everyday-life domains;
- 22 mock service backends;
- 288 tool interfaces;
- 7,453 scripted events;
- 12,261 fine-grained, weighted checks.
Evaluation does not rely on the agent’s self-report. Instead, it reads the observable artifacts the agent leaves behind: backend states, calendar entries, emails, notes, workspace files, bookings, ledgers, and replies. These artifacts reveal whether the task was actually completed.
Dynamic tasks examine how the world changes. Long-horizon tasks examine whether the agent can keep the work moving as those changes accumulate. Together, they distinguish real-world agent evaluation from the conventional one-shot benchmark.
Leaderboard
The strongest model, Claude Opus 5, achieves an avg@3 of just 0.325—32.5 on the paper’s 0–100 scale. Even its best-of-three result, max@3, reaches only 0.412.
All seven frontier models fall within a narrow range of approximately 0.21 to 0.33, with little evidence of a clear performance hierarchy. In other words, being fluent in conversation and capable of calling tools does not mean an agent can manage a real-life task from beginning to end.
Reliability is an even greater concern. Every model has a min@3 of no more than 0.238, and scores fluctuate noticeably when the same task is repeated. Even when a model completes a task correctly once, it may fail to reproduce the result.
For an assistant entrusted with a family member’s travel plans, this instability may be more concerning than a low average score alone.
Performance also varies widely across domains. Even for Claude Opus 5, scores across the ten domains range from approximately 0.218 to 0.511.
The easy-to-hard pattern is highly consistent across models. Shopping, travel, and renovation are relatively tractable, while team building, rental, and exam preparation are among the hardest domains.
These weaknesses are therefore not random noise. They exhibit a stable structure across real-life scenarios.
Long-Term Assistance Is Not Just Tool Use
The ability to serve as a long-term assistant cannot be achieved simply by adding more tools. At least three capabilities must work together.
Proactivity
A capable agent cannot always wait for the user’s next message. It must decide when to inspect the environment, when to remind the user, when to act, and when staying silent is the correct behavior.
VibeLifeBench makes this measurable through silent mutations. Some changes to the environment produce no agent turn and trigger no notification. Only an agent that proactively re-inspects the world can discover that reality has quietly changed.
Persistence
The agent must connect early constraints, intermediate decisions, and later developments into one continuous state. It cannot begin every turn as though it were its first day on the job.
The results show that current models remain weak in persistence and bookkeeping. The problem is not simply whether they write notes. It is whether those notes form durable, auditable, cross-stage artifacts that can be used to track the task over time.
Long-Horizon Coherence
As a task progresses, agents increasingly lose track of the red lines established at the beginning: budget caps, authorization boundaries, privacy requirements, health constraints, and deadlines.
Across every model, per-stage pass rates in the final third of the timeline are 10 to 15 points lower than in the first third. This indicates that long-horizon coherence is an independent capability. It does not emerge automatically from a longer context window.
From ClawMark to VibeLifeBench
The focus on real-world tasks continues our earlier research direction.
Early benchmarks primarily evaluated one-shot tasks: one instruction, one agent, and one execution.
As agent capabilities improved, the duration of agent work began to expand. An agent may now interact with a user over the course of a week, while emails, spreadsheets, calendars, and external services continue to change between interactions.
ClawMark extended evaluation into multi-turn, multi-day, multi-service, and multimodal office scenarios. Deterministic checkers examine whether an agent can continue completing work in a dynamic environment.
Evaluation is no longer limited to whether a single execution succeeds. It must also determine whether the agent can maintain context, process new information, adapt to a changing environment, and ultimately complete the work.
VibeLifeBench pushes this timeline further.
It moves from the office into everyday life, from several days to 20 or 30 days—and sometimes much longer. It also moves from explicit workflows into scenarios with more implicit constraints and risks that are much closer to the individual.
The underlying question remains the same:
Agent evaluation cannot examine only a single execution. It must measure whether an agent can remain reliable across time, state changes, and repeated interactions.
Infrastructure for Long-Horizon, Dynamic Tasks
If ClawMark and VibeLifeBench reveal how difficult a changing world can be, Terrarium is the machinery beneath the stage that keeps that world running.
When a task moves from a single execution in a static sandbox to a world that changes over time, the evaluation infrastructure must acquire a time dimension as well.
Terrarium is a multi-turn data engine for evaluating and optimizing LLM agents in living environments. Researchers can define task programs in pure Python, compose services such as email, databases, calendars, and files, drive agents through multiple rounds of action, and run programmatic checks at any stage.
It addresses three foundational challenges.
First, the environment changes. New emails can arrive, files can appear, and database records can be updated between two agent turns.
Second, the process is not linear. A task can branch or loop depending on the actions the agent has already taken.
Third, proactivity becomes testable. Heartbeats and webhooks allow the agent to decide autonomously whether to inspect the environment, take action, or ignore an event.
Terrarium is therefore more than an environment in which a benchmark runs. It turns long-horizon, dynamic, and proactive task structures into an engineering system.
Through composable services, mutable environments, multi-turn execution, stage-aware checks, and reproducible run configurations, it enables complex tasks to be generated consistently, executed repeatedly, and evaluated reliably.
Once a task acquires a time dimension, the environment and infrastructure supporting it must also be able to handle the changes that time brings.