Terminal-Bench 4.0

Background

Terminal-Bench is an open-source benchmark that measures how well an agent can complete long, realistic pieces of work in a sandboxed terminal. Version 4.0 is a fresh set of 66 community-contributed, maintainer-reviewed tasks, none of which appear in Terminal-Bench 2.1. Each task is built around a real deliverable — a working service, a proof, a CAD model, a trained kernel, a forensic report — and carries an expert time estimate: the median task is estimated at 4 hours of expert work, and the range runs from under an hour to 60 hours.

Where Terminal-Bench 2.x was mostly software engineering and system administration, 4.0 is deliberately wide. The tasks span seven categories:

• Software (18 tasks): databases, distributed systems, frontend performance, data engineering, algorithms
• Science (14 tasks): biology, chemistry, physics, formal mathematics, robotics, linguistics, earth science
• ML (11 tasks): training runs, inference serving, GPU kernels, evaluation harnesses
• Operations (9 tasks): logistics, finance, insurance claims, compliance, supply chain
• Hardware (5 tasks): CAD modeling and RTL design
• Security (5 tasks): cryptography, application security, forensics
• Media (4 tasks): music theory, audio transcription, layout design

We include Terminal-Bench because it is widely reported by model providers, because it reflects the kind of end-to-end terminal work that agentic coding products (Claude Code, Codex, Cursor and others) are expected to do, and because 4.0 raises the ceiling considerably: the tasks are long, multi-step, and graded strictly on the final artifact.

Terminal-Bench is developed by the Terminal-Bench community and the Harbor team as an open-source effort. We thank the community for building the benchmark and for helping us integrate it into our evaluation suite. To learn more or contribute tasks, visit tbench.ai.

Below is an example task from the Operations category (you can find the full task on the Harbor Hub).

Speed up the worker startup for the financial transaction processing pipeline so overdraft notifications are correct, not missed or duplicated, and arrive within 5s during worker respawns and rolling deploys in fresh containers.

Add any extra Python packages to /app/src/worker/requirements.txt.

Methodology

All models were benchmarked using mini-swe-agent, a minimal, model-agnostic agent harness. The model is given the task instruction and a single tool, bash: each turn it issues one or more shell commands, which run in a fresh subshell inside the task sandbox, and it reads back their output. There is no step or cost limit; the agent works until it signals that it has finished or reaches the task’s time limit.

All results reported are avg@3: each model is run through the full benchmark three times, each run scored as pass@1 (one attempt per task), and the displayed score is the mean of the three run-level scores, with error bars showing the standard error across those three runs. Each task is graded by its own verifier, which runs after the agent finishes and inspects the final state of the environment or the artifact the agent produced. A model must pass the verifier’s full test suite to get any credit for a task — there is no partial credit.

Tasks are run in isolated sandboxes on Daytona, following the upstream Harbor setup, including the three tasks that require a GPU. Each task has an 8-hour agent time limit, and each verifier has its own task-specific timeout; an agent that exceeds the time limit, or a verifier that cannot produce a result from the agent’s submission, scores zero for that task, matching Harbor’s semantics.

Comparison to Terminal-Bench 2.1

Our Terminal-Bench 2.1 results used the Terminus 2 harness; Terminal-Bench 4.0 uses mini-swe-agent. The Daytona sandboxes and pass@1 scoring are the same. The task set also changed:

• New tasks. All 66 tasks are new; there is no overlap with the 89 tasks in 2.1.
• Categories instead of difficulty tiers. 2.1 tagged tasks as easy/medium/hard. 4.0 tags them by domain (the seven categories above) and by an expert time estimate. We report scores per category.
• Long time limits. Every 4.0 task allows the agent 8 hours.
• Broader scope. 2.1 was predominantly software engineering. Roughly three quarters of 4.0 is outside traditional software engineering — science, ML systems, operations, hardware, security, and media.

Because the task set is entirely different, scores are not comparable across versions.