TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents
arXiv · HuggingFace · ▲47
Abstract (verbatim)
As large language models and harness frameworks continue to advance, agents operating in terminals are increasingly capable of performing a broader range of general computer-use tasks beyond coding. However, existing benchmarks do not adequately evaluate general-purpose terminal computer-use agents (TUAs): general computer-use benchmarks primarily target graphical user interfaces (GUIs), whereas terminal-based benchmarks largely emphasize technical and programming-centric workflows historically native to the shell. We introduce TUA-Bench, a general-purpose benchmark for terminal-use agents. TUA-Bench includes 120 real-world tasks across five task families, covering routine digital activities-including document editing, email management, and live-web information seeking-as well as scientific and engineering workflows co-designed with PhD-level domain experts that require specialized software. This breadth distinguishes TUA-Bench from prior shell-focused or domain-specific benchmarks. Each task is manually designed, runs in a real terminal with a deterministic setup script, and is evaluated by an execution-based scoring protocol. We find that the strongest frontier agent, Claude Code with Claude Opus 4.8 max reasoning effort, achieves 65.8% overall performance, with substantial gaps across both tracks. By providing a broad and realistic evaluation of terminal-use capabilities, TUA-Bench aims to accelerate the transition from narrow, task-specific assistants to general-purpose agents capable of operating reliably across diverse digital environments.
Background
Background Analysis
1. Technical Context
With the advancement of large language models (LLMs), terminal-use agents have evolved from simple programming assistants to general-purpose tools capable of executing complex, multi-step tasks. These technologies are applied in scenarios requiring efficient computer operation, such as document editing, email management, web information retrieval, and scientific or engineering workflows. Command-line interfaces (CLIs) are naturally suited for LLMs because commands are explicit, feedback is structured, and complex processes can be composed through scripts. However, existing terminal benchmarks primarily focus on technical or programming tasks, failing to evaluate agents’ performance in broader daily and professional contexts.
2. Previous Limitations
Early terminal benchmarks (e.g., Terminal-Bench) were limited to shell-native workflows, neglecting general-purpose use cases. For example, ordinary users might need to edit documents or query web information via terminals, while researchers rely on them for specialized software. These tasks require not only technical skills but also long-horizon planning, tool coordination, and error recovery—capabilities poorly measured by existing tests. Additionally, GUI-based benchmarks, while human-friendly, introduce visual perception challenges that distract from assessing core language model abilities.
3. Proposed Solution
TUA-Bench addresses this gap by designing 120 real-world tasks (covering both daily and professional scenarios). Tasks are co-created with domain experts (e.g., PhDs in biology and engineering) to ensure realism and rigor. Each task runs in a controlled environment with automated verification, and the evaluation protocol focuses on execution-based outcomes rather than subjective judgments. The study tests state-of-the-art models (e.g., Claude Code) and finds even the best agents achieve only 65.8% success, highlighting gaps in planning, tool use, and error handling.
4. Key Differences
Compared to prior work, TUA-Bench stands out for its generality and realism. It expands beyond traditional terminal tasks to include non-technical scenarios (e.g., document editing) and incorporates professional workflows designed with domain experts. Tasks are carefully curated to eliminate ambiguity or triviality, ensuring meaningful evaluation. This makes TUA-Bench a reliable benchmark for advancing general-purpose terminal agents capable of operating across diverse digital environments.
