EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environments
arXiv · HuggingFace · ▲43
Abstract (verbatim)
Autonomous agents are increasingly expected to improve executable policies through feedback, yet existing evaluations often collapse this process into a final score or confound it with open-ended software-engineering progress. We introduce Autonomous Policy Evolution, a controlled evaluation setting in which a harness-model agent repeatedly edits an executable policy system under a fixed interaction budget. We instantiate this setting in EvoPolicyGym, a benchmark built from compact interactive RL environments that evaluates how agents iteratively improve explored policies. On the EvoPolicyGym suite, GPT-5.5 achieves the strongest aggregate rank score and top-two performance on all 16 environments. Beyond leaderboard results, EvoPolicyGym also provides trajectory-level diagnostics that distinguish how agents allocate budget, convert feedback into parametric tuning. These analyses show that strong autonomous policy evolution depends not only on isolated task wins, but on discovering task-appropriate mechanisms and refining policies under bounded feedback.
Background
Background Analysis
1. Technical Context and Real-World Needs
Autonomous agents (e.g., code generation tools or language models) need to continuously improve their behavior through environmental feedback rather than producing fixed outputs. For instance, programming agents must adjust code based on test failures, while language models refine responses via reflection. The core need is to convert feedback into iterative improvements of executable policies within limited interactions, ensuring stability in new scenarios.
2. Limitations of Previous Approaches
Existing evaluation methods suffer from two key flaws:
- Outcome-focused blind spots: Final scores ignore critical issues in the improvement process (e.g., blind retries, overfitting to visible feedback, or neglecting validation set generalization).
- Confounding open-ended engineering: In unconstrained tasks (e.g., continuous software projects), evaluations are distorted by irrelevant factors like specification changes or maintenance quality, making it hard to isolate policy optimization capabilities.
3. Proposed Solution
The paper introduces the Autonomous Policy Evolution framework to address these issues:
- Controlled benchmarking: EvoPolicyGym provides a set of compact interactive RL environments (e.g., robotics, driving simulations) where agents repeatedly edit policies under a fixed budget (e.g., 128 interaction rounds) and optimize based on sandbox feedback.
- Process-oriented evaluation: The benchmark focuses on the policy evolution process itself, not just task performance. Agents submit policy versions and receive training feedback, but validation/test results are computed server-side to ensure focus on improvement efficiency.
- Trajectory-level diagnostics: Full execution-feedback-revision traces are recorded to analyze how agents allocate budgets, diagnose errors, and balance exploration-exploitation.
4. Key Differences from Prior Work
- Process over outcome: Traditional benchmarks (e.g., OpenAI Gym) evaluate final task performance, while EvoPolicyGym measures generalization via hidden validation sets.
- Controlled variables: Unlike open engineering benchmarks (e.g., software maintenance), EvoPolicyGym restricts environment changes to isolate policy optimization.
- Diagnostic insights: Fine-grained trajectory data (e.g., policy revision logs) reveal how feedback translates into concrete improvements (e.g., parameter tuning or structural changes).
This framework enables deeper insights into how autonomous agents iteratively optimize policies, moving beyond simple score comparisons.
