On Policy Annotation: Minimal Human Edits Unlock Massive Gains in LLM Agents
GITHUB GUIDELINE NOTION-Version 1. The Need for a Fast System in Terminal-Based SWE Data Collection In terminal-based SFT data collection for software engineering (SWE) tasks, system responsiveness is a crucial determinant of both data efficiency and data quality. For annotators who are not proficient with command-line interfaces, two fundamental issues—slow typing speed and difficulty recalling commands—create significant friction in the labeling process. First, the interaction cost of terminal input is inherently high. Unlike graphical interfaces that offer affordances such as buttons, menus, or auto-completion, terminals rely entirely on textual command entry. Each operation requires the annotator to recall and retype precise commands and parameters, often under the risk of syntax errors. When annotators frequently make syntax or command errors due to limited terminal proficiency, the repeated cycles of editing, rerunning, and verifying become time-consuming and mentally exhausting, causing substantial inefficiency and fatigue throughout the labeling process. ...
