2026-10-02 — views
AutoSynthData: Generating Training Data for Enterprise Agents
Read this because This system automates the creation of high-quality synthetic data by targeting specific capability gaps, offering a scalable solution for fine-tuning agents in complex environments.
ServiceNow researchers introduce AutoSynthData, a pipeline that converts model failures into validated training tasks for enterprise agents.
Addressing Enterprise Agent Weaknesses
Enterprises require agents that function effectively within their specific environments, shaped by unique systems, rules, and data states. A model may possess broad capabilities yet struggle with particular workflows, tool combinations, or constraints. The primary challenge lies in converting these observed weaknesses into effective training data. An individual failure provides limited insight, but training a model requires many new tasks that exercise the same capability across different situations. These tasks must be completable in the environment, resemble realistic user requests, and include a reliable method to verify success.
The AutoSynthData Pipeline
At ServiceNow CoreAI, researchers developed AutoSynthData to transform capability gaps into training data. The system utilizes a target model’s failures and a stronger teacher’s successes to determine what the model should learn next. It then generates and validates new tasks that exercise those specific capabilities. As the model improves, the curriculum automatically shifts toward areas it still finds difficult. The pipeline is illustrated using EnterpriseOps Gym, leveraging a released dataset to demonstrate the process.
Defining Useful Agentic Tasks
An agentic environment defines the world in which an agent operates, including observable states, modifiable states, tools, APIs, and state transitions. A task is instantiated within this environment and consists of a system specification, user prompt, and verifier. The system specification defines constraints, including instructions, policies, and initialization states. The user prompt specifies the user’s goal and constraints. For a task to be useful for training, it must satisfy three properties: feasibility, ensuring a valid trajectory exists; realism, ensuring the prompt resembles plausible user requests; and difficulty, ensuring the task exposes a current weakness in the agent.
Verification and Curriculum Generation
The verifier determines whether a trajectory successfully completes the task. It must be consistent with the prompt and specification, sound in rejecting invalid trajectories, and complete in accepting all valid solutions. AutoSynthData begins by evaluating the target model in the environment using diagnostic tasks to identify patterns in failures. A stronger teacher helps characterize solvable tasks and successful behavior. These findings are distilled into sanitized capability specification cards. The generator uses these cards to create new tasks with different prompts, states, and solution paths, without receiving original prompts or verifier details.
Scaling and Implementation
The system generates tasks in two phases: target and multiply. The target phase creates core training samples from capability specifications, where workers generate independent tasks in parallel. Each candidate undergoes validation, execution, solver evaluation, and repair before acceptance. The multiply phase expands the dataset by creating novel variants of accepted samples. Each variant has its own user request, environment state, and reference trajectory. A multiplied sample cannot seed another multiplied sample, anchoring expansion to the vetted target set. Implementation details separate generation control from environment-specific execution. A shared controller coordinates quality and coverage, while an adapter handles execution and verification. High-quality synthetic data requires rigorous checks to ensure tasks are executable, solutions work, and verifiers distinguish success from failure.
Practitioner note
Developers should implement rigorous validation loops for every generated task to ensure feasibility and correctness before adding them to training datasets. Monitoring the shift in curriculum as the model improves is essential to maintain training signal relevance and avoid redundant data generation.