2026-08-22 — views
Task Model Induction Recovers Interleaved Workflows from Computer Traces
Read this because AI engineers building desktop agents can use TMI to learn real-world workflows from raw interaction data, enabling more robust and auditable task execution.
New TMI method extracts structured task models from passive computer-use traces, significantly improving agent accuracy over existing baselines.
The Challenge of Naturalistic Computer Use
Naturalistic computer-use traces, which consist of passively recorded screenshots and mouse or keyboard actions, serve as a valuable resource for deriving symbolic, auditable, and reusable models of how everyday work is performed. As computer-use agents begin to enter real-world work environments, the ability to learn how tasks are actually performed becomes critical. Organizations also require the ability to audit and reuse this knowledge effectively. However, inducing such task models presents significant challenges. Activity is typically observed only as low-level events, and real-world work is inherently multi-threaded with interleaved goals. Existing methods generally assume a given task or a single workflow, producing only step-level summaries rather than structured task models.
Introducing Task Model Induction
To address these limitations, researchers have introduced Task Model Induction (TMI). This approach is designed to discover latent tasks within unconstrained traces, effectively disentangling concurrent activity. For each latent task identified, TMI induces a task model that pairs a hierarchical objective model of recursive goal decomposition with a procedure model of the control flow that organized the execution. This dual-model structure allows for a more comprehensive understanding of the task than previous methods.
Performance and Accuracy
The efficacy of TMI has been demonstrated through both intrinsic and extrinsic evaluations. On controlled human and agent trajectories, TMI recovers interleaved tasks with a 0.974 agreement against ground-truth groupings. Furthermore, it reconstructs 74.9% of the observed execution steps, a performance metric that far exceeds that of the strongest workflow induction baseline. In extrinsic evaluations, skills derived from TMI’s task models improve held-out task accuracy by 30.0% over the strongest baseline. These results indicate that TMI provides a significant advancement in the ability to model complex, real-world computer usage patterns from passive data sources.
Practitioner note
Developers building autonomous agents should consider integrating TMI to extract structured task models from raw interaction logs, enabling more accurate task execution. By leveraging the hierarchical goal decomposition and control flow models, engineers can create agents that better understand and audit real-world workflows.