How Automatable Is This Workflow?

A practical framework for Enterprise AI

Worktrace AI Technical Staff · July 2026

Introduction

Enterprise automation has become a defining priority in the AI adoption era. Most organizations want to transform their workflows with more automation, but most do not know where to start. Recent industry surveys suggest that roughly two-thirds of organizations are experimenting with agentic AI, while fewer than one in ten have scaled it to deliver tangible value within even a single business function ^[1]. Our company was founded to help close that gap. Despite being an AI-native organization, we design solutions around human needs, not technology alone. We consider each client's goals, workflows, and comfort level with automation, then recommend practical solutions designed to improve the enterprise KPIs that matter most and enable people to do their best work. To make this decision process transparent, today we introduce a practical automatability framework developed and iterated using more than 500 agent blueprints spanning 14 business functions dominated by support, fulfillment, finance, and back-office operations.

Automation frameworks are not new. Organizations have long used robotic process automation, workflow orchestration, and intelligent document processing to automate predefined steps within stable, repeatable processes. Now with LLM-powered agents, the scope and the nature of that opportunity have changed. These AI agents can interpret unstructured context, use tools across multiple systems, make bounded judgements, and adapt a plan during execution. This shift has prompted new frameworks that classify levels of agent autonomy ^[2,3], assess the risks associated with autonomous operation ^[4], and identify capability and security constraints such as verifiability ^[5], reliable task horizons ^[6], and the lethal trifecta ^[7].

These approaches provide essential building blocks, but they do not by themselves give enterprise leaders and AI engineers an end-to-end procedure for answering a practical workflow-level question: For this specific workflow, how much autonomy should we build, and what must be true before that level can be introduced safely? Our contribution is an operational assessment method that answers that question.

Our framework proceeds in two parts:

• A six-level model of automatability. We define six levels based on how responsibility, judgement, and control shift between the AI agent and the human as automation increases.
• An operational assessment rubric. We introduce seven quality dimensions and a supporting criterion bank for evaluating a workflow's automation potential. The resulting assessment maps each workflow to an appropriate level of automatability and produces an itemized backlog of the capabilities, controls, and process changes required to reach that level.

Automatability levels

We introduce six levels of agent autonomy based on how responsibility, judgement, and control shift between the AI agent and the human as automation increases.

Figure 1. How agent and human roles shift across automatability levels. Click a level to see its role contract and to preview the worked example below at that level.

Our framework is built on the premise that humans should remain meaningfully involved at every level of automation. Its purpose is not to remove workers from organizations, but to reimagine their roles in an AI-enabled workplace. By reducing repetitive and low-value work, automation can help people become more productive and focus on more engaging and consequential projects. Accordingly, our rubric requires a human to retain accountability, oversee exceptions, and maintain audit responsibility for every workflow. We automate execution, not accountability.

The framework therefore stops at managed by exception rather than extending to full autonomy. As autonomy increases, so does risk. Eliminating the final layer of human review would remove a critical safeguard against silent errors, performance drift, and unforeseen edge cases, while often producing only marginal additional labor savings.

Reimagining an automatable workflow

Let's walk through a concrete workflow example: resolving a damaged-order claim in e-commerce customer support. In this workflow, a customer reports that an item arrived damaged; a representative reviews the photos and order history, checks the delivery record, decides a remedy under the refund policy, issues the refund or replacement, replies to the customer, and closes the ticket.

Click through the levels to watch the same workflow, and the human's role in it, transform.

Assessing automatability of a workflow

To determine the appropriate level of automatability for a workflow, we present a rubric of requirements associated with each level of automatability.*

We organize the rubric into two primary categories:

• Capability and reliability (D1–D5): This category asks, Can the agent perform the workflow dependably? A gap in these dimensions represents a technical barrier. Closing it typically requires engineering work, such as improving the agent's capabilities, tools, data, or evaluation performance.
• Risk and governance (D6–D7): This category asks, Should the workflow be delegated, and can the resulting autonomy be governed effectively? Some governance gaps can be closed through safeguards, such as containment mechanisms, monitoring, escalation procedures, or human approval. However, failed governance hard gates are non-negotiable ceilings.

D1. Output verifiability: Can success be checked without a human?

Verifiability is the strongest individual predictor of a workflow's automatability. Unless success can be defined clearly, an AI agent cannot be reliably directed toward the intended outcome or determine whether its work is complete. We evaluate the success specification through five independently answerable properties:

• Automated verifiability (D1.1): Success can be evaluated through an executable, deterministic check.
• Rubric-based verifiability (D1.2): Success can be evaluated using a complete and unambiguous rubric that a human—or an LLM judge—can apply consistently.
• Completion (D1.3): The workflow has a clear definition of what constitutes finished work.
• Evidence (D1.4): The workflow produces sufficient evidence to determine whether the completion criteria have been met.
• Integrity (D1.5–1.7): The workflow has independent, ungameable checks that can be trusted once no human is behind them.

For the purpose of assigning an automatability level, D1.1 and D1.2 provide functionally equivalent verification pathways. D1.1 enables verification through deterministic logic. When D1.1 is not satisfied, a sufficiently clear set of binary success criteria under D1.2 may allow an LLM judge to perform the verification instead.

At Level 5, we require independent and adversarially tested verification, following the separation-of-duties and independent-verifier patterns ^[11,12]. At Level 3, the agent begins operating through its own plan–act–verify loop and must therefore be able to assess the quality, completeness, and ungameability of its output — specification gaming and reward hacking are documented failure modes of goal-driven agents ^[16,17,18]. At Levels 1 and 2, a formal machine-applicable success specification is not required because a human reviews the completed work and authorizes any consequential action. At these lower levels, a clear completion definition (D1.3) and sufficient supporting evidence (D1.4) may be adequate.

D2. Operating envelope: Can the agent reliably determine whether each instance is within its approved scope?

No workflow we observed follows a single, tightly bounded path. Each workflow includes routine cases, exceptions, and instances that fall outside the agent's intended scope. For an agent to operate safely, these boundaries must be defined clearly enough for the system to determine which inputs it should accept and which cases it should handle, escalate, or reject.

This concept is analogous to the operational design domain in driving automation. It assesses whether the workflow's inputs and in-scope versus out-of-scope boundaries are machine-legible, allowing the agent — and the system that assigns work to it — to classify each instance against a declared operating envelope. A well-defined envelope enables a single workflow to serve routine volume efficiently while routing exceptional cases to the appropriate human or process.

D3. Accessibility: Can every action in the workflow be performed programmatically?

A workflow cannot be fully automated if the agent lacks access to critical tools or if authorization requirements create frequent interruptions. Authentication friction, such as two-factor authentication or CAPTCHA challenges, must be handled through a pause-and-resume human handoff, never by having the agent attempt to bypass or solve the challenge itself. The frequency of these interruptions places a practical limit on how much autonomy the workflow can support.

We therefore assess whether each required system is functionally reachable, whether credentials can be provisioned securely, how authentication and authorization are handled, whether suitable test environments are available, and whether rate limits or other access constraints could prevent reliable execution.

D4. Judgement codifiability: Can the decisions be written down?

Nearly every workflow involves some degree of judgement, so a binary question such as "Does this workflow require human judgement?" is not useful — the answer is yes for almost every real-world workflow. Judgement codifiability provides a more meaningful distinction: it separates judgement that can be incorporated into an automated process from judgement that must remain with a human.

The relevant question is not whether judgement is required, but whether that judgement can be codified through explicit rules, decision tables, or established precedents, and whether the boundary for human escalation can be stated precisely. When judgement is delegated to a model rather than encoded deterministically, the reliability of that decision is evaluated separately under D5 below.

D5. Execution reliability: Does the agent execute correctly and consistently across repeated runs?

Some failure modes may remain hidden while a human reviews each step, but they become consequential once the workflow begins operating without continuous supervision at Level 3 and above. The question here is whether these actions can be executed reliably under real operating conditions. We therefore assess:

• Idempotency: Retried, duplicated, or overlapping runs do not apply the same side effect more than once.
• Bounded model judgement: Any decision delegated to a model is constrained to a validated output space before it can trigger a consequential action.
• Input and dependency stability: Required inputs, tools, APIs, and upstream systems remain sufficiently consistent and available for reliable execution.
• Correlated failure containment: A shared change, such as a model, prompt, policy, or tool update, cannot cause uncontrolled failure across many agent instances; deployments are staged, monitored, and reversible to limit the blast radius. A fleet sharing one model or prompt fails in a correlated way ^[15].

D6. Risk, reversibility and security: What is the maximum credible harm, and can it be contained or reversed?

Capability alone does not determine how much autonomy a workflow should receive. In many cases, the autonomy ceiling is set by reversibility: the more difficult an action is to contain, correct, or undo, the greater the need for human oversight.

Containment mechanisms can limit the impact of failure, but they are not sufficient on their own. For example, an agent may remain within a prescribed spending cap while repeatedly making individually permissible decisions that generate cumulative losses, as illustrated by Anthropic's Project Vend experiment ^[9].

In this dimension, we assess:

• Legal and policy authority: Is the organization permitted to automate the workflow and its associated decisions?
• Compliance constraints: Are any decisions subject to mandatory review, approval, or regulatory controls, such as the human-oversight requirements of the EU AI Act ^[10]?
• Worst-case containment: Are plausible adverse outcomes understood, bounded, and detectable?
• Reversibility: Can consequential actions be staged, paused, corrected, or rolled back before harm becomes permanent?
• Data and security scope: Can access to sensitive data, credentials, systems, and actions be limited to the minimum necessary?

D7. Fallback and audit: What happens when it goes wrong, and can we reconstruct what happened?

Agents should halt when predefined conditions are met instead of improvising beyond their approved operating boundaries. Accountability also requires that every action, decision, and handoff be reconstructible after the fact.

This dimension assesses organizational and system readiness. Organizationally, each exception must have a named owner, and the expected exception volume must be manageable for the team to absorb and resolve it — management by exception collapses under alert fatigue ^[14]. In terms of system readiness, halt and escalation conditions must be clearly defined, logs must support full reconstruction of the agent's actions ^[13], and non-escalated outputs must be sampled and reviewed against human judgement on a defined schedule. Because some of these controls can't be visible in workflow traces, we assess them primarily through stakeholder interviews and system documentation.

Automatability scoring

Our rubric is divided into hard gates and core criteria. Hard gates are evaluated first because they represent non-negotiable requirements. If any hard-gate condition for a target level is not satisfied, the workflow is capped at the preceding level, and the remaining core criteria for that target level do not need to be assessed. Once all hard gates for a level are satisfied, the evaluator can proceed to the core criteria, which define the remaining requirements for achieving that level.

The rubric can be read in two directions:

• Reading a column from top to bottom shows the complete set of requirements for a given automatability level.
• Reading a row from left to right shows the level at which a particular condition first becomes required.

We also classify each criterion by the primary source used to assess it:

• A — Automated assessment: The condition can be evaluated automatically from observed workflow traces and the agent-system proposal. If the current system does not yet satisfy a criterion required for the target level, the result is recorded as an enablement gap rather than a definitive "No" and added to the implementation backlog.
• R — Research-based assessment: The condition can't be observed from workflow traces or the proposal alone. It is evaluated through external-document research or targeted experimentation, such as automated red-teaming of the completion check. An LLM may draft the assessment, but a human must review and sign off on it.
• H — Human validation: The condition requires direct input or validation from a human stakeholder.

Figure 2. The Automatability Rubric: hard gates and core criteria. A filled cell in a row's level strip indicates that the condition must be satisfied for the workflow to operate at that level; the outlined cell marks where it first becomes required. Selecting a target level shows that level's complete requirement set — reading a column of the matrix — while each row's strip reads left to right. A failed hard gate immediately caps the workflow at the level preceding the first level at which it is required. "At-least-one" pairs: either D1.1 or D1.2 may satisfy the verification requirement; either D6.3 or D6.4 the corresponding governance requirement. Gates and core criteria are grouped by dimension — expand a dimension to see its conditions. A workflow's ceiling is the highest level for which every required criterion is satisfied.

A worked assessment: the damaged-order workflow

To demonstrate how to use this rubric, we will go over a full assessment for the damaged-order claim workflow. In this scenario, the client currently operates the workflow at L0–L1: representatives investigate, decide, and issue every remedy themselves, with an AI assistant that summarizes claims and drafts replies (L0); in a small pilot, the agent also stages refunds and replies that a representative approves one action at a time (L1). The proposal on the table is an L3 agent under a pre-run contract: given a claim, pursue the goal - correct remedy issued, customer notified, evidence reconciled - and escalate when a rule fires. The assessment answers whether L3 is justified today, and if not, what exactly stands in the way.

Step through the six stages below. The level meter tracks the verdict as the evidence accumulates.

Figure 3. One assessment, end to end. Define the workflow and its envelope; walk the hard gates with evidence; cap at the first failed gate; keep scoring in full-diagnostic mode; emit the backlog; and set the target level the backlog unlocks.

Automatability in practice

In this post, we introduce a six-level automatability framework and an accompanying rubric for assessing the appropriate degree of automation for individual workflows across seven dimensions. The framework is designed around continued human control and incorporates both capability and safety considerations.

We grounded the rubric in real traces of how people currently perform their work across 14 business functions. However, pre-automation traces can indicate only whether a workflow appears ready for automation. A stronger validation requires post-automation evidence from agents operating the workflows in practice. This would allow us to compare predicted automatability ceilings with realized reliability, identify miscalibrated criteria, and revise level assignments based on observed outcomes. We are working with our clients to gather post-automation data to conduct that validation.

Observing agents in production will be essential to improving the framework. As we build reinforcement learning and evaluation environments, and gather feedback from deployments, we plan to reassess automation readiness against actual agent behavior and recalibrate the rubric accordingly.†

At Worktrace, we develop rigorous automation decision processes to help our customers identify reliable and high-ROI automations to build. By publishing our automation decision process, we hope other organizations and agent builders will benefit from our learnings to evaluate automation feasibility in a systematic, and evidence-grounded way.

* The rubric draws on principles used in recent rubric-based benchmarks ^[8]. It is designed to support both human evaluation and automated scoring by an LLM judge. To promote reliable and reproducible assessments, each criterion is expressed as a single, unambiguous yes-or-no question. A well-designed criterion should produce the same result regardless of who performs the evaluation. In other words, two independent human reviewers examining the same evidence should reach the same conclusion. This emphasis on clarity and inter-rater agreement also makes the rubric more suitable for consistent LLM-based evaluation.

† Two additional limitations of this approach:

• Some requirements are not reliably observable in workflow traces or agent proposals and must instead be assessed through stakeholder interviews, system documentation, and direct human validation.
• The current level placements and hard-gate assignments should be understood as priors rather than fixed truths. They reflect the existing literature, current model capabilities, and our engineering experience. As deployment evidence accumulates and frontier capabilities improve, some criteria will move, some gates may relax, and the framework itself will require continued recalibration.

References

[1] McKinsey & Company (2026). The State of Organizations 2026: agentic AI scaling gap. mckinsey.com

[2] Feng, K.J.K., McDonald, D.W., Zhang, A.X. (2025). Levels of autonomy for AI agents. arXiv:2506.12469.

[3] Osmani, A. (2026). Agentic autonomy levels. addyo.substack.com

[4] Satta Chiris, L., Mishra, A. (2025). AURA: An agent autonomy risk assessment framework. arXiv:2510.15739.

[5] Wei, J. (2025). Asymmetry of verification and verifier's law. jasonwei.net

[6] METR (2025). Measuring AI ability to complete long tasks. arXiv:2503.14499.

[7] Willison, S. (2025). The lethal trifecta for AI agents. simonwillison.net

[8] Arora, R.K., Wei, J., et al. (2025). HealthBench: Evaluating large language models towards improved human health. arXiv:2505.08775.

[9] Anthropic (2026). Project Vend: Phase two. anthropic.com/research/project-vend-2

[10] European Parliament and Council, Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 14: Human oversight, Official Journal of the European Union, 2024.

[11] LoginRadius, "Separation of duties for AI agent workflows explained," 2025. loginradius.com

[12] MindStudio, "What is the verifier pattern in multi-agent systems?," 2025. mindstudio.ai

[13] MakerChecker, "Verifiable audit logs and segregation of duties for AI agents," 2026. makerchecker.ai

[14] "Alert fatigue in security operations centres: Research challenges and opportunities," ACM Computing Surveys, 2025. doi:10.1145/3723158

[15] L. Hammond et al., "Multi-agent risks from advanced AI," arXiv preprint arXiv:2502.14143, 2025.

[16] "A sober look at agentic misalignment in automated workflows," arXiv preprint arXiv:2605.24197, 2026.

[17] "Reward hacking in the era of large models: Mechanisms, emergent misalignment, and challenges," arXiv preprint arXiv:2604.13602, 2026.

[18] "Reward hacking benchmark: Measuring exploits in LLM agents with tool use," arXiv preprint arXiv:2605.02964, 2026.