Skip to main content

Run Results Reference

The words used for the result of a single run, and how the KPI figures are counted. Item names follow what the screen shows. How to read the results is covered in the Record Guide, and the thinking behind the figures in Records and metrics.

The words for the result​

How a single run ended is shown as one of these seven.

Shown asMeaningDecided byAgent completion rate
CompletedThe agent ended without reporting that it could not finish, and its final result did not say so either. This includes runs that ended with no report at all, as long as the final result did not say soAgentCounted as a success (not counted when it ended with no change)
Failed (reported by agent)The agent itself reported that it could not finish, or ended with a final result saying so. Without a report, the category is “Other”. A run that wrapped up at the run time limit and could not finish is “Timed out” insteadAgentCounted as a failure (not counted when stopped for a connection confirmation)
FailedIt ended because of an exception or an abnormality in the runtimePlatformCounted as a failure
Timed outIt reached the time limit and was cut off, or it wrapped up at the run time limit and could not finish (including when the agent reported that it could not finish)PlatformCounted as a failure
Failed (stopped)A person stopped it, or the platform cancelled itA person or the platformCounted as a failure
Not allowed by rulesThe request was stopped by the platform's check on its input (the guardrail) and was not carried outPlatformCounted as a failure
Nothing to doThe run never startedPlatformNot counted
  • “Completed” is no guarantee that the work went well. What was done is checked under “What was done” in the run record.
  • A run that ended with no change still has the result “Completed”. It is a run where the agent reported that what it checked had not changed and it did nothing; its reason is “No change” (in Execution history, a series of them is shown as one line, such as “No change (3 times)”). It is not counted in the Agent completion rate.
  • An operation stopped by the execution rules does not change the result. Stopped operations are listed under “What was done” as denials.
  • The check behind “Not allowed by rules” is not the guardrail you write for an agent (an instruction). The run record gives the reason as “Stopped by the guardrail”.
  • A run stopped for a connection confirmation also has the result “Failed (reported by agent)”. It stopped before using a connector set to confirm before connecting, and its reason is “Stopped for connection confirmation”. It is only waiting for a person to confirm, so it is not counted as a failure and is counted under Excluded as “Stopped for connection confirmation”. The run record does not show “Failure summary” or what to review (Connection Guide).

Reasons for “Failed (reported by agent)”​

When an agent reports that it could not finish, it gives what to review and a comment for the person reading it (what it could not do, why, and what to do next). The category is one of the following; where the agent does not choose one, it is derived from what to review. In the run record, the category appears beside the result and as the “Cause” in the “Failure summary”. The comment appears under “Agent's closing report” in the Failure summary. Where the Failure summary points first follows What to review where it narrows things down (the second table below), and otherwise the category (the table below).

CategoryWhere the Failure summary points first
Could not access the connected service (authentication or permission)Check connector settings
The connected service did not respondCheck connector settings
Restricted by the execution rulesCheck execution rules
Required input was missingCheck step settings and inputs
OtherReview what was done

Where What to review narrows things down, the Failure summary points first as follows.

What to reviewWhere the Failure summary points first
Instructions (prompt)Check step settings and inputs
Settings and connections (harness)Check connector settings, Check execution rules, or Check step settings and inputs (which one appears depends on the cause; the cause is written in the explanation under What to review, or in the agent's closing report)
Restricted by an administrator or the systemReview what was done
External serviceReview what was done
Unexpected errorReview what was done

What happens after each result in a workflow​

ResultWhat happens next
CompletedIt moves on to the next agent on the same step, or, if there is none, to the next step (on a step set to “Run at the same time”, to the next step once all of the results are in). If the next step is a review step, it waits for the review there
Failed (reported by agent)By default, a review request is created and the run stops there. Set the step's “When the agent reports it could not finish” to “Proceed anyway” and it moves on as if completed. Set it to “Go to a specific step” and it goes to the chosen later step
Not allowed by rulesBy default, a review request is created and the run stops there. Set the step's “When not allowed by rules” to “Close as failed” and the workflow run ends as failed. Set it to “Go to a specific step” and it goes to the chosen later step
Failed / Timed outThe same step is tried again. Once the retries set in “Retries per step (failed / timed out)” are used up, another failure or timeout ends the workflow run as failed
Failed (stopped)It is not tried again, and the workflow run ends as failed
Nothing to doIt moves on (on a step set to “Run at the same time”, to the next step once the other agents' results are in)

A run stopped for a connection confirmation does not follow this table: it raises a connection confirmation review request and waits. On approval, the same step starts again; when it is rejected or its wait limit passes, it proceeds as in the “Failed (reported by agent)” row above.

When you cancel the workflow run itself, it ends as “Cancelled” (not “Failed”).

In a cycle, no result stops the sweep. Sweeps continue on the schedule you set. A sweep stopped for a connection confirmation runs the sweep of the same placement once more on approval.

Runs still going, and runs with no recorded result​

StateMeaningHow it is counted
RunningIt has not ended yetNot counted (“Running” under Excluded)
Insufficient recordsIt has ended, but no result was recorded. The run record says “no result was recorded”Counted as neither success nor failure (“Insufficient records” under Excluded). Even one of these makes the Agent completion rate provisional. In “Results of this work” on a project's Overview, the rate shows “—” instead of a value, with “Records incomplete: provisional because N runs have no recorded result” below it
Nothing to doThe result “Nothing to do”Not counted (shown under Excluded)
Stopped for connection confirmationA run stopped before using a connector set to confirm before connecting (result “Failed (reported by agent)”, reason “Stopped for connection confirmation”)Counted as neither success nor failure (“Stopped for connection confirmation” under Excluded)

Agent completion rate​

Agent completion rate = Completed ÷ finished

ItemContent
CompletedThe number of runs whose result is “Completed” (leaving out those that ended with no change)
FinishedCompleted plus “Failed / needs action”. “Failed / needs action” is the total of Failed (reported by agent), Failed, Timed out, Failed (stopped) and Not allowed by rules (leaving out runs stopped for a connection confirmation)
The unit countedOne agent execution. Both a workflow step's execution and a cycle's sweep are counted. A retry counts as one too
PeriodRuns that started in the period you chose
Not includedNothing to do, running, insufficient records, runs that ended with no change, and runs stopped for a connection confirmation
When nothing has finished and no record is insufficientIn “Results of this work” on a project's Overview, the rate shows “—” with “No finished runs yet (completed N / finished N)” below it. In the overall totals it shows “No target”, and on the project list card “No data” (meaning there is nothing to count — not the result “Nothing to do”)
When a record is insufficientIt says “Records incomplete: provisional because N runs have no recorded result” (even when nothing has finished)

The KPI items​

These are the items under “Results of this work” in Overview. The figures for each step are under “Metrics” in “Workflow shape” (see “Figures by step” below). You can choose the period (values are in the Settings reference).

Run results​

ItemHow it is counted
Agent completion rateThe formula above
CompletedThe number of runs whose result is “Completed” (leaving out those that ended with no change)
Failed / needs actionThe total of Failed (reported by agent), Failed, Timed out, Failed (stopped) and Not allowed by rules (leaving out runs stopped for a connection confirmation)
Typical duration (p95)The time from start to end of runs that started and ended in the period. Sorted from shortest, it is the value at the 95% position: about 95% of the runs that ended did so within this time. Runs stopped for a connection confirmation are not included
Runs per dayRuns that started in the period, stacked by the day they started as “Completed”, “Failed / needs action” and “Running”. Nothing to do, insufficient records and runs stopped for a connection confirmation are not counted. With more than 1,000 runs in the period, the bars are not shown
Breakdown: failed / needs actionThe count for each kind of failure or needs-action result (Failed (reported by agent), Failed / Timed out, Failed (stopped), Not allowed by rules and so on), and its share. Only the kinds that have runs are listed, most first; Nothing to do and runs stopped for a connection confirmation are not included
ExcludedThe counts of runs with nothing to do, runs still running, runs with insufficient records, and runs stopped for a connection confirmation

Work steps​

Counted from the agent executions that ended in the period you chose. Both workflow step runs and cycle sweeps are included.

ItemHow it is counted
Counted executionsThe number of executions whose result is “Completed”, “Failed (reported by agent)”, “Failed (stopped)” or “Not allowed by rules”. “Failed”, “Timed out”, “Nothing to do” and executions with no recorded result are not counted (their number is shown as “executions not counted”)
Tool callsThe total of the recorded tool calls of the counted executions
StepsThe total of the work steps of the counted executions. The steps of one execution are settled from that execution and its recorded tool calls; failed connector calls and failed external changes are not counted, and there is a cap per execution. Executions with no recorded count are left out of the total, and their number is shown alongside

The tenant-wide total for the current month (counted by UTC) can be seen by a system administrator under “Workflow steps this month” on the “License” screen of the admin console.

Review figures (workflow only)​

These count the review requests created in the period you chose.

Approved by people and send-back rate (for each review step; shown under “Metrics” in “Workflow shape”)

ItemHow it is countedAutomatic action at a deadline
Approved by peopleThe number a person approved ÷ the number decided. The number decided is the number a person approved plus the number sent back or rejectedWhat advanced automatically is in neither the numerator nor the denominator. What was sent back or rejected automatically is in the denominator
Send-back rateThe number sent back ÷ the number decided (as above)What advanced automatically is not in the denominator. What was sent back or rejected automatically is included

Review activity

ItemHow it is countedAutomatic action at a deadline
Reviews (per workflow run)The number of review requests a person decided ÷ the number of workflow runs that started in the periodNot included
Typical waitThe median time from a review request being created to its being decided. Sorted from shortest, it is the middle value; with an even number, the shorter of the two in the middle. Requests not yet decided are not includedIncluded
Send-backs, rejections, cancellationsThe number sent back or rejected, plus the number of workflow runs that started in the period and were cancelledIncluded
Decided on timeOf the review requests with a deadline that have been answered or are past the deadline, the share the reviewer answered by the deadline. Ones settled by an automatic action at the deadline are not counted as on timeIn the denominator (not counted as on time)

The handling of reviews includes connection confirmation review requests too (approvals, rejections and the waiting time; one decided as rejected because its wait limit passed is handled the same as an automatic deadline action).

Figures by step​

Choosing “Metrics” in “Workflow shape” in Overview shows the figures for each step in the step's description box. The period is the same as for “Results of this work”.

StepFigures shown
An agent stepCompletion rate (the formula above, over that step's runs only), Failed / needs action and Typical duration (p95)
A review step (workflow only)The review (“Human review” or “Manual work”), the review method, Approved by people (with the number of review requests below it), Send-back rate, Typical wait and Decided on time. They are counted as in the review figures above, over that step's review requests only. Everyone except people who open the project only through the organization operations role can move to “Execution rules” with “Open settings” in the box (only the owner can change the review method)
An External automation step (workflow only)The numbers sent and failed sends. Only those with a send result are counted, and they are not in the agent completion rate
A Branch stepNot measured

Insights​

“Insights” takes the runs of the projects you are a member of together.

ItemHow it is counted
Agent completion rate, Completed, Failed / needs action, Insufficient records, Typical duration (p95)The same as a project's KPI
Review figuresNot shown

The KPI CSV​

“Export CSV” under “Results of this work” in Overview exports the figures for the period you have chosen. Each line is one item, with the columns metric and value.

metricContent
daysThe period (days)
success_rateThe Agent completion rate (0 to 1). Empty when nothing has finished
success_rate_statusok (final) / incomplete (provisional because a record is insufficient, even when nothing has finished) / none (no record is insufficient and nothing has finished)
succeededCompleted
denominatorFinished
outcome_completed and 6 more linesThe count for each result. completed = Completed, incomplete = Failed (reported by agent), failed = Failed, timeout = Timed out, stopped = Failed (stopped), denied = Not allowed by rules, skipped = Nothing to do. outcome_completed includes runs that ended with no change. No line includes runs stopped for a connection confirmation
excluded_skipped / excluded_running / excluded_missing / excluded_unmeasured / excluded_connectionConfirmThe counts under Excluded (nothing to do / running / insufficient records / always 0 / stopped for a connection confirmation)
duration_p95_minutesTypical duration (p95), in minutes
runs_totalThe number of workflow runs that started in the period
runs_closed_done / runs_closed_failed / runs_closed_rejected / runs_cancelledOf those, the number that ended as done / failed / rejected / cancelled
p95_minutesThe p95 of the time from start to end of a workflow run, in minutes
reviews_per_runReviews (per workflow run)
gate_wait_median_minutesTypical wait (the median time from a review request being created to its being decided), in minutes
interventionsThe number of send-backs, rejections and cancellations
Lines starting with gate_Five lines for each review step: gate_<step number>_name (the step name), gate_<number>_mode (the review method: blocking, Stop and wait for a decision, or notify_only, Notify and proceed after the deadline), gate_<number>_total (the number of review requests created on that step in the period, including ones not yet decided), gate_<number>_approval_rate (approved by people) and gate_<number>_send_back_rate (the send-back rate). The two rates range from 0 to 1, and are empty when the number decided is zero. The number is the step's place in the order of steps, counting disabled steps too, so it can differ from the step number in “Workflow shape” in the overview (enabled steps counted from 1)
Rows starting with step_For each enabled step, step_<number>_name (the step name) and step_<number>_zone (ai for an agent, gate for a review, automation for external automation, branch for a branch); for an agent step _executions, _succeeded, _denominator, _success_rate, _success_rate_status and _duration_p95_minutes; for a review step _gate_wait_median_minutes, _deadline_decisions_total, _deadline_decisions_on_time and _deadline_decision_rate (Approved by people and the send-back rate are in the gate_ rows); for an external automation step _automation_total, _automation_sent and _automation_failed. The number is the same as in the gate_ rows: it counts disabled steps too, so it can differ from the step number in “Workflow shape” in the overview (enabled steps counted from 1)
deadline_decisions_total, deadline_decisions_on_time, deadline_decision_rateDecided on time for reviews: the number of review requests with a deadline that were decided or are past the deadline, the number a person answered on time, and the share. The share is empty when that number is zero
terminated_executions / billable_executions / excluded_executions / tool_calls / steps / steps_unknown_executions / steps_formula_versionWork steps. Executions that ended / counted executions / executions not counted / tool calls / steps / executions with no recorded count / the version of the counting method