# Run Results Reference

The words used for the result of a single run, and how the KPI figures are counted. Item names follow what the screen shows. How to read the results is covered in the [Record Guide](https://developers.fd.agenticstar.tm.softbank.jp/en/docs/saas/project/guide/record-guide.md), and the thinking behind the figures in [Records and metrics](https://developers.fd.agenticstar.tm.softbank.jp/en/docs/saas/project/concepts/records-and-metrics.md).

## The words for the result

How a single run ended is shown as one of these seven.

| Shown as                   | Meaning                                                                                                                                                                                                                        | Decided by               | Agent completion rate                                                         |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------ | ----------------------------------------------------------------------------- |
| Completed                  | The agent ended without reporting that it could not finish, and its final result did not say so either. This includes runs that ended with no report at all, as long as the final result did not say so                        | Agent                    | Counted as a success (not counted when it ended with no change)               |
| Failed (reported by agent) | The agent itself reported that it could not finish, or ended with a final result saying so. Without a report, the category is “Other”. A run that wrapped up at the run time limit and could not finish is “Timed out” instead | Agent                    | Counted as a failure (not counted when stopped for a connection confirmation) |
| Failed                     | It ended because of an exception or an abnormality in the runtime                                                                                                                                                              | Platform                 | Counted as a failure                                                          |
| Timed out                  | It reached the time limit and was cut off, or it wrapped up at the run time limit and could not finish (including when the agent reported that it could not finish)                                                            | Platform                 | Counted as a failure                                                          |
| Failed (stopped)           | A person stopped it, or the platform cancelled it                                                                                                                                                                              | A person or the platform | Counted as a failure                                                          |
| Not allowed by rules       | The request was stopped by the platform's check on its input (the guardrail) and was not carried out                                                                                                                           | Platform                 | Counted as a failure                                                          |
| Nothing to do              | The run never started                                                                                                                                                                                                          | Platform                 | Not counted                                                                   |

- **“Completed” is no guarantee that the work went well.** What was done is checked under “What was done” in the run record.
- **A run that ended with no change still has the result “Completed”.** It is a run where the agent reported that what it checked had not changed and it did nothing; its reason is “No change” (in Execution history, a series of them is shown as one line, such as “No change (3 times)”). It is not counted in the Agent completion rate.
- **An operation stopped by the execution rules does not change the result.** Stopped operations are listed under “What was done” as denials.
- **The check behind “Not allowed by rules” is not the guardrail you write for an agent (an instruction).** The run record gives the reason as “Stopped by the guardrail”.
- **A run stopped for a connection confirmation also has the result “Failed (reported by agent)”.** It stopped before using a connector set to confirm before connecting, and its reason is “Stopped for connection confirmation”. It is only waiting for a person to confirm, so **it is not counted as a failure** and is counted under Excluded as “Stopped for connection confirmation”. The run record does not show “Failure summary” or what to review ([Connection Guide](https://developers.fd.agenticstar.tm.softbank.jp/en/docs/saas/project/guide/connection-guide.md#asking-a-person-before-use-connection-confirmation)).

### Reasons for “Failed (reported by agent)”

When an agent reports that it could not finish, it gives what to review and a comment for the person reading it (what it could not do, why, and what to do next). The category is one of the following; where the agent does not choose one, it is derived from what to review. In the run record, the category appears beside the result and as the “Cause” in the “Failure summary”. The comment appears under “Agent's closing report” in the Failure summary. Where the Failure summary points first follows What to review where it narrows things down (the second table below), and otherwise the category (the table below).

| Category                                                              | Where the Failure summary points first |
| --------------------------------------------------------------------- | -------------------------------------- |
| Could not access the connected service (authentication or permission) | Check connector settings               |
| The connected service did not respond                                 | Check connector settings               |
| Restricted by the execution rules                                     | Check execution rules                  |
| Required input was missing                                            | Check step settings and inputs         |
| Other                                                                 | Review what was done                   |

Where What to review narrows things down, the Failure summary points first as follows.

| What to review                               | Where the Failure summary points first                                                                                                                                                                                      |
| -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Instructions (prompt)                        | Check step settings and inputs                                                                                                                                                                                              |
| Settings and connections (harness)           | Check connector settings, Check execution rules, or Check step settings and inputs (which one appears depends on the cause; the cause is written in the explanation under What to review, or in the agent's closing report) |
| Restricted by an administrator or the system | Review what was done                                                                                                                                                                                                        |
| External service                             | Review what was done                                                                                                                                                                                                        |
| Unexpected error                             | Review what was done                                                                                                                                                                                                        |

## What happens after each result in a workflow

| Result                     | What happens next                                                                                                                                                                                                                                        |
| -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Completed                  | It moves on to the next agent on the same step, or, if there is none, to the next step (on a step set to “Run at the same time”, to the next step once all of the results are in). If the next step is a review step, it waits for the review there      |
| Failed (reported by agent) | By default, a review request is created and the run stops there. Set the step's “When the agent reports it could not finish” to “Proceed anyway” and it moves on as if completed. Set it to “Go to a specific step” and it goes to the chosen later step |
| Not allowed by rules       | By default, a review request is created and the run stops there. Set the step's “When not allowed by rules” to “Close as failed” and the workflow run ends as failed. Set it to “Go to a specific step” and it goes to the chosen later step             |
| Failed / Timed out         | The same step is tried again. Once the retries set in “Retries per step (failed / timed out)” are used up, another failure or timeout ends the workflow run as failed                                                                                    |
| Failed (stopped)           | It is not tried again, and the workflow run ends as failed                                                                                                                                                                                               |
| Nothing to do              | It moves on (on a step set to “Run at the same time”, to the next step once the other agents' results are in)                                                                                                                                            |

A run stopped for a connection confirmation does not follow this table: it raises a connection confirmation review request and waits. On approval, the same step starts again; when it is rejected or its wait limit passes, it proceeds as in the “Failed (reported by agent)” row above.

When you cancel the workflow run itself, it ends as “Cancelled” (not “Failed”).

In a cycle, no result stops the sweep. Sweeps continue on the schedule you set. A sweep stopped for a connection confirmation runs the sweep of the same placement once more on approval.

## Runs still going, and runs with no recorded result

| State                               | Meaning                                                                                                                                                     | How it is counted                                                                                                                                                                                                                                                                                                               |
| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Running                             | It has not ended yet                                                                                                                                        | Not counted (“Running” under Excluded)                                                                                                                                                                                                                                                                                          |
| Insufficient records                | It has ended, but no result was recorded. The run record says “no result was recorded”                                                                      | Counted as neither success nor failure (“Insufficient records” under Excluded). Even one of these makes the Agent completion rate provisional. In “Results of this work” on a project's Overview, the rate shows “—” instead of a value, with “Records incomplete: provisional because N runs have no recorded result” below it |
| Nothing to do                       | The result “Nothing to do”                                                                                                                                  | Not counted (shown under Excluded)                                                                                                                                                                                                                                                                                              |
| Stopped for connection confirmation | A run stopped before using a connector set to confirm before connecting (result “Failed (reported by agent)”, reason “Stopped for connection confirmation”) | Counted as neither success nor failure (“Stopped for connection confirmation” under Excluded)                                                                                                                                                                                                                                   |

## Agent completion rate

**Agent completion rate = Completed ÷ finished**

| Item                                                    | Content                                                                                                                                                                                                                                                                                        |
| ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Completed                                               | The number of runs whose result is “Completed” (leaving out those that ended with no change)                                                                                                                                                                                                   |
| Finished                                                | Completed plus “Failed / needs action”. “Failed / needs action” is the total of Failed (reported by agent), Failed, Timed out, Failed (stopped) and Not allowed by rules (leaving out runs stopped for a connection confirmation)                                                              |
| The unit counted                                        | One agent execution. Both a workflow step's execution and a cycle's sweep are counted. A retry counts as one too                                                                                                                                                                               |
| Period                                                  | Runs that started in the period you chose                                                                                                                                                                                                                                                      |
| Not included                                            | Nothing to do, running, insufficient records, runs that ended with no change, and runs stopped for a connection confirmation                                                                                                                                                                   |
| When nothing has finished and no record is insufficient | In “Results of this work” on a project's Overview, the rate shows “—” with “No finished runs yet (completed N / finished N)” below it. In the overall totals it shows “No target”, and on the project list card “No data” (meaning there is nothing to count — not the result “Nothing to do”) |
| When a record is insufficient                           | It says “Records incomplete: provisional because N runs have no recorded result” (even when nothing has finished)                                                                                                                                                                              |

## The KPI items

These are the items under “Results of this work” in Overview. The figures for each step are under “Metrics” in “Workflow shape” (see “[Figures by step](#figures-by-step)” below). You can choose the period (values are in the [Settings reference](https://developers.fd.agenticstar.tm.softbank.jp/en/docs/saas/project/reference/settings-ref.md)).

### Run results

| Item                             | How it is counted                                                                                                                                                                                                                                                                                                |
| -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Agent completion rate            | The formula above                                                                                                                                                                                                                                                                                                |
| Completed                        | The number of runs whose result is “Completed” (leaving out those that ended with no change)                                                                                                                                                                                                                     |
| Failed / needs action            | The total of Failed (reported by agent), Failed, Timed out, Failed (stopped) and Not allowed by rules (leaving out runs stopped for a connection confirmation)                                                                                                                                                   |
| Typical duration (p95)           | The time from start to end of runs that started and ended in the period. Sorted from shortest, it is the value at the 95% position: about 95% of the runs that ended did so within this time. Runs stopped for a connection confirmation are not included                                                        |
| Runs per day                     | Runs that started in the period, stacked by the day they started as “Completed”, “Failed / needs action” and “Running”. Nothing to do, insufficient records and runs stopped for a connection confirmation are not counted. With more than 1,000 runs in the period, the bars are not shown                      |
| Breakdown: failed / needs action | The count for each kind of failure or needs-action result (Failed (reported by agent), Failed / Timed out, Failed (stopped), Not allowed by rules and so on), and its share. Only the kinds that have runs are listed, most first; Nothing to do and runs stopped for a connection confirmation are not included |
| Excluded                         | The counts of runs with nothing to do, runs still running, runs with insufficient records, and runs stopped for a connection confirmation                                                                                                                                                                        |

### Work steps

Counted from the agent executions that ended in the period you chose. Both workflow step runs and cycle sweeps are included.

| Item               | How it is counted                                                                                                                                                                                                                                                                                                                                     |
| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Counted executions | The number of executions whose result is “Completed”, “Failed (reported by agent)”, “Failed (stopped)” or “Not allowed by rules”. “Failed”, “Timed out”, “Nothing to do” and executions with no recorded result are not counted (their number is shown as “executions not counted”)                                                                   |
| Tool calls         | The total of the recorded tool calls of the counted executions                                                                                                                                                                                                                                                                                        |
| Steps              | The total of the work steps of the counted executions. The steps of one execution are settled from that execution and its recorded tool calls; failed connector calls and failed external changes are not counted, and there is a cap per execution. Executions with no recorded count are left out of the total, and their number is shown alongside |

The tenant-wide total for the current month (counted by UTC) can be seen by a system administrator under “**Workflow steps this month**” on the “License” screen of the admin console.

### Review figures (workflow only)

These count the review requests created in the period you chose.

**Approved by people and send-back rate** (for each review step; shown under “Metrics” in “Workflow shape”)

| Item               | How it is counted                                                                                                                           | Automatic action at a deadline                                                                                                                  |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| Approved by people | The number a person approved ÷ the number decided. The number decided is the number a person approved plus the number sent back or rejected | What advanced automatically is in neither the numerator nor the denominator. What was sent back or rejected automatically is in the denominator |
| Send-back rate     | The number sent back ÷ the number decided (as above)                                                                                        | What advanced automatically is not in the denominator. What was sent back or rejected automatically is included                                 |

**Review activity**

| Item                                  | How it is counted                                                                                                                                                                                                            | Automatic action at a deadline              |
| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- |
| Reviews (per workflow run)            | The number of review requests a person decided ÷ the number of workflow runs that started in the period                                                                                                                      | Not included                                |
| Typical wait                          | The median time from a review request being created to its being decided. Sorted from shortest, it is the middle value; with an even number, the shorter of the two in the middle. Requests not yet decided are not included | Included                                    |
| Send-backs, rejections, cancellations | The number sent back or rejected, plus the number of workflow runs that started in the period and were cancelled                                                                                                             | Included                                    |
| Decided on time                       | Of the review requests with a deadline that have been answered or are past the deadline, the share the reviewer answered by the deadline. Ones settled by an automatic action at the deadline are not counted as on time     | In the denominator (not counted as on time) |

The handling of reviews includes connection confirmation review requests too (approvals, rejections and the waiting time; one decided as rejected because its wait limit passed is handled the same as an automatic deadline action).

### Figures by step

Choosing “**Metrics**” in “Workflow shape” in Overview shows the figures for each step in the step's description box. The period is the same as for “Results of this work”.

| Step                                        | Figures shown                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| An agent step                               | Completion rate (the formula above, over that step's runs only), Failed / needs action and Typical duration (p95)                                                                                                                                                                                                                                                                                                                                                                  |
| A review step (workflow only)               | The review (“Human review” or “Manual work”), the review method, Approved by people (with the number of review requests below it), Send-back rate, Typical wait and Decided on time. They are counted as in the review figures above, over that step's review requests only. Everyone except people who open the project only through the organization operations role can move to “Execution rules” with “Open settings” in the box (only the owner can change the review method) |
| An External automation step (workflow only) | The numbers sent and failed sends. Only those with a send result are counted, and they are not in the agent completion rate                                                                                                                                                                                                                                                                                                                                                        |
| A Branch step                               | Not measured                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |

### Insights

“Insights” takes the runs of the projects you are a member of together.

| Item                                                                                                  | How it is counted           |
| ----------------------------------------------------------------------------------------------------- | --------------------------- |
| Agent completion rate, Completed, Failed / needs action, Insufficient records, Typical duration (p95) | The same as a project's KPI |
| Review figures                                                                                        | Not shown                   |

## The KPI CSV

“Export CSV” under “Results of this work” in Overview exports the figures for the period you have chosen. Each line is one item, with the columns `metric` and `value`.

| metric                                                                                                                                                  | Content                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| ------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `days`                                                                                                                                                  | The period (days)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `success_rate`                                                                                                                                          | The Agent completion rate (0 to 1). Empty when nothing has finished                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `success_rate_status`                                                                                                                                   | `ok` (final) / `incomplete` (provisional because a record is insufficient, even when nothing has finished) / `none` (no record is insufficient and nothing has finished)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `succeeded`                                                                                                                                             | Completed                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `denominator`                                                                                                                                           | Finished                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `outcome_completed` and 6 more lines                                                                                                                    | The count for each result. `completed` = Completed, `incomplete` = Failed (reported by agent), `failed` = Failed, `timeout` = Timed out, `stopped` = Failed (stopped), `denied` = Not allowed by rules, `skipped` = Nothing to do. `outcome_completed` includes runs that ended with no change. No line includes runs stopped for a connection confirmation                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `excluded_skipped` / `excluded_running` / `excluded_missing` / `excluded_unmeasured` / `excluded_connectionConfirm`                                     | The counts under Excluded (nothing to do / running / insufficient records / always 0 / stopped for a connection confirmation)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `duration_p95_minutes`                                                                                                                                  | Typical duration (p95), in minutes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `runs_total`                                                                                                                                            | The number of workflow runs that started in the period                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `runs_closed_done` / `runs_closed_failed` / `runs_closed_rejected` / `runs_cancelled`                                                                   | Of those, the number that ended as done / failed / rejected / cancelled                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `p95_minutes`                                                                                                                                           | The p95 of the time from start to end of a workflow run, in minutes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `reviews_per_run`                                                                                                                                       | Reviews (per workflow run)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `gate_wait_median_minutes`                                                                                                                              | Typical wait (the median time from a review request being created to its being decided), in minutes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `interventions`                                                                                                                                         | The number of send-backs, rejections and cancellations                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| Lines starting with `gate_`                                                                                                                             | Five lines for each review step: `gate_<step number>_name` (the step name), `gate_<number>_mode` (the review method: `blocking`, Stop and wait for a decision, or `notify_only`, Notify and proceed after the deadline), `gate_<number>_total` (the number of review requests created on that step in the period, including ones not yet decided), `gate_<number>_approval_rate` (approved by people) and `gate_<number>_send_back_rate` (the send-back rate). The two rates range from 0 to 1, and are empty when the number decided is zero. The number is the step's place in the order of steps, counting disabled steps too, so it can differ from the step number in “Workflow shape” in the overview (enabled steps counted from 1)                                                                                           |
| Rows starting with `step_`                                                                                                                              | For each enabled step, `step_<number>_name` (the step name) and `step_<number>_zone` (`ai` for an agent, `gate` for a review, `automation` for external automation, `branch` for a branch); for an agent step `_executions`, `_succeeded`, `_denominator`, `_success_rate`, `_success_rate_status` and `_duration_p95_minutes`; for a review step `_gate_wait_median_minutes`, `_deadline_decisions_total`, `_deadline_decisions_on_time` and `_deadline_decision_rate` (Approved by people and the send-back rate are in the `gate_` rows); for an external automation step `_automation_total`, `_automation_sent` and `_automation_failed`. The number is the same as in the `gate_` rows: it counts disabled steps too, so it can differ from the step number in “Workflow shape” in the overview (enabled steps counted from 1) |
| `deadline_decisions_total`, `deadline_decisions_on_time`, `deadline_decision_rate`                                                                      | Decided on time for reviews: the number of review requests with a deadline that were decided or are past the deadline, the number a person answered on time, and the share. The share is empty when that number is zero                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `terminated_executions` / `billable_executions` / `excluded_executions` / `tool_calls` / `steps` / `steps_unknown_executions` / `steps_formula_version` | Work steps. Executions that ended / counted executions / executions not counted / tool calls / steps / executions with no recorded count / the version of the counting method                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
