# Deployment Troubleshooting Guide

This guide explains how to find out what happened, and how to recover, when a marketplace deployment (the managed application deployment) stops partway through. After the infrastructure is provisioned, the application installation deploys the components one by one, in a fixed order, as Helm releases. If it stops partway, the components after that point are never installed, which shows up as symptoms such as "cannot log in" or "the admin console does not open".

> **note**
>
> The commands and screens in this guide are for Azure. If you deployed on AWS, see [On AWS](#aws).

> **caution**
>
> Do not redo the installation by hand on a failed environment. The installer generates new authentication keys and passwords on every run, so running it again on the same cluster makes the values diverge from the components already installed and makes recovery harder. See [Recovering an environment that stopped partway](#recovery) for the recovery procedure.

## Finding the Cause from the Symptom

| Symptom                                                                                                   | What is happening                                                                                                                 | Where to look                                                                                                                                                                                        |
| --------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Login or sign-up shows `Realm not found` / `Realm does not exist`                                         | The initial setup job of the authentication platform (init-platform) has not run, so the realm `agenticstar` has not been created | [init-platform status](#check-init-platform), [Deployment progress](#progress)                                                                                                                       |
| The admin console IP does not respond, or the authentication service logs show `Admin S2S request failed` | The admin console (agenticai-admin) is not installed. The `agentic-star-admin` namespace has no pods                              | [Deployment progress](#progress)                                                                                                                                                                     |
| Pods stay `Pending`                                                                                       | No room on the nodes (`Insufficient cpu` / `Insufficient memory`). Usually the subscription's vCPU quota is too low to add nodes  | [Why a pod is Pending](#check-pending), [Why nodes cannot be added](#check-autoscaler)                                                                                                               |
| The managed application deployment is Failed and the error is only a generic Azure message                | The deployment script (the installer) ended with a timeout or a failure                                                           | [Deployment script logs](#check-deployment-script), [Deployment progress](#progress)                                                                                                                 |
| The deployment fails with `AvailabilityZoneNotSupported`                                                  | The selected availability zones cannot host VMs in this subscription and region                                                   | Deploy again with a different zone combination ([Azure Quickstart](https://developers.fd.agenticstar.tm.softbank.jp/en/docs/marketplace/build/quickstart/quickstart-azure.md#aks-advanced-settings)) |

## Checking Deployment Progress

### Connect to the Cluster

```bash
# The resource group is the managed resource group of the managed application (its name starts with mrg-)
az aks get-credentials --resource-group <managed resource group> --name <AKS Cluster Name>
```

### List the Helm Releases

The marketplace edition uses configmap as the Helm storage driver. **Without `HELM_DRIVER=configmap`, nothing is shown.**

```bash
HELM_DRIVER=configmap helm list -A
```

The installer deploys the releases in the following order. Sort the list by the `UPDATED` time to see how far it got. A release whose `STATUS` is `failed` is where it stopped.

| #  | Release                  | Namespace                | Role                                                                                 |
| -- | ------------------------ | ------------------------ | ------------------------------------------------------------------------------------ |
| 1  | prometheus-operator      | monitoring               | Monitoring platform                                                                  |
| 2  | prometheus-adapter       | monitoring               | Monitoring platform                                                                  |
| 3  | ingress-nginx            | ingress-nginx            | Entry point (LoadBalancer)                                                           |
| 4  | cert-manager             | cert-manager             | Certificates                                                                         |
| 5  | qdrant                   | qdrant                   | Vector database                                                                      |
| 6  | db-migration             | db-migration             | Database initialization                                                              |
| 7  | extauth-service          | proxy-system             | Authentication proxy                                                                 |
| 8  | gate-services            | gate-services            | LLM gateway                                                                          |
| 9  | mail-ingress             | mail-ingress             | Inbound mail                                                                         |
| 10 | agent-sandbox-controller | agent-sandbox-system     | Agent execution platform                                                             |
| 11 | agent-executor           | agent-executor           | Agent execution                                                                      |
| 12 | agenticai-extapi         | agenticai-extapi-service | External API                                                                         |
| 13 | agenticai-datadog        | agenticai-datadog        | Datadog integration                                                                  |
| 14 | pvc-lifecycle-manager    | pvc-lifecycle-manager    | Storage management                                                                   |
| 15 | librechat-auth           | librechat-auth           | Authentication service                                                               |
| 16 | keycloak-extensions      | librechat-auth           | Authentication platform (Keycloak)                                                   |
| 17 | librechat                | agentic-star-front       | Main application                                                                     |
| 18 | agenticai-admin          | agentic-star-admin       | Admin console                                                                        |
| 19 | loki                     | observability            | Log platform                                                                         |
| 20 | alloy-logs               | observability            | Log collection                                                                       |
| 21 | alloy-metrics            | observability            | Metrics collection                                                                   |
| 22 | k8s-monitoring           | observability            | Monitoring                                                                           |
| 23 | agenticai-observability  | observability            | Monitoring                                                                           |
| 24 | init-platform            | librechat-auth           | Initial setup of the authentication platform (realm, clients, initial administrator) |
| 25 | upgrade-controller       | upgrade-controller       | Version management                                                                   |

On AWS, `cluster-autoscaler` (namespace `kube-system`), which adds and removes nodes automatically, is installed before the first release (26 releases in total).

For example, if it stops at the 17th release, the main application (librechat), then the admin console, the log platform, and init-platform are never installed, so both "Realm not found at login" and "the admin console does not open" occur.

### See Why a Release Failed

```bash
HELM_DRIVER=configmap helm history <release> -n <namespace>
```

`DESCRIPTION` shows the reason. `context deadline exceeded` means the pods did not start within the wait time (usually because they stayed Pending).

## How to Investigate

### Why a Pod Is Pending

```bash
# Find pods other than Running / Completed
kubectl get pods -A | grep -v -E 'Running|Completed'

# The reason is in Events (Insufficient cpu / Insufficient memory, and so on)
kubectl describe pod <pod> -n <namespace>
```

### Why Nodes Cannot Be Added

```bash
# Autoscaler status
kubectl get configmap -n kube-system cluster-autoscaler-status -o yaml

# Events for failed node additions
kubectl get events -n kube-system --field-selector reason=FailedToScaleUpGroup

# Free capacity on the nodes (cpu / memory under Allocated resources)
kubectl describe nodes | grep -A 7 'Allocated resources'
```

If a failed node addition event contains something like the following, the vCPU quota is too low.

```text
OperationNotAllowed: Operation could not be completed as it results in exceeding approved standardDDSv4Family Cores quota.
Location: japaneast, Current Limit: 10, Current Usage: 8, Additional Required: 4, (Minimum) New Limit Required: 12.
```

```bash
# Quota usage and limits
az vm list-usage --location <region> -o table

# Quota request history (requires the quota extension: az extension add --name quota)
az quota request list --scope /subscriptions/<subscription ID>/providers/Microsoft.Compute/locations/<region>
```

### Deployment Script Logs

The installer runs as an Azure deployment script (its name starts with `postDeploy-`) in the managed resource group. The logs are deleted one day after the script ends.

```bash
az deployment-scripts list --resource-group <managed resource group> -o table
az deployment-scripts show-log --resource-group <managed resource group> --name <script name>
```

### init-platform Status

```bash
# If the job is Complete, the realm has been created
kubectl get jobs -n librechat-auth

# Check the realm directly (200 means it exists, 404 means it does not)
kubectl port-forward -n librechat-auth svc/keycloak-extensions-keycloakx-http 18080:8080 &
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:18080/realms/agenticstar
```

## Recovering an Environment That Stopped Partway

For an environment whose deployment stopped partway, fix the cause, then **delete the managed application and create it again**. Deleting the managed application deletes the managed resource group (AKS, PostgreSQL, storage, and so on) with it. If you leave it in place, the resources that were created keep incurring charges.

1. Find the cause (see above). If the vCPU quota is too low, request a higher limit as described in [vCPU Quota in the Azure Quickstart](https://developers.fd.agenticstar.tm.softbank.jp/en/docs/marketplace/build/quickstart/quickstart-azure.md#vcpu-quota). Requests for some families are not approved automatically; in that case use the default Standard\_E4ds\_v5
2. Delete the managed application in the Azure Portal
3. Purchase and deploy again with the revised settings, following the [Azure Quickstart](https://developers.fd.agenticstar.tm.softbank.jp/en/docs/marketplace/build/quickstart/quickstart-azure.md)
4. After the deployment, [verify the deployment](https://developers.fd.agenticstar.tm.softbank.jp/en/docs/marketplace/build/quickstart/quickstart-azure.md#verify-deployment)

> **note**
>
> There is no supported way to "redeploy" a failed deployment or to rerun the installer on the failed environment. The installer generates keys and passwords on every run, so rerunning it on the same cluster makes the values diverge from the components already installed.

## On AWS

Environments deployed from AWS Marketplace are created by a CloudFormation stack. If the installation fails, the stack fails and is rolled back, and most of the resources already created (including the EKS cluster) are deleted. However, the S3 buckets may remain because they contain data.

### See Why It Failed

1. Open the stack in the CloudFormation console and find the first failed resource on the **Events** tab. The resource that waits for the installation to complete (WaitCondition) is in a nested stack (`SetupHelperStack`). If the parent stack's Events show a failure of `SetupHelperStack`, open that nested stack and look at its **Events**
2. If the installation failed, the reason of the WaitCondition in the nested stack shows the last output of the installer. Check the name of the release where it stopped and the error message
3. The full output of the installer is saved under `install-logs/` in the private S3 bucket created by the stack. You can read it after a rollback or stack deletion as long as the bucket remains

If nodes could not be added, check the [EC2 vCPU quota in the AWS quickstart](https://developers.fd.agenticstar.tm.softbank.jp/en/docs/marketplace/build/quickstart/quickstart-aws.md#vcpu-quota).

### Recover

Fix the cause (such as an insufficient vCPU quota) and delete the stack. Before creating it again, check whether any S3 buckets remain, and if you do not need them, empty and delete them (versioning is enabled, so delete the old versions too). Then create it again following the [AWS quickstart](https://developers.fd.agenticstar.tm.softbank.jp/en/docs/marketplace/build/quickstart/quickstart-aws.md). Manual recovery of a failed stack and re-running the installer are not provided.

### Investigate After Deployment

In an environment where the deployment has completed, you can use the steps in [Checking Deployment Progress](#progress) and [How to Investigate](#investigate) with `HELM_DRIVER=configmap helm list -A` and `kubectl`, as on Azure. Connect to the cluster with `aws eks update-kubeconfig --region <region> --name <EKS cluster name>`.

## When Nodes Run Out of Room After Deployment

If the deployment completed but pods become Pending as agent executions grow, check that the vCPU quota allows the autoscaler to add nodes. If the quota for the default VM family cannot be raised, you can add a node pool of another family.

```bash
RG=<managed resource group>
AKS=<AKS Cluster Name>

# Use the same subnet and zones as the existing User Pool (apppool)
SUBNET=$(az aks nodepool show -g $RG --cluster-name $AKS --name apppool --query vnetSubnetId -o tsv)
ZONES=$(az aks nodepool show -g $RG --cluster-name $AKS --name apppool --query availabilityZones -o tsv | tr '\n' ' ')

# Example: add a Standard_E8ds_v5 node pool (match the OS disk and other settings to the existing User Pool)
az aks nodepool add -g $RG --cluster-name $AKS --name extrapool \
  --mode User --node-vm-size Standard_E8ds_v5 --zones $ZONES \
  --enable-cluster-autoscaler --min-count 1 --max-count 2 --node-count 1 \
  --os-sku Ubuntu --node-osdisk-type Managed --node-osdisk-size 128 --max-pods 110 \
  --vnet-subnet-id "$SUBNET"
```

Once the node is added, the Pending pods are scheduled automatically.
