Skip to main content

Deployment Troubleshooting Guide

This guide explains how to find out what happened, and how to recover, when a marketplace deployment (the managed application deployment) stops partway through. After the infrastructure is provisioned, the application installation deploys the components one by one, in a fixed order, as Helm releases. If it stops partway, the components after that point are never installed, which shows up as symptoms such as "cannot log in" or "the admin console does not open".

note

The commands and screens in this guide are for Azure. If you deployed on AWS, see On AWS.

caution

Do not redo the installation by hand on a failed environment. The installer generates new authentication keys and passwords on every run, so running it again on the same cluster makes the values diverge from the components already installed and makes recovery harder. See Recovering an environment that stopped partway for the recovery procedure.

Finding the Cause from the Symptom​

SymptomWhat is happeningWhere to look
Login or sign-up shows Realm not found / Realm does not existThe initial setup job of the authentication platform (init-platform) has not run, so the realm agenticstar has not been createdinit-platform status, Deployment progress
The admin console IP does not respond, or the authentication service logs show Admin S2S request failedThe admin console (agenticai-admin) is not installed. The agentic-star-admin namespace has no podsDeployment progress
Pods stay PendingNo room on the nodes (Insufficient cpu / Insufficient memory). Usually the subscription's vCPU quota is too low to add nodesWhy a pod is Pending, Why nodes cannot be added
The managed application deployment is Failed and the error is only a generic Azure messageThe deployment script (the installer) ended with a timeout or a failureDeployment script logs, Deployment progress
The deployment fails with AvailabilityZoneNotSupportedThe selected availability zones cannot host VMs in this subscription and regionDeploy again with a different zone combination (Azure Quickstart)

Checking Deployment Progress​

Connect to the Cluster​

curl
# The resource group is the managed resource group of the managed application (its name starts with mrg-)
az aks get-credentials --resource-group <managed resource group> --name <AKS Cluster Name>

List the Helm Releases​

The marketplace edition uses configmap as the Helm storage driver. Without HELM_DRIVER=configmap, nothing is shown.

curl
HELM_DRIVER=configmap helm list -A

The installer deploys the releases in the following order. Sort the list by the UPDATED time to see how far it got. A release whose STATUS is failed is where it stopped.

#ReleaseNamespaceRole
1prometheus-operatormonitoringMonitoring platform
2prometheus-adaptermonitoringMonitoring platform
3ingress-nginxingress-nginxEntry point (LoadBalancer)
4cert-managercert-managerCertificates
5qdrantqdrantVector database
6db-migrationdb-migrationDatabase initialization
7extauth-serviceproxy-systemAuthentication proxy
8gate-servicesgate-servicesLLM gateway
9mail-ingressmail-ingressInbound mail
10agent-sandbox-controlleragent-sandbox-systemAgent execution platform
11agent-executoragent-executorAgent execution
12agenticai-extapiagenticai-extapi-serviceExternal API
13agenticai-datadogagenticai-datadogDatadog integration
14pvc-lifecycle-managerpvc-lifecycle-managerStorage management
15librechat-authlibrechat-authAuthentication service
16keycloak-extensionslibrechat-authAuthentication platform (Keycloak)
17librechatagentic-star-frontMain application
18agenticai-adminagentic-star-adminAdmin console
19lokiobservabilityLog platform
20alloy-logsobservabilityLog collection
21alloy-metricsobservabilityMetrics collection
22k8s-monitoringobservabilityMonitoring
23agenticai-observabilityobservabilityMonitoring
24init-platformlibrechat-authInitial setup of the authentication platform (realm, clients, initial administrator)
25upgrade-controllerupgrade-controllerVersion management

On AWS, cluster-autoscaler (namespace kube-system), which adds and removes nodes automatically, is installed before the first release (26 releases in total).

For example, if it stops at the 17th release, the main application (librechat), then the admin console, the log platform, and init-platform are never installed, so both "Realm not found at login" and "the admin console does not open" occur.

See Why a Release Failed​

curl
HELM_DRIVER=configmap helm history <release> -n <namespace>

DESCRIPTION shows the reason. context deadline exceeded means the pods did not start within the wait time (usually because they stayed Pending).

How to Investigate​

Why a Pod Is Pending​

curl
# Find pods other than Running / Completed
kubectl get pods -A | grep -v -E 'Running|Completed'

# The reason is in Events (Insufficient cpu / Insufficient memory, and so on)
kubectl describe pod <pod> -n <namespace>

Why Nodes Cannot Be Added​

curl
# Autoscaler status
kubectl get configmap -n kube-system cluster-autoscaler-status -o yaml

# Events for failed node additions
kubectl get events -n kube-system --field-selector reason=FailedToScaleUpGroup

# Free capacity on the nodes (cpu / memory under Allocated resources)
kubectl describe nodes | grep -A 7 'Allocated resources'

If a failed node addition event contains something like the following, the vCPU quota is too low.

OperationNotAllowed: Operation could not be completed as it results in exceeding approved standardDDSv4Family Cores quota.
Location: japaneast, Current Limit: 10, Current Usage: 8, Additional Required: 4, (Minimum) New Limit Required: 12.
curl
# Quota usage and limits
az vm list-usage --location <region> -o table

# Quota request history (requires the quota extension: az extension add --name quota)
az quota request list --scope /subscriptions/<subscription ID>/providers/Microsoft.Compute/locations/<region>

Deployment Script Logs​

The installer runs as an Azure deployment script (its name starts with postDeploy-) in the managed resource group. The logs are deleted one day after the script ends.

curl
az deployment-scripts list --resource-group <managed resource group> -o table
az deployment-scripts show-log --resource-group <managed resource group> --name <script name>

init-platform Status​

curl
# If the job is Complete, the realm has been created
kubectl get jobs -n librechat-auth

# Check the realm directly (200 means it exists, 404 means it does not)
kubectl port-forward -n librechat-auth svc/keycloak-extensions-keycloakx-http 18080:8080 &
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:18080/realms/agenticstar

Recovering an Environment That Stopped Partway​

For an environment whose deployment stopped partway, fix the cause, then delete the managed application and create it again. Deleting the managed application deletes the managed resource group (AKS, PostgreSQL, storage, and so on) with it. If you leave it in place, the resources that were created keep incurring charges.

  1. Find the cause (see above). If the vCPU quota is too low, request a higher limit as described in vCPU Quota in the Azure Quickstart. Requests for some families are not approved automatically; in that case use the default Standard_E4ds_v5
  2. Delete the managed application in the Azure Portal
  3. Purchase and deploy again with the revised settings, following the Azure Quickstart
  4. After the deployment, verify the deployment
note

There is no supported way to "redeploy" a failed deployment or to rerun the installer on the failed environment. The installer generates keys and passwords on every run, so rerunning it on the same cluster makes the values diverge from the components already installed.

On AWS​

Environments deployed from AWS Marketplace are created by a CloudFormation stack. If the installation fails, the stack fails and is rolled back, and most of the resources already created (including the EKS cluster) are deleted. However, the S3 buckets may remain because they contain data.

See Why It Failed​

  1. Open the stack in the CloudFormation console and find the first failed resource on the Events tab. The resource that waits for the installation to complete (WaitCondition) is in a nested stack (SetupHelperStack). If the parent stack's Events show a failure of SetupHelperStack, open that nested stack and look at its Events
  2. If the installation failed, the reason of the WaitCondition in the nested stack shows the last output of the installer. Check the name of the release where it stopped and the error message
  3. The full output of the installer is saved under install-logs/ in the private S3 bucket created by the stack. You can read it after a rollback or stack deletion as long as the bucket remains

If nodes could not be added, check the EC2 vCPU quota in the AWS quickstart.

Recover​

Fix the cause (such as an insufficient vCPU quota) and delete the stack. Before creating it again, check whether any S3 buckets remain, and if you do not need them, empty and delete them (versioning is enabled, so delete the old versions too). Then create it again following the AWS quickstart. Manual recovery of a failed stack and re-running the installer are not provided.

Investigate After Deployment​

In an environment where the deployment has completed, you can use the steps in Checking Deployment Progress and How to Investigate with HELM_DRIVER=configmap helm list -A and kubectl, as on Azure. Connect to the cluster with aws eks update-kubeconfig --region <region> --name <EKS cluster name>.

When Nodes Run Out of Room After Deployment​

If the deployment completed but pods become Pending as agent executions grow, check that the vCPU quota allows the autoscaler to add nodes. If the quota for the default VM family cannot be raised, you can add a node pool of another family.

curl
RG=<managed resource group>
AKS=<AKS Cluster Name>

# Use the same subnet and zones as the existing User Pool (apppool)
SUBNET=$(az aks nodepool show -g $RG --cluster-name $AKS --name apppool --query vnetSubnetId -o tsv)
ZONES=$(az aks nodepool show -g $RG --cluster-name $AKS --name apppool --query availabilityZones -o tsv | tr '\n' ' ')

# Example: add a Standard_E8ds_v5 node pool (match the OS disk and other settings to the existing User Pool)
az aks nodepool add -g $RG --cluster-name $AKS --name extrapool \
--mode User --node-vm-size Standard_E8ds_v5 --zones $ZONES \
--enable-cluster-autoscaler --min-count 1 --max-count 2 --node-count 1 \
--os-sku Ubuntu --node-osdisk-type Managed --node-osdisk-size 128 --max-pods 110 \
--vnet-subnet-id "$SUBNET"

Once the node is added, the Pending pods are scheduled automatically.