Troubleshoot self-hosting

Record the exact:

• Environment
• Namespace
• Time and timezone
• User-visible error
• HTTP status
• Affected Organization and Agent
• Last known successful action
• Recent deployment, migration, configuration, or credential change
• Whether the issue is repeatable

Verify the Kubernetes target

kubectl config current-context

Confirm namespace access:

kubectl auth can-i get pods \
  --namespace agent-farm

Capture workload state:

kubectl get pods,deployments,statefulsets,services,jobs,cronjobs,persistentvolumeclaims,ingress \
  --namespace agent-farm

Capture recent events:

kubectl get events \
  --namespace agent-farm \
  --sort-by=.lastTimestamp

Capture Helm state:

helm list --namespace agent-farm

Inspect a failed release:

helm status RELEASE_NAME \
  --namespace agent-farm

helm history RELEASE_NAME \
  --namespace agent-farm

Inspect a failing pod

kubectl describe pod POD_NAME \
  --namespace agent-farm

Read current logs:

kubectl logs POD_NAME \
  --namespace agent-farm \
  --all-containers \
  --tail=200

If the container restarted, read the terminated instance:

kubectl logs POD_NAME \
  --namespace agent-farm \
  --all-containers \
  --previous \
  --tail=200

The standard local command is:

./run.sh

It performs these operations in order:

1. Checks Docker.
2. Validates required .env values.
3. Starts local LiteLLM and its PostgreSQL database.
4. Starts or adopts the k3d cluster.
5. Writes host and container kubeconfigs.
6. Loads Hermes and OpenClaw images into k3d.
7. Starts application PostgreSQL and Redis.
8. Applies Alembic migrations.
9. Starts Product API, Ingest, Communications, worker, and UI.

The first failed stage normally identifies the subsystem to inspect.

Docker is unavailable

Check the engine:

docker info

Inspect Compose services:

docker compose -f compose.yml ps

Read local service logs:

docker compose -f compose.yml logs \
  --tail=200 \
  db redis api worker ui

Common causes include:

• Docker Desktop is not running
• Docker is in Windows-container mode instead of Linux-container mode
• The Docker VM lacks memory or disk space
• Another stack already owns ports 3000, 8000, 8001, 7070, or 16443
• Native make dev-* services are running alongside the Compose stack

Do not run native API/UI services and the full containerized stack on the same ports.

.env is missing

If .env does not exist, run.sh copies .env.spec and exits. Fill the required values, then rerun it.

If startup reports missing values, correct each one instead of bypassing validation.

Keep these stable across restarts:

SECRET_SIGNING_KEY
AGENT_TOKEN_ENCRYPTION_KEY
LITELLM_MASTER_KEY

API startup data initialization fails

A common startup error is:

500: Error while initializing startup data

Read the earlier API log lines for the underlying exception.

For a new database, confirm PLATFORM_ADMIN_CREDENTIALS uses:

email:password

The password must contain at least:

• Eight characters
• One uppercase letter
• One lowercase letter
• One digit

Verify local k3d

Use the host kubeconfig:

export KUBECONFIG=.k3d/kubeconfig-host.yaml
kubectl get nodes
kubectl get pods --all-namespaces

The default cluster is:

agentfarm-dev

Host tools use:

.k3d/kubeconfig-host.yaml

The containerized API uses:

/app/.k3d/kubeconfig-internal.yaml

Kubeconfig certificate error

A local cluster created before host.docker.internal was added to the API-server certificate can produce:

x509: certificate is valid for 127.0.0.1, not host.docker.internal

Confirm the error is from the local containerized API. If so, recreate the local cluster:

./stop.sh --clean
./run.sh

Invalid kubeconfig path

If Agent start fails with:

Invalid kube-config file. No configuration found.

Verify the mounted path inside the API container:

docker exec aai_api \
  ls -l /app/.k3d/kubeconfig-internal.yaml

For native API development, K8S_KUBECONFIG_PATH must be absolute or resolve relative to api/, because make dev-api runs from that directory.

Native API creates an Agent that cannot answer

For native development, set the API’s LiteLLM URL:

LITELLM_BASE_URL=http://127.0.0.1:7070

If it is empty, Agent creation can complete without minting a LiteLLM key, leaving the Agent unable to make model requests.

Agent pods must use the host-facing address:

AGENT_LITELLM_BASE_URL=http://host.docker.internal:7070

A loopback address inside an Agent pod points back to that pod, not to the host.

Start with:

helm list --namespace agent-farm
kubectl get jobs,pods --namespace agent-farm

Agent Barn uses Helmfile release dependencies. A failed sync can leave earlier releases updated while later releases remain unchanged.

Inspect each affected release rather than assuming the entire platform rolled back.

API pre-upgrade hooks

The API release runs these important hooks:

1. API and registry Secret resources
2. LiteLLM virtual-key Job
3. Alembic migration Job
4. API and worker rollout

Successful hook Jobs are removed. Failed Jobs remain.

LiteLLM key hook fails

Inspect:

kubectl describe job agentbarn-api-litellm-key \
  --namespace agent-farm

kubectl logs job/agentbarn-api-litellm-key \
  --namespace agent-farm

Check:

• LiteLLM pod readiness
• LiteLLM master key
• agent-farm-user ServiceAccount permissions
• Access to create or update litellm-api-key
• Network access to http://litellm:4000
• The existing agentbarn-api key alias

The key Job deletes and recreates the API’s LiteLLM virtual key. If a later hook fails, existing API or worker pods may still hold the deleted key until they are replaced with pods that load the updated Secret.

Migration hook fails

Inspect:

kubectl describe job agentbarn-api-migrate \
  --namespace agent-farm

kubectl logs job/agentbarn-api-migrate \
  --namespace agent-farm

Check the current revision:

kubectl exec \
  --namespace agent-farm \
  deployment/agentbarn-api \
  -- sh -c 'cd /app/api && alembic current'

Do not retry until you know whether the migration transaction committed any schema or data changes.

A Helm rollback does not downgrade Alembic.

Deployment times out

Helmfile uses a 600-second timeout. Inspect:

• Jobs still running
• Pods waiting for readiness
• Image pulls
• PVC binding
• Database connectivity
• Certificate challenges
• Resource quota failures
• LiteLLM readiness

Preserve failed hook logs before another deployment. The next hook execution can remove the previous failed Job through its before-hook-creation policy.

List non-ready pods:

kubectl get pods \
  --namespace agent-farm

Pending

Describe the pod:

kubectl describe pod POD_NAME \
  --namespace agent-farm

Look for:

• Unschedulable CPU or memory requests
• Namespace quota exhaustion
• Unbound PVCs
• Missing ServiceAccounts
• Missing Secrets
• Node selectors or taints
• Admission failures

ErrImagePull or ImagePullBackOff

Inspect the pod events and image reference:

kubectl get pod POD_NAME \
  --namespace agent-farm \
  -o jsonpath='{.spec.containers[*].image}{"\n"}'

Confirm the registry Secret exists:

kubectl get secret agentbarn-api-registry-pull-secret \
  --namespace agent-farm

Check:

• Image repository
• Image tag
• Registry hostname
• Registry credentials
• Pull Secret name
• Registry reachability
• Whether the image was published

Do not decode or share the pull Secret as part of ordinary diagnosis.

Local Agent image pull failure

Local Agent images use IfNotPresent and are normally imported into k3d.

Reload them:

bash docker/k3d/k3d-load-images.sh

Or reload one runtime:

TARGET=hermes bash docker/k3d/k3d-load-images.sh

TARGET=openclaw bash docker/k3d/k3d-load-images.sh

The image tags in .env must match the respective runtime VERSION files.

A local warning about FailedToRetrieveImagePullSecret can be harmless when the image is already imported. It becomes material when accompanied by ErrImagePull or ImagePullBackOff.

Imported images can be garbage-collected when the Docker host is under disk pressure. Inspect:

docker system df
docker stats --no-stream

Then reimport the runtime image.

CrashLoopBackOff

Read both current and previous logs:

kubectl logs POD_NAME \
  --namespace agent-farm \
  --all-containers \
  --tail=200

kubectl logs POD_NAME \
  --namespace agent-farm \
  --all-containers \
  --previous \
  --tail=200

Check termination state:

kubectl describe pod POD_NAME \
  --namespace agent-farm

If the exit code is 137 or the reason is OOMKilled, inspect memory pressure.

For local development, the Docker VM may be exhausted even when the Agent pod has no explicit memory limit. Increase Docker Desktop memory or reduce the number of active workloads.

API health fails

Port-forward the API Service:

kubectl port-forward \
  --namespace agent-farm \
  service/agentbarn-api \
  8000:8000

Then request:

curl --fail http://localhost:8000/api/v1/health

If this fails, inspect:

kubectl logs deployment/agentbarn-api \
  --namespace agent-farm \
  --tail=200

Inspect application PostgreSQL:

kubectl get pod,service,endpoints,persistentvolumeclaim \
  --namespace agent-farm

kubectl logs statefulset/postgres-app \
  --namespace agent-farm \
  --tail=200

The API health endpoint primarily verifies PostgreSQL connectivity. It does not prove that Redis, workers, Agents, LiteLLM, Firecrawl, or providers are healthy.

Authentication fails after a password change

Changing the PostgreSQL Secret does not change the password stored in an already initialized PostgreSQL data directory.

If a deployment value was changed accidentally:

• Restore the previous configured password
• Restart only the workloads that need to reload it
• Verify connectivity

For an intentional rotation, follow a coordinated database credential migration.

PVC remains Pending

Inspect:

kubectl describe persistentvolumeclaim PVC_NAME \
  --namespace agent-farm

Check:

• StorageClass name
• Provisioner availability
• Access mode
• Requested capacity
• Namespace quota
• Node topology

Changing STORAGE_CLASS does not move data from an existing PVC. StatefulSet volume claim templates are not an in-place storage migration.

Database or Agent workspace data is missing

Stop state-changing operations and establish:

• Whether the original PVC still exists
• Whether a replacement PVC was created
• Which node or storage backend held the data
• Which backup corresponds to the failed environment
• Whether stable encryption keys are available

Use port-forwarding to separate application health from ingress health.

Test API without ingress

kubectl port-forward \
  --namespace agent-farm \
  service/agentbarn-api \
  8000:8000

curl --fail http://localhost:8000/api/v1/health

Test UI without ingress

kubectl port-forward \
  --namespace agent-farm \
  service/agentbarn-ui \
  3000:3000

Open:

http://localhost:3000

If port-forwarding works but the public hostname does not, inspect DNS, Traefik, ingress, and TLS rather than the application container.

Inspect ingress and certificates

kubectl get ingress,certificate,challenge \
  --namespace agent-farm

kubectl describe ingress agentbarn-api \
  --namespace agent-farm

kubectl describe ingress agentbarn-ui \
  --namespace agent-farm

Check:

• DNS resolves to the ingress address
• Hostnames match the configured environment
• Traefik is available
• The configured ClusterIssuer exists
• HTTP-01 challenges can reach the cluster
• Certificate hostnames match the ingress
• TLS Secrets exist

The API ingress intentionally exposes only /api. A public request to /metrics should not work; Prometheus scrapes it through the internal Service.

Service has no endpoints

Inspect:

kubectl get service,endpoints \
  --namespace agent-farm

A Service without endpoints usually means:

• Its selector does not match a pod
• The expected pod does not exist
• The pod is not ready
• The target port name is incorrect

UI loads but API calls fail

Check:

• agentbarn-api Service and endpoints
• UI server logs
• API logs
• UI backend URL used when the image was built
• Browser network response status
• The public API hostname and TLS
• Whether UI and API images belong to the same release

The UI proxies API traffic from inside the namespace. A browser-visible UI does not prove that the UI server can resolve or reach the API Service.

Login fails for every user

Check:

• API health
• Application database
• SECRET_SIGNING_KEY
• Browser cookies and configured application URL
• UI and API hostname consistency
• Recent key or domain changes

Changing SECRET_SIGNING_KEY invalidates assumptions behind existing signed sessions and tokens.

Initial Platform Administrator login fails

For a fresh database, inspect API bootstrap logs and verify:

PLATFORM_ADMIN_CREDENTIALS=email:password

The password requires eight or more characters with uppercase, lowercase, and a digit.

Do not expect changing the environment value to behave like an ordinary user password-reset workflow for an established deployment.

Password reset or invitation email does not arrive

Check:

• CLOUDFLARE_ACCOUNT_ID
• CLOUDFLARE_API_TOKEN
• SENDER_EMAIL
• Email Sending permission
• Verified sending domain
• Provider quota
• API logs

When email configuration is absent, Agent Barn logs the condition and performs a no-op instead of making API health fail.

One user receives 403 or 404

Check:

• Active Organization
• Membership
• Organization Role
• Agent Access Role
• Explicit Agent Access
• Agent General Access
• Whether the resource belongs to another Organization

Platform Administrator status does not automatically provide Organization authority through ordinary Organization routes.

The main API can remain healthy while background delivery is unavailable.

Inspect the worker:

kubectl get deployment agentbarn-api-worker \
  --namespace agent-farm

kubectl logs deployment/agentbarn-api-worker \
  --namespace agent-farm \
  --tail=200

Inspect Redis:

kubectl get pod,service,endpoints \
  --namespace agent-farm

kubectl logs deployment/redis \
  --namespace agent-farm \
  --tail=200

Inspect reconciliation:

kubectl get cronjob agentbarn-api-event-reconciler \
  --namespace agent-farm

kubectl get jobs \
  --namespace agent-farm

Platform Administrators can review Event Deliveries at:

/dashboard/platform/event-deliveries

Interpret Event Delivery state

Reconciliation republishes eligible pending and stale deliveries. It does not automatically replay DEAD_LETTERED deliveries.

Check:

• Worker readiness
• Redis connectivity
• Reconciliation schedule and last run
• attempt_count
• Bounded last_error
• Dead-letter reason
• Whether the handler supports the event name and schema version

Do not modify Event Delivery rows directly as a routine repair.

Begin in Agent Barn with:

• Agent status
• last_error
• Health
• Current logs
• Historical log snapshots
• Runtime
• Selected model
• Template or Override version
• Skill versions
• Runtime Communications adapter and internal reachability

A successful Agent start clears the previous persisted error.

Inspect Agent Kubernetes resources

kubectl get deployments,services,persistentvolumeclaims \
  --namespace agent-farm \
  --selector agentbarn.io/component=agent \
  --show-labels

Inspect the selected Agent pod:

kubectl describe pod AGENT_POD \
  --namespace agent-farm

kubectl logs AGENT_POD \
  --namespace agent-farm \
  --all-containers \
  --tail=200

If it restarted:

kubectl logs AGENT_POD \
  --namespace agent-farm \
  --all-containers \
  --previous \
  --tail=200

Agent status is RUNNING but no pod exists

Establish whether:

• The pod was manually deleted
• The Deployment was deleted
• The Agent resources exist in another namespace
• The API kubeconfig targets the wrong cluster
• The staging API created resources in production
• A release or cleanup operation removed the workload

After correcting the infrastructure boundary, stop and start the affected Agent deliberately so the API rebuilds its resources.

Agent remains STARTING

Check:

• Pod scheduling
• Image availability
• Runtime startup logs
• Health-server readiness
• Runtime-local completion endpoint availability
• Internal reachability to Communications
• Per-start Communications credential and protocol version
• LiteLLM reachability

Agent becomes ERROR

Read last_error before retrying. Common causes include:

• Kubernetes authorization failure
• Missing runtime image
• Invalid generated Runtime configuration
• Runtime Communications adapter failure
• LiteLLM or model-provider failure
• PVC or scheduling failure

Agent does not respond in a shared channel

Inspect the affected Connection’s policy and provider-specific settings. Mention, direct-message, membership, allowlist, and event-subscription behavior varies by Platform; do not treat one provider’s rule as universal.

Activity combines two independently written datasets. Conversation Messages are written through Communications from provider ingress and Runtime replies, scoped by Communication Connection and provider location, and identified by (connection_id, provider_message_id). If a Conversation is missing, inspect the Connection, policy decision, Delivery, and Communications journal.

Tool Calls are written by the Agent Runtime through Ingest on port 8001, authenticated with the Agent ID and its per-start ingest key. Tool Call persistence is independent of Conversation persistence: if messages work while Tool Calls are missing, inspect Ingest rather than Communications.

Test deployed ingest

Port-forward the ingest endpoint:

kubectl port-forward \
  --namespace agent-farm \
  service/agentbarn-api \
  8001:8001

Then request:

curl --fail http://localhost:8001/ingest/v1/openapi.json

A successful response proves the process is reachable through the Service. Runtime event submission also requires the Agent’s per-start ingest key.

Check:

• Ingest process and Service endpoint
• Agent INGEST_BASE_URL
• Agent ingest Secret
• Runtime telemetry plugin
• API logs for ingest authentication failures
• Agent logs for event delivery failures
• Whether the Agent was restarted with current telemetry configuration

Test local ingest from an Agent-like pod

kubectl run ingest-check \
  --rm \
  --interactive \
  --tty \
  --restart=Never \
  --image=curlimages/curl \
  -- curl -sS -o /dev/null -w '%{http_code}\n' \
  http://host.docker.internal:8001/ingest/v1/openapi.json

Expected:

200

If it cannot connect:

• Confirm make dev-api or make dev-ingest is running
• Confirm ingest binds to 0.0.0.0
• Confirm INGEST_BASE_URL uses host.docker.internal
• On native Linux, confirm CoreDNS and the host firewall allow the k3d bridge to reach port 8001

Activity exists but costs are empty

Costs use a separate path. Background synchronization imports LiteLLM spend logs into stored cost records and recovers eligible missing OpenRouter charges.

Check:

• Agent LiteLLM key exists
• Key alias and Agent association
• LiteLLM database
• LiteLLM spend metrics
• Time range and Organization
• Cost permissions
• Whether the model request actually reached LiteLLM

Model requests fail

Check LiteLLM:

kubectl get pod,service,endpoints \
  --namespace agent-farm

kubectl logs deployment/litellm \
  --namespace agent-farm \
  --tail=200

Port-forward it:

kubectl port-forward \
  --namespace agent-farm \
  service/litellm \
  4000:4000

Check readiness:

curl --fail http://localhost:4000/health/readiness

Common model failures include:

Also verify:

• LITELLM_MASTER_KEY remained stable
• The API’s litellm-api-key Secret exists
• The Agent restarted after a runtime or key repair
• Selected model is allowed by AGENT_MODEL_ALLOWLIST
• Default model uses litellm/openrouter/<model-slug>
• OpenRouter supports the requested model

Firecrawl calls fail

Inspect:

• Firecrawl API pod
• Playwright service
• RabbitMQ
• Redis
• Firecrawl PostgreSQL
• Firecrawl API key
• Agent-level credential override
• Cluster egress

Use Agent logs and the Tool Call result to determine whether the failure occurred during authentication, scraping, browser execution, or result ingestion.

Google Workspace authentication fails

Check:

• Google client ID and secret
• Authorized redirect URI
• Exact WEB_APP_URL
• Callback:

<WEB_APP_URL>/api/v1/integrations/google/callback

• Google consent-screen access
• Requested scopes
• Existing credential ownership

Communication Connection credentials fail

Use the Connection management validation action, then inspect Connection status, safe diagnostics, and its journal.

Check:

• Slack bot and app tokens
• Telegram bot token
• Discord bot token and permissions
• Teams application configuration
• Channel, guild, chat, team, or user restrictions
• Whether the credential belongs to the configured platform account

These credentials belong to Connections rather than Agents. Do not paste tokens directly into logs or diagnostic messages.

Stored credentials cannot decrypt

This usually indicates the application database and AGENT_TOKEN_ENCRYPTION_KEY do not belong to the same recovery set.

Restore the matching encryption key. Replacing provider credentials one by one does not repair other encrypted data.

If Grafana is available but empty, inspect Prometheus targets.

Port-forward Prometheus:

kubectl port-forward \
  --namespace agent-farm \
  service/monitoring-prometheus-server \
  9090:80

Open:

http://localhost:9090/targets

Expected jobs include:

agentbarn-api
litellm
agent
kube-state-metrics
Prometheus self-scrape

Agents are absent from monitoring

Agent discovery requires:

agentbarn.io/component=agent

Inspect:

kubectl get services \
  --namespace agent-farm \
  --selector agentbarn.io/component=agent \
  --show-labels

Agents created before monitoring support may need to be stopped and started once.

If only the legacy component label is missing, patch Services without restarting:

kubectl label services \
  --namespace agent-farm \
  --selector agentfarm.io/component=agent \
  agentbarn.io/component=agent \
  --overwrite

Slack alerts do not arrive

Inspect Alertmanager:

kubectl port-forward \
  --namespace agent-farm \
  service/monitoring-alertmanager \
  9093:9093

Open:

http://localhost:9093

Check:

• Alert is firing in Prometheus
• Alertmanager received it
• SLACK_ALERTS_WEBHOOK_URL is valid
• The Secret is mounted
• The webhook can post to #alerts
• Environment label is correct
• No active silence suppresses it

Resource or certificate alert is missing

The built-in stack does not alert on:

• Node health
• CPU or memory capacity
• cAdvisor metrics
• PVC capacity
• Certificate expiry
• Backups
• Redis
• Worker readiness

Use external infrastructure, storage, certificate, backup, and synthetic monitoring for those signals.