Configure a production deployment

Choose the cluster and namespace where this installation will run. Record its API and web application hostnames, storage class, image registry, and credential owners.

If you maintain separate test and production installations, give each an explicit configuration and identify which data and credentials belong to it. The supplied deploy.sh uses bootstrap resources for agent-farm; choosing another namespace requires matching bootstrap resources and permissions.

Use Deployment configuration to set the inputs for your installation and Deploy to Kubernetes for the deployment procedure.

Keep each environment on its own deployment target, with its own namespace, credentials, hosts, and kubeconfig. Do not share signing keys, encryption keys, or database passwords between a test environment and production. Namespace isolation is not a separate cluster or provider security boundary: use distinct application and LiteLLM database passwords, signing and encryption keys, bootstrap administrators, kubeconfigs, hosts, and sender subdomains.

Verify that each installation's API receives its own release namespace as K8S_NAMESPACE; otherwise a test installation can create Agent workloads in the production namespace.

kubectl get deployment agentbarn-api \
  --namespace <your-namespace> \
  -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="K8S_NAMESPACE")].value}{"\n"}'

Expected: the namespace of the installation you are checking.

Use an authenticated image registry you control and pin production versions rather than deploying from a moving tag.

AAI Labs' cluster names, release workflows, and PUBLIC_* or STAGING_* variables describe its hosted environments. Maintainers of those installations should use AAI Labs hosted-service operations.

Keep authoritative platform deployment Secrets in a protected secret manager: database passwords, signing and encryption keys, registry password, LiteLLM master key, OpenRouter and Firecrawl keys, Platform Administrator bootstrap credentials, Grafana administrator password, and alerting webhooks. These are distinct from Communication Connection credentials.

Registry, OpenRouter, Cloudflare, Google OAuth, Slack webhook, and Grafana credentials are usually independently rotatable. Test emergency rotation without changing the stable recovery set.

Secret isolation and Connection credentials

Give each installation its own PostgreSQL passwords, signing and encryption keys, registry password, LiteLLM master key, Firecrawl key, Platform Administrator credentials, Grafana administrator password, alerting webhook, and model-provider key where quota isolation is required. Share an email account and token, OAuth application credentials, database usernames and names, model defaults, or allowlist configuration between installations only deliberately.

Slack bot and app-level tokens, Microsoft Teams App ID/client secret/Tenant ID, Telegram bot tokens, and Discord bot tokens are Communication Connection credentials. They are created through Agent Barn after deployment, encrypted in PostgreSQL, never returned by read APIs, and are not GitHub Actions deployment Secrets, Agent Runtime variables, Agent Secrets, or Shared Credentials.

AGENT_TOKEN_ENCRYPTION_KEY must be a valid Fernet key, stored securely, stable across ordinary deployments, shared by Product API and Communications, and included in recovery planning. Rotation is a data migration because Agent Secrets, Connection credential envelopes, and Runtime-related encrypted values depend on the current key.

Bootstrap administrator

PLATFORM_ADMIN_CREDENTIALS creates an administrator only when none exists. Updating the Secret later does not reset an existing account password. Create at least two independently controlled, named Platform Administrators and avoid routine use of the bootstrap account.

Use different identities for deployment and API runtime orchestration.

The API identity needs namespaced access to create Agent Deployments, Services, Secrets, ConfigMaps, and PVCs, and to inspect Pods, health, logs, exec, and port-forward operations. It should not access staging, unrelated namespaces, cluster Secrets, or nodes.

Prepare the hook's ServiceAccount

The LiteLLM key setup hook runs under a configured Kubernetes ServiceAccount. In the supplied chart, its default name is agent-farm-user. Your cluster administrator must provision the account and the Secret permissions required by the hook in the installation's namespace.

This hook identity is separate from the kubeconfig used by the API to manage Agent workloads. Follow Arrange Kubernetes access before deployment; do not assume that provisioning the hook account creates the API credential or all of its workload permissions.

Verify that the account configured through LITELLM_KEY_SERVICE_ACCOUNT exists in the installation's namespace.

kubectl get serviceaccount \
  --namespace agent-farm \
  agent-farm-user

Audit effective permissions

kubectl auth can-i create deployments --namespace agent-farm
kubectl auth can-i create services --namespace agent-farm
kubectl auth can-i create secrets --namespace agent-farm
kubectl auth can-i create persistentvolumeclaims --namespace agent-farm
kubectl auth can-i get pods --namespace agent-farm
kubectl auth can-i get pods/log --namespace agent-farm

Repeat the checks with impersonation or the intended kubeconfig, then confirm that the same identity cannot read another namespace.

The default deployment requests at least 19 GiB before any Agents: 5 GiB for application PostgreSQL, 2 GiB each for LiteLLM and Firecrawl PostgreSQL, and 10 GiB for Prometheus. Every Agent PVC requests another 1 GiB.

A node-local StorageClass such as local-path keeps data on a single node. Choose storage that meets the production durability target. Changing STORAGE_CLASS on an existing PostgreSQL StatefulSet is not an in-place migration because its volume claim template is immutable.

Set STORAGE_CLASS to a replicated, node-independent StorageClass for production. Durable PostgreSQL storage must survive node failure; treat local-path as temporary where one node is a failure domain. Communication data durability follows application PostgreSQL persistence, while Agent workspace PVC behavior remains separate from Communication Delivery persistence.

Back up the complete recovery set

Agent Barn does not install a database or Agent PVC backup controller. Integrate an external backup system, define frequency, retention, off-cluster storage, encryption, access, RPO/RTO, restore order, verification, and failure ownership.

Redis has no persistent volume in the current chart. Durable Domain Event intent remains in PostgreSQL, and reconciliation can republish eligible Event Deliveries when Redis returns.

Public hosts and Communications paths

The Product API’s API_EXTERNAL_URL resolves from the public API host. Connection webhook URLs use https://<public-api-host>/communications/v1/webhooks/<connection-id>; never configure provider webhooks against the UI hostname or an internal ClusterIP address.

Expose only /communications/v1/webhooks from Communications. Runtime Delivery and Platform Driver routes, /health, /metrics, and Ingest routes remain internal. Provider webhooks require public DNS, valid certificates, stable ingress routing, the correct public API origin, provider authentication, and Connection-scoped paths. Microsoft Teams currently depends on this authenticated webhook path.

UI_HOST=agentbarn.example.com
API_HOST=api.agentbarn.example.com
GRAFANA_HOST=grafana.agentbarn.example.com
WEB_APP_URL=https://agentbarn.example.com

Confirm DNS resolution, ports 80 and 443, certificate issuance and renewal monitoring, API-to-Service routing, and restricted Grafana authentication.

Introduce network isolation in staging

The charts do not create NetworkPolicies. If the cluster enforces them, allow only required UI-to-API, service-to-database, worker-to-Redis, LiteLLM-to-provider, Firecrawl, Agent, and Prometheus flows. Test egress carefully: a bad policy can leave Agents running but unable to reach model or messaging providers.

OpenRouter and LiteLLM

Use a production-dedicated OpenRouter key when isolation is required. Assign its budget, credit limit, rate limits, billing alerts, owner, rotation path, and emergency replacement procedure. The OpenRouterCreditsLow alert is useful only when the provider key has a credit limit; an unlimited key reports infinite remaining credit.

Keep LITELLM_MASTER_KEY stable and separate from the OpenRouter provider key.

Transactional email

SENDER_EMAIL=noreply@mail.agentbarn.example.com

Transactional email requires the Cloudflare account ID, API token, and sender address, plus a verified sending domain. Verify invitations, password recovery, lifecycle notifications, reputation, and quota. Use a different sender subdomain for staging.

Production and staging share the Cloudflare account, token, and account quota in the current workflow, so rotation and quota exhaustion can affect both.

Google Workspace

https://agentbarn.example.com/api/v1/integrations/google/callback

Register the exact production callback. If an OAuth client is shared with staging, register that callback too; separate clients provide stronger failure and consent-screen isolation.

Firecrawl

Treat the platform Firecrawl key as a production credential. Its Services remain internal, but Firecrawl makes outbound requests, so enforce appropriate egress, destination, abuse-prevention, and capacity policies.

API, UI, Communications, worker, LiteLLM, Redis, all PostgreSQL databases, each Firecrawl component, Prometheus, Grafana, and Alertmanager run as single replicas by default.

Each Agent adds a Deployment, pod, Service, 1 GiB PVC, CPU and memory demand, model usage, platform API traffic, and monitoring traffic. Agent resource profiles are not configurable through .env.deploy.

Load-test representative Hermes and OpenClaw workloads in staging, including concurrent Agents, peak tools, large model responses, browser or Firecrawl work, Template and Skill loading, and restart behavior.

Production Platform messaging requires a separate Communications Deployment and ClusterIP Service on port 8002, with /health probes, internal /metrics, a provider ingress supervisor, and an outbound Delivery worker. It uses the same API image tag as Product API while running a separate process.

Communications replicas coordinate supervised provider ingress through PostgreSQL-backed leases: one replica owns a supervised Connection at a time, webhook traffic may load-balance, and lease expiry enables failover. Every replica shares the same database and encryption configuration. Use queue depth, oldest queued age, and provider status to guide scaling; Connection revision changes reconcile provider sessions without restarting Agents.

If the availability target requires multiple replicas, replication, or failover, treat that as custom architecture and validation work, not an environment-variable toggle.

SLACK_ALERTS_WEBHOOK_URL=REPLACE_WITH_PRODUCTION_WEBHOOK
GRAFANA_ADMIN_PASSWORD=REPLACE_WITH_STRONG_PASSWORD
GRAFANA_HOST=grafana.agentbarn.example.com

For every alert, document severity, on-call owner, response target, diagnostic link, escalation, recovery action, and user-notification threshold. Trigger a controlled staging alert before go-live.

Understand monitoring scope

Prometheus requests a 10 GiB PVC and retains 15 days by default. The namespace-scoped configuration does not collect cluster-wide node, kubelet, or cAdvisor metrics.

Add cluster monitoring for nodes, filesystems, volumes, container CPU and memory, the control plane, ingress, and cert-manager. Grafana dashboards are operational views, not audit logs or business records.

Prometheus should scrape Communications internally from port 8002. Monitor error and degraded Connection counts, dead-lettered Delivery outcomes, queue growth, oldest queued age, Delivery latency, reconnect frequency, policy-rejection changes, and Communications pod readiness and restarts.

• agentbarn_communication_connection_status
• agentbarn_communication_delivery_outcomes
• agentbarn_communication_queue_depth
• agentbarn_communication_oldest_queued_age_seconds
• agentbarn_communication_delivery_latency_seconds
• agentbarn_communication_reconnects
• agentbarn_communication_policy_dispositions

These metrics intentionally exclude Organization, Agent, Connection, Conversation, and User identifiers. Prometheus and alerts detect platform-wide changes; Communication diagnostics supports Agent-scoped investigation, the Communications journal provides content-free stage history, Agent logs describe Runtime behavior, Conversation history contains accepted message content, and Security Audit records selected recovery and health events. Do not place message content, provider tokens, or raw exception text in metrics or alerts.

Write and schedule a recovery exercise using this minimum sequence:

1. Provision an isolated recovery namespace or cluster.
2. Restore stable keys and the three PostgreSQL databases.
3. Restore required Agent PVCs and compatible deployment artifacts.
4. Deploy without changing the restored keys and verify the expected schema.
5. Verify authentication, LiteLLM virtual keys, and a model request.
6. Verify Shared Credentials and Agent credentials can decrypt.
7. Start a test Agent and verify messages, Tool Calls, and costs.
8. Verify Event Delivery reconciliation, monitoring, and alerts.

Preserve version compatibility

Record the git commit, API/UI and runtime image tags, chart versions, Alembic revision, PostgreSQL and LiteLLM versions, backup time, and secret-manager references with every recovery set. Do not restore into an arbitrary application version.

Plan migration rollback

Database migrations run as a Helm pre-install/pre-upgrade hook. Helm rollback changes manifests and images; it does not reverse Alembic migrations. Before schema changes, review compatibility, back up the application database, test upgrade and recovery in staging, and decide whether rollback uses an older compatible image or a database restore.

Rollback planning also covers release-tagged Product and Communications images, Runtime base-image versions, Communications protocol compatibility, Platform Plugin behavior, pending and dead-lettered Deliveries, and provider-session reconciliation. API and Communications normally roll back together because they use the same API image tag; durable Deliveries remain in PostgreSQL across replacement. Do not delete Connection or Conversation data, reuse a release tag for different image contents, or assume database migrations are reversible.

In staging, verify API and UI health, administrator login, Organization isolation, invitations and email, OAuth, Agent creation, Hermes and OpenClaw startup, messaging platforms, LiteLLM requests, Activity, Tool Calls, logs, costs, Shared Credentials, Event Delivery processing, dashboards, alerts, migrations, and restart recovery.

Use dedicated staging chat applications, credentials, and data. Do not test with production workspaces or customer conversations.

Immediately before production

kubectl config current-context
kubectl get pods --namespace agent-farm
kubectl get pvc --namespace agent-farm
kubectl get certificate --namespace agent-farm

Confirm the production backup completed, is readable, and is associated with the matching stable-key set.

Production Communications canary

1. Confirm migrations and Product, Ingest, Communications, worker, and UI readiness.
2. Confirm Communications /health and Prometheus scraping internally, then verify Runtime DNS to its ClusterIP Service.
3. Create or use a dedicated canary Agent and add a canary Communication Connection.
4. Confirm provider credential validation, start the Agent, and send an inbound message that passes policy.
5. Confirm the pipeline advances from Provider observed through Provider delivered, with the Conversation under the correct Connection.
6. Confirm a Tool Call appears through Ingest separately and no queue-age, dead-letter, or Connection-error alert is introduced.
7. Where the Platform uses webhooks, confirm the public provider webhook path works.

A Product API health check alone is not a sufficient canary. Hermes and OpenClaw share the Communications protocol: test at least one supported Runtime after Runtime or adapter changes, both after shared protocol changes, the affected Platform Plugin after provider changes, supervised ingress after Slack/Telegram/Discord changes, webhook ingress after Teams or ingress changes, and outbound delivery after provider-client changes.

Verify releases, workloads, storage, ingress, certificates, and the public API.

helm list --namespace agent-farm

kubectl get deployments,statefulsets,pods,jobs,cronjobs \
  --namespace agent-farm

kubectl get pvc,ingress,certificate \
  --namespace agent-farm

curl --fail https://api.agentbarn.example.com/api/v1/health

Then sign in with a named Platform Administrator, select the production Organization, invite a controlled test user, verify email enrollment, and start a non-sensitive test Agent. Send a message and verify the response, Activity, Tool Calls, cost attribution, Agent metrics, and alert state. Retire the test Agent according to policy and record the deployed commit and verification result.