Upgrade Agent Barn

Overview

An Agent Barn upgrade can change the API, UI, worker, database schema, supporting services, monitoring, Agent builders, and runtime images.

GitHub branch deployments and versioned release bundles use the same Helmfile release graph. Neither path automatically rebuilds existing Agent workloads, so platform deployment and runtime rollout remain separate operations.

Understand the upgrade boundaries

Changes that are not routine upgrades

• Changing PostgreSQL passwords on initialized volumes.
• Rotating SECRET_SIGNING_KEY, AGENT_TOKEN_ENCRYPTION_KEY, or LITELLM_MASTER_KEY.
• Changing a StatefulSet’s existing StorageClass.
• Renaming agent-farm or agent-farm-staging.
• Moving persistent data to another cluster or storage provider.

Choose an upgrade path

Cluster:         dedicated Talos cluster
Workflow:        .github/workflows/deploy-public.yml
Release tag:     vX.Y.Z
API/UI tags:     vX.Y.Z (pinned)
Registry:        registry.agentbarn.dev
Configuration:   PUBLIC_ prefixed values and secrets

Before you begin

Assign a release owner, database and backup owner, verification owner, incident decision-maker, Agent restart owner, and communication owner.

• Confirm staging, production monitoring, DNS, certificates, storage, registries, and providers are healthy.
• Confirm there is no active incident or overlapping automated or manual deployment.
• Verify operators can reach Kubernetes, Helm history, application logs, and backups.
• Confirm every required image exists before approval.

Prepare the target release

1. Select the target release from Agent Barn releases and read its release notes.
2. Download its source ZIP or tar.gz and extract it into a separate working directory. If a matching deployment bundle has been provided instead, extract that separately and use its included configuration.
3. For a source archive, copy .env.deploy.spec to .env.deploy. Carry forward your installation's infrastructure settings and credentials, including stable signing and encryption keys. Account for settings added or changed in the target release.
4. Obtain the image locations and component versions intended for the target release. Preserve separate API/UI and runtime version selections; source download availability does not establish image availability.
5. Continue with the backup and migration preparation below before deploying.

API and UI use the selected product release tag. Runtime image tags remain separate; do not replace every component tag with the product tag or with latest.

Verify the Communications upgrade surface

Communications is a separately served process. Its Deployment, internal ClusterIP Service, port 8002, /health, /metrics, Runtime protocol base URL, Connection-scoped public webhook ingress, database-backed ingress leases, durable Deliveries, and journal retention must be checked independently. Product API readiness does not prove the Communications rollout or end-to-end provider delivery succeeded.

Check the Communications rollout and process health

Run these commands from a terminal with kubectl configured for the cluster you just upgraded.

The examples use Agent Barn's standard namespace, agent-farm, and API Helm release name, agentbarn-api. The Communications Deployment and Service are both named agentbarn-api-communications.

If your installation uses a different namespace, replace agent-farm. If you changed the API release name, replace agentbarn-api-communications with <your-api-release-name>-communications.

1. Wait for the Communications Deployment

kubectl rollout status deployment/agentbarn-api-communications \
  --namespace agent-farm

Wait for Kubernetes to report that the Deployment successfully rolled out. If it reports a failure, inspect the Communications workload before continuing with the upgrade checks.

2. Confirm that the Service exists

kubectl get service agentbarn-api-communications \
  --namespace agent-farm

The output should list the Communications Service with port 8002/TCP.

The Service is internal to the cluster. Its existence alone does not confirm that the application is healthy.

3. Check process health from inside the cluster

The following command creates a temporary pod in the same namespace, calls the internal health endpoint, and removes the pod when it exits.

Your Kubernetes account needs permission to create and attach to a pod in this namespace. The cluster also needs to be able to pull the curlimages/curl image.

kubectl run communications-health-check \
  --namespace agent-farm \
  --rm -i \
  --restart=Never \
  --image=curlimages/curl \
  -- curl --fail --show-error \
  http://agentbarn-api-communications:8002/health

A successful response contains:

{"status":"ok"}

If you use a different API Helm release name, update the hostname in this command as well as the resource names in the earlier commands.

This response confirms that the Communications process is reachable and responding. It does not confirm that every Slack, Microsoft Teams, Telegram, or Discord connection is working.

4. Check an actual connection

After the process check succeeds:

1. Open Agent Barn and select a running Agent with a configured chat connection.
2. Review that connection's health.
3. Send a message from a location and user allowed by the connection's settings. Mention the bot if its policy requires it.
4. Confirm that the Agent replies through the same connection.

If the health endpoint succeeds but messaging fails, inspect the affected connection and its delivery diagnostics. See Communication Diagnostics.

Also confirm on the Communications surface

• /metrics remains internally scrapeable.
• Webhook ingress exposes only the required Communications prefix; webhook Connections retain their generated Connection-specific URLs.
• Supervised Connections reconcile, statuses do not remain unexpectedly PENDING, CONNECTING, DEGRADED, or ERROR, and ingress leases do not show persistent contention.
• Queue depth, oldest queued Delivery age, dead letters, latency, reconnects, and policy dispositions remain within the normal profile.

After migration

Confirm existing Connections remain scoped to their correct Agents and Organizations, encrypted credentials remain readable without exposure, enabled Connections are reconciled, and status transitions resume. Account for pending or processing Deliveries, dead-letter history, and journal entries; keep Conversation Messages scoped by Connection and provider location. Regenerate Runtime protocol credentials through normal Agent start behavior when required; do not move Connection credentials into Agent Secrets or Runtime configuration.

Replica coordination

Communications replicas coordinate supervised provider ingress through database leases. Expect leases to transfer while old replicas terminate and new replicas become ready; watch for reconnect spikes, sustained duplicate sessions, lease contention, and Connections stuck in CONNECTING or ERROR. Do not manually assign provider sessions to pods or run independent Telegram pollers for the same bot outside Agent Barn.

Rollback and recovery

Application and database rollback are separate decisions

Before any rollback, confirm the previous API image operates against the migrated schema, the migration is backward-compatible, and new Connection, Delivery, journal, Template, Skill, or Agent data remains readable. Also confirm the previous Runtime image supports the active Communications protocol and the previous Platform Plugin understands persisted Connection settings and schema versions. Preserve the deployed tag and diagnostic evidence. If schema compatibility is uncertain, stop and assess the migration rather than blindly downgrading Alembic or restoring the database.

k3s branch rollback

k3s API and UI use moving latest tags. A Helm rollback can restore an old manifest revision while still pulling the current image. Prefer a reviewed corrective or revert commit that rebuilds, republishes, rolls out with a new commit annotation, and passes verification.

Hosted public and release-bundle rollback

Where schema compatibility permits, redeploy the previous immutable API and UI tags; restore compatible Hermes and OpenClaw versions if they changed. Retain stable secrets unless restoring matching data and keys, then recheck Product API, Ingest, Communications, and workers independently. Confirm Connection reconciliation and Delivery processing resume, using the Communications journal to identify Deliveries that require operator action.

Database rollback

Helm rollback does not execute alembic downgrade. Use a tested, data-safe downgrade only when compatible with the restored application; otherwise use a forward fix or restore the database recovery set.

Agent runtime rollback

Restoring the old platform runtime reference does not change Agents already restarted. Identify affected Agents, verify one canary on the restored reference, then proceed in batches.

Inspect partial Helmfile state

helm list --namespace agent-farm

helm status RELEASE_NAME --namespace agent-farm
helm history RELEASE_NAME --namespace agent-farm

Releases have independent histories. Do not assume they share a Helm revision or upgrade status.

Recover the correct component

• Restart or roll Communications: replaces the service processes.
• Request a Connection reconnect: increments that Connection’s revision and recreates its provider session.
• Retry a Delivery: requeues an eligible dead-lettered outbound Delivery only after its provider, credential, policy, configuration, or Runtime failure is corrected.
• Restart an Agent: replaces Runtime resources and protocol credentials.

Use the smallest recovery action that addresses the fault. Do not restart every Agent for Connection-only changes.

Troubleshooting

Upgrade checklist

• Before staging

  • The release scope is understood.
  • New variables and provider requirements are documented.
  • Database migrations were reviewed.
  • Runtime changes were identified.
  • Stable-key and storage changes are excluded or separately planned.
  • Backups completed and restore procedures were tested.

• Before production

  • Staging deployed the complete release.
  • Runtime and Platform Plugin canaries were selected independently.
  • Migration duration and compatibility were verified.
  • Monitoring and alert delivery are healthy.
  • Production backups completed.
  • Current releases, images, and Alembic revision were recorded.
  • Rollback and recovery decisions are documented.
  • No other deployment is active.
  • Operators and incident owners are available.

• After production

  • Helm releases are healthy.
  • Expected images are running.
  • Database revision matches the release.
  • Product, Ingest, Communications, UI, worker, and providers are healthy.
  • Existing data and access boundaries remain correct.
  • Monitoring targets and alerts are healthy.
  • A canary Agent passed.
  • Required Agent restart batches are tracked.
  • Release and recovery records are complete.

Next steps

Keep production configuration, Kubernetes deployment, migrations, self-hosting Communications, and monitoring available during the rollout. For operational follow-up, use Communication Connections, Communication diagnostics, Runtime deployment, and the Slack, Microsoft Teams, Telegram, and Discord setup guides.

Roll out initiated delivery

Initiated delivery depends on compatible application schema/code, runtime-generated scripts, and runtime image hooks. Roll the application through its normal migration workflow, deploy the matching runtime images, and recreate affected Agent runtime configuration through the supported stop/start flow. Existing pods do not acquire new image hooks or generated scripts merely because a new application image was built. Preserve the Agent's persistent state when performing a normal restart.

Hermes session continuity

Agent Barn's Hermes adapter sends resume_session: true with a stable Connection/location/thread session identity. The shipped Hermes image patches its run endpoint to reload persisted conversation history, including compaction lineage and tool-call metadata. Deploy the compatible patched image and adapter together when adopting this behavior.

Existing session state remains on the Agent's persistent volume. New sessions can legitimately have no history. Do not recreate storage as a routine response to missing continuity, and do not assume updating the application automatically replaces every running Agent pod or image. Recreate the affected runtime through the normal Agent lifecycle after selecting the intended compatible runtime image.

Runtime images have their own version sources. A product release tag is not a substitute for the Hermes or OpenClaw image version. Preserve persistent state when rolling runtime configuration.

See Choose an Agent Runtime for transport and replacement boundaries.

Client image release

The manual Client Release workflow publishes customer API, UI, Hermes-base, and OpenClaw-base images without deploying a customer cluster. It is separate from k3s and public hosted deployments. See Client image release for registry settings, independent Runtime versions, and backend topology assumptions.