Skip to main content

High Availability (HA) Deployments

To ensure continuous service delivery and eliminate service interruptions during updates, the vps-apps stack implements a Zero-Downtime Blue-Green High Availability (HA) Deployment pattern.

This system guarantees that when an application is updated, a healthy instance of the new version is fully running and registered with the Traefik reverse proxy before the old version is stopped and removed.

Architecture Overview​

diagram

1. Generic Blue-Green Deployments (Stateless Apps)​

All generic, stateless applications in this repository use a unified Zero-Downtime HA orchestrator script located at scripts/deploy-app.sh.

How the Orchestrator Works​

diagram

When deploy-app.sh is executed for a project directory:

  1. Active Stack Detection: The script inspects running Docker containers to see if the canonical name ${PROJ} is already running.
    • If running, the Active Stack is Blue (${PROJ}), and the Next Stack is defined as Green (${PROJ}-next).
    • If not running (or if ${PROJ}-next is running), the Active Stack is Green (${PROJ}-next), and the Next Stack is Blue (${PROJ}).
  2. Compose Cleaning (clean-compose.py): To avoid container name and port mapping collisions while both stacks overlap:
    • It strips any hardcoded container_name fields.
    • It strips host ports mapping definitions (e.g. 8080:80).
    • It outputs a temporary compose file: docker-compose.ha-temp.yml.
  3. Concurrent Start: The script starts the Next Stack using the cleaned compose file and a custom project name (-p "$NEXT_PROJ").
  4. Initialization & Health Wait: The script allows the new stack to boot and initialize for 10 seconds, during which Traefik auto-detects it via Docker labels and routes traffic.
  5. Teardown: Once the new stack is online, the orchestrator gracefully stops and removes the old stack ($ACTIVE_PROJ).

2. Custom Blue-Green Orchestrations​

Deeply integrated or multi-container applications (like admin and devex) require custom orchestration for APIs, frontends, or internal networks. They use dedicated deploy.sh scripts in their respective directories instead of the generic orchestrator.

The admin Stack​

The admin app consists of a React frontend and a Node.js API sidecar.

  • Frontend Swap (admin-ha): The new React frontend is started inside container admin-next on the internal network. Once healthy, it is connected to traefik-net using the --alias admin-ha network alias. The old frontend is disconnected, stopped, and removed, and the new container is renamed to admin.
  • API Swap (admin-api-ha): The Node API is sequenced to prevent overlaps. It starts inside admin-api-next, waits for health checks, attaches to the monitoring network, attaches to traefik-net, and re-attaches to the internal network using --alias admin-api-ha before tearing down admin-api.

The devex Stack​

The devex developer platform consists of the main FastAPI application and a docs-site sidecar container.

  • Main App Swap (devex-ha): Starts devex-next container, waits for health checks, attaches it to traefik-net with the --alias devex-ha network alias, disconnects/removes the old devex container, and renames devex-next to devex.
  • Docs Swap (devex-docs-ha): Starts devex-docs-next, waits for health checks, attaches it to traefik-net with --alias devex-docs-ha routing alias, and replaces the old devex-docs container.

3. Stateful Infrastructure (Exclusions)​

Stateful services that manage data volumes or configuration endpoints are not suited for Blue-Green concurrent swaps, as multiple concurrent containers writing to the same volumes could cause data corruption or mount lockouts.

The following services bypass Blue-Green and deploy using standard in-place upgrades (which are also handled automatically when invoked via deploy-app.sh):

  • PostgreSQL (postgres/)
  • Traefik Reverse Proxy (traefik/)

Emergency Routing Alias Recovery​

If a raw docker compose up -d is run on production for an HA application, the routing alias is stripped, resulting in 502 Bad Gateway errors from Traefik.

To restore service immediately without restarting the containers, manually re-attach the routing aliases:

For devex​

docker network disconnect traefik-net devex 2>/dev/null || true
docker network connect --alias devex-ha traefik-net devex

docker network disconnect traefik-net devex-docs 2>/dev/null || true
docker network connect --alias devex-docs-ha traefik-net devex-docs

For admin​

# Re-attach frontend alias to Traefik network
docker network disconnect traefik-net admin 2>/dev/null || true
docker network connect --alias admin-ha traefik-net admin

# Re-attach API alias to internal Admin network
docker network disconnect admin_admin-internal admin-api 2>/dev/null || true
docker network connect --alias admin-api-ha admin_admin-internal admin-api