Lifecycle

Holden uses webhooks and polling to maintain a queue of apps that need reconciliation. A worker processes them one at a time.

The Queue

The queue is an in-memory, deduplicated list of apps. If an app is already queued, pushing it again is a no-op. This prevents redundant work.

The worker processes the queue sequentially:

1. Take an app from the queue
2. Fetch its config from git
3. Reconcile the app (including its needs)
4. Repeat

What Pushes to the Queue

Trigger	What gets queued	When
Webhook (/webhook/:app)	That app	Push to app’s git repo
Boot	All apps	Holden starts
Poll timer	All apps	Every poll_interval seconds (default: 300)

flowchart LR
    W["Webhook (:app)"] --> Q[Queue]
    B[Boot] --> Q
    P[Poll timer] --> Q
    Q --> Worker --> Reconcile

Maintenance doesn’t push to the queue directly. Instead, it pauses the queue, processes all apps, then reboots Holden. The fresh Holden instance queues all apps on boot.

Boot Flow

When Holden starts:

1. Check if Overseer is needed (no labels, labels changed, stuck in -old state)
2. If yes → spawn Overseer, idle until replaced
3. If no → queue all apps

This ensures all apps converge to desired state after any restart—whether from a fresh install, server reboot, or Overseer update.

The nightly maintenance window triggers the Overseer:

flowchart LR
    H[Holden] --> O[Overseer] --> N[New Holden] --> Boot

Failure Handling

If reconciliation fails for an app:

1. Retry 3 times with backoff (1s, 5s, 30s)
2. Log the failure, move on to the next queued app
3. Docker’s restart policy handles container crashes
4. The next trigger (poll timer, webhook, or maintenance) will retry

No explicit “failed” state is tracked. The system is self-healing through its regular triggers.