2serverDocumentationGitHub
2server / DocumentationRead Markdown (.md) View source

Locks & concurrency

The lock protects a conflicting operation, not a command or an SSH connection. Connected deployments of independent image apps can overlap image pulls, preDeploy, startup, readiness, post-switch observation and draining. This applies to source App files and deploy/reload/rollback/scale app NAME.

Current boundaries

ResourceProtected workDuration
app:NAMEApp contract, secrets, templates, generation choice, preDeploy, rollout/rollbackValidated mutation through final snapshot commit; source image preparation runs before it and the previous generation's drain after it
Compose service identity, container names, upstream fileExplicit physical bindings that can alias across app namesSame app operation; separate services in one Compose project can overlap
domainsDomain ownership, certificates, Cloudflare policy read/modify/write, edge site release and its whole-release rollbackDomain phase only, through snapshot commit
Caddy kernel lockChange upstream/site files and reload (the reload validates and keeps the running config on rejection); restore a failed routeActual switch/restore only, about one caddy reload; app readiness, observation and drain run outside it
Control kernel mutexAcquire/release/break reservations and compare-and-swap currentShort local filesystem transaction; no upload, rollout or provider request
serverSetup/bootstrap/restore, whole-manifest changes, extension/dependency lifecycle and operations without a scoped contractWhole operation; conflicts with every reservation

Reads and plans take no operation reservation and persist no revision/history. Source image App apply/deploy --apply validates dependencies, bindings, secret references and domain conflicts in a read-only session, then downloads the image before acquiring its operation reservation. Explicit digests are downloaded too. Tag resolution uses the digest returned by its own docker pull, not a later lookup of the mutable local tag. Docker owns cache coordination. A fresh session then acquires the reservation, reloads VM configuration/secrets and repeats those checks before rollout. Changed server identity or incompatible bindings fail closed; concurrent independent changes survive snapshot merge.

Under its app lock the runtime pulls a digest only when the VM no longer holds it (for example, it was pruned after preparation); a digest already present is not asked of the registry again. Tags still pull there, preserving registry failure behavior. Migration, readiness, observation and rollback remain inside the app operation. Draining the previous generation does not (see below). Named resource commands, extensions and legacy whole-manifest deployments retain their existing preparation boundaries.

App secret updates reserve only their app. Server secret updates are exclusive because all apps and domain operations may consume them. Extension and legacy whole-manifest operations retain server scope: they can alter shared providers, networks, authentication and routes. They must be decomposed into explicit resource contracts before narrowing them; they are not assumed independent just because their CLI names differ.

A source App with domains first reserves its app and rolls it out. When its domains are unchanged, the deploy does not take domains at all: a read-only check confirms that the committed domain specs equal the file, the stored origin certificate would be reused (not issued or renewed), the live edge release holds byte-identical site, certificate and key files (compared by digest on the VM) and publishes every host, and Cloudflare already has the DNS records, strict SSL and cache rules the phase would write. It then verifies the origin and public routes, as the phase would, and reports Domains unchanged; verified origin and public routes without the domains lock. Any difference, failed read or failed verification falls back to the full phase below and names the reason.

Otherwise it commits the healthy app, then acquires domains (waiting at most 120 seconds), refreshes the committed snapshot/certificate state, and reconciles domains. Another app can continue its rollout during that wait. Domain errors preserve the committed app and any issued certificate state, and report failure. Standalone Domain files and individual domain CRUD use the same domains reservation.

Domain operations remain serialized with each other, even for different zones: Cloudflare policies are zone-wide read/modify/write, and origin-check failure currently rolls back the entire edge site release. Per-zone locks alone would not make that rollback safe. Narrower domain locking requires per-site rollback first. Apps switching their independent upstream snippets need only the short Caddy kernel lock, so two different apps deploying at once (unchanged domains or none) wait on each other only if their route switches coincide, for at most one reload.

A custom preDeploy command may write a database shared by multiple apps. App names cannot identify that external write set. Such migrations must use their migration engine's database lock or an explicit shared database lock in the migration command. The CLI does not assume that different app names imply independent database schemas.

Snapshot commits

Each session remembers its baseline. At commit, it fetches the latest revision and applies only its changes: apps/domains by name, app secrets by app, portable state by path, and other configuration by top-level resource. Unchanged entries come from the latest snapshot. A conflicting edit fails rather than overwriting another writer; values are not included in the error.

Uploads happen outside the control mutex. Inside the mutex the writer checks all held reservation tokens and the expected current revision, then atomically switches the symlink. If the revision changed during upload, only merge/upload/ commit is retried (at most 20 attempts); migrations and rollout are never rerun. A failed final commit retains the private recovery snapshot as before.

Crash recovery and compatibility

Reservations persist across operator crashes or SSH loss: external work may still be running, so neither TTL expiry nor an operator PID proves safety. server lock lists each scoped reservation's resources, owner and lockId. server unlock --lock-id ID --apply archives only that reservation. It cannot release another app's lock. Revoked writers cannot commit, including writers whose domain reservation was revoked after their app rollout.

Scoped reservations occupy the old control/lock gate directory, so older CLIs fail closed on acquire. An old global lock blocks new scoped writers. Use the updated CLI for inspection/unlock of scoped locks; do not use an old unlock implementation that treats the gate as a single reservation. Never mix legacy local-manifest mutations with connected sessions on the same VM.

The implementation retains the existing per-app VM flock across rollout and drain. This also protects runtime work still executing after an SSH disconnect. Manually breaking a control reservation does not cancel that work.

Once the route is committed, the previous generation only drains in-flight requests (drainSeconds) and stops. That runs in a detached VM process that inherits the app flock, so the rollout returns and the operator's control reservation is released without waiting for it. A server-scoped operation is no longer held behind another app's drain. The next rollout, rollback, scale or retire of the same app waits for the flock up to 120 + drainSeconds + stopTimeoutSeconds seconds, then fails as before. A generation that fails to stop is logged to the VM journal under tag 2server and is left running, as before.

Timing and contention diagnostics

Connected mutation commands write [2srv lock] JSON events to stderr. acquired includes operation, resources and waitMs; released includes heldMs. Times use the local monotonic clock and include SSH round trips. They measure control reservations, not individual runtime flock waits or rollout phases.

Most of a reservation used to be connection setup and probe overhead, not work. Each remote step now reuses one OpenSSH master per transport (see the operator guide), so a step costs a round trip instead of a fresh connection, which took 4-7 seconds through an IAP tunnel. Readiness probes run curl from the host to the container address when the network is not internal, instead of starting a curl container per probe. Extension image tags are resolved concurrently. No revision/history is written for these diagnostics.

An initial conflict fails immediately, with a blocked event. Set TWO_SERVER_LOCK_WAIT=SECONDS (0-3600) to let an app or other scoped mutation wait that long for the conflicting reservation instead, for example an app deploy that meets a brief server operation. Server-scoped operations never wait: they would refuse the revision the blocker commits. The source domain phase waits up to its existing 120-second budget. A waiting acquire emits waiting once, then acquired or blocked, and polls from 250ms backing off to 2s. Conflict events include the blocking reservation's lockId, resources and creation time when available; use 2srv server lock for its operator details. Tokens and snapshot/secret values are excluded. SSH, ownership and invalid-response errors report acquire-failed separately from contention. release-failed is not evidence that the reservation was freed.

heldMs ends when release returns; an interrupted CLI may have no final event. Reservations still do not expire automatically. Use normal inspection/recovery, not timing alone, to decide whether an abandoned reservation can be broken.