Locks & concurrency
The lock protects a conflicting operation, not a command or an SSH connection.
Connected deployments of independent image apps can overlap image pulls,
preDeploy, startup, readiness, post-switch observation and draining. This applies
to source App files and deploy/reload/rollback/scale app NAME.
Current boundaries
| Resource | Protected work | Duration |
|---|---|---|
app:NAME | App contract, secrets, templates, generation choice, preDeploy, rollout/rollback | Validated mutation through final snapshot commit; source image preparation runs before it and the previous generation's drain after it |
| Compose service identity, container names, upstream file | Explicit physical bindings that can alias across app names | Same app operation; separate services in one Compose project can overlap |
domains | Domain ownership, certificates, Cloudflare policy read/modify/write, edge site release and its whole-release rollback | Domain phase only, through snapshot commit |
| Caddy kernel lock | Change upstream/site files and reload (the reload validates and keeps the running config on rejection); restore a failed route | Actual switch/restore only, about one caddy reload; app readiness, observation and drain run outside it |
| Control kernel mutex | Acquire/release/break reservations and compare-and-swap current | Short local filesystem transaction; no upload, rollout or provider request |
server | Setup/bootstrap/restore, whole-manifest changes, extension/dependency lifecycle and operations without a scoped contract | Whole operation; conflicts with every reservation |
Reads and plans take no operation reservation and persist no revision/history.
Source image App apply/deploy --apply validates dependencies, bindings, secret
references and domain conflicts in a read-only session, then downloads the image
before acquiring its operation reservation. Explicit digests are downloaded too.
Tag resolution uses the digest returned by its own docker pull, not a later
lookup of the mutable local tag. Docker owns cache coordination. A fresh session
then acquires the reservation, reloads VM configuration/secrets and repeats those
checks before rollout. Changed server identity or incompatible bindings fail
closed; concurrent independent changes survive snapshot merge.
Under its app lock the runtime pulls a digest only when the VM no longer holds it (for example, it was pruned after preparation); a digest already present is not asked of the registry again. Tags still pull there, preserving registry failure behavior. Migration, readiness, observation and rollback remain inside the app operation. Draining the previous generation does not (see below). Named resource commands, extensions and legacy whole-manifest deployments retain their existing preparation boundaries.
App secret updates reserve only their app. Server secret updates are exclusive because all apps and domain operations may consume them. Extension and legacy whole-manifest operations retain server scope: they can alter shared providers, networks, authentication and routes. They must be decomposed into explicit resource contracts before narrowing them; they are not assumed independent just because their CLI names differ.
A source App with domains first reserves its app and rolls it out. When its
domains are unchanged, the deploy does not take domains at all: a read-only
check confirms that the committed domain specs equal the file, the stored origin
certificate would be reused (not issued or renewed), the live edge release holds
byte-identical site, certificate and key files (compared by digest on the VM)
and publishes every host, and Cloudflare already has the DNS records, strict SSL
and cache rules the phase would write. It then verifies the origin and public
routes, as the phase would, and reports Domains unchanged; verified origin and public routes without the domains lock. Any difference, failed read or failed
verification falls back to the full phase below and names the reason.
Otherwise it commits the healthy app, then acquires domains (waiting at most
120 seconds), refreshes the committed snapshot/certificate state, and
reconciles domains. Another app
can continue its rollout during that wait. Domain errors preserve the committed
app and any issued certificate state, and report failure. Standalone Domain
files and individual domain CRUD use the same domains reservation.
Domain operations remain serialized with each other, even for different zones: Cloudflare policies are zone-wide read/modify/write, and origin-check failure currently rolls back the entire edge site release. Per-zone locks alone would not make that rollback safe. Narrower domain locking requires per-site rollback first. Apps switching their independent upstream snippets need only the short Caddy kernel lock, so two different apps deploying at once (unchanged domains or none) wait on each other only if their route switches coincide, for at most one reload.
A custom preDeploy command may write a database shared by multiple apps. App names cannot identify that external write set. Such migrations must use their migration engine's database lock or an explicit shared database lock in the migration command. The CLI does not assume that different app names imply independent database schemas.
Snapshot commits
Each session remembers its baseline. At commit, it fetches the latest revision and applies only its changes: apps/domains by name, app secrets by app, portable state by path, and other configuration by top-level resource. Unchanged entries come from the latest snapshot. A conflicting edit fails rather than overwriting another writer; values are not included in the error.
Uploads happen outside the control mutex. Inside the mutex the writer checks all held reservation tokens and the expected current revision, then atomically switches the symlink. If the revision changed during upload, only merge/upload/ commit is retried (at most 20 attempts); migrations and rollout are never rerun. A failed final commit retains the private recovery snapshot as before.
Crash recovery and compatibility
Reservations persist across operator crashes or SSH loss: external work may
still be running, so neither TTL expiry nor an operator PID proves safety.
server lock lists each scoped reservation's resources, owner and lockId.
server unlock --lock-id ID --apply archives only that reservation. It cannot
release another app's lock. Revoked writers cannot commit, including writers
whose domain reservation was revoked after their app rollout.
Scoped reservations occupy the old control/lock gate directory, so older
CLIs fail closed on acquire. An old global lock blocks new scoped writers.
Use the updated CLI for inspection/unlock of scoped locks; do not use an old
unlock implementation that treats the gate as a single reservation. Never mix
legacy local-manifest mutations with connected sessions on the same VM.
The implementation retains the existing per-app VM flock across rollout and
drain. This also protects runtime work still executing after an SSH disconnect.
Manually breaking a control reservation does not cancel that work.
Once the route is committed, the previous generation only drains in-flight
requests (drainSeconds) and stops. That runs in a detached VM process that
inherits the app flock, so the rollout returns and the operator's control
reservation is released without waiting for it. A server-scoped operation is no
longer held behind another app's drain. The next rollout, rollback, scale or
retire of the same app waits for the flock up to
120 + drainSeconds + stopTimeoutSeconds seconds, then fails as before. A
generation that fails to stop is logged to the VM journal under tag 2server
and is left running, as before.
Timing and contention diagnostics
Connected mutation commands write [2srv lock] JSON events to stderr. acquired
includes operation, resources and waitMs; released includes heldMs.
Times use the local monotonic clock and include SSH round trips. They measure
control reservations, not individual runtime flock waits or rollout phases.
Most of a reservation used to be connection setup and probe overhead, not
work. Each remote step now reuses one OpenSSH master per transport (see the
operator guide), so a step costs a round trip instead of a fresh connection,
which took 4-7 seconds through an IAP tunnel. Readiness probes run curl from
the host to the container address when the network is not internal, instead
of starting a curl container per probe. Extension image tags are resolved
concurrently.
No revision/history is written for these diagnostics.
An initial conflict fails immediately, with a blocked event. Set
TWO_SERVER_LOCK_WAIT=SECONDS (0-3600) to let an app or other scoped mutation
wait that long for the conflicting reservation instead, for example an app
deploy that meets a brief server operation. Server-scoped operations never wait:
they would refuse the revision the blocker commits. The source domain phase
waits up to its existing 120-second budget. A waiting acquire emits waiting
once, then acquired or blocked, and polls from 250ms backing off to 2s. Conflict events include the blocking reservation's
lockId, resources and creation time when available; use 2srv server lock for
its operator details. Tokens and snapshot/secret values are excluded. SSH,
ownership and invalid-response errors report acquire-failed separately from
contention. release-failed is not evidence that the reservation was freed.
heldMs ends when release returns; an interrupted CLI may have no final event.
Reservations still do not expire automatically. Use normal inspection/recovery,
not timing alone, to decide whether an abandoned reservation can be broken.
