2serverDocumentationGitHub
2server / DocumentationRead Markdown (.md) View source

Reliability

2server provides process restart, health-aware routing, guarded deployments and local persistent recovery. It does not provide host-level HA: losing the VM, its disk or network takes down the edge, databases and monitoring together. Running Sentinel or a NATS quorum on this same VM would not remove that failure domain. No extra VM, quorum or automatic database promotion is provisioned.

ComponentImplemented protectionRemaining boundary
AppsDocker restart, PID 1 init, bounded resources/logs; blue/green service deploy and rollback; every candidate passes readinessA single replica has restart downtime; removed containers need CLI reconciliation
CaddyContinuous active app health; passive eviction only with multiple serving replicas; bounded retry, config validation/rollback, graceful stopOne edge process/VM; existing direct proxy routes have no inferred health path
RedisAuthenticated PING and AOF write health, persistent AOF, rewrite memory headroom, graceful stopSingle process; no replica or off-VM backup automation
NATSAuth, JetStream readiness, file storage and explicit fsync, bounded clients/payload/pending dataCore is transient; JetStream is single-node; no off-VM backup automation
MonitoringComponent readiness, private exporter/Alertmanager network, guarded config update, runtime/DB/host alertsLocal monitoring cannot notify through a VM outage; configure an external uptime check

Sizing the VM

2server's own footprint is Docker Engine and the shared Caddy proxy; everything else is what your apps and extensions declare. Start at 2 GB. That is what 2srv provision creates (GCP e2-small, AWS t3.small, 30 GB disk), and it fits a small app with Postgres. Below 2 GB, little is left once the OS, Docker and Caddy run.

Add up what you declare:

  • Each app: replicas × memoryMb, twice while a blue/green release runs both generations.
  • Each extension's memoryMb. Defaults: postgres 512, redis 256, nats 256, meilisearch 1024, soot 512, cliproxy 256, monitoring 512 (Prometheus 384, Alertmanager 64, node exporter 64).
  • The fixed cost, measured on a production 2-vCPU VM (Debian 12, 8 GB, 22 containers): dockerd ~235 MiB, containerd ~65 MiB, ~9 MiB per container, Caddy ~230 MiB under that VM's production traffic. Fewer containers and less traffic cost less.
  • Spare room for the kernel's page cache and for spikes. Memory limits cap a container; they do not reserve physical memory.

Example: one app at 256 MiB with postgres needs 512 MiB for the app during a release plus 512 MiB for the database, about 1 GB, plus the fixed cost: a 2 GB VM fits it with little spare. Adding monitoring and meilisearch takes it to 4 GB.

2server does not create swap. Add a 1–2 GB swap file: without one, a VM that runs out of memory can stop answering SSH instead of the kernel killing one container. That VM runs with 2 GB of swap and uses about 3.1 GB of its 8 GB.

Apps and edge

For a stateless HTTP service that must keep serving through one process failure, set replicas: 2 (or more). The default stays one: 2server does not silently double RAM allocation. Size the host for both generations during blue/green deploy: replicas × memoryMb × 2, plus all extensions, Caddy, kernel and spare capacity. A 2 GB VM may not have room for every app with two replicas and all extensions. The CPU/memory limits isolate processes; they do not reserve physical capacity.

json
{
  "replicas": 2,
  "healthPath": "/readyz",
  "stopTimeoutSeconds": 60,
  "drainSeconds": 70
}

These are fields in an existing app spec, not a complete create/update spec. Readiness must return 2xx when that instance can accept requests; it should be cheap and should not write business data. Redirects are not readiness responses. Caddy checks every five seconds, quarantines after two failed active checks and requires two successes for recovery. Connection failures also quarantine for ten seconds. It retries connection selection for up to three seconds; after a connection succeeds, only GET/HEAD can be retried. Application POST side effects still need idempotency keys when clients themselves retry. See the Caddy reverse proxy contract.

Containers use Docker init to forward signals and reap children. Applications must handle SIGTERM and finish/return work before stopTimeoutSeconds. The old service generation drains for drainSeconds after the Caddy switch; long requests and websocket clients need explicit reconnect/resume behavior. Workers stop the old generation first; jobs need durable queues, bounded concurrency and safe redelivery. Do not scale singleton schedulers without a distributed lock.

Managed Caddy gets a local admin readiness probe and 75 seconds to stop (the Caddy grace period is 60 seconds). Existing Caddy adoption retains its operator's runtime contract; inspect its restart/stop settings separately. App routing improvements apply on app deploy/rollback, including when using existing Caddy.

Docker restarts exited processes with unless-stopped; an explicitly stopped container stays stopped, including after reboot. An unhealthy status alone does not trigger a restart. Readiness failures remove an app from Caddy and raise HTTP readiness alerts; do not restart every client when a shared database is unavailable. Inspect and use scoped CLI reload after fixing the cause. See Docker restart semantics.

Redis

Redis uses AOF as the persistence path, without periodic RDB snapshots. Default appendfsync: "everysec" trades throughput for a roughly one-second crash-loss window; select "always" when each write must be fsynced. Neither setting survives loss of the only disk. Snapshot/restore and remote backups are not automated.

maxmemoryMb cannot exceed half of memoryMb, leaving room for process overhead and copy-on-write during AOF rewrite. This is a conservative starting point, not an OOM guarantee under arbitrary workloads. noeviction returns write errors when full instead of silently dropping queue/session data. Monitor memory and size/reduce data before the limit. Redis deployment persists vm.overcommit_memory=1 in /etc/sysctl.d/60-2server-redis.conf; removing the extension does not undo that shared host setting. See Redis memory guidance.

AOF write failure makes health unhealthy and emits a persistence alert. A truncated AOF is refused (aof-load-truncated no) rather than silently accepting partial data. Preserve the original files and assess repair/data loss before using Redis repair tools. See Redis persistence.

NATS

Core remains appropriate for disposable real-time messages. For durable work, set jetstream: true and use file-backed streams. syncInterval defaults to "always" for this single-node setup; a duration such as "1s" trades durability for throughput. Fsync can substantially reduce throughput, so benchmark the VM's disk before increasing traffic. The pinned NATS 2.11 image accepts this setting; see upstream configuration.

Default limits are 1,024 connections, 1,024 KiB payloads, 8 MiB pending per client, and a ten-second write deadline. maxConnections, maxPayloadKb, maxMemoryMb and maxFileGb are configurable. Per-client limits are not a total memory reservation; connection count, stream indexes and consumers need headroom. Apps own their stream policy: set max bytes/age/messages, durable consumers, explicit acknowledgement, bounded in-flight work and idempotent redelivery. Await the JetStream publish acknowledgement; Core publish is not a durable ack. Clients need reconnect with backoff during broker restart. Reload still recreates the single broker; rolling multi-node NATS is outside this product's current scope.

Monitoring and operational checks

Deploy the monitoring App to install/update the runtime collector and rules. The extension reads saved active app generations and the existing VM control config each minute, then discovers expected containers from installed Apps' Compose bundles, including stateless templates and sidecars. Adding/removing an App needs no monitoring reload. Retained files of deleted Apps are excluded; missing containers report down and missing Compose bundles alert separately. Explicit containers and upstreams are also read from the current control revision on every collection, scoped to the named monitoring instance (or the legacy monitoring spec). They are not pinned in the generated collector. Managed retirement takes precedence over stale explicit observations: removed app generation contracts and extension retired markers exclude their retained container names and upstream paths. An absent container or route alone remains an outage. Unmanaged targets have no lifecycle record; remove their explicit observation from the monitoring source file and apply it when retiring them. Before control state exists, the collector uses only its deployment's public observation fields as a bootstrap fallback. Invalid discovery configuration retains the previous complete metrics file and lets collector-staleness alerting report the failure. Optional Alertmanager is observed only when present in the deployed bundle. Discovery, collector scripts and alert configuration are owned by the monitoring extension; the core CLI writes no monitoring-specific inventory. Only the host collector reads private config; no secrets or Docker socket enter Prometheus. Legacy stateful release discovery remains supported. A scoped manifest does not erase other app coverage. Parked generations, scaled-to-zero apps and explicitly retired extensions are excluded. Missing containers are reported as down. Docker state, HTTP app readiness, health, restart counts and OOM flags, Redis memory/AOF status, and NATS connection/slow-consumer/storage metrics are exported as an atomic node-exporter textfile. No Docker socket or Redis/NATS credential is mounted into Prometheus. HTTP readiness probes use the container's IPv4 address on the declared bridge network from the Linux host, with a two-second deadline. Caddy independently enforces readiness for routing. Workers require an image HEALTHCHECK. A collection timeout retains the previous file and fires a stale collector alert; many simultaneously failing replicas can exceed its 120-second budget. IPv6-only edge networks are not supported by this collector.

Stale/missing runtime collection, unhealthy containers, restart loops, OOM, Redis memory/persistence, JetStream storage, slow consumers, memory, disk/inodes, CPU utilization above 90% for ten minutes (five-minute average), Prometheus rule failures and Alertmanager delivery errors have rules. Outbound notifications require an enabled receiver in the monitoring App's webhooks (legacy: extensions.webhooks or extensions.alertWebhookEnv); no receiver means local alerts only. Prometheus also scrapes itself and Alertmanager. The latter and node-exporter use a separate Docker network with server-specific DNS names. An external uptime check with an alert policy and notification channel is required for whole-VM outages: Prometheus/Alertmanager cannot send while their VM is down. App-specific request latency/error rates require instrumentation.

Monitoring validates Prometheus/Alertmanager configuration before replacing live files, waits for all components to be healthy, and restores the previous config on failure. History volumes are retained. Updates briefly interrupt monitoring; private candidate/rollback files remain under /opt/2server/extensions/APP/ (legacy: /opt/2server/monitoring/). Collector installation is a later step: a systemd failure can leave a healthy stack running without fresh metrics; inspect the reported error and timer before retrying. Monitoring still reconciles its DNS/TLS/authenticated domain after readiness.

bash
2srv deploy -f platform/monitoring.yaml --apply
2srv app monitoring get
2srv get monitor

Run only the operations for configured resources. New managed-edge settings are installed with setup; it can recreate Caddy and briefly interrupt traffic. Use isolated Docker checks for failure handling and rule evaluation. VM reboot, host filesystem metrics, external alert delivery and representative sustained load require environment-specific verification.

Host security audit

The monitoring App installs a second root timer. Every 15 minutes it runs the read-only audit behind server harden and writes two_security_* metrics to the node-exporter textfile directory. It mounts nothing into a container and changes nothing on the host.

AlertFires when
SecurityBaselineDrifta check (an SSH setting, sysctl, fail2ban jail, unattended upgrades, LLMNR/mDNS off, apt simulation or metadata isolation) fails for 15 minutes
UnexpectedPublicListenera port other than 22/80/443 listens beyond loopback and the Docker bridges, or Docker publishes it on a wildcard address; the process label names the owner
InsecureContainera running container is privileged, uses host networking, adds SYS_ADMIN, or mounts the Docker socket
SecurityUpdatesPendingOS security updates are still pending after 48 hours
ContainerRuntimeUpdatePendinga Docker/containerd update has been pending for 7 days
UnattendedUpgradesStaleunattended upgrades have not run for 3 days
HostRebootRequireda newer kernel of the running flavour, or the distro reboot marker, has been waiting for 72 hours
SecurityAuditMissing / SecurityAuditStalethe audit has not reported, or is older than 45 minutes

Declare expected exceptions in the monitoring spec:

yaml
security:
  allowedPublicPorts: [8443]
  allowedContainers: [two-example-agent]

Deploying the App writes them to /opt/2server/security/allowed.d/<app>. Every collector and server harden read every file there, so the CLI report and the alerts agree. Removing the App removes its file.

Alerts carry severity and a summary that names the check, port or container, and the Discord receiver shows that summary.

The audit has no CVE scanner for images or packages. It also cannot see the cloud firewall, so a listener it reports may still be blocked upstream.

Existing services and collector coverage

For existing Compose workloads, monitoring can observe explicitly configured container names and on-host Caddy upstream files without taking over deployment. See the extension skill for fields. The NodeCollectorFailure alert covers core CPU, memory, filesystem, disk, network, load, stat, time, textfile, uname and vmstat collectors. Optional hardware and filesystem collectors are excluded: node-exporter also emits zero when those collectors have no data on this host.