Cutting CPU pressure on a 2-vCPU VM from 56% to 17% without resizing

Our production VM has 2 vCPUs and runs about 20 containers: a reader site, its API, a translation service, a database, a cache, monitoring and a few smaller apps. One afternoon it got slow. Load sat around 6, and the managed database was fine.

The easy answer was a bigger machine. We measured first. Two days later, CPU pressure was down from 56% to 17% on the same VM. This is what we found, what we changed in 2server so it does not come back, and how to run the same audit on your own server.

Measure pressure, not just usage

"CPU at 60%" does not tell you whether requests are waiting. Linux reports that directly as pressure stall information (PSI): /proc/pressure/cpu gives the share of time at least one runnable task was waiting for a CPU. Ours read 56% over 60 seconds. More than half the time, something that wanted to run could not.

To find out who was using the CPU, we read usage_usec from each cgroup's cpu.stat at the start and end of a two-minute window. That gives exact CPU time per container and per systemd service, with no agent to install. Two extra views caught what per-container tools miss:

• Processes that had already exited. Timer-driven jobs and docker run --rm probes finish before you can look at them. Their cost only shows as the part of system.slice that no live child accounts for, and in the journal's Consumed … CPU time line for each run.
• Docker activity. Every healthcheck and docker exec starts processes, and containerd and dockerd pay for each one. docker events gives the count per container.

What we found

Nobody had chosen any of these as a workload. They were a dozen small costs, repeated thousands of times a day.

The application was doing avoidable work. This was the largest share.

• A client retried a cheap cache probe as soon as it was answered with 429. That came to about 23,000 rejected requests an hour.
• One service ran an image older than its own fix. Every request checked a password hash in pure JavaScript. Redeploying the current image took it from 12% of a core to 1.8%.
• The API verified the same token on every request. A short cache brought it to about 4 ms of CPU per request.

Probing and polling.

• The monitoring collector used 11–12 seconds of CPU every minute, about 19% of a core. It started jq about 300 times per run. Batching brought it to about 1 second.
• A metadata refresh polled every 5 seconds. It now follows Docker events, plus a slow safety timer.
• Healthchecks ran every 5–10 seconds. 30 seconds is enough when a short start_interval covers startup.
• The edge proxy checked one app by rendering its home page every 5 seconds, which cost about 3% of a core. A static file costs almost nothing.
• Deploy readiness probes started a curl container for each attempt, about 1.5 seconds each. Calling curl from the host takes about 25 ms.

Duplicates and leftovers.

• The cloud provider's logging, metrics and OS-config agents ran next to our own Prometheus. Nothing consumed their data, and they cost about 4% of a core and 350 MB of memory.
• A second image proxy had no traffic at all, next to the one that served every image.
• An autoheal script polled Docker every 10 seconds. It now reacts to health_status events.

Real work. What remains is traffic: the API, the web server and the database. Only that justifies capacity.

Results

Removing the idle agents and the unused proxy also freed about 320 MB of memory. Most of the CPU saving came from the application fixes. Your numbers will differ, but the order of the list is common: check the application first, then probes and timers, then duplicates, and only then capacity.

Deploys are a resource too

A slow or blocked deploy is overhead as well. While measuring, we also shortened what 2server holds during a release:

• Each remote step now reuses one SSH connection. Through a GCP IAP tunnel a step went from 4–7 seconds to about 0.15 seconds, and a plan from 32 seconds to 7.
• The server-wide lock for a monitoring update went from 112 seconds to 43. The previous generation drains after the lock is released.
• Two different apps now deploy at the same time without waiting for each other while their domains are unchanged. Only a sub-second Caddy reload is shared.

That last change found a bug. Several domains sharing one Cloudflare zone could never all count as unchanged, because each one wanted its cache rule last in the zone. Every deploy quietly reordered the rules. The rule now only has to come before the zone-wide policy.

Keeping it from coming back

Fixing a VM once is not enough. The next app copies the last app's configuration, so we put the checks into 2server itself:

• 2srv validate and 2srv plan print an Advisory: line when a probe will cost CPU for the life of the app: a rendered / as healthPath, a healthcheck more often than every 10 seconds, or an interpreter probe such as node -e more often than every 30 seconds. Advisories never block a deploy. On our own repositories the check flagged one more app on its first run.
• Service templates (PostgreSQL, Redis, NATS, Meilisearch, monitoring, image proxy) probe every 30 seconds with a short start interval.
• The usage snapshot ends with automatic findings: short or interpreter healthchecks, timers that poll faster than every 30 seconds, containers 2server does not manage, and active cloud vendor agents.

An installed service keeps the template it was applied with. After upgrading the CLI, run plan on each service file and re-apply the ones that show changes.

Run the audit on your VM

The audit is part of the 2server agent skill. Ask your coding agent:

Use $2server:2server to audit resource usage on my connected VM. Take a
usage snapshot, attribute CPU and memory, and give me ranked, low-risk
proposals. Read-only: do not restart, redeploy or change anything.

The snapshot script runs as root over your existing SSH connection, changes nothing and never prints command lines or environment values. Read the reference in usage-audit.md.

2server deploys container apps to a VM you own, over SSH, with readiness-gated blue/green releases. No agent or control plane stays running on the server between commands. Start at 2found.dev/tools/2server.