Project Battleship
Project Battleship
How I Hardened 28 Containers in a Single Day
with AI and a Very Long Night
This morning I had 28 Docker containers with zero security hardening.
Tonight I have a warship.
Here's something nobody tells you when you're building a self-hosted security operations center from a fifth wheel RV: the stack will grow faster than your discipline. You'll spin up containers because the project demands it. Then more containers because the security tools demand it. Then more because the content platform demands it. And one morning you'll look at docker ps and realize you have 28 containers running with zero capability restrictions, zero resource limits, and zero image version pinning.
That was this morning.
By tonight, 22 of those containers were locked down with cap_drop: ALL, selective capability restoration, resource limits, process count caps, and digest-pinned images. The other 6 are either already hardened or intentionally left elevated because they need raw packet access to do their job.
This is how that happened.
Audit First. Always.
The temptation with infrastructure hardening is to start changing things. Open compose files, start dropping capabilities, see what breaks. That's how you take down DNS for your entire network at 2am and spend three hours figuring out which container needed which capability to start.
Instead, I made a decision that saved the entire project: audit first, harden second. Two separate sessions. The first one touches nothing — pure reconnaissance. Map every container's image digest, capability set, network attachment, resource consumption, and security posture. Then — and only then — start the surgery with a known baseline and a rollback path for every change.
I call this Project Battleship. The goal: turn a native village fishing boat into a battle-hardened warship. Not by guessing. By knowing exactly what you have before you start changing it.
What the Reconnaissance Found
Containers with cap_drop: 0 of 28
Containers with no-new-privileges: 1 of 28
Containers with resource limits: 0 of 28
Images running unpinned :latest tags: 17 of 28
Containers with Docker socket access: 2 (both read-only)
UFW firewall exposures: 1 — OpenWebUI on all interfaces
Network topology drift: zero — clean
One container out of twenty-eight had no-new-privileges set. One. Everything else was running with full default Docker capabilities — which means every container could potentially escalate privileges, spawn child processes without limits, and consume as much memory and CPU as it wanted.
The firewall finding was worse. OpenWebUI — the chat interface for our local AI stack — was accessible on all network interfaces. Not just LAN. Not just the VPN. Everything. That got fixed first.
Seventeen containers were running :latest image tags with no version pinning. In March 2026, after supply chain attacks hitting Docker Hub directly, that's not just sloppy — it's a trust decision you're making every time the container restarts and pulls a fresh image you haven't verified.
cap_drop: ALL and the Art of Breaking Everything
Docker's default capability set gives containers far more privilege than most services need. The hardening approach is simple in concept: drop every capability, then add back only what each service actually requires. In practice, this means things break — and the breaking teaches you exactly what each container needs to function.
The first stack went down in crash loops immediately. Postgres, Mosquitto, and InfluxDB all start as root and then drop to a service user during initialization. That setuid call requires the CHOWN, SETUID, and SETGID capabilities. Without them, the container starts, tries to switch users, fails, and restarts. Forever.
But here's the thing — Grafana, Node-RED, and Blinko all run with bare cap_drop: ALL and zero added capabilities. They start as their own user and never need privilege transitions. The pattern became clear:
cap_add: [CHOWN, SETUID, SETGID]
# Container starts as its own user?
cap_drop: ALL # That's it. Nothing else needed.
# Container uses chroot sandboxing? (unbound)
cap_add: [SYS_CHROOT, MKNOD, SETUID, SETGID, CHOWN]
# Container reads mounted config through cap_drop?
cap_add: [DAC_READ_SEARCH]
That pattern — deploy with bare cap_drop: ALL, check logs for permission errors, add back minimum required capabilities — became the methodology for all seven compose files. Every crash loop was a lesson. Every lesson was documented.
Six containers can't use no-new-privileges at all — they rely on setuid transitions during startup. Ghost uses gosu for root-to-node switching. Unbound uses chroot sandboxing. CrowdSec needs privilege transitions for its SQLite database. These aren't gaps — they're informed exceptions with documented reasoning.
When the Fix Reveals What Was Never Broken
The last session of the night was the most satisfying. For days, our MCP connection — the pipeline between Claude AI instances and the CORTEX brain on the stack — had been dropping intermittently. Multiple sessions diagnosed it as client-side session corruption. "Just open a new window" became the standard fix.
That diagnosis was wrong.
When the compose surgery recreated the MCP server container, Docker assigned it a new IP address. The Cloudflare tunnel routing that container's traffic was still pointing at the old IP. Every request from every AI station was hitting a dead address and returning a 502 before it ever touched the container.
"Open a new window" appeared to work because it reset the client state, and occasionally a container restart would land the server back on the same IP by chance. Confirmation bias did the rest. Nobody traced the full request path until tonight.
cloudflared: dial tcp 172.20.0.4:7778 → connection refused
container IP: 172.20.0.2
# The fix — one field in the tunnel config
mcp.mpdc.dev: http://172.20.0.4:7778 → http://localhost:7778
# Standing order: all tunnel routes use localhost
# Container IPs drift. localhost doesn't.
One PUT to the Cloudflare API. One field change. A bug that persisted across multiple sessions — killed in a single command.
The lesson: when a workaround "usually works," the real fix hasn't been found yet. Trace the full path. Every hop. Every layer. The answer is always in the part of the chain nobody checked.
Five Sessions. One Day.
What This Looks Like as a Solo Operator
I'm one person in a 40-foot fifth wheel with a Dell T3600, a cellular connection, and an AI crew that follows a governance protocol I wrote. The AI doesn't make decisions. It doesn't ship changes. It doesn't approve its own work. Every infrastructure change goes through a chain of command — research, present options, wait for explicit approval, execute with logging.
The audit-first methodology isn't just about being careful. It's about the AI's fundamental limitation: every session starts fresh. A new instance with the protocol loaded but zero memory of what the last instance discovered. If the first session makes changes and the second session needs to debug those changes, the second session is working blind.
But if the first session only gathers data and stores it in the persistent brain, the second session walks in already knowing the territory. That's the difference between a junior engineer guessing at 2am and a senior engineer working from a documented baseline.
This is what AI governance actually looks like. Not a whitepaper. Not a framework diagram. A real stack, real containers, real hardening, with a human making every call and an AI doing the legwork.
The Ship Is Hardened. Now It Fights.
Compose surgery is complete. The next phase is detection infrastructure — container runtime monitoring, vulnerability scanning across all images, and a single-pane security dashboard that visualizes everything the stack is already collecting. The data is flowing. The visualization layer comes next.
And in about two weeks, I'm walking into BSidesOK with this story.
Not a vendor pitch. Not a product demo. A war story from a solo operator who built a 28-container security operations center from a fifth wheel RV, hardened it in a day with AI assistance, and documented the whole thing in real time so nobody else has to learn it the hard way.
That's the whole point of The Paranoid~R.V. — build it, break it, fix it, write it down.