Project Battleship

Project Battleship
As you can see, my young apprentice, your friends have failed. Now witness the firepower of this fully ARMED and OPERATIONAL battle station!
Build Journal — Article 14

Project Battleship
How I Hardened 28 Containers in a Single Day
with AI and a Very Long Night

This morning I had 28 Docker containers with zero security hardening.
Tonight I have a warship.

2026-03-26  |  Chris Sholmire  |  The Paranoid~R.V.

Here's something nobody tells you when you're building a self-hosted security operations center from a fifth wheel RV: the stack will grow faster than your discipline. You'll spin up containers because the project demands it. Then more containers because the security tools demand it. Then more because the content platform demands it. And one morning you'll look at docker ps and realize you have 28 containers running with zero capability restrictions, zero resource limits, and zero image version pinning.

That was this morning.

By tonight, 22 of those containers were locked down with cap_drop: ALL, selective capability restoration, resource limits, process count caps, and digest-pinned images. The other 6 are either already hardened or intentionally left elevated because they need raw packet access to do their job.

This is how that happened.

// The Decision

Audit First. Always.

The temptation with infrastructure hardening is to start changing things. Open compose files, start dropping capabilities, see what breaks. That's how you take down DNS for your entire network at 2am and spend three hours figuring out which container needed which capability to start.

Instead, I made a decision that saved the entire project: audit first, harden second. Two separate sessions. The first one touches nothing — pure reconnaissance. Map every container's image digest, capability set, network attachment, resource consumption, and security posture. Then — and only then — start the surgery with a known baseline and a rollback path for every change.

I call this Project Battleship. The goal: turn a native village fishing boat into a battle-hardened warship. Not by guessing. By knowing exactly what you have before you start changing it.

// The Audit

What the Reconnaissance Found

# The tactical pre-flight audit revealed:

Containers with cap_drop: 0 of 28
Containers with no-new-privileges: 1 of 28
Containers with resource limits: 0 of 28
Images running unpinned :latest tags: 17 of 28
Containers with Docker socket access: 2 (both read-only)
UFW firewall exposures: 1 — OpenWebUI on all interfaces
Network topology drift: zero — clean

One container out of twenty-eight had no-new-privileges set. One. Everything else was running with full default Docker capabilities — which means every container could potentially escalate privileges, spawn child processes without limits, and consume as much memory and CPU as it wanted.

The firewall finding was worse. OpenWebUI — the chat interface for our local AI stack — was accessible on all network interfaces. Not just LAN. Not just the VPN. Everything. That got fixed first.

Seventeen containers were running :latest image tags with no version pinning. In March 2026, after supply chain attacks hitting Docker Hub directly, that's not just sloppy — it's a trust decision you're making every time the container restarts and pulls a fresh image you haven't verified.

Before — 0600 Hours
cap_drop ALL: 0
no-new-privileges: 1
Resource limits: 0
Images pinned: 11
Credential isolation: 0 stacks
UFW exposures: 1 critical
After — 0400 Hours
cap_drop ALL: 22
no-new-privileges: 15
Resource limits: 22
Images pinned: all digest-pinned
Credential isolation: 2 stacks migrated
UFW exposures: 0
// The Surgery

cap_drop: ALL and the Art of Breaking Everything

Docker's default capability set gives containers far more privilege than most services need. The hardening approach is simple in concept: drop every capability, then add back only what each service actually requires. In practice, this means things break — and the breaking teaches you exactly what each container needs to function.

The first stack went down in crash loops immediately. Postgres, Mosquitto, and InfluxDB all start as root and then drop to a service user during initialization. That setuid call requires the CHOWN, SETUID, and SETGID capabilities. Without them, the container starts, tries to switch users, fails, and restarts. Forever.

But here's the thing — Grafana, Node-RED, and Blinko all run with bare cap_drop: ALL and zero added capabilities. They start as their own user and never need privilege transitions. The pattern became clear:

# Container starts as root, drops to service user?
cap_add: [CHOWN, SETUID, SETGID]

# Container starts as its own user?
cap_drop: ALL # That's it. Nothing else needed.

# Container uses chroot sandboxing? (unbound)
cap_add: [SYS_CHROOT, MKNOD, SETUID, SETGID, CHOWN]

# Container reads mounted config through cap_drop?
cap_add: [DAC_READ_SEARCH]

That pattern — deploy with bare cap_drop: ALL, check logs for permission errors, add back minimum required capabilities — became the methodology for all seven compose files. Every crash loop was a lesson. Every lesson was documented.

Six containers can't use no-new-privileges at all — they rely on setuid transitions during startup. Ghost uses gosu for root-to-node switching. Unbound uses chroot sandboxing. CrowdSec needs privilege transitions for its SQLite database. These aren't gaps — they're informed exceptions with documented reasoning.

22
Containers Hardened
7
Compose Files
5
Sessions
0
Downtime
// The Neural Pipeline

When the Fix Reveals What Was Never Broken

The last session of the night was the most satisfying. For days, our MCP connection — the pipeline between Claude AI instances and the CORTEX brain on the stack — had been dropping intermittently. Multiple sessions diagnosed it as client-side session corruption. "Just open a new window" became the standard fix.

That diagnosis was wrong.

When the compose surgery recreated the MCP server container, Docker assigned it a new IP address. The Cloudflare tunnel routing that container's traffic was still pointing at the old IP. Every request from every AI station was hitting a dead address and returning a 502 before it ever touched the container.

"Open a new window" appeared to work because it reset the client state, and occasionally a container restart would land the server back on the same IP by chance. Confirmation bias did the rest. Nobody traced the full request path until tonight.

# The diagnosis
cloudflared: dial tcp 172.20.0.4:7778 → connection refused
container IP: 172.20.0.2

# The fix — one field in the tunnel config
mcp.mpdc.dev: http://172.20.0.4:7778 → http://localhost:7778

# Standing order: all tunnel routes use localhost
# Container IPs drift. localhost doesn't.

One PUT to the Cloudflare API. One field change. A bug that persisted across multiple sessions — killed in a single command.

The lesson: when a workaround "usually works," the real fix hasn't been found yet. Trace the full path. Every hop. Every layer. The answer is always in the part of the chain nobody checked.

The AI does 70% — research, drafting, analysis, options generation. I provide the 30% that matters — decisions, judgment, and the willingness to say "we're not hardened enough." The machine watches. The human decides. — The 70/30 Principle
// The Day

Five Sessions. One Day.

Session 1 — Bridge
Project Battleship Launched
Gap analysis across the entire stack. Identified the hardening deficit. Built the audit methodology and work orders for every subsequent session.
Session 2 — Engineering
Tactical Pre-Flight Audit
Ten audits against the live stack. Version inventory, capability posture, network topology, resource baselines, socket exposure, firewall rules. Zero changes. Pure reconnaissance.
Session 3 — Engineering
First Compose Hardened
Application stack — 6 containers locked down. Discovered the cap_add patterns through controlled crash loops. Credentials migrated from hardcoded compose to environment files.
Session 4 — Engineering
Remaining Six Stacks
Security stack (15 containers — the big one), Ghost, Traefik, OpenWebUI, LiteLLM, voice services. Every stack verified post-surgery. 28 containers running. Zero restarting.
Session 5 — Engineering
Neural Pipeline Restored
Traced the MCP tunnel failure to its real root cause. Fixed a bug that had been misdiagnosed across multiple sessions. Every AI station now has live eyes on the stack.
// The Methodology

What This Looks Like as a Solo Operator

I'm one person in a 40-foot fifth wheel with a Dell T3600, a cellular connection, and an AI crew that follows a governance protocol I wrote. The AI doesn't make decisions. It doesn't ship changes. It doesn't approve its own work. Every infrastructure change goes through a chain of command — research, present options, wait for explicit approval, execute with logging.

The audit-first methodology isn't just about being careful. It's about the AI's fundamental limitation: every session starts fresh. A new instance with the protocol loaded but zero memory of what the last instance discovered. If the first session makes changes and the second session needs to debug those changes, the second session is working blind.

But if the first session only gathers data and stores it in the persistent brain, the second session walks in already knowing the territory. That's the difference between a junior engineer guessing at 2am and a senior engineer working from a documented baseline.

This is what AI governance actually looks like. Not a whitepaper. Not a framework diagram. A real stack, real containers, real hardening, with a human making every call and an AI doing the legwork.

// What's Next

The Ship Is Hardened. Now It Fights.

Compose surgery is complete. The next phase is detection infrastructure — container runtime monitoring, vulnerability scanning across all images, and a single-pane security dashboard that visualizes everything the stack is already collecting. The data is flowing. The visualization layer comes next.

And in about two weeks, I'm walking into BSidesOK with this story.

Not a vendor pitch. Not a product demo. A war story from a solo operator who built a 28-container security operations center from a fifth wheel RV, hardened it in a day with AI assistance, and documented the whole thing in real time so nobody else has to learn it the hard way.

That's the whole point of The Paranoid~R.V. — build it, break it, fix it, write it down.

Get the build journal.

Behind-the-build documentation. No spam. No selling out.
Your data never leaves my stack.

Subscribe

The Paranoid~R.V.  —  40ft of Infrastructure. Zero Fixed Addresses.