The Watchtower Had No Windows

The tool is What's Up Docker. WUD for short, 8.3.1 at the time all this went down. It's notify-only, which is the entire reason it made the cut. It watches the registries, compares what I'm running against what's out there, and lights up a little tile when something new exists. It doesn't pull anything and it doesn't restart anything. It taps me on the shoulder and then it waits, because nothing on this rig moves unless I move it. If you've been reading here a while, you already knew that part.

So it goes in, and after some cleanup the dashboard settles into this beautiful quiet wall of tiles. Everything accounted for, almost nothing asking for attention. And I wrote a whole piece a while back about green lights and what they're actually worth, so you'd think the alarm would have gone off in my head a lot sooner than it did.

Day zero

Let me back up to the noise first, because the quiet only makes sense after it.

Day one, WUD had no labels to work with. It ships assuming you'll tell it what matters, and I hadn't told it anything yet, so its suggestions were coming straight off the raw registry firehose. It offered my reverse proxy a Windows Server image, which is a bold thing to suggest to a Linux box living in an RV. It offered a database a beta build like we were old drinking buddies. It offered another container a build for an ARM processor I don't own. And my personal favorite, it suggested one container downgrade itself, because somewhere in the sorting a plain string compare decided 9 comes after 19. Anybody who ripped their CD collection back in the day and watched track 9 land after track 19 in the folder knows that trap. Some problems just follow us around forever.

None of that is a knock on the tool, and it isn't a bug report either. A registry carries every architecture, every base image, and every release candidate somebody pushed at 2am, and if you don't filter it you get all of it. The tool was doing exactly what I'd configured, which was nothing. So the first evening went to writing tag filters, a little regex leash on each service that says only show me things that look like version numbers:

wud.tag.include=^v?\d+\.\d+\.\d+$

Some flavor of that pattern went onto seventeen services in one stack alone, and while I was in there I pinned six containers that had been riding :latest to the versions they were actually running. The :latest tag is its own article someday and I'm not opening that can tonight.

Counting to fourteen

Here's the part I actually want you to steal.

Once the noise was gone and the dashboard looked calm, I counted. Fifty containers on the roster, so how many of them were producing an update signal... and I don't just mean how many were green, I mean how many were producing any signal whatsoever.

Thirty-six.

Fourteen containers, better than a quarter of the fleet, were saying nothing at all. And the design decision that turns that into a trap is that WUD stays quiet about anything it can't read. There's no red tile for "I couldn't check this one." An image it can't see looks exactly like an image that's perfectly current, and from the ground you cannot tell the difference. The watchtower had no windows on that side of the field.

An empty update list is not evidence that everything's current. Sometimes it's evidence that nobody's looking.

Three kinds of quiet

The fourteen broke into three buckets, and I'll take them in order of annoying.

One container was silent on purpose. wud.watch=false, my label, my reasons, and that's the only kind of silence I'm okay with... the kind I signed off on myself.

Six were on tags that don't look like version numbers at all, rolling tags and names instead of digits. WUD can handle those by watching the digest, meaning the actual content hash, and flagging when the tag quietly starts pointing at something new. The catch is that digest watching ships disabled. There's no warning banner about it, just a log line at startup that scrolls by once and never comes back, and after that the container simply never has news. The fix is one label:

wud.watch.digest=true

The last seven, which is fully half the blind spot, were images from one publisher hosted on a registry that wants a GitHub token before it'll even read you the manifest. And I don't mean a token to pull the image, I mean a token just to ask what versions exist. WUD would knock, get nothing back, log "No Registry Provider found" one time, and move on forever.

Now the ten-minute fix there is obvious. Mint a token, feed it to WUD, done. I didn't do it, and I want to walk through why, because the reasoning matters more than the fix. A token is a credential, and a credential is a thing you now have to store and rotate and worry about for the rest of its natural life. I've written plenty here about secrets and their care and feeding, and adding a brand new permanent one just to solve a read problem felt like buying a gun safe to store a library card. Turns out the same publisher mirrors the same images, same tags and everything, on GitHub's own container registry, where reading is free and anonymous. Seven lines changed in one compose file, the same images kept running from a hostname that actually answers questions, and there's no new secret for me to babysit for the next decade.

The update that waited

There's a fourth flavor of invisible I have to own separately, because this one was self-inflicted in the most responsible way possible.

Two containers on this stack were pinned by digest. Full sha256, immutable, the gold-star posture everybody tells you to have. And they were completely invisible to the watcher, structurally, always, because a bare digest has no tag and a watcher watches tags. There is nothing for it to compare. I nearly made it worse, too... those two were sitting on my remediation list marked for digest-watch labels, and that label would have applied clean, thrown zero errors, watched absolutely nothing, and the list would have read as complete. That one still makes my eye twitch a little.

What actually fixed it was repointing them at the semver tag that the digest's own metadata named. The pull reused every layer in about a second and a half, which is Docker's way of telling you it's the same bits under a new name, so nothing about the running software changed. And the very first watch after the repoint lit up with a new version available. It had been available for a while. That update sat out there for the entire life of the pin and I had no way of knowing it existed.

So the rule got carved into the hull: digest pinning and update watching are mutually exclusive, per container, pick one. You get immutability or you get visibility, and any tooling that lets you write labels implying you can have both will happily let you feel covered while covering nothing.

The lists lied

By this point I had lists. A pinning list, a label list, a digest list. Lists feel like progress, right, you cross off the last item and the project's done.

I ran the count against the roster one more time anyway, docker ps -a on one side and containers actually reporting on the other, and found one more carrying no labels of any kind. It wasn't on any of my lists, because the lists had all been built from each other, and a container that never made the first list is invisible to every list descended from it. This is the second time that exact kind of gap has bitten this stack, so it's a permanent rule now: audit against the roster, never against your own paperwork. Your lists only contain what you remembered to put on them.

One more from the trenches while I'm here, because it'll save somebody a bad hour someday. Depending on how the labels got quoted in a compose file, grepping for - wud. will sail right past half of them and cheerfully report a coverage gap that doesn't exist. Ask the tool what it can actually see, and trust that over your grep.

The key under the mat

Last one, and it's the one that named this article.

WUD needs to talk to Docker to know what's running, which means mounting the Docker socket into the container, and if you know anything about Docker you know that socket is the whole f*cking ballgame. Write access to it is root on the host wearing Groucho glasses. My own hardening doc caps socket consumers on this stack at two, read-only. I wrote that doc and I meant it. And the week WUD went in, the count stood at four consumers with at least one of them holding read-write.

The read-write one was WUD, because that's how the example compose mounts it, that's how half the internet mounts it, and in the blur of that week it went in the way the quick-start said to. Three days later my own audit caught it, and the fix was one edit, :ro on the end of the mount line, followed by the test that actually mattered: a full manual watch across all fifty containers on the read-only socket. Container registration, event listening, registry queries, the whole UI came through with zero permission errors. Notify-only never needed write access in the first place. The tool I stood up to watch everything spent three days holding a key to the building, for a job that never once required opening the door.

Nobody used the key, and before you say it, I know that's not the point. The standing rule on this rig is that privilege gets justified by function, and "it came that way" has never been a function. For the record, one read-write consumer does stay on that socket, and it's the stack manager, because starting and stopping and recreating containers is its entire job and you can't do that through a read-only mount. That's what a justified key looks like... there's a reason, the reason is written down, and I can say it out loud without wincing.

The tool I stood up to watch everything spent three days holding a key it never needed.

What the quiet was hiding

So what do you take home from all this, besides a pile of regex and a registry swap you're welcome to steal?

For me it's how every single one of these failed. The unreadable registry, the digest watching that ships off, the container with no labels, the socket mount... not one of them made a sound, and not one of them was malfunctioning. Every piece was working exactly as configured, which is why there was no red light anywhere. The dashboard wasn't lying to me. It just wasn't saying anything, and I'd been sitting there hearing "all clear."

The only way I know to catch that is to count. Roster in one column, containers actually producing a signal in the other, then go chase down everything that isn't in both. Took me ten minutes once I finally did it, and it should have happened the day the tool went in. Learn from my August.

Two caveats before I let you go, one for the checklist and one for the tinfoil.

The checklist one: WUD has an auto-update mode where it recreates containers itself when new versions land. The moment you flip that switch it needs the socket write access back, and everything I just said about the read-only mount quietly un-happens. If you flip it, re-run your audit. Better yet, don't flip it. The entire reason this setup works on a read-only socket is that a human fires every update, and on this rig that human is me.

The tinfoil one, because the thought genuinely crossed my mind while writing this: doesn't naming your tooling in public hand somebody a map? Every tool I mention is one more upstream that could get poisoned, one more tag somebody could shove a surprise behind. Fair thought, and I'm not going to pretend supply chains don't get poisoned, because that's just the world now. But everything in this piece lives behind the wall and none of it answers to the internet, and more to the point, the watcher can't touch anything. It sees and it suggests. Knowing what I watch tells you nothing about what runs here, because nothing runs until I read the suggestion, decide it smells right, and fire it with my own two hands.

Same as it ever was. DiY or DiE, and that includes not handing the keys to the robot you built to hold the flashlight.

Cheers,
Chris