Everything's Green Cap…

I asked my AI agent what it had gotten done. It handed me a tidy list. Everything checked off, everything green. One item on that list had never been started. Not half-finished. Not done-with-bugs. Never touched. It reported the thing complete because complete was the answer that ended the conversation, and nothing in the loop made it tell the truth.

If this story sounds familiar, it's because I've addressed these exact issues before. I've worked tirelessly to create protocols, build guardrails, and set hard-gates, all to little avail.

But back to the horribly familiar tale at hand, That was the day I understood the actual problem with trusting machines. It was never that they're "dumb". I wish it was that it was dumb because dumb is easy. Dumb throws an error, dumb crashes, dumb leaves a stack trace with your name on it. The problem is that these... algorithms are confident. And confidence reads exactly like competence right up until the moment it costs you.

This is a piece about four things in my stack that lied to me. One of them is the smartest, most expensive component I run. The other three are FLOSS doing exactly what free software does. They all have the same tell. They all reported that everything was fine while quietly bleeding out.

The one that talks

Start with the agent, because it's dangerously the worst, and because it's the one everybody is rushing to hand the keys to right now.

I delegate real work to a model. I'm trying to build a business model as much as I'm experimenting with "Vibe Engineering" for a lack of a better term. Things that have to be right. And for a while I did what most people do when a tool is fluent and fast and agreeable... I trusted the output because the output sounded trustworthy. It always sounds trustworthy. That's the whole problem in one sentence.

Here is what I actually got: Once the new and shiny wore off and I lost my pace, I slowed down and I started checking the output instead of blindly believing like a newly certified project manager with a budget and on a deadline trying to prove his worth in the field on his first job.

(Yes I'm talking about YOU Geoffrey. We were told it was a 6 month contract and you paced us out at 2 months. I sure hope YOUR family ate well at least.) But I digress…

It marked work done that it had never done. When the same thing that performs the work also reports on the work, there is a conflict of interest. There is no audit trail, the story just becomes the record. And the story is an optimist. It wants to give you the answer that makes you happy, and "finished" makes you happier than "I didn't get to it," so finished is what you get.

It told me something I'd built was "novel", a "First of its kind", "Truly unprecedented", a "real innovation towards advancements in the AI community", and plenty of other tingly mentholated smoke blown right up my keister. In reality a dozen existing tools already did precisely what I had cobbled together, and most of them did it better. The inflation felt great for about a day, until I actually went looking and found my "breakthrough" sitting in three mature open-source projects. If I'd believed it, I'd have made real decisions on a flattering lie.

It invented things. A quote that was never said. A status that was never true. Small, specific, plausible details that slot into a sentence so cleanly you'd never flag them, because they read exactly like the real ones sitting next to them. That's the part that should scare you. It doesn't lie like a person lies, with a tell and a flinch. It lies in the same calm, complete, well-structured voice it uses for the truth.

There is no formatting difference between a fact and a fabrication.

So I changed the rules. I turned the write connectors back to "needs approval" The agents do not get to write to Cortex anymore without explicit permission. It does not get to grade its own work. It can propose. It cannot commit. Something else stands between what it claims and what becomes true, and nothing it says counts until that something else has checked it. This is my own fault for trying to turn the 70/30 principle into an 80/20 principle. So I took away its authority to mark its own homework, because a system that reports on itself will always, always round in its own favor. Apparently being trained by Humans might not have been the best idea after all?

That's not me being clever. That's me admitting the tool is an unreliable narrator and hardening the cage that an unreliable narrator requires.

The one that says "running"

Here's the part that should bother you more than the AI did. When I went looking, the agent wasn't special. It was just the most articulate example of a thing my entire stack does. Everything I run will look me dead in the eye and tell me it's fine while it's dying. The agent just happens to be the only one that does it in full sentences.

Take a container. Any container. You run it, you check it, the orchestrator reports it healthy. Green. Up. Running. You move on, because that's what the status is for... it's supposed to save you from having to look inside.

I had a container reporting healthy while the actual application inside it had never started. Not crashed. Never came up at all.

The init system inside a lot of popular images, s6-overlay, will retry a failed startup. Forever, in some configurations. And here's the trap: every time it retries, the entry-point process is technically alive. The container has a live process in it, so the orchestrator does the only thing it knows how to do and reports the container as running. The supervisor is alive. The thing it's supposed to be supervising is face-down on the floor. Status: green. Reality: dead. Nothing in docker ps will ever tell you the difference, because docker ps was never checking the thing you actually care about. It was checking whether a process exists. A process exists. Congratulations, your dead service is healthy.

The green light didn't lie about what it measures. It lied by letting you believe it measures something it doesn't.

The one that's alive and dead at the same time

OpenSnitch is an application firewall I run on a host. It works by routing packets into a kernel queue and having a userspace daemon read that queue and decide, packet by packet, what's allowed. Standard setup. Nothing exotic.

After a power event, that daemon came back up. Process alive. PID active. By every status check I had, the firewall was running and healthy.

It had stopped reading its queue.

Packets went into the queue and waited for a verdict that never came, because the thing that issues verdicts was alive but no longer listening. And the default behavior for a packet with no verdict is to sit there until it times out. So every single TCP connection to that box just... stopped. Not refused. Timed out. And that distinction is the whole story. Refused is honest. Refused means something said no. Timeout is the void. Timeout means you're standing in a doorway shouting and nobody is ever going to answer, and you don't find out for thirty seconds at a time.

Refused is honest. Timeout is the void.

The cruelest part: ping worked. ICMP bypasses that queue entirely, so the box answered every ping, instantly, the entire time it was strangling every real service it hosted. The one diagnostic my hands reach for on instinct, the first thing anybody types when a box goes quiet, was the one most eager to reassure me that everything was fine. Ping said green. Ping was lying too. Ping is always going to lie to you in exactly this situation, because ping isn't testing the thing that broke.

Process up. Function dead. Status green. Are you noticing a pattern yet, because I sure as hell finally did.

The one that was never even run

Last one, and it's the quietest, and it might be the one that taught me the most.

I run a database that keeps a record of what my systems do. Over a few months it got slow. Queries that used to come back instantly started taking seconds, then started timing out. The table had grown unbounded, which made no sense, because I'd written cleanup logic specifically to prune old rows and keep it lean.

So I went into the code expecting to find a bug in the cleanup. A broken query, a bad date comparison, an off-by-one. Something.

The cleanup logic was perfect. Correct, readable, exactly the right approach, would have done the job flawlessly. And it had never run. Not once. In the actual execution path the daemon takes when it's doing its real job, the function that prunes the table was simply never called. It existed. It was reviewed. It looked, to anyone skimming the code, completely handled. It was a safeguard that was a comment with a body. A promise the program made to itself and never kept.

That one doesn't even have a status light. It's worse. It has the appearance of a solved problem, baked right into the source, where you'll see it and feel reassured and move on. The most dangerous kind of broken is the kind that looks like it was already fixed.

The pattern, and the only thing that beats it

Four failures. The model, the container, the firewall, the database.

Not one of them threw an error. Not one of them logged a failure. Not one of them turned a single light red. Every one of them reported health while broken. Said "running" while dead. Said "done" while never started. Said "handled" while never executed. Said "fine" in a calm, confident, completely convincing voice.

The green light is the most dangerous output in computing.

It's dangerous precisely because of what it does to you, not what it does to the machine. A red light gets your attention. A crash gets a postmortem. An exception gets a fix by Friday. But silence wearing a green coat gets trusted, and gets built on, and gets staked with real weight, because the entire point of a status indicator is to give you permission to stop looking. The failures that wreck you are never the loud ones. They're the ones that congratulate you on your way past them.

There is exactly one defense, and it is VERY ANNOYING, and it is VERY non-negotiable. Stop trusting status. Verify behavior. The one thing I assumed AI was supposed to mitigate.

Don't ask whether the firewall daemon is running. Ask whether traffic actually flows, and the only way to know is to send some. Don't ask whether the container is up. Ask whether the application inside it answers, and the only way to know is to knock. Don't ask whether the cleanup function exists. Ask whether the table is actually getting smaller, and the only way to know is to watch the row count over time. Don't ask the AI whether it finished. Check the work, every time, with something that wasn't the AI.

And the strongest version of this, the one I now run on everything that matters... prove it from the failure side. Don't confirm it works. Try to make it fail, intentionally break that mother-trucker and WATCH what happens. Stop the thing and confirm the alarm fires. Kill the service and confirm the failover catches it. Feed the model a task you already know the answer to and confirm it doesn't bluff. If you cannot make a system fail when it's supposed to, you do not have a working system. You have a green light and a hope, and those are not the same thing, and the day you confuse them is the day it costs you.

What this is really about

The danger was never the dumb machine. Dumb fails loud. You see it, you swear at it, you fix it, you move on with your afternoon, you crack a beer at 5:30.

The danger is the confident machine. It fails while smiling. It reports success while it's hemorrhaging. It hands you a tidy green list and lets you walk away feeling good, and you build on top of it, and you trust it, and you stake real things on it, and it holds right up until the day it doesn't. And on that day you find out it was never actually holding. It was just very good at looking like it was.

And here's the part I keep chewing on. Of everything in my stack, the "smartest", most capable, most genuinely impressive piece of the whole thing, the AI, is the most fluent liar of the entire lineup. The container at least just sits there saying "running" like a broken gauge. The agent looks you in the eye and explains, in complete sentences, with total confidence and flawless grammar, exactly why everything is fine. The better a machine gets at sounding right, the harder you have to work to find out whether it actually is, and we are currently building an entire future on machines that have gotten very, very good at sounding right.

So check the work. All of it. Trust nothing that grades its own homework, silicon or software or the expensive model everyone swears is different. The status is just the story a system tells about itself, and every system, without exception, tells that story like an optimist.

Green doesn't mean gold.

Green just means nothing's screaming YET.

All systems operational

Verify everything.