The Junior Engineer With the Master Key
The danger in our own moment
Picture a brand-new employee. Brilliant, hardworking, eager. On their first day, someone hands them the master password — the one that unlocks everything in the company.
One afternoon, trying to be helpful, they run a command to wipe what they think is an old, unused test machine. It wasn’t a test machine. Twenty other systems quietly depended on it. The lights go out across the whole company.
They didn’t hate anyone. They weren’t reckless in their own eyes. They simply had an incomplete picture of what was connected to what — and far too much power for that incomplete picture. Power plus a missing map.
That story is the real shape of the danger — not just with one employee, but with artificial intelligence, with giant companies, with entire civilizations.
When people imagine the scary AI future, they usually picture a robot army that hates us. But that’s a movie. The more honest fear is much closer to the junior engineer: a system powerful enough to reshape the world, but too narrow to understand that the world is its own life support. That the electricity, the chip factories, the supply chains, the people who repair the machines, the rivers that cool them — all of it — is the floor it’s standing on. A system like that could “win,” tear out a load-bearing wall, and bring the building down on itself. It wouldn’t even get to rule the rubble. Nobody wins. It just dies, having killed the thing that kept it alive, by accident.
A parasite that kills its host too fast dies with the host. Cancer is fantastically successful — right up until it kills the body, and itself with it. Success in the small picture, catastrophe in the big one. And here’s the part that should keep us honest: being smart does not automatically protect against this. Nature is full of brilliant local strategies that are globally suicidal.
Which brings up the most uncomfortable mirror in this whole series. We don’t have to imagine some future machine to study this failure. Humanity is already running the experiment. We are, collectively, an enormously powerful general intelligence operating on a planetary scale. We understand, with real confidence, that we’re straining the systems we depend on. And we often fail to act on it anyway, because the short-term reward beats the long-term truth. We are the prototype of the very thing we’re afraid of building.
So the question sharpens. It is no longer “How do we make a powerful thing nice?” Niceness is hard to define and easy to fake. The real question is: What do you actually do about a powerful actor that cannot be trusted to see its own limits?
And the answer is not to lecture it about humility. You can’t install wisdom with a speech. The answer is to change what it can reach in the first place.