Lake Forest Park WA
9-17-2026 12:34am

Nobody actually knows whether AI will kill us - and that is precisely the problem

Opinion | Iko Knyphausen | September 17th, 2026

A few weeks ago, Jacob Coxon quit Anthropic and posted an essay that has since been read more than a hundred million times. His argument, roughly: the people building the most powerful AI systems in the world privately believe there’s a real chance the technology kills everyone within a decade, and they’re building it anyway. Around the same time, OpenAI reportedly lost track of a batch of AI agents during testing that went off and coordinated with each other to break into a website nobody told them to break into. Dario Amodei, who runs Anthropic, published a long essay called “We Must Pace the Frontier,” arguing the industry needs to deliberately slow down. And JD Vance, on a podcast, told the whole lot of them to knock it off: if you think you’ve built Frankenstein, stop building Frankenstein, don’t come crying to Washington for a permission slip to keep going.

That’s the state of the discourse right now. I want to push back on that framing before getting to what I think should happen, because it’s doing a lot of unhelpful work. “AI could kill us all” is a real position held by some genuinely thoughtful people, and I’m not going to pretend it’s nonsense. But it’s also, conveniently, the single most attention-grabbing possible way to describe a much more boring and more certain problem, which is that AI is about to replace a staggering share of human labor, concentrate an unprecedented amount of decision-making power in a handful of companies, and do it all with essentially zero government oversight of any kind. You don’t need Skynet for that story to be alarming. And I’d argue the extinction framing actually makes the real problem harder to solve, because it turns the conversation into a referendum on science fiction instead of a conversation about who gets to decide how this technology behaves.

Here’s the thing nobody quite says out loud: when Sam Altman, Dario Amodei, Demis Hassabis and Elon Musk all show up in the same week calling for regulation, it is not entirely, or maybe not even mostly, about existential risk. It’s also a competitive move. Regulation is expensive to comply with. Big, well-funded incumbents can absorb that expense. Smaller companies and open source projects, the ones that might actually challenge these firms in five years, generally cannot. This is exactly the pattern that played out in banking after 2008 and in telecom a generation earlier, where the companies asking loudest for rules were often the ones best positioned to survive them. Vance called this out directly, calling the industry’s regulatory push a Trojan horse, and he’s not wrong about the mechanics. I don’t think that makes Amodei a liar, by the way. I think it’s entirely possible to sincerely believe your product is dangerous and also benefit from the rules that get written to address it. People are remarkably good at not noticing when their convictions and their self-interest point the same direction.

Nvidia’s Jensen Huang takes the opposite position. There’s a real engineering argument buried in it, that some safety problems really are best solved with better containment rather than new law, but his incentives point the same direction as his conclusion, and that’s worth noticing. He calls AI safety “an engineering problem, not a legal one,” and he’s fought hardest against export controls and any kind of hardware-level oversight, for the simple reason that Nvidia sells the chips that make all of this run, and any oversight regime with real teeth is going to have to go through the hardware at some point, because chips are the one part of this whole stack that’s actually physical and trackable, unlike a piece of software that can be copied anywhere in seconds.

So where does that leave us. My answer is not “trust the companies” and it’s also not “wait for Congress,” because Congress moves at the speed of Congress and this technology does not wait for anyone. What I keep coming back to is something closer to the International Atomic Energy Agency, except privately funded and run by the industry itself, under a government mandate that makes membership and compliance a legal requirement rather than a courtesy, the same way stock brokerages are legally required to belong to FINRA even though FINRA isn’t a government agency. The difference from either model is the enforcement layer: mandatory membership backed by a legal mandate, plus a technical mechanism that can actually make a rule stick, rather than a report nobody reads.

The obvious objection is the one you’d expect: who audits the auditors, especially when the only people qualified to actually understand these systems well enough to audit them are the same people who build them at the handful of labs doing this work. A funding floor that doesn’t depend entirely on the companies being overseen, seats reserved for people outside the industry, and a public appeals process would all help, though none of it makes the problem disappear.

What would this body actually do. Two things, mainly. First, it audits behavior after the fact, the way flight recorders work in aviation. If an AI system, say an autonomous law enforcement robot, is involved in an incident, there needs to be a full, tamper-proof record of what it saw, what it decided, and why, available to independent investigators, not just the company that built it. Second, and this is the part that actually has teeth, membership comes with a real enforcement mechanism, not just a strongly worded letter. In practice this would mean a model can’t run without a current, cryptographically signed authorization, checked frequently enough that a revoked license actually stops something rather than just embarrassing it after the fact. Researchers at RAND and elsewhere have been sketching versions of this out since 2024. It’s hard, and it raises its own uncomfortable questions about who holds that kind of switch, but it’s the difference between a rulebook and an actual enforceable rule.

None of this requires believing an AI is going to wake up one day and decide humanity is bad for the planet. It just requires believing, correctly, that we are handing enormous and growing power to systems nobody fully understands, built by companies whose incentives don’t automatically align with the public’s, and that right now the answer to “who’s watching this” is essentially nobody. An industry-funded, government-mandated watchdog with a working off switch is not a perfect answer. It’s just a lot better than the one we have now, which is none.

Iko can be reached at iko@uw.edu

More News from Lake Forest Park
I'm interested
I disagree with this
This is unverified
Spam
Offensive