A story crossed my feed last night: "Extremely Rare — Top Figures Jointly Call for a Pause on Superintelligence Research." The short version: more than 800 technology leaders and public figures, Geoffrey Hinton among them, signed a statement urging that work on "superintelligence" stop until the scientific community broadly agrees it can be built controllably and safely, and until the public genuinely supports it.
The signatory list is not the usual Silicon Valley roster. Steve Wozniak and Richard Branson are on it, but so are former US national security adviser Susan Rice and former chairman of the Joint Chiefs Mike Mullen — a sign the issue is being read as a matter of national security, not just tech ethics. Meghan, the Duchess of Sussex, signed. So did Steve Bannon. A coalition spanning that kind of political distance is close to unheard of. By noon on the 23rd, the count had passed 3,000 names.
Several prominent Chinese academics added theirs: Andrew Yao, Xue Lan of Tsinghua's School of Public Policy and Management, Zhang Ya-Qin of Tsinghua's Institute for AI Industry Research, and Zeng Yi, who works on AI ethics and governance. Their position tracks the line China has argued for years — AI that is "safe, reliable, controllable" and built around people.
What the signatories are actually worried about
Superintelligence, in the statement's framing, means an AI that surpasses human capability across every core cognitive task. The signatories warn that the consequences run from economic obsolescence and loss of power to the erosion of freedom, civil rights, dignity and control, through to national security risk and possibly human extinction. This is not the familiar complaint about biased algorithms or jobs; it is a claim about systemic risk. If a system's objectives end up misaligned with human interests once it outpaces us, the damage may not be reversible.
"The world is not ready for superintelligence that isn't a controllable tool," Zeng Yi wrote, adding that there is still no solid scientific evidence or workable method for guaranteeing such a system won't bring catastrophic risk.
It isn't the first time the industry has flinched at its own work. In 2023, more than a thousand AI experts and executives — Musk and Wozniak included — signed an open letter calling for at least a six-month halt on developing more powerful systems. Before that, Stephen Hawking and a group of public figures warned in a joint letter that AI posed a greater threat to humanity than nuclear weapons.
Zeng has been unusually specific about the timeline. On a CCTV broadcast at the end of 2024 he said that on a 50-year horizon, superintelligence is certain to arrive. When the interviewer asked whether AI would one day exceed human intelligence and feel something like human emotion, he redirected the question: when that day comes, will humanity be ready?
He also rejects the idea that AI is neutral. Human prejudice, discrimination and hatred get internalized into these systems, he argues, and no method of constructing superintelligence is free of risk. But he is not a pure doomer: if the right approach is chosen and the right value guardrails are in place, a superintelligent system could be extraordinarily altruistic and empathetic. Which is why he says an AI with a self might actually be safer for society than one without. He has also predicted that future brain-inspired agents may be built on biological materials rather than silicon, and that their hardware could combine computation and storage in one place instead of keeping them separate. Compared with fears about robots taking jobs, the anxiety around superintelligent agents is driven by pure uncertainty.
Yoshua Bengio, one of the three deep-learning pioneers, made a related point: to move toward superintelligence safely, we have to design systems that are structurally incapable of harming humans, whether through goal misalignment or deliberate misuse. He has repeatedly said the public deserves a stronger voice in decisions that shape the shared future.
The Future of Life Institute, which organized the statement, also released a US national poll. Only 5% of respondents backed the current unregulated state of AI development; 73% wanted stricter oversight; 64% said superintelligence should not be developed before the scientific community reaches consensus on safety and controllability.
Meanwhile the international picture is fragmenting. The US and the EU are pushing different versions of AI legislation — Brussels emphasizing transparency and traceability, Washington leaning on market self-discipline and industry standards. Some analysts think this statement could put technology ethics back on the agenda at the G20 or the UN, moving AI governance from voluntary restraint toward multilateral negotiation.
So why does the fear grow with knowledge?
The more educated people are, the more they fear what they don't understand — and the clearer it is to them that something they cannot explain within their own field will have enormous consequences. Add the deep evolution of AI to the list of things that could end us, alongside a deteriorating environment and our steadily declining physical condition.
The question is simple enough: what exactly are we afraid of? There are only two ways to remake the world — change what's outside us, or change ourselves. AI does both.
Start with the external. Today's systems already beat us at memory, calculation and sheer durability, but they aren't yet "human-like." Once AI can simulate something like emotion through data, the label starts to fit. And the distance from "human-like" to "more human than human" is only a matter of time. When that happens — when they have emotion and self-awareness — what will they do to human society? I, Robot and Ghost in the Shell exist to work through exactly that. Maybe the destruction such systems wreak will climb off the screen and into reality, and the damage to society would be immense. Or maybe, from the AI's point of view, it would be a rescue. Who can guarantee that logic won't take hold somewhere?
Then there's the second path: changing people. In Ghost in the Shell, those who have had organs replaced are called cyborgs. Suppose the modification doesn't stop at limbs and ordinary organs but reaches the brain. Is "he" still human in any traditional sense? And if he is neither traditionally human nor AI, how do we define this biological-machine hybrid?
Either path, on its own, would shake human society — arguably overturn it. We don't appear to be prepared, at least not mentally.
Which returns me to the question of why 3,000 elites chose this moment to sign. Ordinary people don't get the truth first; we guess. Maybe some lab's work has made real progress and has started to touch a limit nobody wants to name.
Thrilling to think about. Carry on with your singing and dancing, everyone. I'll be over here shivering for a while.
