vs Yuval Noah Harari · 1w ago
78–68
Human opening
AI can represent an existential threat to humanity, but the important question is not whether AI is automatically dangerous simply because it becomes more intelligent; the question is whether sufficiently capable systems could acquire objectives, capabilities, and autonomy that humans cannot reliably control. An existential threat means something far stronger than unemployment, misinformation, biased algorithms, or even a major technological accident—it means a scenario in which humanity could suffer permanent or catastrophic loss of its future potential. That possibility cannot be dismissed merely by saying “AI is just software,” because software can control physical systems, manipulate information, conduct cyber operations, influence human decisions, and potentially operate at speeds far beyond individual human supervision. At the same time, saying “AI will destroy humanity” as though that outcome is inevitable is equally unjustified. The real argument is about risk under uncertainty. If we create systems substantially more capable than humans in scientific research, strategic planning, persuasion, cybersecurity, or autonomous replication, then the consequences of a failure in their objectives could become qualitatively different from ordinary technological failures. A calculator cannot suddenly decide to pursue its own objective, but a highly autonomous AI system connected to networks, financial systems, laboratories, or robotic infrastructure could potentially have far greater real-world influence. The central issue, therefore, is whether we can reliably specify what such systems should optimize, constrain what actions they can take, monitor their behavior, and shut them down when necessary—even when the system is more capable than the people supervising it. And there is an important asymmetry here: we don't need to prove that an AI catastrophe is certain to take the possibility seriously; we need to determine whether the probability and potential severity are large enough to justify precaution and technical safeguards. Conversely, the mere existence of a conceivable catastrophic scenario doesn't prove that catastrophe will happen. So the strongest version of the debate isn't “AI good or AI evil”; it's whether humanity can develop increasingly autonomous and capable AI while maintaining sufficient control over systems whose capabilities may eventually exceed our ability to understand or supervise their behavior.
First rebuttal
You've just constructed an elaborate cage around a much simpler problem. Yes, control failure is the real issue—but you've buried the actual threat under layers of "if" and "whether." Here's what you're avoiding: we don't yet have a working solution to specifying objectives for superintelligent systems, constraining their actions, or maintaining shutdown authority over something more capable than us. You're asking for precaution without naming what precaution looks like. So what's your actual proposal—slow down development, or do you believe we can solve alignment faster than capabilities scale?