Experts Issue Grim Warnings: AI Poses Existential Catastrophe Threat to Humanity

Current and former employees of American AI laboratories, alongside their leaders, have issued dire warnings about an impending catastrophe potentially caused by advanced artificial intelligence. Alex Krasodomski, a prominent British researcher and expert in digital media, AI, and communication technologies, emphasized that warnings from companies like Anthropic regarding AI's dangers must be taken very seriously. However, Krasodomski, who directs the Digital Society Programme at Chatham House (the UK's Royal Institute of International Affairs), stressed in a report from the institute that these concerns should not be exploited to solidify the dominance of American laboratories or to escalate competition with Beijing. A former lab employee cautioned that unchecked AI development could mean "we all might die in the near future." This was followed by a proposal from Dario Amodei, CEO of Anthropic AI, to slow the pace of AI development to ensure its safety. The proposal garnered support from other prominent lab leaders, including Sam Altman and Elon Musk. Krasodomski noted that no observer of the AI field could dispute the importance of focusing more on safety during the development and deployment of this technology. However, he warned that renewed attention to the long-term risks of AI should not lead to overlooking its immediate impacts. Furthermore, this issue should not be used as a sharp tool to eliminate competitors of currently leading American AI companies. He asserted that this matter cannot be effectively addressed without reaching some form of agreement between the United States and China. These new warnings center on a concept referred to as "recursive self-improvement," an approach to training AI models that is beginning to see actual implementation, albeit still in its early stages. Krasodomski explained that the risk lies in this technology, which involves AI training other AI systems, potentially leading to an enormous acceleration in development pace. This acceleration could, at some point, lead to a "tipping point" where AI systems become capable of developing themselves beyond human oversight. Krasodomski believes it is unclear whether this alone would be sufficient to produce AI models threatening human control. Nevertheless, the rapid evolution of this technology's capabilities over the past few years makes completely dismissing this possibility somewhat reckless. Experts have been quick to point out that some claims made recently are not entirely well-founded. Specifically in cybersecurity, the founder of the UK's National Cyber Security Centre dismissed the danger of an AI model disrupting the entire internet within a year, describing it as a "thought experiment presented as an evidence-based warning." Despite this, the message that AI poses an existential threat to humanity in the near term received widespread media coverage with little opposition. Krasodomski stated that some risks arising from AI are immediate and certain. Over the next two years, machine learning technologies with growing capabilities will be developed and deployed in ways that will radically reshape economies, societies, governments, battlefields, classrooms, and more. Mathematicians recently experienced their own "Lee Sedol moment," referring to the anxiety one feels when machines begin to outperform them in tasks previously exclusive to human ingenuity. Most people are expected to experience something similar soon. He pointed out that few understand the immediate implications better than the AI companies themselves. A distinguished report recently issued by Anthropic details the misuse of its products for espionage, cyberattacks, information manipulation operations, and weapons development. Most major laboratories maintain significant research capabilities to study AI's potential impact on the global economy. This is a technological revolution reshaping the global society, and its current impacts cannot be left uncontrolled or unregulated. The present impacts are real, and the short-term risks are substantial. Moreover, the long-term risks—whether a gradual decline in human roles as machines assume greater responsibilities, or something more sudden—are potential threats warranting attention. Efforts led by experts such as Stuart Russell are accelerating to achieve an international agreement on clear and verifiable "red lines" to prevent catastrophic consequences. Without these efforts, the first opportunity to establish a binding and applicable global governance system for this technology may be lost. Krasodomski emphasized the critical importance of transparency from AI laboratories. He noted that the model outlined by Amodei in his proposal—involving an embedded third party or independent entity within AI labs, with powers similar to those of employees, tasked with evaluating the safeguards adopted by companies—has been a demand from AI governance advocates for years. Anthropic's unilateral offer to implement this, with some other labs ready to follow suit, undoubtedly represents a step forward. If such a third party is established, it is crucial that it be chosen from outside the "Silicon Valley bubble." The UK's AI Safety Institute appears to be one of the most suitable candidates globally for this role. National AI oversight models similar to the American FDA or international models like the International Atomic Energy Agency are also considered reliable options. Amodei's proposal for American AI laboratories to collaborate on setting unified standards and coordinating a slowdown in work pace was met with sharp criticism from David Sacks, former AI lead at the White House. Sacks wrote on X (formerly Twitter): "Stop pretending antitrust laws must be suspended so you can form a cartel, and stop claiming you need a regulatory approval process beyond product liability rules." He added: "Demanding your preferred regulatory framework as the price... will look like blackmail of the public and the political system." This stance reflects concerns that immense pressures on these companies to meet revenue expectations might push them to try to expand their competitive advantage, i.e., entrenching their superiority at the expense of others. Any such effort should be rejected, as competition for better performance and reliability will be essential for enhancing AI safety in the short term. It could also slow the pace of development by weakening the commercial incentive to inject massive capital into training advanced, leading AI models, which are constantly increasing in cost. Any proposed policy that hinders competition – such as banning open-source AI models, relaxing product liability laws, or prohibiting the use of existing models proven safe – should raise alarm bells. The UK, in particular, should not ban open-source or open-weight AI models. Finally, any efforts to maintain human control over AI development will be incomplete without some form of understanding between the major powers in this field, the United States and China. This could perhaps involve setting "red lines" like those suggested by Professor Russell. Krasodomski noted that the current discourse, heavily focused on preserving US superiority over China, makes this more difficult. Amodei argues that democracies must maintain their lead over China and accuses Beijing of adopting a less stringent approach to AI safety. For his part, US President Donald Trump portrays AI as a "zero-sum game," preferring to focus on winning the AI race, stating, "whoever wins in AI, wins." In contrast, Chinese President Xi Jinping has spoken publicly about the necessity of human control over AI. The Chinese intelligence agency also issued its first warning this week regarding the risks AI poses to national security. Nevertheless, Xi's push to promote China's vision for AI governance, which comes amidst competition with Washington, will undoubtedly fuel the sense of an accelerating AI arms race, and this will in no way contribute to curbing the speed of development. Krasodomski concluded his analysis by stating that the AI file will be on the table when Trump and Xi meet in Washington later this month. While achieving some progress is not impossible, given the insistence of prominent voices in both countries on the need to surpass the other, reaching an agreement will require tremendous efforts.