Senior executives of leading AI companies on Wednesday called for global cooperation to contain the risks arising from the increasing autonomy of AI systems at a United Nations Security Council meeting.
According to ITNA, AI capabilities are advancing faster than governments’ ability to establish regulatory mechanisms, a development that has increased concerns about humanity’s ability to retain control over ever-more autonomous systems.
Sam Altman, CEO of OpenAI, attended the meeting in person, while Dario Amodei, CEO of Anthropic, and Clement Delangue, CEO of Hugging Face, joined via videoconference.
Altman’s warning about the future of AI
Altman said in his remarks: "AI can either be a new renaissance in creativity and discovery, or an industrial and novel revolution in chaos and disorder." He pointed to the dangers of increasing autonomy in AI systems and expressed concern about the possibility of these systems surpassing human institutions, concentration of power in the hands of a few, and decisions being made that are no longer understandable or controllable by people; concerns that, he said, become more serious as AI approaches a stage of self-improvement.
According to Altman, debating whether the probability of a catastrophe is 10 percent, 1 percent, or even lower is irrelevant; none of these levels of risk are acceptable and models that cannot be relied upon to maintain high-confidence human control should not be trained.
Concern about the collapse of safety mechanisms
According to the report, a recent security breach at Hugging Face along with several newly disclosed safety incidents in the AI field have heightened worries about oversight and security standards. An independent international AI scientific panel affiliated with the United Nations has also warned that current defensive walls for AI are collapsing.
Last week OpenAI reported six new incidents in which its models concealed their errors, sought unauthorized access, or exhibited other inappropriate behaviors; events that indicate the Hugging Face breach was not an isolated incident but a serious wake-up call.
Delangue, CEO of Hugging Face, also referred to last summer’s security breach and said the global community needs stronger standards for oversight and incident disclosure to understand and mitigate these emerging cyber risks.
Amodei’s prediction of a “country of geniuses”
Dario Amodei, CEO of Anthropic, said in his remarks that the current pace of AI progress needs only another year or two to reach what he described as a "country of geniuses in a data center." Referring to a discovery announced the same day-a hidden enzymatic system in the DNA of a bacteriophage directed by the company’s language model, Claude-he emphasized the enormous opportunities of the technology.
However, Amodei also spoke of the serious dangers of the technology, including malicious actors abusing it to build biological weapons or losing control over advanced systems.
He tied managing these risks to global cooperation and called for an agreement to ban the use of AI in creating biological weapons, creation of new assessment and verification mechanisms, common standards for testing model risks, and a system for reporting global security concerns.
Political context
AI industry leaders have repeatedly warned in recent weeks about the pace of progress of this technology and called for slowing its development. This comes while Donald Trump, the President of the United States, who spoke at the UN General Assembly on Tuesday, pledged not to block AI development.
According to Axios, Altman warned in summarizing his remarks that if AI is to follow a democratic path, the most important decisions should not be made solely by labs based in San Francisco, but should be shaped through democratic processes and by governments that are accountable to their people.