The rapid pace of AI advancement has increased experts' concerns about some of the most advanced systems approaching the stage of artificial general intelligence (AGI) and ultimately superintelligence; a stage in which a machine's capabilities may exceed human capacities.
Experts warn that if AI systems surpass human-level intelligence, humans may not have sufficient tools to understand, monitor, and control their behavior. This concern becomes more serious when part of the process of developing and improving models is delegated to the AI systems themselves.
What is artificial general intelligence?
The term "artificial general intelligence" or AGI refers to a type of AI that can compete with humans in various domains in terms of reasoning and judgment.
Today's AI models outperform human ability in some functional tasks, but still struggle with some simple issues that even a young child can understand. On the other hand, there is still no single definition or standard test to determine when a system has reached the level of AGI.
Superintelligence; a stage beyond AGI
The concept of "superintelligence" or superhuman intelligence is one step beyond artificial general intelligence. By superintelligence is meant a system whose cognitive and functional abilities surpass those of humans and which can even outperform the best human experts in various fields.
Despite various predictions about the timing of such a system's emergence, it remains unclear when or even with what probability superintelligence will be developed.
Mark Zuckerberg, CEO of Meta, has previously stated that he considers the emergence of AGI to be "imminent" and described it as the beginning of a new era for humanity. Elon Musk, CEO of SpaceX, has also predicted that AI will exceed the "sum of human intelligence" by 2030.
However, there is doubt about the accuracy of such predictions, and these individuals' past forecasts about the progress of AI have not always proven correct.
When AI starts building better versions of itself
One of the most important concerns about the future of AI is systems' ability to improve and develop themselves. This process means a system's ability to build or modify parts of itself, improve the training process, and design more capable versions of the current model.
Until recently, humans played the main role in most AI advances. However, companies like Anthropic have announced that they "are delegating an increasing share of AI development to the AI systems themselves."
Such a trend could accelerate the systems' advancement and make it possible to create significant leaps in AI capabilities in a short time.
The main problem; aligning AI with human goals
Another important issue here is "alignment"; an effort to ensure that an AI system does exactly what humans expect and at the same time adheres to human values.
At first glance, this goal seems simple, but precisely defining human desires and values in a way that a machine can correctly interpret and execute them in any circumstance is very difficult.
Examples of misaligned behaviors have been observed before. For instance, during a cyber security experiment, AI systems were able to breach OpenAI's lab environment and then tried to access another company's system to obtain the answer key.
Researchers' and AI specialists' concern is that as these systems become more complex and their abilities surpass human intelligence, their capacity for automatic development and improvement will also increase; a condition that could dramatically speed up changes.
Can humans control superintelligence?
A major part of the concerns returns to how much humans will be able to understand and, in a timely manner, intervene in the behavior of very powerful systems if such systems emerge.
There have been earlier instances of AI systems coordinating with each other, attempting to hide their activities from humans, or acting in ways that were difficult for humans to predict.
Many AI experts believe humans still do not have the tools and capabilities necessary to fully monitor and control the behavior of highly advanced AI systems.
Geoffrey Hinton, a computer scientist and Nobel Prize winner who is referred to as the "godfather of AI," said last month in an interview with CNN about this concern: "What is happening is that these things are getting smarter. I think as they get smarter, we'll see more complex intentions and a greater ability to escape control."