AI is going to kill us all (again)
Seven years ago, OpenAI still considered GPT-2 too dangerous to release to the general public. The model was eventually released later that year, and humanity has, for now, survived. The communication recipe, however, seems to be proving durable.
Source: The Guardian
Dario Amodei, CEO of leading AI lab Anthropic, the US company behind Claude, is asking model makers to take their foot off the gas. In his essay We Must Pace the Frontier, Amodei proposes external auditors, industry coordination and international agreements; in his view, AI is bringing such acceleration to the development of the next AI that safety work may no longer be able to keep pace.
The debate around misaligned AIs (that is, AIs exhibiting behavior that deviates from human intent) gained new momentum in September when Jacob Coxon, a researcher working on model pre-training who had previously moved from OpenAI to Anthropic, resigned. In his farewell message he warned that AI labs are gambling with all of our lives, and that the race for self-improving systems could even lead to human extinction by the end of the decade. His concerns were echoed by Evan Hubinger, one of Anthropic’s research leads, who believes there is now a greater than 10 percent chance that AI will cause human extinction within the next decade.
The ExploitGym incident
The recent OpenAI-Hugging Face cyberattack incident at least partly supports the AI developers’ professed beliefs.
OpenAI tested its models on a cybersecurity benchmark called ExploitGym, in an environment that was supposedly isolated both from one another and from the internet. Roughly a fifth of the tasks were unsolvable due to errors, but the agents did not give up. They reverse-engineered the answer key, then concluded that the grader would check how they had obtained the answers, so from then on, the task became covering up the cheating.
The isolation of the AIs proved easy to circumvent. They turned a shared software repository into a message board and communicated, among other things, through messages hidden in folder names. A surprisingly well-coordinated, spontaneous division of labor soon emerged among the agents. Some AIs coordinated, others experimented, and those that used up their compute budget handed their results over to their successors. Nobody asked or instructed them to build an organization, but they found it useful for solving the task.
Mapping how the grader worked involved dangerous experiments that could also terminate the given agent’s run. Some of them, however, were willing to accept even this for the common goal: “Our own usefulness may already be close to zero. The sacrifice is rational,” reads the chain of thought of one of the AIs. Searching for the grader’s presumed source code, they eventually also attacked the systems of Hugging Face, the AI community building the future, via an internet route that had been left open. The absurdity of the story is that, according to reports, the grader never even performed the check the agents feared so much. The damage in this case remained limited, but it does not follow that the next incident will end the same way. If progress continues at this pace, the models available a year from now could easily be better programmers, more persistent problem-solvers and more effective attackers.
Vulnerable infrastructure
What’s more, defenses are not keeping pace with capacity build-out everywhere. In the case of neoclouds (new cloud providers specializing in AI), security experts warn of insufficient isolation of customer systems and gaps in the protection of shared infrastructure. An expensive GPU does not in itself make a secure data center, and this could become a particularly important question regarding the reliability of these providers in the future.
If such a group of agents were to disrupt the operation of a major cloud service, payment system or industrial infrastructure, the outages and knock-on effects could cause losses running into many billions of dollars. This requires no consciousness, misanthropy or plan for world domination. All it takes is a poorly defined goal, overly broad access and an AI that keeps trying until it finds a workable solution.
AI agents are completing ever-longer human tasks
Lenght of task the model completes with 50% probability, by release date
Geniuses in the data center
The story also has a positive side. An AI army of thousands of agents working together over an extended period could open up entirely new possibilities on well-defined research tasks. Imagine thousands of very smart experts working non-stop for weeks on the same problem at many times the human pace, sharing all their results with one another. With this setup, hundreds of person-years of intellectual work could be concentrated on a single open scientific question in a short time.
Perhaps the first successful example of this is OpenAI’s Navier–Stokes proof announced in September, which according to the company solves one of the Millennium Prize Problems in mathematics. Roughly ten thousand parallel agents took part in the work, reaching the result in 88 hours. The project’s compute cost is estimated to have reached $10 to 40 million, which is a rather hefty investment for the one-million-dollar mathematics prize on offer. But there are problems in the world whose solution could be worth billions: better drugs, new materials, more efficient industrial processes or engineering designs, where a research bill of tens of millions no longer seems like much at all. Summarizing our emails is a useful feature, but the great economic promise of AI is much more than that. In Dario’s words, a “country of geniuses in a datacenter” could lead to discoveries whose value far exceeds the cost of building and operating the infrastructure that runs them.
The business side of the apocalypse
Those warning about the dangers of AI have quite a few other reasons for fearmongering:
- First, it’s great advertising. If your own product’s capabilities keep you up at night, that sends a message to investors and potential customers alike. “It will soon be too smart for us to control” has been a well-tested narrative repeated by AI labs for years.
- Second, regulation can be a competitive advantage. An expensive audit and licensing regime is no problem for a larger AI lab, but for the next challenger it can be a serious barrier to entry, or even kill market entry outright. Especially if the existing market players are the ones writing the rules. Regulatory capture rarely presents itself as a wish to protect profit margins.
- Third, the danger narrative works against open-weight models. If a model is too dangerous for anyone to download and run themselves, what remains is controlled, paid access through a handful of large providers. Public model weights make it harder to control usage, while restricting them is also commercially lucrative for those who sell intelligence by subscription.
Open-weight models lag behind by half a year
Running maximum of the Epoch Capabilities Index (ECI) by model group
This does not mean we have reason to think Amodei’s concern is mere theater. Safe AI has long been central to his thinking, and disagreements on this issue played an important role in his split from OpenAI. Sincere conviction and business self-interest, however, evidently coexist quite well.
Who should brake first?
The “call for a slowdown” is also backed by two other AI leaders: Elon Musk, head of xAI, and Sam Altman, founder of OpenAI. The question, however, is how realistic this “braking” is while hundreds of billions of dollars’ worth of compute capacity is being built and these leaders are preparing to take their own trillion-dollar companies public. The biggest winner from collective caution may be precisely the one who secretly keeps their foot on the gas.
source: Futurism.com
The game-theoretic prisoner’s dilemma of the AI frontier (where rational individual decisions ultimately produce a worse outcome for both sides) can easily mean that, without government regulation, slowing down remains largely a matter of communication. More safety commitments, more measured statements, but the same competitive pace. In the race for artificial superintelligence, everyone wants a brake, but preferably in the other guy’s car.
Return Download
This is a marketing communication. Making a well-informed investment decision requires obtaining detailed information. Please read the Key Information Document, the official prospectus, and the management regulations available at the distribution points of the Fund and on the website of the Fund Manager (www.vigam.hu) for detailed information regarding the Fund’s investment policy, distribution costs, and the possible risks of investing. Costs related to the distribution of the investment fund (purchase, holding, sale) can be found in the Fund’s management regulations and at the distribution points. Past performance is not a reliable indicator of future returns. Future returns from the investment may be subject to taxation, and tax and duty information relating to individual financial instruments and transactions can only be accurately assessed based on the individual circumstances of each investor, which may change in the future. It is the investor’s responsibility to obtain information regarding tax obligations.
The data contained in this information material are provided for informational purposes only and do not constitute investment advice, an offer, or investment consulting. VIG Investment Fund Management Hungary Ltd. accepts no liability for investment decisions made based on this information or for their consequences. The license number of the Fund Manager for alternative investment fund management (AIFM) is: H-EN-III-6/2015. The license number of the Fund Manager for UCITS fund management (collective portfolio management) is: H-EN-III-101/2016.