AI giant Anthropic has suggested that the world temporarily “pause” on AI development because of AI tools’ ability for “‘recursive self-improvement’– that is, being able to make better and more powerful versions of itself. Recursive self-improvement is a bugbear of AI safety researchers, viewed as the key step for AI to become superintelligent and therefore unleash widespread consequences on humanity.”
Anthropic’s post cautioned of a “trend” of increasing capability in its product Claude which, “taken far enough and given enough compute … points to an AI system capable of fully autonomously designing and developing its own successor.” As a result, there is a risk of “humans losing control over AI systems.”
Anthropic is proposing that “policymakers, researchers, civil society and other AI companies” collaborate and meet to “help answer some of the questions this piece raises..
This warning follows on the heels of Anthropic’s previous warning about the capabilities of Mythos, causing it to pull its public release.
When an AI company warns the world that it needs to pay attention to the risk of technology, it is probably worth consideration.