A well-known test designed to measure whether an artificial intelligence (AI) can think and learn like a human is nearing a solution, but the people who created the test believe this is more about problems with the test itself than actual progress in AI development.
In 2019, Francois Chollet, a prominent figure in the AI community, came up with a test called the ARC-AGI benchmark, which stands for “Abstract and Reasoning Corpus for Artificial General Intelligence.” This test aims to see if an AI can learn new skills that it wasn’t specifically trained on. Chollet argues that ARC-AGI is unique because it is the only test that effectively measures how far we’ve come toward creating an AI with general intelligence, even though other tests have been suggested.
Before this year, the most advanced AI systems could only solve about 30% of the tasks in the ARC-AGI test. Chollet believes this is due to the AI industry focusing too much on large language models (LLMs), which he argues lack true reasoning abilities. He explained on social media that LLMs primarily rely on memorization of data rather than genuine understanding.
Read Also: Google Launches $20B Renewable Energy Initiative To Power AI
Chollet stated, “LLMs struggle with generalization, because they depend entirely on memorizing patterns.” For example, LLMs can predict what comes next in a sentence based on patterns they’ve seen before, like knowing that “to whom” often comes before “it may concern” in an email. However, Chollet believes that while LLMs can remember certain reasoning patterns, they aren’t capable of creating new reasoning for new situations. He argued that if an AI needs to see many examples of a pattern to understand it, then it’s simply memorizing rather than truly learning.
To encourage research that goes beyond LLMs, Chollet and Mike Knoop, the co-founder of Zapier, launched a competition in June with a prize of $1 million for anyone who can create an open-source AI that can beat the ARC-AGI test. Out of nearly 18,000 entries, the best-performing AI managed to solve 55.5% of the tasks, which is an improvement over last year’s top score but still falls short of the 85% success rate that would indicate “human-level” intelligence.
Despite this progress, Knoop cautions that we are not necessarily 20% closer to achieving artificial general intelligence (AGI). In a blog post, he pointed out that many submissions to the ARC-AGI competition seemed to solve problems through brute force rather than genuine understanding, suggesting that many tasks in the test might not effectively measure true intelligence.
The ARC-AGI benchmark consists of puzzle-like challenges where an AI must create a correct answer grid using colored squares. These tasks were intended to test whether an AI can adapt to new problems it hasn’t encountered before, but it’s unclear if they are accomplishing that goal.
Knoop admitted that the ARC-AGI test has not changed since it was first introduced in 2019 and acknowledged that it has flaws. Both Chollet and Knoop have faced criticism for promoting ARC-AGI as a reliable measure of progress toward AGI, especially since there is ongoing debate about what AGI actually means. For instance, a staff member from OpenAI recently suggested that AGI might already exist if we define it as an AI that performs better than most humans at most tasks.
To address these concerns, Knoop and Chollet plan to release an updated version of the ARC-AGI test and hold another competition in 2025. Chollet stated that they aim to guide the AI research community toward solving what they consider the most significant challenges in AI and to speed up the timeline for achieving AGI.
However, making these improvements will likely be challenging. The difficulties encountered with the first ARC-AGI test show that defining intelligence for AI might be just as complicated and contentious as it is when we try to define intelligence for humans.
