If you’ve ever tried to use ChatGPT as a calculator, you’ve likely noticed its difficulty with math. It’s not alone in this, though—many AI systems struggle with numbers. For example, Anthropic’s Claude often stumbles on basic word problems, Google’s Gemini has trouble with quadratic equations, and Meta’s Llama faces challenges with simple addition.
So, why can these models produce intricate soliloquies yet get tripped up by elementary arithmetic?
A big part of the issue is tokenization. This process divides data into smaller chunks or tokens (for example, splitting “fantastic” into “fan,” “tas,” and “tic”). While this helps AI efficiently process information, tokenizers don’t always understand numbers correctly. For instance, the number “380” might be treated as a single token, but “381” could be broken into two tokens (“38” and “1”), disrupting the numerical relationships.
However, tokenization isn’t the only reason AI struggles with math. These systems are primarily statistical models trained to identify patterns in vast amounts of data. They predict based on past examples, which works well for language but not as much for precise math. For instance, given the multiplication problem 5,789 x 1,283, ChatGPT might guess that a number ending in “7” multiplied by one ending in “2” will produce a product ending in “4.” But it may struggle with the rest, leading to incorrect results.
Read Also: Facebook Unveils Redesign Aimed at Capturing Gen Z Audience
Yuntian Deng, an assistant professor at the University of Waterloo, benchmarked ChatGPT’s multiplication abilities earlier this year. His study found that GPT-4o struggles with multiplying numbers beyond four digits each, with accuracy dropping to under 30% for such problems. According to Deng, errors in any intermediate step of a calculation often compound, leading to incorrect final results.
Yuntian Deng is optimistic. In his study, he and his colleagues also tested OpenAI’s newer “reasoning” model, o1, which recently became part of ChatGPT. Unlike GPT-4o, the o1 model “thinks” through problems step by step before answering. This approach significantly improved performance, with the model correctly solving nine-digit by nine-digit multiplication problems about half the time.
“The model might be solving problems in ways different from how we do it manually,” Deng explained. “It raises curiosity about the model’s internal methods and how they differ from human reasoning.”
Deng believes this progress suggests that certain types of math problems—multiplication being one—could eventually be “fully solved” by AI like ChatGPT. “It’s a well-defined task with known algorithms,” he said. “We’re already seeing significant improvements from GPT-4o to o1, showing clear advancements in reasoning capabilities.”
