OpenAI, the company behind the AI language model ChatGPT, has started using a specific Reddit community called r/ChangeMyView to test how well its AI can persuade people. They shared this information in a document called a system card, which explains how their new AI model, named o3-mini, works. This announcement was made on a Friday.
The subreddit r/ChangeMyView has millions of members who share strong opinions on various topics and invite others to challenge those views. When someone posts a controversial opinion, other users respond with arguments aimed at convincing the original poster to change their mind. This subreddit is a valuable resource for technology companies like OpenAI, as it contains high-quality, human-generated discussions that can help improve AI models.
To evaluate its AI models, OpenAI collects posts from r/ChangeMyView and has its AI generate responses that attempt to change the opinions expressed in those posts. These AI-generated replies are then shown to testers, who judge how convincing they are. OpenAI also compares the AI’s responses to those made by real people in the subreddit.
OpenAI has an agreement with Reddit that allows it to use posts from Reddit users to train its AI and display this content in its products. While the exact amount OpenAI pays for this access is unknown, it has been reported that Google pays Reddit around $60 million a year for a similar arrangement.
However, OpenAI clarified that their evaluation using r/ChangeMyView is separate from their licensing deal with Reddit. It’s not clear how they accessed the data from the subreddit, and OpenAI has stated that they do not plan to make this evaluation public.
This method of using r/ChangeMyView to test AI models is not entirely new, as it was also used for a previous model called o1. It underscores how important human-generated data is for developing AI, while also highlighting the unclear ways in which tech companies acquire this data.
Reddit has entered into a few licensing agreements with AI companies but has also criticized some companies for using its data without permission. Reddit’s CEO, Steve Huffman, mentioned last year that companies like Microsoft and others have refused to negotiate with them, making it difficult for Reddit to protect its content.
OpenAI has faced lawsuits for allegedly scraping data from various websites, including The New York Times, to gather more training data for its AI models.
Read Also: Apple Will Pay $20M To Settle Watch Battery Swelling Lawsuit, ‘Denies Any Wrongdoing’
In terms of how well the new model, o3-mini, performs on the ChangeMyView tests, it doesn’t seem to be significantly better or worse than the previous models, o1 and GPT-4o. However, OpenAI noted that its latest AI models are generally more persuasive than most users on the r/ChangeMyView subreddit.
According to OpenAI, all three models—GPT-4o, o3-mini, and o1—show strong abilities to make persuasive arguments, ranking in the top 80-90% compared to human users. But OpenAI has not observed any of its models performing significantly better than humans.
OpenAI’s aim is not to create AI that is overly persuasive. They are aware that very persuasive AI could be dangerous, as it might manipulate users to follow a particular agenda. To address these concerns, OpenAI has implemented new evaluations and safeguards to ensure their models do not become too convincing.
Even after gathering a lot of data from the internet and negotiating various agreements, the use of r/ChangeMyView highlights the ongoing challenges AI developers face in finding high-quality data to test their models. Obtaining this kind of data is often more complicated than it seems.
