AI has passed a new benchmark, scoring better than the average human on a recognized creativity test involving 100,000 people. But there's more to the story than the results, underpinning how difficult it is to put "creativity" in a measurable box.
Université de Montréal researchers led a large-scale study that pit 100,000 humans against the current leading generative AI models in an attempt to assess the creative power of both. It's the largest comparative study of its kind ever conducted.
In order to measure what most of us would instantly consider a subjective field, the team used divergent linguistic creativity tasks to score the latest LLMs including ChatGPT, Claude and Gemini, as well as the humans.
“We developed a rigorous framework that allows us to compare human and AI creativity using the same tools, based on data from more than 100,000 participants,” said Professor Karim Jerbi, from the Department of Psychology at the Université de Montréal.
The first caveat should be made here: It's obviously very hard to quantify human creativity in a way that can be compared with a LLM. So while this is a massive study, it's still defined by the measures and constraints that the scientists employed.
The team used the Divergent Association Task (DAT), something used in psychology to measure a specific type of creativity. Essentially, it asks someone to come up with 10 words in four minutes, and the less related the words are, the more creative the list is considered to be. Then the scientists had the AI models do the same.
What they found was that while LLMs demonstrated more creativity – as measured by the DAT – than a large number of humans, around half of the participants fared better than AI, and the top 10% far exceeded the performances of their computer challengers.
So yes, while some people failed to show more "divergent creativity" than Claude, for example, a whole lot of people didn't. And this pulls into sharp focus just how difficult it is for even today's most advanced machines to replicate the output of the human brain – even after their creators have scraped what feels like every word in every language on Earth.
“Even though AI can now reach human-level creativity on certain tests, we need to move beyond this misleading sense of competition,” said Jerbi. “Generative AI has above all become an extremely powerful tool in the service of human creativity: It will not replace creators, but profoundly transform how they imagine, explore, and create – for those who choose to use it.”
So while LLMs are better than some humans when it comes to specific creative tasks, the same can be said when assessing a group of people. And this study highlights how complex and nuanced measuring human traits are – and how LLM benchmark scores aren't really solid indicators to use in comparative analyses.
"Even though AI can now reach human-level creativity on certain tests, we need to move beyond this misleading sense of competition,” said Jerbi. “Generative AI has above all become an extremely powerful tool in the service of human creativity: It will not replace creators, but profoundly transform how they imagine, explore, and create – for those who choose to use it.”
The researchers also investigated how AI models compared with humans when it came to creative writing tasks, including haikus, film synopses and short stories. Once again, the most creative humans outperformed the machines – even if LLMs overall scored better than the average participant.
And it's worth noting that the LLMs expressed the most creativity when they were guided well – by humans. So it appears we are still a long way off from being replaced. And while AI has infiltrated our daily lives, there's a growing pushback on AI slop and using technology that exploits artists. Recently, some 800 artists have banded together to campaign against the use of AI-generated content in a broad range of creative fields.
In this study, the researchers note that rather than think of it as a "human versus machine" investigation, the work should instead highlight AI's ability to assist people in creative endeavors.
“By directly confronting human and machine capabilities, studies like ours push us to rethink what we mean by creativity,” added Jerbi.
The study was published in the journal Scientific Reports.
Source: Université de Montréal
