Customers find it difficult to tell the difference between product reviews written by a person or ChatGPT, according to new research*.

The researchers also found that AI-reviews were much more overwhelmingly positive, but these reviews were often less-detailed and more likely to include false information.

These findings come from a study by Yingting Wen, Professor of Marketing at emlyon business school, alongside her colleague, Sandra Laporte, also a Professor of Marketing at Toulouse School of Management.

The researchers wanted to examine whether or not generative-AI tools, like ChatGPT, were able to create narratives that resonate with humans to a similar level that human marketers were – focusing on marketing content, such as social media posts, product reviews and ad copy.

To do so, the researchers conducted three separate studies comparing generative-AI content to human-created content. The researchers focused on the sensory experiences of two products; whisky and chocolate.

The first study used a text analysis tool to compare AI-generate product reviews to human-generated reviews, finding that human-written content was more genuine, and less positive than AI-generated content.

Then, in the second study, the researchers used human raters to review the content too, as opposed to a text analysis tool, of which the raters came to the same conclusion.

In the final study, the researchers reviewed social media posts; of which half were branded content, and half unbranded. Some of the posts were written by ChatGPT, and some by human social media marketers – with human raters reviewing the content, judged the sentiment of the post and reviewed the engagement too.

The researchers found that AI still struggles with creating rich, varied, and detailed content compared to humans. However, AI is effective at creating content with positive emotions and persuasive appeal, and can also be more convincing to customers when it comes to social media posts.

The researchers also found that ChatGPT 4 was much more effective than ChatGPT 3.5, when it came to creating effective and persuasive content.

“As generative AI tools like ChatGPT are increasingly used in marketing, they help automate tasks like crafting social media posts and responding to customer comments, resulting in higher engagement and increased purchase intent.”, says Professor Wen.

“However, research shows that while AI-generated content can be effective, it still lacks the nuanced understanding and authentic voice that human creators bring to marketing, therefore human input is still needed in the process”.

The researchers state that these findings clearly show that aspects of a marketer’s role that AI can be hugely beneficial in creating impactful content for their company. However, there are still clearly challenges around misinformation – such as ChatGPT inventing information – that need to be considered in the process.

*Research from emlyon business school and Toulouse School of Management.