Researchers discover surprising flaws in ChatGPT content. Here’s how to catch them.
Highlights
- Why low-quality ChatGPT content passes human review.
- ChatGPT is comprehensive even when it should be concise.
- Professor cites a flaw that ruins ChatGPT-generated essays.
- ChatGPT fails the fictional Voight-Kampff android detection test and the surprising reason why this matters.
ChatGPT produces content that is comprehensive and plausibly accurate.
But researchers, artists, and professors warn of shortcomings to be aware of which degrade the quality of the content.
In this article, we’ll look at 11 disadvantages of ChatGPT content. Let’s dive in.
1. Phrase Usage Makes It Detectable As Non-Human
Researchers studying how to detect machine-generated content have discovered patterns that make it sound unnatural.
One of these quirks is how AI struggles with idioms.
An idiom is a phrase or saying with a figurative meaning attached to it, for example, “every cloud has a silver lining.”
A lack of idioms within a piece of content can be a signal that the content is machine-generated – and this can be part of a detection algorithm.
This is what the 2022 research paper Adversarial Robustness of Neural-Statistical Features in Detection of Generative Transformers says about this quirk in machine-generated content:
“Complex phrasal features are based on the frequency of specific words and phrases within the analyzed text that occur more frequently in human text.
…Of these complex phrasal features, idiom features retain the most predictive power in detection of current generative models.”
This inability to use idioms contributes to making ChatGPT output sound and read unnaturally.
2. ChatGPT Lacks Ability For Expression
An artist commented on how the output of ChatGPT mimics what art is, but lacks the actual qualities of artistic expression.
Expression is the act of communicating thoughts or feelings.
ChatGPT output doesn’t contain expressions, only words.
It cannot produce content that touches people emotionally on the same level as a human can – because it has no actual thoughts or feelings.
Musical artist Nick Cave, in an article posted to his Red Hand Files newsletter, commented on a ChatGPT lyric that was sent to him, which was created in the style of Nick Cave.
He wrote:
“What makes a great song great is not its close resemblance to a recognizable work.
…it is the breathless confrontation with one’s vulnerability, one’s perilousness, one’s smallness, pitted against a sense of sudden shocking discovery; it is the redemptive artistic act that stirs the heart of the listener, where the listener recognizes in the inner workings of the song their own blood, their own struggle, their own suffering.”
Cave called the ChatGPT lyrics a mockery.
This is the ChatGPT lyric that resembles a Nick Cave lyric:
“I’ve got the blood of angels, on my hands
I’ve got the fire of hell, in my eyes
I’m the king of the abyss, I’m the ruler of the dark
I’m the one that they fear, in the shadows they hark”
And this is an actual Nick Cave lyric (Brother, My Cup Is Empty):
“Well I’ve been sliding down on rainbows
I’ve been swinging from the stars
Now this wretch in beggar’s clothing
Bangs his cup across the bars
Look, this cup of mine is empty!
Seems I’ve misplaced my desires
Seems I’m sweeping up the ashes
Of all my former fires”
It’s easy to see that the machine-generated lyric resembles the artist’s lyric, but it doesn’t really communicate anything.
Nick Cave’s lyrics tell a story that resonates with the pathos, desire, shame, and willful deception of the person speaking in the song. It expresses thoughts and feelings.
It’s easy to see why Nick Cave calls it a mockery.
3. ChatGPT Does Not Produce Insights
An article published in The Insider quoted an academic who noted that academic essays generated by ChatGPT lack insights about the topic.
ChatGPT summarizes the topic but does not offer a unique insight into the topic.
Humans create through knowledge, but also through their personal experience and subjective perceptions.
Professor Christopher Bartel of Appalachian State University is quoted by The Insider as saying that, while a ChatGPT essay may exhibit high grammar qualities and sophisticated ideas, it still lacked insight.
Bartel said:
“They are really fluffy. There’s no context, there’s no depth or insight.”
Insight is the hallmark of a well-done essay and it’s something that ChatGPT is not particularly good at.
This lack of insight is something to keep in mind when evaluating machine-generated content.
4. ChatGPT Is Too Wordy
A research paper published in January 2023 discovered patterns in ChatGPT content that makes it less suitable for critical applications.
The paper is titled, How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection.
The research showed that humans preferred answers from ChatGPT in more than 50% of questions answered related to finance and psychology.
But ChatGPT failed at answering medical questions because humans preferred direct answers – something the AI didn’t provide.
The researchers wrote:
“…ChatGPT performs poorly in terms of helpfulness for the medical domain in both English and Chinese.
The ChatGPT often gives lengthy answers to medical consulting in our collected dataset, while human experts may directly give straightforward answers or suggestions, which may partly explain why volunteers consider human answers to be more helpful in the medical domain.”
ChatGPT tends to cover a topic from different angles, which makes it inappropriate when the best answer is a direct one.
Marketers using ChatGPT must take note of this because site visitors requiring a direct answer will not be satisfied with a verbose webpage.
And good luck ranking an overly wordy page in Google’s featured snippets, where a succinct and clearly expressed answer that can work well in Google Voice may have a better chance to rank than a long-winded answer.
OpenAI, the makers of ChatGPT, acknowledges that giving verbose answers is a known limitation.
The announcement article by OpenAI states:
“The model is often excessively verbose…”
The ChatGPT bias toward providing long-winded answers is something to be mindful of when using ChatGPT output, as you may encounter situations where shorter and more direct answers are better.
5. ChatGPT Content Is Highly Organized With Clear Logic
ChatGPT has a writing style that is not only verbose but also tends to follow a template that gives the content a unique style that isn’t human.
This inhuman quality is revealed in the differences between how humans and machines answer questions.
The movie Blade Runner has a scene featuring a series of questions designed to reveal whether the subject answering the questions is a human or an android.
These questions were a part of a fictional test called the “Voigt-Kampff test“.
One of the questions is:
“You’re watching television. Suddenly you realize there’s a wasp crawling on your arm. What do you do?”
A normal human response would be to say something like they would scream, walk outside and swat it, and so on.
But when I posed this question to ChatGPT, it offered a meticulously organized answer that summarized the question and then offered logical multiple possible outcomes – failing to answer the actual question.
Screenshot Of ChatGPT Answering A Voight-Kampff Test Question
The answer is highly organized and logical, giving it a highly unnatural feel, which is undesirable.
6. ChatGPT Is Overly Detailed And Comprehensive
ChatGPT was trained in a way that rewarded the machine when humans were happy with the answer.
The human raters tended to prefer answers that had more details.
But sometimes, such as in a medical context, a direct answer is better than a comprehensive one.
What that means is that the machine needs to be prompted to be less comprehensive and more direct when those qualities are important.
From OpenAI:
“These issues arise from biases in the training data (trainers prefer longer answers that look more comprehensive) and well-known over-optimization issues.”
7. ChatGPT Lies (Hallucinates Facts)
The above-cited research paper, How Close is ChatGPT to Human Experts?, noted that ChatGPT has a tendency to lie.
It reports:
“When answering a question that requires professional knowledge from a particular field, ChatGPT may fabricate facts in order to give an answer…
For example, in legal questions, ChatGPT may invent some non-existent legal provisions to answer the question.
…Additionally, when a user poses a question that has no existing answer, ChatGPT may also fabricate facts in order to provide a response.”
The Futurism website documented instances where machine-generated content published on CNET was wrong and full of “dumb errors.”
CNET should have had an idea this could happen, because OpenAI published a warning about incorrect output:
“ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers.”
CNET claims to have submitted the machine-generated articles to human review prior to publication.
A problem with human review is that ChatGPT content is designed to sound persuasively correct, which may fool a reviewer who is not a topic expert.
8. ChatGPT Is Unnatural Because It’s Not Divergent
The research paper, How Close is ChatGPT to Human Experts? also noted that human communication can have indirect meaning, which requires a shift in topic to understand it.
ChatGPT is too literal, which causes the answers to sometimes miss the mark because the AI overlooks the actual topic.
The researchers wrote:
“ChatGPT’s responses are generally strictly focused on the given question, whereas humans’ are divergent and easily shift to other topics.
In terms of the richness of content, humans are more divergent in different aspects, while ChatGPT prefers focusing on the question itself.
Humans can answer the hidden meaning under the question based on their own common sense and knowledge, but the ChatGPT relies on the literal words of the question at hand…”
Humans are better able to diverge from the literal question, which is important for answering “what about” type questions.
For example, if I ask:
“Horses are too big to be a house pet. What about raccoons?”
The above question is not asking if a raccoon is an appropriate pet. The question is about the size of the animal.
ChatGPT focuses on the appropriateness of the raccoon as a pet instead of focusing on the size.
Screenshot of an Overly Literal ChatGPT Answer
Screenshot from ChatGPT, January 2023
9. ChatGPT Contains A Bias Towards Being Neutral
The output of ChatGPT is generally neutral and informative. It’s a bias in the output that can appear helpful but isn’t always.
The research paper we just discussed noted that neutrality is an unwanted quality when it comes to legal, medical, and technical questions.
Humans tend to pick a side when offering these kinds of opinions.
10. ChatGPT Is Biased To Be Formal
ChatGPT output has a bias that prevents it from loosening up and answering with ordinary expressions. Instead, its answers tend to be formal.
Humans, on the other hand, tend to answer questions with a more colloquial style, using everyday language and slang – the opposite of formal.
ChatGPT doesn’t use abbreviations like GOAT or TL;DR.
The answers also lack instances of irony, metaphors, and humor, which can make ChatGPT content overly formal for some content types.
The researchers write:
“…ChatGPT likes to use conjunctions and adverbs to convey a logical flow of thought, such as “In general”, “on the other hand”, “Firstly,…, Secondly,…, Finally” and so on.
11. ChatGPT Is Still In Training
ChatGPT is currently still in the process of training and improving.
OpenAI recommends that all content generated by ChatGPT should be reviewed by a human, listing this as a best practice.
OpenAI suggests keeping humans in the loop:
“Wherever possible, we recommend having a human review outputs before they are used in practice.
This is especially critical in high-stakes domains, and for code generation.
Humans should be aware of the limitations of the system, and have access to any information needed to verify the outputs (for example, if the application summarizes notes, a human should have easy access to the original notes to refer back).”
Unwanted Qualities Of ChatGPT
It’s clear that there are many issues with ChatGPT that make it unfit for unsupervised content generation. It contains biases and fails to create content that feels natural or contains genuine insights.
Further, its inability to feel or author original thoughts makes it a poor choice for generating artistic expressions.
Users should apply detailed prompts in order to generate content that is better than the default content it tends to output.
Lastly, human review of machine-generated content is not always enough, because ChatGPT content is designed to appear correct, even when it’s not.
That means it’s important that human reviewers are subject-matter experts who can discern between correct and incorrect content on a specific topic.
