ChatGPT and GPT-3: One Notch Above Useless
As of January 2023, AI writing tools are not very good. Understand what their limitations are and whether you should still be using them for your marketing content
The amount of noise being generated about ChatGPT is obscene. But is it actually useful for B2B marketing?
TL;DR: Not really.
In the last year, I’ve spent dozens of hours trying to generate content with OpenAI’s GPT-3, and lately also ChatGPT. I’ve used these tools both on pet projects and on client work (with the client’s approval) - and the results have been uniformly underwhelming.
And all the breathless social media posts about how AI tools are going to revolutionize marketing? I’m convinced that 90% of the people writing them are not actually using these tools for any real-life use case. The other 10% are spammers.
Caveats
The use cases I tested were mostly around generating short-form and long-form marketing copy, so can’t comment on how useful it is for, e.g., writing code.
I work with B2B clients, typically in more obscure and technical niches. I haven’t tested OpenAI extensively for simpler use cases, although I suspect it would be plagued by the same problems.
This is written in January 2023. Perhaps things will change. I will try to update this article if they do.
With that out of the way, let’s get into the details of my experience with ChatGPT and GPT-3 (I’ll occasionally refer to these as “GPT tools” since they are essentially the same), and why I came to the conclusion that they’re somewhere between mostly and completely useless.
GPT content is riddled with factual mistakes, and the need to double check everything slows you down.
The GPT tools are really good at creating text that’s grammatically acceptable and has a consistent internal logic. The problem is they are often just plain wrong, and in weird and unexpected ways..
Example 1: ChatGPT writes a HubSpot tutorial for an alternate reality
I asked ChatGPT to write a guide implementing lead scoring in HubSpot. This is a topic I know very well, as I’ve done this process myself multiple times. Here’s what I got a back:
Here is a guide to implementing lead scoring in HubSpot:
First, log in to your HubSpot account and navigate to the Contacts menu.
Next, click on the Lead Scoring button, which can be found in the top-right corner of the page.
Once in the lead scoring page, click on the Create Scoring Model button.
Give your lead scoring model a name and choose the properties that you want to use for scoring (ex: job title, company size, industry, etc.).
Assign point values to each property, based on how important it is to your business. For example, a lead from a large enterprise may be worth more points than a lead from a small business.
Next, you can assign different point ranges to different lead scores. For example, a lead score of 0-50 may be considered a "cold" lead, while a score of 51-100 may be considered a "warm" lead.
You can also use scoring rules to give or take away points based on certain conditions. For example, you can give extra points to leads that have a specific job title or have visited a certain page on your website.
Test the scoring model in your HubSpot account by creating some test contacts, and monitor the score and points they receive.
Once the lead scoring model is set up, you can use it to prioritize leads and target specific groups of leads with different marketing campaigns.
As needed, refine the scoring model based on how it performs and how your contacts are evolving.
Note: This is a high level guidance, for more detailed instructions on how to use lead scoring in HubSpot, it's recommended to refer to Hubspot’s official documentation and training materials.
Pretty good, right? There’s just one issue: the practical parts are all made up. There is no Lead Scoring button in the top right of the page. There is no button to Create Scoring Model. God knows where ChatGPT brought these instructions from, but steps 1-4 are all completely detached from the actual process of scoring leads in HubSpot (here’s their documentation if you don’t believe me).
Steps 5-10 are reasonably accurate, although they are more like general instructions on how to set up lead scoring models - you can find countless such articles on Google - rather than a guide to implementing lead scoring in HubSpot.
Example 2: ChatGPT acknowledges its mistake, then immediately gives an equally wrong alternative
Maybe this is a fluke? I started a new chat and asked the same question:
Lead scoring is a process of assigning a numerical value to a lead based on their level of engagement and the likelihood that they will become a paying customer. In HubSpot, you can use lead scoring to prioritize leads and focus your efforts on the most promising prospects. Here is a general guide to implementing lead scoring in HubSpot:
Define your lead scoring criteria: Determine the characteristics that define a high-quality lead for your business. These can include things like job title, company size, industry, website behavior, and engagement with your content.
Create a lead scoring model: In HubSpot, go to the "Contacts" menu, select "Lead Scoring," and click on "Create Lead Scoring Model." This will allow you to assign points to…
It’s repeating the same nonsense again. This strikes me as worse than making up new nonsense. I tried correcting it:
Eran: Step 2 is wrong. HubSpot does not have a button called "lead scoring" or "create lead scoring model".
ChatGPT: You're correct, my apologies for the confusion. HubSpot does not have a built-in "Lead Scoring" feature. However, you can still implement lead scoring by using a combination of custom properties, contact lists, and workflows. Here's an updated guide on how to implement lead scoring in HubSpot:
Define your lead scoring criteria: Determine the characteristics that define a high-quality lead for your business. These can include things like job title, company size, industry, website behavior, and engagement with your content.
Create custom properties: You will need to create custom properties to track the criteria you've defined. For example, you may create a custom property called "Company Size" that will allow you to assign points based on whether a company is small, medium, or large.
Assign points to actions: In HubSpot, you can assign points to different actions, such as visiting a specific page on your website, downloading a whitepaper, or attending a webinar. This allows you to track engagement and assign points based on the level of interest a lead has shown.
Create contact lists: Create a contact list for each stage of your lead scoring process, so you can segment your leads based on their score. For example, you may create lists for "Hot leads," "Warm leads," and "Cold leads."
Create a workflow: Create a workflow that will automatically assign points to leads based on their behavior and move them to the appropriate contact list. You can also use workflows to perform actions like sending an email or notifications to specific team member, when a lead reaches a certain score threshold.
Monitor and adjust: Lead scoring is an ongoing process, so it's important to monitor the results and adjust your scoring criteria and thresholds as needed. You can monitor the performance of your lead scoring model by analyzing the data in the "Contacts" section of HubSpot.
Finalize the lead scoring based on your business needs, set up scoring criteria and thresholds, by using the hubspot lead scoring model you can now easily identify the leads that are ready to engage with your sales team or those that may need further nurturing.
Keep in mind that, this process is not built-in feature in Hubspot and it's an integration of multiple feature of hubspot like custom properties, lists, and workflows. But, with this guide you will be able to create a work around for lead scoring.
This is still completely wrong. You don’t use workflows and contact lists to do lead scoring in HubSpot; you use a scored property and set up the scoring criteria in that property (here’s that link to HubSpot’s documentation again). A workflow can’t assign points, and the use of lists in this case is pointless. ChatGPT is flat out getting it wrong.
Now, to be fair, this was a specific question that the AI really stumbled over (although it was the first marketing-related one I asked it). The tool doesn’t get everything wrong. In fact, it gets most things right… But “most” is really not good enough - and the mistakes are far too frequent for comfort. I read some claims that ChatGPT is 97% accurate. This, in my experience, is bullshit. It will struggle to write a 500 word essay without one or two glaring mistakes.
Note that lead scoring in HubSpot is a topic I know really well first-hand. What would happen if I asked it to cover a topic I didn’t really know much about? As it stands, I would have to fact check every single claim ChatGPT makes. This after-the-fact research slows down the entire process (and makes it much more annoying).
Of course, humans make mistakes too; but a human would be far less inclined to just make up a button that doesn’t exist in the interface; and a skilled editor can suss out when a writer doesn’t know what they’re talking about. The way ChatGPT weirdly remixes information it has in its training data to confidently spout nonsense makes its brand of BS uniquely hard to detect.
Bottom line: You still need to do the research when using GPT tools - which means you’re not saving quite as much time as you thought.
The writing is bland and generic.
I am truly perplexed when I see people ranting and raving about how ChatGPT generates great copy. I feel like we’ve really gotten used to some piss-poor copywriting if the dull, uninspired, but grammatically correct prose that ChatGPT spits out is considered good.
Example 3: ChatGPT writes the world’s most generic sales email
Eran: Our company has launched a new analytics tool called TimeCharter. It connects to tools such as Outlook and Google Calendar and creates a dashboard displaying how employees are spending their working hours. We want to run an email outreach campaign targeting VPs at midmarket companies, where the goal is to get them to schedule a demo of the new product. Write the first email.
ChatGPT: Subject: Introducing TimeCharter - The Solution to Maximizing Productivity in Your Midmarket Company
Dear [VP's name],
I hope this email finds you well. I wanted to take a moment to introduce you to TimeCharter, a new analytics tool that connects to popular calendar applications such as Outlook and Google Calendar. The tool creates a dashboard that displays how your employees are spending their working hours, allowing you to identify areas of inefficiency and improve productivity.
TimeCharter is specifically designed for midmarket companies like yours, and we believe it can have a significant impact on the performance of your business. We would love the opportunity to show you a demo of the product and answer any questions you may have.
Would you be available for a demo next week? We can schedule a time that works best for you.
Thank you for your time, and I look forward to hearing from you.
Best regards,
[Your Name]
Again, you can be impressed by the fact that ChatGPT managed to tease out the more obvious benefits - which were not provided in the prompt - and include them in the email. I’m not saying this isn’t impressive! I’m just saying that the email itself sucks. It has no personality, no pacing, and no hook to keep the reader going. It reads like a generic sales email written by the world’s most average marketer. Is it better than nothing? Yes. But send this email to an outreach list and you’re burning perfectly good prospects…
Plagiarism is an issue.
I’ve been running the GPT outputs through a plagiarism detector. In many cases, there are no issues at all. In others, up to 30% of the content is copied verbatim from elsewhere on the internet.
It’s similar to the problem with factual accuracy. The tools screw up, and you don’t know when they’re going to screw up, which means you need to check every single text. Not a deal breaker on its own, but the whole idea of using these tools is to save time - and we’re getting further and further away from that.
The workflow is tedious.
What we get from all of the above is that if you care about the quality of what you’re publishing, you need to take the output you got from ChatGPT and:
Review every factual claim for accuracy
Run it through a plagiarism checker, and revise plagiarised parts if there are any.
Spruce up the bland writing.
This can still save time and in some cases might be easier than starting from scratch. But it’s hardly as hands-free and instant as the hype implies.
Would you use this to write something that means a lot to you?
Finally, let me ask you - if you had to write some very important piece of text, would you use ChatGPT for it? Something like:
The cover letter for your dream job
An email to your boss detailing why you deserve a raise
The copy for an expensive billboard ad
I’m going to go out on a limb here and say you would absolutely not use the output from ChatGPT or GPT-3 for these scenarios, because you know that it’s not good enough.
And yet for some reason, you think it’s fine to waste your prospects’ time with this bland, inaccurate, and derivative content when they’re reading your blog posts, your emails, or your About page. I hate to break it to you, but this isn’t such a great idea. If most of what you’re writing falls into the “who cares, nobody’s going to read it anyway” category, maybe you should take a step back and re-evaluate your marketing strategy.
When is it actually useful?
Above I’ve covered (most of) the bear case for AI writing in its current state, but it can be useful in some scenarios. These include:
Spammy SEO. Google has made some statements implying they would crack down on AI-generated content. In practice I see a lot of AI sites dominating the SERPs, especially if they had some modicum of authority before they went full-AI. While most serious companies (correctly) view it as too risky to execute at scale, this does work for the time being. If you’re the type of person who wants to make a quick buck by littering the internet with big globs of text, and don’t really care if they’re particularly good or accurate, this is a viable path.
Summarization: This only kind of works, but I’ve found some success in using the AI tools to summarize reputable sources - and more limited success in feeding summaries from a few sources back to the AI and asking it to write something new. There are still major issues and the tools will absolutely miss important points from the sources they summarize (they are significantly worse at summarization than a human with a highlighter).
Bland and generic content on topics the AI has plenty of training data about. I personally would not recommend my clients write this kind of content, but if they want to publish dozens of “what is [technical concept]” articles and are willing to risk angering the Google overlords, GPT tools can be a good solution.
Foreseeing some of the objections to this article
“Your prompts weren’t good enough.”
This piece is long enough as it is, so I didn’t include every single variation of prompt. You’ll have to believe me when I say changing the prompts didn’t help with the core issues. Better ‘prompt engineering’ has great results occasionally, but like with all things OpenAI - it’s impossible to predict when it will have a massive positive impact and when it will do nothing (or make things worse).
“But all these people say they’re using it to do all these things!”
Yeah, I don’t really buy it. ChatGPT is the ultimate engagement-bait. You get some impressive looking result, take a screenshot, write a LinkedIn post about how you’re using it to save 76 hours a week, hit publish, and bask in the warm thrill of Likes. It’s really easy to get something that looks great for a specific prompt but will completely fail the next time you try to use it. Most of the people claiming they’re using it every day are just bullshitting.
This thread is worth reading:
“A freelancer from Fiverr could also make these mistakes!”
I am not advocating you hire one.
“Google isn’t any better.”
It often is.
You can at least see what the source is.
I’m not advocating you use the first results in Google as your main source for writing about complex topics.
“You’re just afraid it will take your job.”
Not really? It’s true that I make most of my money these days from writing, but I’m pretty well-positioned to do other things, or to become an AI-wielding marketer if that’s what the future market will dictate. I’m not too worried.
“But what it CAN do is amazing!”
I agree it’s amazing. I’m just saying it’s not very useful right now.
“It will get better.”
Okay, I promise to give it a fair shake when it does.
Bottom line: Should you use AI writing tools?
You might think my answer would be a resounding no, but really it’s more nuanced than that. In their current state and for my specific set of use cases, I found these tools to be almost entirely useless; you might have a different experience – they’re better for some scenarios and industries than others.
My advice would be to give it a try and see how it works out for you. I’m still going to try and use these tools and wait to see if they get any better — or maybe I’ll get better at finding ways to incorporate them in my workflows.
What I wouldn’t do is just take the stuff they spit out and use it as-is. Treat it as a first draft by a writer who is both a bit stupid and a pathological liar. In some cases you’ll find this is a helpful thing to have; in others, you’ll find it’s not worth the frustration. Use your judgement — and definitely don’t take the word of two-bit influencers chasing engagement on Twitter or LinkedIn.
At the end of the day, marketing is all about etching a tiny mark in your prospect’s brain. This will never be achieved by spitting out endless paragraphs of the same ‘educational’ content that all your competitors have, regardless of whether it’s human or machine-generated. Focus less on filling web pages with words and more on creating something that your prospects will care about.
P.S., if you do decide to use AI in your writing, just use OpenAI’s tools directly (ChatGPT and GPT-3). Don’t pay for overpriced subscriptions to products like Jasper that just wrap the same technology in a UI you don’t need.