It’s been touted as a revolutionary timesaver – but what about the errors?
Almost overnight, ChatGPT has become a rare tech innovation that transcends boundaries and becomes a cultural phenomenon. Some have called its release the iPhone moment for artificial intelligence (AI) because of the way that it has cut across industries, jobs, and sections of society.
One of those areas is its potential use as a coding tool. Plenty of blogs have sprung up touting the use of the large language model-based AI chat tool as a coding assistant, or even an outright programmer to craft new code. Many include advice on what prompts to provide the tool in order to get it to write the best possible code for you.
But is the situation more complicated? Can ChatGPT really spell the end of the developer? Not everyone is convinced.
The case against ChatGPT
Developer blogger Scott Hanselman – who works for Microsoft, which recently invested a rumored $10 billion into OpenAI, the company behind ChatGPT – warned that the tool should be used with caution.
Yes, it could make developers 10 times faster, Hanselman tweeted – provided they understand what the model is outputting. “Otherwise it will make the code wrong, 10x faster,” he wrote. (Microsoft turned down a request to speak to Hanselman for this story.)
Joe Youngblood, who describes himself not as a developer by training but as a marketing consultant and futurologist, has tried to use ChatGPT to clean up code that he and his team of developers has already written, looking for incremental improvements in his company’s codebase.
He does that by asking ChatGPT to write the same code as he already has, and looks to see if the generated code works better. “So far, we’ve found it’s frequently much, much worse,” Youngblood says. “It works really well when you’re a developer and you know what you’re looking for and what you’re asking,” he explains. “It doesn’t work as great if you say: ‘Hey, I would like code written for this.’”
Youngblood says that ChatGPT can spit out convincing enough code, but scratch beneath the surface and you realize it’s “frequently very incorrect and very inaccurate”.
For example, he has asked ChatGPT to write a WordPress plugin, but found that it’s not possible to trigger the plugin. “Right now, it’s a good assistant,” he says. “It’s similar in utility to GitHub’s Copilot, but it’s not replacing the need to hire a developer just yet. If you’re not a developer, it’s a pretty useless tool.”
The case for ChatGPT
“My experience has been that if you don’t know what you’re doing, and you want to do something extremely simple, ChatGPT will get you like 80 to 85% of the way there,” says Pietro Schirano, design lead at financial company Brex.
For instance, the developer wanted to build a tool that would translate code from one language to another, and ChatGPT managed to get the tool working almost perfectly, with just minor manual tweaks needed from Schirano. “Its ability to break down a problem for an engineer or designer is really good,” he says. Schirano is a believer in ChatGPT as a coder’s friend – a sounding board, of which you can ask questions and get what is likely to be a correct (or correct enough) answer.
“It makes the process of finding things a whole lot easier, and a whole lot more fluid,” says Sridhar Ramaswamy, CEO of large language model-powered search engine Neeva, and former head of Google’s advertising business. “One of the issues with modern tech is that because of good tooling, but also because of an explosion of things like libraries, it’s impossible for one person to keep everything that you might potentially need in your head,” he says.
When Ramaswamy started at Google, he found the idea of needing to have all of these details at his fingertips frustrating. “There’s so much stuff available that it simply isn’t feasible for you to remember everything,” he says.
Which is why he uses ChatGPT as a memory jogger. While talking, he demonstrates how he uses the generative AI chatbot: Ramaswamy knew that it was possible to code something in Python that read a CSV file and processed the columns, but couldn’t remember what module to import. So he typed in a query asking ChatGPT to jog his memory – which it did.
“It gave this code and off I go,” he says. “Because I know this area well, I don’t have to worry about things like: ‘Is this buggy?’”
Don’t quit your day job
That is the inherent issue in tools like ChatGPT, and why Ramaswamy believes they can be a useful additional tool for, but not replacement of, developers.
“If you make it write complete programs, you always have to worry the thing can’t quite get it right,” he says.
ChatGPT and other generative AI tools can hallucinate answers that aren’t based in reality, including when working with code. That is why StackOverflow has banned the use of any AI-generated code on its popular developer Q&A platform. In its explanation, the site’s admins said: “Because the average rate of getting correct answers from ChatGPT is too low, the posting of answers created by ChatGPT is substantially harmful to the site and to users who are asking and looking for correct answers.”
Still, Ramaswamy says that the tool makes things “magically faster” if used right. It’s not and should never be considered a standalone programmer, he says. But it could be considered a helper. “The programmer that’s doing the work is able to apply their judgment about how they’re using it,” says Ramaswamy.
ChatGPT provides a tool that can smooth out that process. “It replaces your manual pages and endless trawling through StackOverflow, and a whole pile of really bad SEO sites for simple questions,” he says. “It replaces all of them, it’s much nicer to use, and it can do custom work for you as you go along.”
