My experience with ChatGPT

Published on: Tue Dec 27 2022

Introduction

What is ChatGPT? Actually, I’m going to let ChatGPT introduce himself to you:

Hi there! I’m ChatGPT, a chatbot tool developed by OpenAI that uses the GPT language model to generate human-like text. I’ve been trained on a massive dataset of human conversation, so I can produce responses that are natural, coherent, and relevant to the conversation at hand. With me, developers can easily create chatbots that can hold engaging and informative conversations on a wide range of topics. These chatbots can handle tasks like answering questions, providing recommendations, and engaging in casual conversation, all while providing a seamless and engaging user experience. Whether you’re looking to build a customer service chatbot or simply want to create a chatbot for fun, I’m a powerful and versatile tool for creating chatbots that can hold intelligent and engaging conversations.

As you can see, ChatGPT is the next step in actually useful natural language AI. It was built upon the GPT3 language model (so it’s sort of like a GPT 3.5) with additional manual human feedback. The result is an AI that exceeded all of my exepectations of what we can accomplish with natural language AI in the year 2022. Don’t get me wrong, GPT3 and its predecesors were already incredible. However, as you will see in this article, ChatGPT will make you excited and scared for the future and the limits of human productivity.

Disclaimer!: ChatGPT only has access to knowledge up to 2021. So it is not aware of any newer information past 2021. I hope that in the future they will make an architecture that can continuously get updated, but this is what we have for now.

Non-Technical related topics

As you can imagine, I will mostly go into what ChatGPT can do for software related topics. But before we get into those topics, I want to share what I was able to acquire on non-technical topics. This will give you a better picture of how broad ChatGPT’s knowledge is.

Omega 3

Cooking A Goose

Teaching Korean

Legal Documents

As you can see, ChatGPT is only limited by your thirst of knowledge. Ever wondered what the differences between the Harry Potter books and movies are? Ever needed help with a recipee? Need gift ideas for a friend? Why German college is mostly free? Whatever it is, you’ll certainly get something useful out of ChatGPT. And even if it’s not on the first prompt, you can steer it gently in the right direction over a conversation.

Now, let’s get to the technical prompts and how they might make your job of developing software easier.

Research

This is, in my experience, ChatGPT’s strongest side: Its ability to provide useful and condensed information about a new topic.

Instead of googling X, opening 30 tabs and scrolling through forums, you can save yourself a lot of time by just asking ChatGPT and continously prompting more information.

A practical example: I acquired a project where another software company wanted to modernize their legacy windows desktop application, that was literally written in 1994 and last updated 2007. It was a collection of flash animations and interactive elements. So to minimize effort I was wondering if we could do something with the existing flash files and not have to rewrite everything in HTMl5 Canvas and JavaScript. This is my conversation with ChatGPT:

Honestly this blew my mind. I had already done some manual google research the week before, since I hadn’t gotten to use ChatGPT at that time. But the results it gave me in a matter of minutes were equally, if not better, useful than my 3 hours of searching through articles, YouTube and Google search result pages.

Code Generation

This one is very interesting. I have gotten incredible, mediocre and completely results in this category.

A very impressive feat I got was when I asked ChatGPT about generating code for a Mandelbrot set. It’s one of many fractals and looks very pretty.

I copied the HTML it gave me 1:1 and put it on the following page: mandelbrot

This honestly blew me away… first try and it just works. And the most amazing part is that you can now go into a dialog with ChatGPT and ask it to make any modifications you’d like. “Make it less colorful”, “Use TypeScript”, “Rewrite it with React”, “Make it compatible with IE9”, … All of this will be executed beautifully by ChatGPT.

Now to take away some excitement, there were some flaws when I asked it to do more fancy things. For example I wanted it to do a version using WASM and C. So it would generate the mandelbrot points in C and send them over to JavaScript by compiling it to WASM and rendering it in the canvas.

It honestly failed miserably. It gave me incorrect instructions to install emscripten on my linux system. It provided me wrong C code, that didn’t export anything to WASM. It gave me wrong JavaScript code that didn’t import the .wasm file correctly and gave an error in the Browser. In this case I needed to go back to googleing again and had to fall back to my own coding knowledge to fix most of what ChatGPT gave me.

So the tl;dr is that you might get better results if you break down the task for ChatGPT and don’t ask it to do big tasks in one step. For example I could’ve asked it first to generate a function in C to generate the mandelbrot set, then ask it how to export and compile a C function in WebAssembly, then how to import a WebAssembly module in the browser and then how to call it in JavaScript. That way you might get better individiual results that you can then yourself piece together. However you’ll need good knowledge of the task and technologies at hand, so this will probably not work well in an area that you never touched before.

Advent of Code

This one blew my mind. I had already solved the AOC22 Day 11 problem in TypeScript, but wanted to see what ChatGPT would make out of it.

Your puzzle input for this problem was a list of monkeys that looks like the following:

Monkey 0:
  Starting items: 79, 98
  Operation: new = old * 19
  Test: divisible by 23
    If true: throw to monkey 2
    If false: throw to monkey 3

Monkey 1:
  Starting items: 54, 65, 75, 74
  Operation: new = old + 6
  Test: divisible by 19
    If true: throw to monkey 2
    If false: throw to monkey 0

...

This is what ChatGPT came up with to parse the monkey object:

Prompt: I've got a text with a description of objects that I need to parse into a TypeScript object. Here's what it looks like in raw text form:

Monkey 0:
 Starting items: 79, 98
 Operation: new = old * 19
 Test: divisible by 23
  If true: throw to monkey 2
  If false: throw to monkey 3

class Monkey {
  startingItems: number[];
  operation: string;
  test: {
    divisibleBy: number;
    throwTo: number;
  };

  constructor(startingItems: number[], operation: string, test: { divisibleBy: number, throwTo: number }) {
    this.startingItems = startingItems;
    this.operation = operation;
    this.test = test;
  }
}

function parseText(text: string): Monkey {
  // Split the text into lines
  const lines = text.split('\n');

  // Extract the starting items, operation, and test information from the lines
  const startingItems = lines[1].trim().split(':')[1].trim().split(',').map(Number);
  const operation = lines[2].trim().split(':')[1].trim();
  const testDivisibleBy = Number(lines[3].trim().split('by')[1].trim());
  const throwToTrue = Number(lines[4].trim().split(':')[1].trim().split('monkey')[1].trim());
  const throwToFalse = Number(lines[5].trim().split(':')[1].trim().split('monkey')[1].trim());

  // Create a new Monkey object with the extracted information
  const monkey = new Monkey(startingItems, operation, { divisibleBy: testDivisibleBy, throwTo: throwToTrue });

  // Return the Monkey object
  return monkey;
}

const text = 'Monkey 0:\n  Starting items: 79, 98\n  Operation: new = old * 19\n  Test: divisible by 23\n    If true: throw to monkey 2\n    If false: throw to monkey 3';
const monkey = parseText(text);

After asking it to get the text from a file, it gave me this code:

const fs = require('fs');

function parseTextFromFile(filePath: string): Monkey {
  // Read the text from the file
  const text = fs.readFileSync(filePath, 'utf8');

  // Use the parseText function from the previous example to parse the text and create a Monkey object
  return parseText(text);
}

// Example usage
const monkey = parseTextFromFile('path/to/file.txt');

As you might’ve noticed, the monkey object is actually not being parsed correctly. Currently it only has one “throwTo” property, but we need to save another one for the false case.

Thankfully ChatGPT is pretty intelligent and corrects this issue after nudging it into the right direction:

Prompt: "We are missing a property for the monkey. You need to save which monkey to throw to for both the true and false case

class Monkey {
  startingItems: number[];
  operation: string;
  test: {
    divisibleBy: number;
    throwTo: number;
    throwToFalse: number;
  };

  constructor(startingItems: number[], operation: string, test: { divisibleBy: number, throwTo: number, throwToFalse: number }) {
    this.startingItems = startingItems;
    this.operation = operation;
    this.test = test;
  }
}

function parseText(text: string): Monkey {
  // Split the text into lines
  const lines = text.split('\n');

  // Extract the starting items, operation, and test information from the lines
  const startingItems = lines[1].trim().split(':')[1].trim().split(',').map(Number);
  const operation = lines[2].trim().split(':')[1].trim();
  const testDivisibleBy = Number(lines[3].trim().split('by')[1].trim());
  const throwToTrue = Number(lines[4].trim().split(':')[1].trim().split('monkey')[1].trim());
  const throwToFalse = Number(lines[5].trim().split(':')[1].trim().split('monkey')[1].trim());

  // Create a new Monkey object with the extracted information
  const monkey = new Monkey(startingItems, operation, { divisibleBy: testDivisibleBy, throwTo: throwToTrue, throwToFalse });

  // Return the Monkey object
  return monkey;
}

It even corrected the function that parses the text!

Now this is very impressive, it can parse out puzzle input. But can it actually solve the advent of code problem?

Let’s find out.

After each monkey inspects an item but before it tests your worry level, your relief that the monkey's inspection didn't damage the item causes your worry level to be divided by three and rounded down to the nearest integer.

The monkeys take turns inspecting and throwing items. On a single monkey’s turn, it inspects and throws all of the items it is holding one at a time and in the order listed. Monkey 0 goes first, then monkey 1, and so on until each monkey has had one turn. The process of each monkey taking a single turn is called a round.

When a monkey throws an item to another monkey, the item goes on the end of the recipient monkey’s list. A monkey that starts a round with no items could end up inspecting and throwing many items by the time its turn comes around. If a monkey is holding no items at the start of its turn, its turn ends.

Figure out which monkeys to chase by counting how many items they inspect over 20 rounds.

function countInspections(monkeys: Monkey[]): number[] {
  // Initialize an array to store the counts for each monkey
  const counts = Array(monkeys.length).fill(0);

  // Initialize an array to store the items that each monkey is holding
  const items = Array(monkeys.length).fill([]);

  // Simulate 20 rounds
  for (let round = 0; round < 20; round++) {
    // Loop through the monkeys in order
    for (let i = 0; i < monkeys.length; i++) {
      // Get the current monkey and its items
      const monkey = monkeys[i];
      const monkeyItems = items[i];

      // Loop through the monkey's items
      for (let j = 0; j < monkeyItems.length; j++) {
        // Get the current item
        const item = monkeyItems[j];

        // Increment the count for this monkey
        counts[i]++;

        // Divide the worry level by three and round down to the nearest integer
        let worryLevel = Math.floor(item / 3);

        // Check if the item passes the test
        if (item % monkey.test.divisibleBy === 0) {
          // If the test passes, throw the item to the monkey specified in the "throwTo" property
          items[monkey.test.throwTo].push(worryLevel);
        } else {
          // If the test fails, throw the item to the monkey specified in the "throwToFalse" property
          items[monkey.test.throwToFalse].push(worryLevel);
        }
      }

      // Clear the monkey's items after its turn
      items[i] = [];
    }
  }

  // Return the counts for each monkey
  return counts;
}

const monkeys = [/* Array of Monkey objects */];
const counts = countInspections(monkeys);

Believe it or not, but this is correct! I simulated it and it returned the correct answer.

Let’s try to make it solve the second part of the advent of code problem:

You're worried you might not ever get your items back. So worried, in fact, that your relief that a monkey's inspection didn't damage an item no longer causes your worry level to be divided by three.

Unfortunately, that relief was all that was keeping your worry levels from reaching ridiculous levels. You’ll need to find another way to keep your worry levels manageable.

At this rate, you might be putting up with these monkeys for a very long time - possibly 10000 rounds!

Worry levels are no longer divided by three after each item is inspected; you’ll need to find another way to keep your worry levels manageable.

Sadly, the code it gave me was a lot of nonsense. Its plan was to reduce the worry level by a fixed amount like subtracting 50 of taking off a fixed amount of 10%. I tried to nudge it in the right direction, since I knew that the answer was to find the greatest common denominator of all division tests and modulo % the current item by that to keep the division test correct but the numbers low.

But it wasn’t smart enough to figure this out on its own, even after giving it some hints.

Funnily enough, when I opened a new chat with ChatGPT and specifically asked it to solve an abstract version of this problem, it gave the correct answer:

So again, you might get stuck with more complicated and bigger tasks. But breaking them into smaller problems or starting a new chat with ChatGPT often can still get you correct results.

Nim

This was a short, but pleasant prompt. I was learning the nim programming language.

All I wanted to do was to split a string and reverse the order. Honestly it’s impressive how bad the nim docs and the google results are, just based on this use case. I couldn’t find anything to help my case for literally an hour.

Frustrated I went to ChatGPT and asked “How do I split a string and reverse the order in nim”. Here’s some of the code I got back:

import strutils

let s = "Hello, World!"
let words = s.split() # split the string into a sequence of words
let reversedWords = rev(words) # reverse the order of the words

# print the reversed words
for word in reversedWords:
    echo word

let s = "Hello, World!"
let words = s.split(",") # split the string on commas
let reversedWords = rev(words) # reverse the order of the words

# print the reversed words
for word in reversedWords:
    echo word

Literally only took one minute. It’s just a much better alternative to scrolling endlessly on StackOverflow and just copy and pasting something.

Bottom Line

The bottom line is that ChatGPT is very useful for a variety of tasks. Research, code snippets, debugging errors, solving math equations, and much more. In the short span of time that I had the fortune to use it, it helped me with actual real world problems, that I was facing as someone who develops software and runs a company.

ChatGPT is one of the most incredible technologies I’ve seen this decade. And the fact that this is only the first version of this conversational AI is kind of scary. I can’t wait for it to become more specialized in many areas so it become your lawyer, doctor, software developer, teacher, cook or architect. I think we only saw a glimpse of what it’s going to be able to do in a couple years further down the line.