Watch and Learn | Get Structured Response From Open AI Using Function Calls

With function calling you can still get some hallucinations from the system, but at least those are going to be structured hallucinations.

Using Function Calling we will build an app called Movie Nerd. You just enter the name of the movie and the API will return release date of the movie, actors, director, summary of the movie, most memorable lines from the movie and to add a bit weirdness to this whole process we will ask Chat GPT to generate description of the movie poster, that we will send to DALL-E to generate an Image of the poster which almost always gives a weird and kind of creepy result.

So let’s get started.

Setup

For this demo I’m using Next.js 13 with Tailwind installed for a bit of styling. You will also need Open AI API key if you wanna follow along with this tutorial. That key needs to be saved to the .env file at the root of your Next.js app.

We are going to be using using openai npm package to talk to ChatGPT and DALL-E. Just be sure to use at least version 3.3.0 of the module since that is the one that supports Function Calling.

If you are having troubles with Open AI setup you can checkout this post for more information about this setup. And of course everything we do here will be available on Github, the link is in the description below.

Only note worthy thing is index.js in pages folder that is currently just displaying an input box and a button.

The code for the file looks like this:

import Head from 'next/head'
import {useState} from "react";

export default function Home() {
  const [loading, setLoading] = useState(false)
  const [data, setData] = useState([])


  const handleSubmit = async (e) => {
    e.preventDefault()

    setLoading(true)

    const movieName = e.target.movieName.value

    const res = await fetch(`/api/get-movie-data?movie_name=${movieName}`)
    const data = await res.json()

    console.log(JSON.parse(data));

    setData(JSON.parse(data))

    setLoading(false)
  }

  return (
    <>
      <Head>
        <title>Movie Nerd</title>
        <meta name="description" content="Generated by create next app"/>
        <meta name="viewport" content="width=device-width, initial-scale=1"/>
        <link rel="icon" href="/favicon.ico"/>
      </Head>
      <main className="max-w-3xl mx-auto py-10">
        <div>

          <form className="mt-1 mb-10" onSubmit={(e) => handleSubmit(e)}>
            <label htmlFor="movieName" className="mb-2 block text-lg font-medium text-gray-500 prose">
              Enter the movie
            </label>

            <input
              type="text"
              name="movieName"
              id="movieName"
              className="block w-full rounded-md border-gray-500 shadow-sm focus:border-indigo-500 focus:ring-indigo-500 sm:text-sm p-5"
              placeholder="Enter name of your movie"
            />

            <button
              type="submit"
              className="mt-5 w-1/2 mx-auto flex w-full justify-center rounded-md border border-transparent bg-indigo-600 py-2 px-4 text-sm font-medium text-white shadow-sm hover:bg-indigo-700 focus:outline-none focus:ring-2 focus:ring-indigo-500 focus:ring-offset-2"
            >
              Get me the movie data
            </button>
          </form>

          {loading && <div className="text-white">Please wait...</div>}

        </div>
      </main>
    </>
  )
}

We have our form, a loading state, and handleSubmit function that is going to trigger get-movie-data endpoint and send it the name of the movie. Of course, if you try clicking the “Get me the movie data” button right now you will get a 404 error. Because that route still doesn’t exist.

Let’s fix that.

Defining Open AI Call

In the pages/api folder create a file called get-movie-data.js.

Next, initialise Open AI API like this:

const {Configuration, OpenAIApi} = require("openai");
const configuration = new Configuration({
  apiKey: process.env.OPENAI_KEY,
});
const openai = new OpenAIApi(configuration);

Now we are going to export default handler function, that receives request and a response.

export default async function handler(req, res) {
  
}

In the handler function we are going to make our request to the Open AI API. Like this:

const response = await openai.createChatCompletion({
	model: "gpt-3.5-turbo-0613",
   messages: [
	    {"role": "system", "content": "You are a movie database assistant."},
	    {"role": "user", "content": `Give me information about movie called: ${req.query.movie_name}`}
    ],
    functions: [
        {name: "get_movie_data", "parameters": schema}
    ],
    function_call: {name: "get_movie_data"},
    temperature: 0,
});

Let’s go through this code for a second. We are using createChatCompletion function to talk to ChatGPT, this is the function that supports Function Calls.

Next we are defining a model - for this to work you need to use either gpt-3.5-turbo-0613 or gpt-4-0613 but since I don’t have access to GPT-4 I will use 3.5.

In the previous AI video on this channel we were sending a prompt to the Chat GPT. We are not going to do that today. Instead we are going to send messages array. The first one primes Chat GPT to be our movie database assistant. And the next one is our prompt. So we say “Give me information about movie called:”, and then we send in the movie name from the request object.

Next, we are defining functions. Meaning you can send more than one function to the API. But for this use case one function is enough. We are giving it a name of get_movie_data and then sending in parameters. The parameter in this case is going to be a schema that we are going to build out during this video.

After that we just do a function_call and call our get_movie_data function.

This is all the code we need to talk to Open AI and get our data. The only thing we need to worry about now is defining our schema to get the structured response from Open AI.

And to send that data to our frontend we just need to add this one liner:

res.status(200).json(response.data.choices[0].message.function_call.arguments)

And this should send the response from Open AI to our App so we can display it.

Defining the schema

Now we need to define a schema, this is the most important part. But luckily it’s very easy to do.

Our schema is going to be of type object and is going to have some properties.

const schema = {
	type: "object",
	properties: {
	
	}
}

First property that we are going to add is release_date it is going to be of type string . We can also provide a description so that Chat GPT has context of what we wanna get. We can just write something like: “Release date of the movie in Day.Month.Year format”. You can think of it like a mini Chat GPT prompt.

const schema = {
	type: "object",
	properties: {
		release_date: {
	      type: "string",
	      description: "Release date of the movie in Day.Month.Year format"
	   },
	}
}

There is one more thing that we need to do, to get a response is to define required fields. Like this:

const schema = {
	type: "object",
	properties: {
		release_date: {
	      type: "string",
	      description: "Release date of the movie in Day.Month.Year format"
	   },
	},
	required: ["release_date"]
}

We just add a required array to the end of the schema and fill it with fields that we wanna get from the API.

Ok, so now let’s see if this works. Go to your browser, and enter a name of the movie in the input, for example Inception. Click “Get me the movie data” button.

Since we didn’t setup any data displays on our page, we will check our console to see what we are getting from the API.

And after a few seconds we get back an object with the release_date property and the movie release date.

Nice.

So now let’s get the actors for the movie. We follow the same steps as for release_date but this time our property type is going to be an array. So we have to set it up a bit differently. Since this is an array we also need to define how the objects in that array will look.

actors: {
	type: "array",
   description: "Get the names of first five most important actors in this movie",
	 items: {
        type: "object",
        properties: {
          first_name: {type: "string"},
          last_name: {type: "string"}
        }
	 }
},

For that we use items property and in it define that our item is going to be an object that is going to consist of two fields first_name and last_name both of those fields are going to be strings.

And that is it.

Now just add actors to the required array and we can test this out.

required: ["release_date", "actors"]

Enter Inception again and let’s see what we get from the API.

We are getting an array of first five actors for this movie. Great!

Now I will add a few more fields to our schema, so that now it looks like this:

const schema = {
  type: "object",
  properties: {
    release_date: {
      type: "string",
      description: "Release date of the movie in Day.Month.Year format"
    },
    actors: {
      type: "array",
      description: "Get the names of first five most important actors in this movie",
      items: {
        type: "object",
        properties: {
          first_name: {type: "string"},
          last_name: {type: "string"}
        }
      }
    },
    director: {
      type: "string",
      description: "Director of the movie"
    },
    summary: {
      type: "string",
      description: "Provide a brief summary of the movie remove any double quotes from the text."
    },
    quotes: {
      type: "array",
      description: "Provide three most memorable lines from this movie",
      items: {
        type: "object",
        properties: {
          quote: {
            type: "string",
          },
          actor: {
            type: "string",
            description: "Actor that is saying the line."
          },
          person: {
            type: "string",
            description: "Character in the movie saying the line."
          }
        }
      }
    },
    poster_description: {
      type: "string",
      description: "Accurate description of the movie poster for this movie in seven sentences."
    }
  },
  required: ["release_date", "actors", "director", "summary", "quotes", "poster_description"]
}

I added a director, summary of the movie, quotes array that is going to provide us with 3 most memorable lines from the movie, character in the movie saying the line and the actor playing it.

And lastly we defined poster_description field that is going to ask ChatGPT to provide a description of the movie poster in seven sentences, so that we can send that to DALL-E later on.

Also note that I instructed ChatGPT to avoid double quotes for the movie summary because that could screw up our response.

Let’s test this out with the movie Big Lebowski since this will get us some nice quotes.

We are probably going to have to wait a bit longer for this response since we are requesting much more data.

Ok it works, “this code really ties the app together”.

Before we implement those creepy poster images for the movie, let’s display the data on our page.

Display response on the page

I’m just going to add this code to below loading indicator:

{Object.keys(data).length > 0 && !loading && (
	<ul className="text-white mb-8">
	  <li className="mb-4"><strong>Release date:</strong> {data.release_date}</li>
	  <li className="mb-4">
		 <strong>Actors:</strong>
		 <ul className="ml-4 mt-4">
			{data.actors.map((actor, index) => (
			  <li key={index}>{actor.first_name} {actor.last_name}</li>
			))}
		 </ul>
	  </li>
	  <li className="mb-4"><strong>Director:</strong> {data.director}</li>
	  <li className="mb-4"><strong>Summary:</strong> {data.summary}</li>
	  <li className="mb-4">
		 <strong>Quotes:</strong>
		 <ul className="ml-4 mt-4">
			{data.quotes.map((quote, index) => (
			  <li key={index} className="mb-4">
				 <figure>
					<blockquote>
					  {quote.quote}
					</blockquote>
					<figcaption>
						—{quote.person}, <cite>played by {quote.actor}</cite>        </figcaption>
				 </figure>
			  </li>
			))}
		 </ul>
	  </li>
	</ul>
 )}

Here we are just checking if we are getting something from the API, and if we do we are displaying data inside of unordered list.

Since actors and quotes are arrays we are mapping through them. Nothing complicated.

Let’s test this out with a movie Goodfellas.

Nice, we are getting the data on the page. Now lets implement those creepy images.

Setup Movie Image Endpoint

Create get-movie-image.js inside the API folder, initialise Open AI API and define handler function. Just like we did for get-movie-data.js

const {Configuration, OpenAIApi} = require("openai");  
const configuration = new Configuration({  
apiKey: process.env.OPENAI_KEY,  
});  
const openai = new OpenAIApi(configuration);

export default async function handler(req, res) {
  
}

In the handler function we are going to call createImage function from openai module.

const response = await openai.createImage({
	
})

We need to define prompt for this API, which is going to be poster_description that we got from ChatGPT. Number of images, in our case 1, and the size. I’m going with ‘512x512’. And lastly we are just returning response from the API to our frontend.

const response = await openai.createImage({
	prompt: `${req.query.poster_description}`,
	n: 1,
	size: "512x512"
 })


 res.status(200).json(response.data.data[0].url)

And that is it for the API. We just need to implement image creation on our page.

Displaying DALL-E image on the page

In index.js add two states, for image loading and image URL.

const [imageLoading, setImageLoading] = useState(false)  
const [imageUrl, setImageUrl] = useState(undefined)

After that we will define getImage function that will get the image and set it to the imageUrl state.

const getImage = async (movieDescription) => {
 setImageLoading(true)
 const resImage = await fetch(`/api/get-movie-image?poster_description=${movieDescription}`)
 const image = await resImage.json()

 setImageUrl(image)

 setImageLoading(false)
}

Now we will use useEffect that is going to have data dependency. So when the data changes a.k.a we get the movie data response we will trigger getImage function and send it poster_description.

useEffect(() => {
 if(data.poster_description) {
	getImage(data.poster_description)
 }
},[data])

Below movie data I will add loader for imageUrl.

{imageLoading && <div className="text-white">Image loading, please wait...</div>}

And below that we display our image if we get anything from the DALL-E API.

{imageUrl && !imageLoading && (
	<div>
	  <img src={imageUrl} alt="" />
	</div>
)}

And that should be it. Save it and let’s test it in the browser. Write Top Gun in the box for example.

This response is going to take a bit longer since first we are contacting Chat GPT to get movie data and after that we will contact DALL-E API to get the image.

That image is weird, but at least we are getting something, so that is nice.

It would probably be better to send that description to Midjurney API, I’m sure that we would get much better results. But for the sake of brevity and simplicity we are using Open AI APIs all the way in this demo.

Note that if you don’t send anything to the API or send movie name that doesn’t exist it will make stuff up for you, so you can use this as an imaginary movie generator or something like that.

Error Handling

Just one note before I conclude this video - It’s good to wrap your requests to the API in a try catch block, so that you get some useful info if something breaks.


try {
 const response = await openai.createImage({
	prompt: `${req.query.poster_description}`,
	n: 1,
	size: "512x512"
 })


 res.status(200).json(response.data.data[0].url)

} catch (error) {
 if (error.response) {
	console.log(error.response.status);
	console.log(error.response.data);
 } else {
	console.log(error.message);
 }
}

While doing the prep for this post I would sometimes get the message from the API that the description for the poster is using unsafe words, and then it wouldn’t generate the image. I wouldn’t know what the problem was if I didn’t wrap it in a try catch block. So keep that in mind.

Conclusion

As you can see using AI to be your backend of sorts can be very powerful, and this is just scratching the surface. You can do much more complicated things with this approach. But that’s a topic for another post.

Get Structured Response From Open AI Using Function Calls