OpenAI Vision API Crash Course - Chat with Images (Node)
OpenAIAging📅 2024-03-13
Integration Flow
┌─────────────────────────────────────────────────────────────────────┐
│ INTEGRATION ARCHITECTURE │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ USER REQUEST │
│ │ │
│ ▼ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ │ ────► │ │ ────► │ │ │
│ │ OpenAI │ │ API │ │ Output │ │
│ │ │ ◄──── │ │ ◄──── │ │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │ │ │
│ │ │ │ │
│ └────────────────────────┴───────────────────────┘ │
│ │ │
│ ▼ │
│ ┌───────────┐ │
│ │ RESULT │ │
│ └───────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘Integration Points
- API authentication
- Request handling
- Error management
Transcript
[00:00] The OpenAI Vision API has been available for a while now, but I haven't seen anyone talk about one of the most important use cases for this API in production applications. If you're unfamiliar with the Vision API, it allows you to pass images to an LLM like GPT-4, and you can then ask the model questions about these images. For example, I can ask the model to describe this image, and the model will respond with an accurate description of the image. But what I found recently is that this
[00:30] model is actually really good at extracting unstructured data from images as well. Take this menu for example. This is simply a PNG image with a lot of unstructured data in it, like the name of the restaurant, we've got different courses, we've got the contact details for the restaurant, the address, etc. I can actually use the Vision API to extract any information that I want from this menu and return it in a JSON structure, which we can use in our applications.
[01:01] This can be incredibly useful in applications where you want to analyze things like invoices, receipts, and statements from JPEG or PNG images. And in the terminal, I can actually show you an example of what that would look like. So in this example, I simply ask the model to return all the information on the menu in a JSON structure. So here we can see the restaurant name. We can also see all the menu items, and we also get the individual items along with their prices. And we also receive the contact
[01:33] information like the phone number and address in this structure as well. So in this video, we will have a look at calling the OpenAI Vision API in a node environment, so a fundamental understanding of JavaScript would be beneficial. But please let me know in the comments if you would like to see a Python version of this video as well. To get started, let's open a code editor like dscode. Let's initialize a node environment by opening the integrated terminal, and let's run npm
[02:04] init -y. This should create this package.json file. And if this didn't work for you, then please ensure to install node.js on your machine by going to nodejs.org and then downloading this LTS version of node. Let's open the package.json file, and just below main, let's add a new property called type, and let's set that equal to module. That will allow us to use the import syntax instead of the require syntax when importing packages. Let's also install a few dependencies.
[02:36] So let's run npm install openai as well as .env. We will use a .env file to store our OpenAI API key. In fact, let's do that now. Let's create a new file, and let's call it .env. In this file, create a new environment variable called openai API key and set that equal to your openai API key. I actually have a dedicated video on creating openai API keys,
[03:07] but in a nutshell, go to this URL, then click on create new secret key, give it a name, click on create secret key, copy this key value, and then add it to this variable, like so. I will delete this key after this recording, so please ensure to use your own API key. We can now close this file, and let's create a new file, let's call it app.js, and we can now start writing our code. Let's start by importing a few things.
[03:38] First, I want to make our environment variable available to this code. Let's start by importing all as .env from .env and then .env.config. There's only one other import that we need, and that is openai. Now we only need to import the openai class. Let's import something from openai, and what we need is the openai class.
[04:10] Let's instantiate the new instance of openai by creating a variable called openai, which is equal to new openai, like so. We are now able to interact with the openai APIs using this object. Let's test this out by calling openai.chat.completions.create. This is how we can pass messages and prompts to an openai model. This method takes in an object as input,
[04:42] and we can specify the model which we would like to use. And just to test this out, let's just go with the GPT 3.5 Turbo model. We will change this to the vision model in a second, but let's go with this model simply to ensure that this is working. We also have to pass a messages property, which is an array of values, as we are able to pass in multiple messages, like the system message, a user message, etc. Each message is an object, and each of
[05:12] these objects have a role property, which can be a system, user, function, or tool. It simply pass in a user message, and this also takes in a content property, which is also an array of values, and you will see why this is an array in a minute. But for simple instructions like this, we only need one object with a type of text, and for the text, we can simply pass in something like "hello". Once we start working with the images, we
[05:44] will actually pass more than one structure. One will contain the text, like describe the image, and the second object will actually contain the image itself. But for a simple prompt, this is good enough. Now this method is asynchronous, so we do have to await its response, and let's assign the response to a variable called "response", like so. Now let's simply console.log "response.choices", and we'll simply grab the first value in "choices",
[06:14] and let's go ahead and run this in the terminal. Let's enter "node app.js", and we can see that this is working since we are receiving a response from our model. Now let's move on to the fun stuff. Let's see how we can pass images into this model. We will have a look at two implementations of this in this video. First we will pass in a URL to an image, and second we will pass in a local image which we will store on our server. Let's first have a look at the URL.
[06:45] So the first thing we need to do is to change this model to the GPT vision model, which is called... In fact if I press control spacebar, I can see all the available values, and what we want is this GPT4 vision preview model. Now we just need to change these messages slightly. We will still pass in a role of user, since we are passing this instruction to the model, but let's change the text to something like "describe this image". Now we also have to pass the URL to the
[07:17] image by adding another object to this array. Let's add the type, and let's set this to image URL. Then we also have to pass an image_url property, and this takes an object as input, and the first property that we have to pass in is the URL, and optionally we can also pass in this detail property, but let's leave that for now. Now you can go ahead and find any image that is hosted online and grab the URL from that
[07:49] image. You will simply go with this image over here, right click, copy image address, and then paste that into this value. So if you want, you can actually pass in this detail property, and this will simply resize the image, since larger images will use up more tokens, and if possible it's recommended to use a lower version of the image instead. And by setting detail to low, this API will first resize the
[08:19] image before passing it to the model. Let's go ahead and test this in the terminal. We do get a response back, which seems accurate, but you might notice that this text does get cut off halfway through the sentence, and that is because the max token limit for the vision API is extremely low by default. We can increase this token limit by going back to this create method, and just below messages, which is just below this final array, we can add another property called
[08:50] max tokens, and we can now increase this value, and if we run this again in the terminal, the result should be greatly improved, which it is. So before we move on to loading images locally from our machine, I do want to ask you a massive favor. Please hit the like button, subscribe to my channel, and leave a comment. Now let's have a look at loading images from the server. So I've created this new folder called images, and within images, I have this menu.png file, and this simply contains some information
[09:21] about the business, the different courses, the prices for the items, etc. So let's have a look at how we can change this code to work with local images. So basically how this works is within this URL property, we are able to pass in a URL to an image, or we can pass in base64 encoded image data instead. So we will grab the image from the server, convert it into a base64 representation, and then pass that into
[09:52] this URL property over here. Don't worry, it's actually easier than you think. First I'm going to remove this URL over here, and before we call this create method, let's load this image. In order to load images from the file system, we have to import FS from FS. There's no need to install this as a dependency, as this is natively supported by node. So above response, let's load the image by calling FS.readFileSync, and now we have
[10:25] to pass in the location to that image, which is images slash menu dot PNG. This method takes in a second property, which is an object, and here we can pass in the encoding as base64. Now all we have to do is assign this to a variable, which I will call base64Image. So this variable will now contain the base64 encoded representation of this image, which we can pass to our model.
[10:56] So we can actually go down to this URL property. So you might be tempted to simply pass in the base64 image as a value, but in order to tell the model what we're working with, we have to pass in a few more information. So instead of using a string, I'm actually going to pass in bactics and change this to a template string, which starts with data colon image slash, I'm just going to hardcode this as PNG. But of course, feel free to
[11:26] programmatically get the extension from the file, semicolon base64 comma, and now we can pass in the name of that variable, which was base64Image. And since we are dealing with text, I'm actually going to remove flow to improve the quality of the response. And now all we have to do is change this prompt text to something like create a JSON structure for all the items on the menu, return only the JSON structure.
[11:57] And now let's go ahead and test this in the terminal. And we do get this JSON structure coming back. And we can actually improve this output in two ways. First, this changes console log to be a bit more specific. So let's add dot message dot content. And we can also improve the quality of the response by adding a system message to this list of messages as well. So let's add an object is also add a comma, let's add a role as system.
[12:27] And for the content, it's add an object with type text. And it's also add the content property. And for content, we will pass in an array with one object. And this object takes in the type, which is text. And for the text value, we can pass in something like return a JSON structure based on the requirements of the user only returned the JSON structure. Nothing else did not return quote, quote,
[12:57] quote, JSON is run this again in the terminal. And we can now see that the response has been greatly improved. I hope you can see the potential in using this API to fetch unstructured data from images. Now that we have the response in a JSON format, we can convert the text into a JavaScript object and work with it in our applications. Another fascinating approach is to combine this technique with a framework like Lang chain, we can use output parsers to create a consistent JSON structure using this data,
[13:29] check out my Lang chain JS series over here.