Automate Your Browser with AI! Build a Computer Using Agent (OpenAI API)
OpenAI Responses API Tutorial (Python)Recent📅 2025-04-02
Build Process
┌─────────────────────────────────────────────────────────────────────┐
│ BUILD AI AGENT WORKFLOW │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ PLANNING │
│ ──────── │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Define │───►│ Choose │───►│ Setup │ │
│ │ Scope │ │ Stack │ │ Project │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │
│ ▼ │
│ DEVELOPMENT │
│ ─────────── │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Build │───►│ Add AI │───►│ Test & │ │
│ │ Features │ │ Logic │ │ Debug │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │
│ ▼ │
│ DEPLOYMENT │
│ ────────── │
│ ┌─────────────┐ ┌─────────────┐ │
│ │ Deploy │───►│ AI Agent │ ◄── LIVE! │
│ │ to Cloud │ │ Running │ │
│ └─────────────┘ └─────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘Key Components
- Project architecture
- Core features implementation
- Best practices
Transcript
[00:00] In this video, you will learn how to add computer use to your AI agents. Computer using agents, or CUAs for short, are agents that can use applications on your machine to perform tasks on your behalf. For example, we could ask our agent to find an image of Emma Watson, and the agent will automatically open up the browser and navigate to something like Google Images, perform a search, and return the result. So, in a nutshell, how this works is you would set up some kind of environment
[00:30] that the agent can interact with. For example, we could set up something like a browser environment, which we'll do in this tutorial, and the agent would use frameworks like Playwright to navigate to different browser pages. Alternatively, you could set up a virtual machine, and the agent could interact with the machine by executing commands. In this video, we'll simply set up a browser environment, and then ask our agent to go online and perform certain tasks for us. Effectively, how this works is we would
[01:01] initiate our agent by asking a question, like, "Find an image of Emma Watson." The agent will respond with some kind of action. For instance, if we look at the initial landing page of our browser, and then decide to maybe enter a value or click on a button, we then programmatically need to implement this actual click function, and then execute that click on this browser environment. We will then take a snapshot of what the browser currently looks like, and then pass that back into the model.
[01:32] The model will then re-evaluate where it is in the browser, and then decide on the next course of action. This is called the CUALoop, and trust me, it seems more complicated than it is, and we will implement all of this during the course of this tutorial. Here, I've set up some very basic boilerplate code, and of course, you can download all of this code for free by using the link in the description of this video. All we're doing here is we've set up some input messages, and this message simply says, "Find an image of Emma Watson."
[02:04] We don't have any tools at the moment. We're extenstiating our OpenAI class, and then sending a message to GPT40 Mini. So if we run this in the terminal, which simply says that, "I can't browse the Internet for images." Right, this is because we're not using a computer use model at the moment. So the first thing we need to change here is to use a computer use model. So for model, enter computer use preview. Now, we also have to assign a special
[02:35] kind of tool to this agent, which will allow it to interact with a browser. So in our tools list, let's add a new dictionary, and let's enter a key called type, which is equal to computer use preview. Now, we have to set the display width. There is something like 1024. So all we're doing now is we're just kind of explaining what the environment looks like. So this would be a browser with a width of 1024, a height of 768, and for the environment,
[03:06] it will be set as browser, as we will be building a browser use agent. Of course, if you were using a virtual machine or a terminal, you would just have to adjust this environment. Now, on the create method, we just have to add two keys. First, we have to add reasoning, which is equal to a dictionary, and here we have to pass in generate summary as concise. At the time of recording, concise seems to be the only option that works. If I use detailed, I will receive an
[03:38] error saying that this model currently does not support detailed reasoning summary. So let's simply select concise. And then finally, let's add another property called truncation, which is equal to auto. Okay, so let's go ahead and actually run this, as I do want to show you what this output looks like. So far, we're only asking our agent to find an image of Emma Watson, but because we're now passing in a computer use model, it's no longer telling us that it can't use the internet.
[04:09] It's not giving us a response saying response computer tool call, and in here, we get an action, which is off type screenshot. This will become important as we have to implement each of these actions in our code. Then we also get this call ID, which we will use, and we also get this type called computer call. So effectively, what this agent is telling us at this stage is it can't see the browser. So please, can you send me a screenshot of what the browser window currently looks like?
[04:40] Now, at this stage, we haven't set up the browser at all. So we can't even take a screenshot at this stage. And this is why we're going to set up Playwright, because Playwright will create this little virtual browser environment that we can play with. So what we first need to do is install Playwright. So in the terminal, run pip install and Playwright. And this will now download Playwright and install all of these dependencies. So now at the top of our code,
[05:11] let's enter from Playwright.syncapi. Let's import sync Playwright. So what we can do in our code now is to start up a little browser interface using Playwright. And ideally, I want to do this before we call our agent. So what I'm going to do is create a new function called main. And then with in main, I'm just going to add all of this to main.
[05:41] And then with in main, I'm going to type with sync Playwright as p. We can then create a new variable called browser, which is equal to p.chromium, which is referring to a chromium browser.launch. And this takes in a few arguments. First, we will set headless to false, because we actually want to see the interface. And then we also want to set chromium sandbox to true.
[06:14] And for env, we simply want to pass in an empty dictionary. And finally, we want to pass in a few arcs, which is equal to a list. And we just have to pass in two entries. The first is dash dash disable extension. And for the second one, we want to pass in disable dash file dash system. Now, so far, this won't do much, as we only have an instance of the browser.
[06:45] And now the first thing we want to do is to actually open up a new page. So it's like a new tab in the browser. Let's do that by entering page, which is equal to a browser dot new page, like so. Let's also set the viewport size of this browser, so that it's the same as the display with the night that we passed in our tool. So let's enter page dot set underscore viewport size,
[07:15] which takes in the width and height. And these values are the same as the value sent to our model in the tool. Now, we can actually navigate to the first page. So let's say navigate to initial URL. So let's say page dot go to. And just to keep things interesting, let's go to HTTPS colon. And we'll just go to bing dot com. Everyone's favorite landing page, right?
[07:45] I'm going to set the second argument, which I didn't see in the open AI documentation. And without this, I actually ran into plenty of issues. Effectively, what would happen is we would take a screenshot before the page finished loading, which meant we took a screenshot of a partially rendered page and send that back to our agent. And the agent would then make all sorts of mistakes. So let's add wait until. And the value that we want to pass in is DOM content loaded. So this means we will now wait until the
[08:16] DOM has finished rendering. Now, we also want to add our responses API call into this logic as well. So it's also nested within this play right code block. I'm also just going to add a comment above this responses API just to say create initial response. Now finally, all we have to do is call our main function. All right, so let's go ahead and run this in the terminal. And the first time you run this, you will get this error message. And this is simply telling us that we
[08:47] have to install play right. So all we have to do is copy this command play right install and then run that in our terminal. And this will download and install all the dependencies for using play right. Okay, so now off the play rights installed, we can try to run the script again. And when you run this, you will actually see the browser popping up, but it goes away extremely quick. So it's really hard to show during this recording. It briefly shows up and then it closes again. We will fix that in a second.
[09:17] But basically what's happening is when we run this main function play right instantiates this browser instance, it then opens up a new tab. It then sets the viewport size and it navigates to Bing. Then the window will stay open until we receive the response from the AI. And when this completes that window closes again at the moment. These two things are working independent. We've got play right that allows us to interact with the browser and at the moment this agent doesn't have a view of that play right window.
[09:49] It's simply asking us to share a screenshot of that window so that they can figure out what to do next. So if I just show you this diagram again, we've implemented this capability of calling this computer use model and we've set up our computer environment or that browser environment over here. Now our agent is asking us to show us a screenshot of that environment so that it can decide which action to take next and we will then call that action interact with environment and then send back the latest screenshot.
[10:21] And as per this diagram, all of this needs to happen in a loop. Let's go ahead and implement this feedback loop. For this, I'm going to create a new function just above mine and let's call this function computer use loop. This will take in a couple of arguments. First, we want to pass in the browser, also the page, as well as the response from this initial LLM call. Now you might be wondering why we need the browser and not just the page.
[10:52] Well, I found a very interesting situation when using this in a real example. Certain actions might spawn off different browser tabs. So what could happen is the agent might click on a link that opens up a new tab and now the agent no longer has a view of that new tab. It will only see the initial page that it generated. So if for some reason a new tab opens up, we want to be able to interact with that tab as well. Let's say while true. So we'll run this loop until we write
[11:23] some code to break out of it. And the first thing I want to check is if the agent passed in any computer call related actions. If not, then we want to break out of this loop. So let's get all the computer call actions from the output by creating this variable called computer calls, which is equal to. And now I'm going to write some crazy code, but this is simply generating a list from the output where any output type is equal to computer call. So let's say item for
[11:55] item in response dot output. If the item dot type is equal to computer call. So you might recall when we had a look at the output that was generated, the output contained a list of values. And for the value that we had a look at the type was computer call. And all we're saying here is grab all of the values in that list where the item type equals computer call
[12:26] and then add it to this list over here. And now we can simply add a check to say if not computer calls, which means we no longer have any further computer calls, which means effectively we are done. Then I'll simply print no more computer calls. And instead of exiting, let's say output from model like so. And just below this, we can simply show all the output from the model. So let's say for item in
[12:58] response output, simply print item. And finally, we can break out of this loop. So I'll just add a comment saying exit with no computer calls are issued. All right. So this piece of code will simply exit out of this loop once we're done processing. So for the next step, let's take that screenshot of the browser window and let's pass it back to our agent. Now in order to generate a screenshot, all we have to do is call page.screenshot like this.
[13:28] Now, I think it's a good idea to add this to its own function. So just before our loop, I'm going to create a function called get screenshot, which takes in the page and all this will do is return page.screenshot. Cool. So what we can do now is generate that screenshot and use it in our code. So let's say screenshot bytes is equal to our screenshot function and we'll send our page to this function. Now this will return the screenshot in bytes
[13:59] and what we really want is the screenshot in base64 format. So we can do that by creating a variable called screenshot underscore base64, which is equal to base64, which we have to import. So at the top of the code, let's also import base64 and then in our code, let's call base64.b64 encode and we should use the encode function and
[14:31] not the decode function and what we want to encode is screenshot bytes. And then on this result, we want to call decode and we'll set UTF-8. Now that we have the screenshot in base64 format, we can pass it to the responses API. If you're new to passing images to the responses API, then I do have a dedicated video on working with images with in this responses API. But in summary, we can send URLs or we can send base64 encoded images.
[15:03] Now let's call our agent again and this time we'll pass the screenshot to the agent. Let's say response is equal to client.responses.create and for the model, we'll use computer use preview again. Then we also have to pass in the conversation history, which we can do by calling previous response ID is equal to response.id. We already covered previous response ID in a previous video as well.
[15:35] So you might want to check out that video. This simply means we don't have to manually keep track of the conversation history. Then we'll pass in our list of tools again, which is equal to tools. Then for the input, we will pass in a list and this will take in a dictionary and here we have to pass in a few properties as well. We need to pass in the call ID and we will populate this in a second. We also have to set the type as computer call output
[16:10] and then we also have to specify an output property. This is a dictionary with two properties. The first one is type, which is equal to input image and secondly, we'll pass in the image URL and this is an F string with data colon image slash PNG semi colon base 64 comma and we'll pass in our screenshot
[16:40] and the base 64 version of the screenshot. Now let's take a moment to explain this code. Now at this stage, we would have performed some action. So maybe we took a screenshot or we click the button or we enter text, something happened. That instruction from the agent contained a call ID and we then have a look at that property earlier on in this video. Now what the agent wants us to do is to send this input response with that same call ID and then to attach this image, the
[17:12] screenshot so that it can figure out what to do next. Now we can get this call ID from our list of computer calls. So each of these computer call instructions will have an associated call ID and what the agent wants to do is for us to call the action per computer call and return the response of that action in this input field. Now it's very easy to get that ID. So at the top of this code just below our if statement,
[17:42] I'm actually going to write this comment that says we expect at most one computer call per response. So in my experience, I've typically only received one entry in computer calls. So the agent would say something like take a screenshot or click a button or enter text, never more than one instruction at a time. So I'll go with the assumption especially for this tutorial that we'll only have one computer call. So let's create a variable called
[18:13] computer underscore call, which is equal to our computer calls list and we'll simply grab the first entry. Now let's also get a variable called last call ID, which is equal to computer underscore call dot call ID. We can now grab this variable and pass that in as our call ID. Right, let's continue with our create method. After input, let's also add truncation,
[18:45] which is equal to auto. And finally, outside of our loop, I'm simply going to return response. And then back in our main function, we can now call that loop. So let's do final underscore response is equal to computer use loop. We'll pass in the browser, the page and our initial response. And then it's also print this response. So it's a final response and we'll just
[19:16] send final response dot output underscore text. And after we've printed the final output, we can simply close the browser window. This will happen automatically, but I prefer to ensure that it does get closed correctly. All right. Well, this should run in an infinite loop, but let's start this in any case. I'll just stop it halfway through execution. So let's run the script. This brought up the browser window and is actually closed because our agent responded with a type message.
[19:48] And it said I'm unable to help with that request. This does happen from time to time. Sometimes you just have to restart the script. I see it happened again. So let's adjust this prompt slightly. Let's say open Bing and search for images of Emma Watson. Then click on the first image that you found. All right. Let's see if that made any change. And I can see it's opened the browser and looking at the console,
[20:19] we can see that we did receive a computer call type message and for the action, it asked us to send a screenshot. All right. So the script was running for a long time. So I think this infinite loop was causing a lot of delays. So within our computer use loop directly after we call this create method, I'm simply going to print the output from this call and immediately break this loop. We will remove this break in a minute, but just to test this
[20:49] out, let's run the script. So we get our browser window and first off, we can see this type of computer call and an action of screenshot. So hopefully this code has not taken the screenshot and it's busy sending it to our agent. All right. This is way better. Let's have a look at this. The first response was the one that asked us for the screenshot. So this type was screenshot and then we took the screenshot and we send it back to the agent and this
[21:20] type is still computer call. Now let's have a look at this action. This action involves a button click. So it's a left mouse click and this is of type click. We can also see the X and Y coordinates of that mouse click. So clearly the agent looked at the screenshot and decided to make a left mouse click at this position. I want to show you a very cool trick for troubleshooting this. At the moment, we can't see the screenshot, right? It was sent to the agent and
[21:50] the agent simply responded. But what if we wanted to see what the agent saw? We can do that by creating another utility function. So maybe just above get screenshot. Let's add show image. This takes in the base 64 version of the image. Then it calls the base 64 library and it then decodes the image and then it uses this image library to show the image. So we just have to install and import this as well.
[22:22] So in the terminal, let's run pip install and pillow. And then at the top of the code, let's enter the following from pil pillow import image. And we also have the import bytes IO. So at the top of the file, let's enter from IO import bytes IO. Now what we can do is after we've taken the screenshot, we can now call show image and then pass in the base 64
[22:54] version of the screenshot. Now for interest sake, if I run this in the terminal, our browser window opens without taking the screenshot, which actually generated this image. So now I can see exactly what the agent sees. And then finally, the agent wants to perform a mouse click at these coordinates. And if I had to assume it's probably trying to click on the input box, the search box. Okay, so this does mean that we have to implement these actions. So we have to give our agent the ability
[23:24] to now actually click on the page to handle all of these different actions. I am going to create a new helper function called handle model action. And this also takes in the browser, the page and the action that the agent is trying to perform. This action object contains things like the type of action and the arguments for that action. In our code, let's grab the action type, which we can get from action dot type. Now I do want to mention that a lot of this code has more to do
[23:54] with how playwright works. And it's not really related to the responses API in particular. So I'm going to copy and paste a lot of these actions across because this video is going to be too long if I also have to explain how playwright works. But we will implement one or two of these actions together. First, let's write these in a try catch block. So if something goes wrong with one of these actions, we will have some traceability of it. And as I mentioned earlier, we might run into a situation with a browser open up a new tab.
[24:25] So let's deal with any new tabs. So let's say all pages is equal to browser dot contexts and we'll just grab the first value dot pages. So this will give us all the tabs that are available in our browser at the moment. Then let's say if length all pages is greater than one and all pages and we'll simply grab the last page that's open is
[24:56] not equal to the current page. Then we want to switch our active page over to the last active tab. So what would happen if I just run through this manually is the agent might search for something like Emma Watson and it might then actually go to something like images. And the moment I clicked on images, it opened up a new tab. Now, if we don't switch to the new tab, all the screenshots that we send to the agent will still reflect this page and
[25:28] you will end up in a bit of an infinite loop. As the agent will try to click on this link repeatedly, not knowing that it's simply creating new tabs the whole time. So all we're doing here is switching over to the last active tab. Now just that we can see what's going on, we will print switch to a new tab to the terminal. Cool. Now we know we are interacting with the current active tab. Now to perform these actions, let's add a case statement on action type.
[26:00] And the first example we saw was on the click action, that mouse click. So what we can say is case if it's a click action, then we want to get those X and Y coordinates. So we can get those from action dot X and action dot Y. Then it's also grab button from action dot button. This will tell us if this is a left or right mouse click. Let's also print this. So we'll just say
[26:31] clicking at these coordinates with button that. And now all we have to do is start playwright to now click in that position. So these are both in functions within playwright. So let's say page dot mouse dot click. And all we have to do is pass in the X and Y coordinates as well as the button that should be clicked like the left mouse button. We will implement additional actions in a minute, but let's simply test this left mouse click for now. So outside of this match
[27:03] statement, let's return page. So this will simply return the new active page back to our loop. All right. So let's go to our computer use loop. And one thing we have to do is get the action. So maybe below last call ID, let's create a new variable called action, which is equal to computer call dot action. And now we can actually go ahead and perform that action. So let's do this. Let's call handle model action. This function takes in our browser.
[27:36] It also takes in our page and our action. And because this function returns the new active page, we'll just change our page variable as well. It can take some time for this action to be executed. So it might make sense to add a short wait statement. So let's add time dot sleep and we'll just wait one second after each action. And we just have to import time as well. So at the top of the code, let's import time. And I think that's all we need. So let's
[28:07] try this now. Let's run the script and we can see our browser window. And this also generated this image and this ran and it simply closed the browser. Oh, yes. And that is because we have this break statement. So let's actually remove this break statement. Cool. So we went to terminate the processing after the first call. And let's run this again. This might cause an infinite loop again, but we'll simply terminate this during processing. So here we can see that reasoning
[28:38] summary. It's waiting for the action of type screenshot. It's also generated this image for me and something very interesting just happened. This dropdown came up and I did not click this by the way. This was our agent clicking in this input button. And it's also generated this new screenshot and this screenshot was passed to our agent. I'm just going to terminate processing here as we will be running in a loop. But looking at the actions first, it asks us to send a screenshot. Then on the second call, it asks us to
[29:10] perform an action by clicking the left mouse button in these coordinates. And that put the focus on the input field and now saying for the action enter the text in my Watson and the type now is now type. So we tried to enter some text into the field. Now we don't yet have an action for type. So that's what we need to implement next. And in fact, to speed things up, I am going to paste in a whole bunch of additional actions. Now again, these are more related to
[29:41] playwright itself and not really the responses API. So if you would like to learn more about using playwright and all the different actions, then I'm sure there are plenty of documentation and tutorials on playwright. But to get this agent to work, we can add a scroll action. So it might need the ability to scroll up and down the page. Let's also add this action called key press. Let's also add the action for type, which is the action they tried to perform, right? They tried to enter the words in my
[30:12] Watson. So now it should be able to do that. And in some instances, the agent might actually ask us to wait. So it is also able to identify scenarios that might need some time to process. And then finally, I'll just add this catch all for any actions that are not recognized. So with these few actions, we should now have an agent that should be able to look up images of Emma Watson and then click on the very first image. Of course, enter any prompt that you want. So you can go to websites, maybe
[30:45] navigate to products, etc. Now take note that I'm not doing anything. My hands are actually off the keyboard and mouse and let's see what this does. So it's clearly clicked on the input field. It's entered Emma Watson. And in the terminal, we can see those actions as well. It's clicked on the search button, which brought up these results. Let's see if it's able to go to images, which it does. And hopefully you just saw what happened. It opened up a new tab. So if we didn't add the logic to deal with new tabs, the agent would only be
[31:17] able to see the results from the very first tab. Most likely forcing you into an infinite loop. Okay, so the agent clicked on the first image that it found. And as this is running, I can see all the screenshots being generated off screen. So I can see exactly what the agent sees, which goes a long way in troubleshooting this. Trust me. And there you have it. You now know how to add computer use to your AI agents. So if you followed along, then thank you for going on this journey with me.
[31:48] If the scores helped you at all, then please consider hitting the like button and subscribing to my channel. I'll see you in the next one.