How I Add REAL-TIME Web Scraping to Any AI Agent (Bright Data MCP)

FlowiseAI v3 TutorialFresh📅 2025-06-25

Watch on YouTube ↗

Integration Flow

┌─────────────────────────────────────────────────────────────────────┐
│                      INTEGRATION ARCHITECTURE                       │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   USER REQUEST                                                      │
│        │                                                            │
│        ▼                                                            │
│   ┌─────────────┐         ┌─────────────┐         ┌─────────────┐  │
│   │             │  ────►  │             │  ────►  │             │  │
│   │   Google   │         │    OpenAI   │         │      API    │  │
│   │             │  ◄────  │             │  ◄────  │             │  │
│   └─────────────┘         └─────────────┘         └─────────────┘  │
│        │                        │                       │           │
│        │                        │                       │           │
│        └────────────────────────┴───────────────────────┘           │
│                                 │                                   │
│                                 ▼                                   │
│                          ┌───────────┐                              │
│                          │  RESULT   │                              │
│                          └───────────┘                              │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Integration Points

• Agent configuration
• Tool integration
• Memory management

Transcript

[00:00] Hello and welcome back. In today's video, I'm going to show you how to add realtime web scraping to your AI agents. We can do all sorts of really cool things with our agents. We can assign tools, get them to answer questions from a custom knowledge base, and much much more. But in order to get these agents to retrieve realtime data from websites can be quite a challenge. One way to retrieve information from the web is to assign a web search tool. On this channel, we've had a look at a few of those different tools. We could assign

[00:31] Brave Search, Tavly API, or perhaps something like SER API, which we've used a lot on this channel, but these tools have their own limitations. All these tools really do is perform a Google search, and they're not really scraping real-time information from websites. Let me show you. Let's ask the agent to summarize the first three articles on this web page. So we're expecting it to summarize the introduction to codeex, the introduction to open AI 03 at04 mini

[01:02] and the introduction to GPT4.1 in the API. Right? So back in flowwise, let's say go to openai.com/news and summarize the first three articles. Let's send this. And we're getting this message saying I'm unable to directly access or summarize the current three articles from OpenAI as my search results are only returning a general site description and not a specific article. And we can see the agent used

[01:33] this tool to perform a couple of Google searches. But this is really just the equivalent of doing a search on Google and then looking at the search results only. For the most part, this can be very useful as some of the search results might return the information that we want. But the search tools do not allow the agents to actually open up the resulting websites and then scrape the data from these pages. You will find the same limitation when asking for product information from Amazon as an example. So, if we had to ask our agent

[02:04] to go onto Amazon and find laptops with RTX5080 GPUs, all we'll get back is the general search results from the search engine and not the information from Amazon.com directly. So, how can we actually get our agent to scrape data in real time? If you've ever dealt with web scrapers before, you probably know what a challenge it can be. Websites can block web scrapers in many different ways. As an example, it can block your IP address or block the script with things like captures, dynamic content,

[02:36] or pop-ups. Now, the tool I'm going to show you today is able to access any website and scrape its content, and it can bypass all of those challenges all together. So, for this, we will be using Bright Data's MCP server. Now, if you're unfamiliar with what an MCP server is, it's a simple way of assigning tools to agents. And on the website, they call it a limitless web data infrastructure for AI and BI. The server comes with some very useful tools. It also allows our

[03:07] agent to search the web. It can use the unlocker API to extract information from any website and it also gives us a headless browser for agents. In short, this server gives your agent access to all the tools it needs to interact with the web. Let's start by signing up for our account. You can find the link in the description. From the dashboard page, we don't have to do much. All we want is to generate an API key. To do that, go to account settings. Then under API keys, simply click on these three

[03:39] dots and select refresh key. Then simply copy this key, note it down somewhere, and click on okay. Now let's go ahead and add the server to Flowwise. So back in our agent under tools, let's add a new tool. And in the list, let's search for custom MCP. Then under parameters, we need to paste in this MCP servers config. To get that configuration, we can go to the GitHub page, which I'll link in the description as well. And

[04:11] then scrolling down, let's look for this quick start guide. And from this example, all we have to copy is everything from this starting curly brace after bright data. So everything from here down to about the second curly brace over there. Let's copy this and let's add it to flowwise. Now let's add our API key. So we'll replace this with our API token. Then for the web unlocker zone and this is the thing that will

[04:41] allow our agent to extract data from any website. If we don't specify this value, it will default to MCP unlocker. Now to ensure that this name is correct, we can go back to this bright data website. Then let's go to proxies and scraping. And then you can find the zone name over here. Mine is actually called web unlocker one. So I'm going to add that to my web unlocker zone. And if you wanted to use the headless browser, you can simply add it over here. I'm not

[05:11] going to use it in this video, but if you wanted to set it up, simply go to proxies and scraping, then click on browser API. You can then give it a name and click on add. I'm not going to use it, but just remember whatever name you give it here, just use that same name in Flowwise. I'm going to remove it. So, let's remove all of these. And that's actually all we need. Let's click on save. Then, let's refresh these actions. And now we can see all the tools that

[05:41] are available in the server. And this also means that we've successfully connected to the server as well. So I'm going to select a few of these for this tutorial. But there are plenty of tools here. So scrape as HTML and markdown. You'll use these to scrape data from any website. So I'm actually going to add scrape from markdown. Then let's have a look at what else we have access to. We also have access to a search engine. So let's add that. And this also means we can actually remove SER API as we'll be

[06:14] using this MCP server as our search engine as well. All right, continuing with this list, we can retrieve data for Amazon products. So I'm actually going to do that in this tutorial. So let's add that tool. And I also want to perform Amazon product search. So I'll add that as well. So, I'll only be using these tools in this tutorial, but just to give you an idea of how many different tools there are. When we scroll down, we can retrieve information from the Apple App Store. We can scrape Best Buy, Booking, Crunch Base, eBay,

[06:47] Etsy, Facebook, GitHub, Google Maps, Home Depot, Instagram, and this list just keeps going and going and going. So, let's actually give these actions a spin. And let's save this flow. And in the chat, let's try the same example again. I'm going to clear the chat. And now let's say go to openai.com/news and summarize the first three articles. All right. Now, let's see if this is different to using SER API. I can already tell that we are receiving the

[07:18] correct information back. Just looking at the OpenAI website again, we are expecting the first three articles to be the one on codeex 03 and 04 mini and GPT 4.1 in the API. And looking at flowwise, we've got the article on codeex openai 03 and 04 mini as well as GPT 4.1. Let's try another example. Let's scrape data from amazon.com. Let's say go to amazon.com and search for RTX5080

[07:48] laptops. Return the product name, price, and specs. Let's send this. At the time of recording, this tool does seem to have a small bug, but this is how you can resolve this issue very easily. This message is simply saying that this parameter pages to search expects a string value and not a number. So what we can do is open our agent. Then let's add a system message and let's say when using the Amazon search tool the pages

[08:21] to search parameter should be a string and not a number. Cool. Let's save this. Then let's try this again. Go to amazon.com and search for RTX 5080 laptops. Return the price, product name, and specs. Let's send this. And this time we're getting results back. So, this is just a small

[08:51] tip for resolving MCP related issues. Looking at the results, we can see this tool was indeed called and we're getting these laptops back. This also seems to correspond to the listings that I received as well. And what you can do is click on these links to open them in a new tab. I hope you found this video useful. If you did, consider hitting the like button and subscribe to my channel for more FlowWise content. I also want to thank Bright Data for making this video possible. And I highly recommend checking out this MCP server. And by the way, YouTube thinks that you'll enjoy

[09:22] this next video. So, click on the card and I'll see you in the next one. Bye-bye.