Make Your RAG Agents Actually Work! (No More Hallucinations)
Flowise AI (2024) TutorialAging📅 2024-11-07
Build Process
┌─────────────────────────────────────────────────────────────────────┐
│ BUILD CHATBOT WORKFLOW │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ PLANNING │
│ ──────── │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Define │───►│ Choose │───►│ Setup │ │
│ │ Scope │ │ Stack │ │ Project │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │
│ ▼ │
│ DEVELOPMENT │
│ ─────────── │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Build │───►│ Add AI │───►│ Test & │ │
│ │ Features │ │ Logic │ │ Debug │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │
│ ▼ │
│ DEPLOYMENT │
│ ────────── │
│ ┌─────────────┐ ┌─────────────┐ │
│ │ Deploy │───►│ Chatbot │ ◄── LIVE! │
│ │ to Cloud │ │ Running │ │
│ └─────────────┘ └─────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘Key Components
- Agent configuration
- Tool integration
- Memory management
Transcript
[00:00] In this video, we will push the boundaries of what is possible with llama 3.2, a tiny open-source model that you can run on your own machine by building this advanced multi-stage RAG agent. This flow might look intimidating, but let's simplify it before we jump into the detail. With a traditional RAG agent, you would have a user sending a question to an AI agent. So the agent will use an LLM to try and answer the user's question and respond with some answer.
[00:30] Now of course, we can also add RAG to this flow by adding some sort of additional knowledge to the flow, like a vector store, a database, a PDF file, or it could just be pretty much anything else. So the agent will try to do a semantic search to retrieve the most relevant information from the data source and inject that into the LLMs prompt. Now this agent will work fine for very simple use cases, but it can also introduce some problems. For example, when the agent reaches out to the vector store, it's very
[01:02] possible that the documents returned from the database are not of a very high quality or not related to the user's question. The accuracy of the documents returned can depend on quite a few things, like the vector store being used or the embedding model that was used. So bad data can lead to hallucinations, and that is where the agent will kind of make up its own answers, which are not factually rooted in what is being returned from the knowledge base. And secondly, the agent will try its best to
[01:33] use the data returned from the knowledge store to come up with some sort of answer, and that response might actually not answer the user's original question. So what we can do instead is build agentic RAG systems. These workflows execute several steps that will ensure that the quality of the responses are greatly improved. This implements several concepts that are being used today to build these advanced RAG agents, first of which is routing. With routing, we can direct the user's
[02:04] question down different paths to retrieve the data that's the most relevant to the user's question. For example, for certain questions, we might want to go to our vector store or our knowledge base, or for certain questions, we might want to do a web search instead, for example, to retrieve live information. The second technique that we will implement is called fallback. This means that after we've retrieved the data from the vector store, we will use another LLM to first check if the documents returned are actually
[02:36] relevant to the user's question, and if the documents are not relevant, we can try to do a web search instead. And lastly, we have a technique called self-correction. This means that after we've retrieved the most relevant information, we will use an LLM to try and answer the user's question from the context, but before showing the answer to the user, we will use another LLM to check the answer for any hallucinations and that the answer is actually relevant to the user's question. And
[03:06] if it's not, we can attempt to rephrase the question or simply respond with something like, "I couldn't find the answer." Now, let's visualize these three different techniques, and afterwards, we'll build this flow in Flowwise. As with the normal RAG agent, our user will send the question to our RAG pipeline. Now, this first agent will actually not try to answer the user's question outright, but instead, it will actually look at the user's question and try to determine what the correct data source would be. We
[03:37] will call this a router agent. It could decide to go to the knowledge base like a vector store, or it might determine that a web search would be the best approach. So we will provide some instructions in this agent system prompt to help it make this decision. So for certain information, go to the knowledge base, and for other info, just do a web search. And of course, this knowledge base could be a vector store, a database, a file, or whatever else. Or if we do a web search, we can use something like Davily or SERP API or any of
[04:09] those services, effectively allowing our agent to do a Google search. In respective of which route we decided to take, we should have some information available. So you would think that it would make sense to generate an answer at this point, but we still have to risk that the information coming back from the vector store is still not relevant to the user's question. So what we can do instead is break this connection and then add in another agent into the mix. And this agent is responsible
[04:39] for checking the relevance of the documentation returned from the knowledge base and then provide it with some sort of grade. If these documents are semantically similar to the user's question, then we can go ahead and generate an answer. Or if this agent determines that these documents are not related to the user's question, then it can fall back to this web search instead. So that means we will discard the documents that we retrieved from the knowledge base and attempt to find the
[05:09] answer from the web instead and then try to generate an answer. So that covers routing and fallback. Now let's have a look at the final technique and that's called self-correction. So what we can do is add another agent to this flow that's responsible for checking the answer generated from this LLM to ensure that there are no hallucinations and that the answer actually is relevant to the user's question. If the answer is relevant, we will pass that response back to the user. Or if the answer is not relevant to
[05:42] the user's question or it contains hallucinations, then our agent can simply respond with something like "I don't know". And of course you can take all the techniques that you learned in this video to take this a step further. Where this agent will first try to go to a different data source or even try to rephrase the question for the web search and try again a certain number of times before eventually responding with "I don't know". So in today's video you will learn a lot. And this is a slightly more
[06:12] complex topic but it's definitely worth learning if you want to take your agent game to the next level. So let's finally build this out in Flow Wise. I do want to mention that this example is inspired by this article written by the Lang Chain team where they do something extremely similar but coding it out using Lang Graph in Python. So instead of writing this in Python, we will try to build this using a no-code platform called Flow Wise. And by the way, Flow Wise is also using Lang Graph behind the scenes
[06:43] to make this all work. I also want to mention that you can download this flow for absolutely free and you will find a link to it in the description of this video. This video was a lot of work to create so if you do enjoy it or find any value in it then please hit the like button and subscribe to my channel. So to get started go to Agent Flows and click on Add New. Let's save this flow by giving it a name something like "Multistage RAG Agent". Let's save this. I will assume that you know the basics of using Agent Flows so if you are new to Agent
[07:15] Flows and sequential agents then check out my crash course over here and then come back to this video. Let's start by adding a start node to the canvas. So under sequential agents let's go to start and let's add this node to the canvas. Let's start by adding a chat model. Let's go to Add Nodes, Chat Models, and let's add a chat-alama node because we do want to demonstrate the ability of using small models like Lama 3.2. But of course you're more than welcome to use OpenAI,
[07:45] Anthropic, or any other model. If you're unfamiliar with Olama it's an awesome tool that allows you to run open source models on your own machine. Check out my dedicated video on setting up Olama especially this one that shows you how to build a RAG chatbot using Olama and the Lama 3.2 model. So for the model name let's enter Lama 3.2 and we'll be using the 3 billion parameter model. It's set to temperature to a lower value like 0.2 and let's assign the chat node to the
[08:16] start node. We're not going to add ancient memory in this tutorial but you're free to do it in your version as well. I'll simply add state so under Add Nodes let's add the state node and let's assign it to the start node as well. At the moment the state node does not contain any values but we will use this to control the state and the behavior of this application during the course of this video. So just to test that the LLM is actually up and running I'll simply add an LLM node like so. I'll
[08:46] simply call it LLM and finally let's connect an end node just to complete this process. Let's save this flow and in the chat let's simply enter hey and I do get a response back meaning that the connection to Olama and the Lama 3.2 model is working. I just added this node for testing so I'm going to delete it and instead let's have a look at what we want to build. First we want to receive the user's question and then use an LLM to route the question
[09:17] down different parts. Either we want to retrieve data from the knowledge base or simply perform a web search. So let's start by building this router. Let's go to Add Nodes then under sequential agents we want to do some routing. So either we can use the condition node or the condition agent node. The difference is that with the condition node we can simply look at a specific value and then hard code the path that we need to follow. Because we're not looking at a very simple value we have to intelligently
[09:48] look at the question that the user is sending and use an LLM to decide where to route to. So let's add the condition agent node. Let's attach the start node to this condition agent node and let's call this the router agent and under additional parameters let's set the system prompt as well as a human prompt. In the interest of time I'm actually going to paste in the prompts but as a reminder you can download the entire flow for free from the description of this video. You can then easily
[10:19] import that flow into your flow voice instance and copy across my prompts. But in a nutshell we're saying that if the question is related to things like agents, prompt engineering and adversarial attacks then reach out to the vector store or if the question is not related to those topics then perform a web search. We're also telling this agent to return a JSON structure with a single key called data source which contains the values web search or vector store. We will use
[10:52] these values to figure out where to route the question to and this is simply telling the agent not to try and answer the question itself but to simply return that single value called data source. For the human prompt we can simply create a variable called question and we can then assign a value to that variable by clicking on format prompt values. Then let's click on this edit button, let's click on this field and let's select the question from the chat box. Now of course we want this node to produce a JSON
[11:22] structure with this data source key so I'll actually copy that value then under JSON structured output we can tell this node to return a JSON structure as the response and we can define the exact fields that we want to get back. So I am expecting one field called data source which is of type enum so enum allows us to provide specific enum values that we expect like vector store and web search and for the description
[11:53] I'll simply enter data source. Let's close this pop-up and now all we have to do is set these routing conditions. So we could set them within this table which is easy enough to do but what I don't like about this approach is that we will always have this end output over here. I simply want the routes to either be vector store or web search so under condition I'm actually going to switch over to the condition code menu but don't worry we won't be writing any complex code in this video.
[12:23] Let's click on see example and what we can do is replace this content part with the value that we are outputting in this JSON structure. So just to have a look at that value again we are creating a property called data source so what we can do in the code is to replace content with data source so result will now give us whatever value is captured in data source and we can now see if result includes any of those values. So if result includes vector
[12:54] store then we want to go down the vector store route. I like to keep these two values the same then let's also copy this and paste it below and for this one we want to see if result includes web search and if it does we want to go down the web search path and I do not want this end route. Let's save this and now on this note we can see that we only have the vector store and web search paths. To test this out let's add two LLM nodes to this canvas
[13:26] so I'll add one over here let's call this one the vector store LLM and this also attaches end node for now and it's also attaches vector store route to this LLM node. Let's copy this node let's move it down here let's also copy the end node let's attach it to this LLM node let's attach this web search path to the LLM node as well and let's rename this to web search LLM. Now at the moment these LLM nodes won't do much because we simply want
[13:57] to see if this is actually working so let's give an instruction like respond with vector store route and let's do the same thing for the web search node so we'll just say respond with web search route. Okay let's save this and in the chat let's see if this is working what is a react agent so because this question is referring to agents we should expect this flow to go to the vector search LLM which it does and we can see that our router agent determined that the vector store should
[14:28] be called let's try something else like what is the current weather in New York let's send this and this time we've been sent down the web search path great so that means our router is working so let's go ahead and set up this vector store logic and afterwards we'll work on the web search logic so the first thing we need to do is to set up our knowledge base let's go back to the dashboard let's go to document stores so in the interest of time i've already set up a document store for us and in this document store
[14:59] i'm scraping information from these web pages and i didn't set up the config for this document store to use the olama embedding model and i'm using the faius vector store that simply creates the vector store database on my local machine if you're new to document stores then definitely check out my dedicated video on setting up document stores the right way so back in our agent flow let's improve this LLM node i'm actually going to disconnect this end node and in this vector store LLM node let's do the following for
[15:30] the system prompt we can simply enter use the provided tool to answer the user's question and in the human prompt let's enter a variable called question and in format prompt values let's assign a value to question and that will be the question coming in from the chat window then let's add another note to this canvas and that will be the tool node and then let's assign this LLM node to the tool node i'll simply call this tool node vector store tool now let's go ahead and design our vector store retriever
[16:00] tool so under tools let's add the retriever tool let's assign this to the tool node then for the retriever name i'll call this a knowledge base retriever and for the description we can enter the vector store contains documents related to agents prompt engineering and adversarial attacks finally let's attach our document store so under nodes go all the way down to vector stores and add the document store node and attach it to the retriever tool from the drop
[16:31] down select your document store and i'm also going to enable return source documents so that the user can see where this information was retrieved from at this point let's attach this tool node to this end node let's save this flow and let's test this what is a react agent so in these messages we can see that the vector store tool was indeed called this node called our knowledge base retriever tool passing in the keyword react agent and this is the response that came back from our document store
[17:04] we can also click on this button and this will take us directly to that web page that we scraped there is one change that i do want to make to this tool node the reason for this will make sense as we progress with this video but i do think it's a good idea to set this up now as you saw in the chat this tool node retrieves documents from our document store and then those documents could be passed along to the next node but what i want to do instead is actually write the documents to a state value and we can then use the state
[17:34] value to continue with our process that will also allow other nodes in this flow to override the state value let's go back to our state node let's add a new property called documents and for the operation let's select replace and initially this will have no value so what we can do now in the tool node is once we get a response back we can update the state value by adding a new item then under keys let's select documents and
[18:05] for the value let's select the flow output tool output this property contains the output from calling our vector store tool and we can now close this we are now done setting up the knowledge base retrieval so let's move on to the second route which is the web search so let's go to this web search allo m let's click on additional parameters for the system prompt let's enter use the provider tool to answer the user's question and for the human prompt let's enter a
[18:35] variable for the question then let's click on format prompt values under question let's select the question from the chat box now let's also assign a tool node so under sequential agents let's add the tool node like so let's attach the allo m node to the tool node and let's call this tool node web search tool now thankfully setting up web search is really easy let's go to add nodes then under tools let's add SERP API and let's attach it to the tool node now if this is your
[19:06] first time using SERP API then simply go to SERP API.com and register or sign into your account and you can then simply copy your private key from the dashboard and then within this drop down click on create new you can give this credential any name and then paste in your API key now on the tool node click on additional parameters then under update state click on add item then let's select documents and as the value let's select the tool output this is exactly the
[19:36] same thing we did with the vector store tool node and this will simply store the results from the web search in the document state now to test this out let's connect the tool node to this end node let's save this flow and in the chat let's ask something like what is the current weather in New York and if I expand this we can see that the router decided to go down the web search path which is correct we can see that the web search tool called the SERP API tool which in turn searched
[20:07] online for the latest weather and returned the following results great so at this point we are receiving documents back from our vector store but it's very possible that these documents are not really relevant to the user's question and that takes us to the second technique that we can implement in our RAG solutions and that is this fallback technique with fallback we can add another LLM to the chain and this agent will be responsible for checking the relevance of the retrieved documents in relation to the user's question if these documents
[20:39] are not relevant then we'll fall back to a web search otherwise we'll continue to generate an answer so what does this look like in flow wise first let's scoot over this end node and let's go to add nodes and let's add another conditional agent but instead of connecting the tool node directly to the condition agent node we're actually going to add another LLM node in between these two nodes so I'll simply add the LLM node and then I'll connect the tool node to this LLM node and I'll then add this
[21:10] LLM node to the condition agent so why am I doing this this actually resolves an issue with inflow wise whereby if you connect a tool node to any node before or after it that node will have access to the stool node so that means that if we didn't have this LLM node in between then this conditional agent might actually attempt to call the tools provided by the stool node which is not what we want so in order to make the tools
[21:40] inaccessible to this node we will simply add this dummy LLM node which actually won't do anything it's simply there to create a buffer between the tools and our condition agent node you can of course try to omit this node but if you do experience funny issues whereby this node is trying to call this tool node then simply add the spanning in between I hope that makes sense for this dummy node I'll simply give it a name like vector store dummy and for the additional parameters I'll simply add a prompt do
[22:12] not respond do not call tools and that's really it now for the condition agent let's give it a name like grader LLM after all this agent is going to do grading on the documents and in return either yes or no so under additional parameters let's add a system prompt and a human prompt for the system prompt let's enter you are a grader assessing relevance of the retrieved documents to a user's question if the document contains keywords or semantic meaning related to
[22:43] the question then write it as relevant and do not try to answer the question yourself simply write the relevance of the document to the question then carefully and objectively assess whether the document contains at least some information that is relevant to the question then very important return a jason with two keys a score which is either yes or no to indicate whether the document contains some information related to the question and the second key is reasoning where you need to
[23:14] explain the reasoning behind your decision here is the retrieved document we will then inject the documents from our state object into this variable called context then for the human prompt let's simply add the user's question let's save this then under format prompt values for the context we will be using the value that we stored in state so let's select flow dot state this one with replace with key then click on edit and let's replace this placeholder with documents let's
[23:45] submit this and as a reminder we are referring to this documents property that we are storing in the state object then let's change the question to the question from the chat window now we also want to return two properties from this node first is the score which is of type enum and for the values we are expecting yes or no and for the description we can simply enter content relevance score let's add another item
[24:15] called reasoning which is of type string and for the description we'll enter reason for your decision and this is actually a very handy tip as we won't see this value when executing this flow but if you are analyzing this flow with something like langsmith then you will be able to troubleshoot your flow using these values let's open up condition let's go to code let's click on see example and the value that we want to check is score then in the if condition let's change this
[24:46] value to yes and let's change the return to yes as well then let's copy this if statement let's place it just below let's change this now to no and let's return no as well let's also remove this end route let's save this and now we get these yes or no conditions so if the documents are not relevant to the user's question then we can attach this no output to the web search node over here or if the documents are
[25:18] relevant then we can attempt to generate an answer let's actually do that now as well so let's add another llm node like so and i think to keep things organized let's actually start adding sticky nodes so let's go to add nodes let's search for sticky and i'll add it just above this node over here and let's enter generate and answer so if the documents are relevant then we'll try to generate an answer and let's do the same thing for the web search nodes
[25:48] so instead of just calling this end node we want to take the results from our web search and also attempt to generate an answer so we would be tempted to attach the tool node to this llm node but as i mentioned earlier this could cause this llm node to try and call this tool node as well so the workaround is just to add another dummy llm node so i'll simply copy this one let's add it next to the tool node as a buffer i'll attach the tool node to this llm node let's rename this to
[26:19] web search dummy and we'll then connect this llm node to the generate answer node and of course we also need to provide a name to our generate answer llm node and i'll just call it generate answer let's actually add another sticky just above this grider node with a note of check if the documents are relevant and if not route to web search now let's wrap up this generate answer node so let's start by attaching the end node let's pick on additional parameters and in the
[26:49] system prompt let's add answer the user's question using the below context keep the answer to about three sentences max and we will attach the documents from the state object in a minute but let's first save this and in the human prompt let's simply enter the user's question with the question variable then let's scroll down to format prompt values for the context let's select this flow state let's change this placeholder to refer to the documents instead and for the question let's select the
[27:20] question from the chat window so this node will now take the context that we retrieved either from web search or from the vector store and then try to answer the user's question from that context now what i also want to do is take the generated answer and store that in a state variable as well so let's go back to our state node let's add another item called generation you could also call it answer or whatever you want i'll just call it generation and for the operation let's
[27:51] call it replace with the default value then let's go back to our answer node under additional parameters let's scroll down to update state let's add an item let's select generation and for the value we'll simply select the allen node output great let's save this and we should be able to test this flow now let's open up the chat window let's clear the chat now let's first try the vector search path by entering what is a react agent we can see that the router used the vector store
[28:22] path which is correct our vector store tool retrieved the following information from the knowledge base then our grader decided to generate an answer because it found the retrieved context to be relevant to the user's question and therefore we get a generated answer now let's try a scenario where the flow will attempt to go to the vector store and then find that the documents are actually not related to the user's question let's ask what is flow wise ai and how does it implement rag so this rag text will trick the
[28:54] application to go to the vector store it should then see that the documents are not relevant and then do a web search instead so let's have a look at this as expected the router decided to use the vector store and our vector store tool retrieved the following from the knowledge base our grader then decided to use the web search instead so clearly it determined that the documents are not relevant so now we are performing a web search and now we are getting the correct answer back excellent so what we'll do as a final step is what
[29:26] is called self-correction so this agent will be responsible for checking if there are any hallucinations or factual errors in the answer that we got back from this agent if the answer is correct then we will return that answer to the user if we find that the answer is not correct then we can simply respond with something like i don't know try rephrasing your question of course you can take this as far as you want maybe you want to fall back to a third data source or attempt to call the web search in a loop to try
[29:57] and find a different answer that's really up to you let's go back to flow wise let's move this end node and let's perform this hallucination check under add nodes let's add another condition agent node let's call this hallucination check let's attach the allyl end node to this agent and just for completeness sake let's also add a sticky note with a text like check for hallucinations under additional parameters let's change the system prompt and the human prompt now feel free to pause this video or simply copy this
[30:29] prompt from my agent flow but this is simply telling the agent to check that the answer that we received is correct and it needs to return a score which can be either yes or no let's save the system prompt and then for the human prompt let's add facts so these would be the documents that we received from either the web search or the vector store search along with the answer that we received from the generation node and of course we are expecting to receive two keys a
[30:59] grade which can be either true or false and reasoning that contains the explanation for the score let's save this then under the json structured output let's add the grade which is of type enum then for the values it can be either true or false and for the description we'll enter student grade let's add another item this one is called reasoning which is of type string with a description of the teacher's reasoning behind the grade let's close this pop-up then let's click on
[31:31] condition let's go to condition code let's click on c example i'm going to remove this line to return end then we want to check the value of grade so if this includes true then we will return true let's actually copy the statement then let's change this to false and let's return false let's save this now we have true or false parts so let's start with the true path so this means that the answer was correct for this let's add an allol
[32:02] m node i'm actually also going to copy a sticky across let's call this final responses let's call this allol m node summary allol m let's connect the true path to our summary allol m and let's also attach our end node under additional parameters let's set the system prompt of you are a helpful ai assistant use the following generation and format into a friendly and concise response to the user do nothing do things like here is a concise response and for the human
[32:33] prompt will simply pause in the generated answer then let's go to format prompt values for generation let's click on edit within this field let's select this state value let's change this to flow dot state dot generation like so and that should allow this node to generate an answer for us and i think i might have forgotten to set these values in the hallucination node so let's go to additional parameters let's pick on format prompt values for the context we will select the
[33:04] documents from state so let's click on state let's edit this and this will refer to flow dot state dot documents let's do the same thing for generation we will select our state and let's rename this to flow dot state dot generation like so right so we can now close this pop-up and this means that this note will now be able to check for hallucinations and then direct the flow down this true path which will summarize and provide an answer to the user if this
[33:35] node finds that there are hallucinations then we can go down this false path for this i'll simply copy this allot end node i'll also copy the end node let's rename this to something like no response let's connect the end node as well and let's connect this false route to this node let's change the system prompt to something like you are a helpful ai assistant and for the human prompt we can just enter simply say i'm sorry but i couldn't find the answer to your question you could try rephrasing
[34:06] the question and try again now let's save this flow and let's test this let's try a super weird question like based on your knowledge based on rag what is the current weather in cape down so what happened here was we decided to do a web search which brought back the current weather in cape down so we then generated an answer from that then the hallucination check correctly identified that the response does not actually answer the user's question and therefore we are getting the message saying i'm sorry but
[34:37] i couldn't find the answer to your question so i do hope that you found some value in this video not just around building agents with flow wise but also how to include more advanced techniques within your rag applications if you found this video useful then please hit the like button and subscribe to my channel also check out these other flow wise videos over here i'll see you in the next one bye bye