How To Auto Update Your Knowledge Bases | FlowiseAI
Flowise AI (2024) TutorialAging📅 2024-10-17
RAG Architecture
┌─────────────────────────────────────────────────────────────────────┐
│ RAG (Retrieval Augmented Generation) │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ DATA INGESTION │
│ ────────────── │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │Documents │───►│ Loader │───►│ Splitter │───►│Embeddings│ │
│ │PDFs, URLs│ │ │ │ │ │ │ │
│ └──────────┘ └──────────┘ └──────────┘ └────┬─────┘ │
│ │ │
│ ▼ │
│ VECTOR STORE ┌───────────────────┐ │
│ ──────────── │ Vector Database │ │
│ │ (Pinecone, etc) │ │
│ └─────────┬─────────┘ │
│ │ │
│ QUERY PIPELINE │ │
│ ────────────── │ │
│ │ │
│ ┌──────────┐ ┌──────────┐ │ │
│ │ User │ │Retriever │◄─────────────────┘ │
│ │ Question │────────►│ │ │
│ └──────────┘ └────┬─────┘ │
│ │ │
│ ▼ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Answer │◄────────│ LLM │◄────────│ Context │ │
│ │ │ │ │ │ │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘RAG Components
- Core concepts explained
- Step-by-step implementation
- Practical examples
Transcript
[00:00] In this video, I will show you how to automate the process of keeping your document stores up to date. Document stores are a great solution for managing all your data sources in a single container. This way, you can easily add or remove information from the knowledge base, without having to change the AI chat flow itself. But one thing you guys have been asking me in the comments is how to refresh the document store automatically when a document loader uses dynamic data. For instance, you might be scraping information from a
[00:30] website that changes often, or you might be pulling data from a database, like Airtable. In those situations, you might want the document store to be refreshed as soon as the data is changed or like every hour, once a day, etc. I really want to give you the best value possible in this video, so I thought it would be fun to build a real world application together. Here, we have an AI powered business directory. This is similar to the Yellow Pages, but instead of searching businesses using keywords,
[01:01] we can simply have a natural conversation with an AI chatbot. The business listings are actually just stored in an Airtable database, but this could be any type of data source, like you could be scraping data from a website or pulling from a database. Now, the issue is if we add a new listing to this database, for example, let's add Oak End Barrel, which is a fictitious restaurant, we'll just enter some address over here, like so. Let's just enter some sort of phone number,
[01:32] let's enter a description like "a restaurant that sells steaks and sushi". Now, I've committed the changes to this table, and if we go back to our chatbot and we ask something like "I'm looking for a sushi restaurant in Johannesburg", we simply get a message saying that it can't find that latest entry. And that brings us to the challenge that you guys have been facing with document stores. Whenever the data source changes, you manually have to execute the following steps. And in this video, we will automate this
[02:03] process, but let me show you how this works. First, you have to go to Actions, click on Preview and Process, then we need to click on Process to load the data and chunk it, and now we get 17 documents back, which means that the latest entry has been pulled in. Now we have to upsell this data into the vector store by going to Upset Config, and then we need to click on Upset, and now we can finally see that that one record has been added and 16 records were skipped. So if we go back to
[02:33] our chatbot now, and we ask the same question, we should get the correct response back, and we do. It's very possible that you might have a situation where you want to manually refresh the data, but in many instances, like with this business directory, we want the data to be fresh maybe once a day, or every few hours. So let's have a look at automating this process. You are going to learn a lot, because in this video, I will show you how to set up the chat flow, the document store, and we will use Airtable as our
[03:06] database in this video. We will also use a workflow automation tool called n8n to run this automation on our behalf, and you will also learn how to use the new Flow-wise API endpoints for updating the document store. First, let's create our chat flow. I'm using a cloud instance of Flow-wise, but you're more than welcome to use a local instance to follow along with this tutorial. On your local machine, simply open up the command prompt or terminal and enter npx flow-wise
[03:37] start. If this is your first time running Flow-wise, then you will be prompted to install Flow-wise, so simply enter y and press enter, and afterwards, you should see a message like this, and after that, you can access Flow-wise by going to localhost 3000, and you should see a dashboard like this. Now let's create the chat flow by clicking on add new. Let's give it a name like ai-business-directory, and let's save this. Let's add a new node. Let's go to agents,
[04:07] and let's add the tool agent node. Let's start by adding an AI model to the canvas. So under add nodes, let's go to chat models, and I'll be using the chat open AI model. Now I've got plenty of videos showing you how to use chat olama to run models locally for free on your own machine, or you could use grokchat which is also free, and you'll also find the video on my channel for using grokchat. So let's add our chat model node. Let's attach
[04:37] this to our agent. Let's select our credentials. Let's select our model as gpt40-mini, and I'm going to lower the temperature to 0.2, as I want our assistant to strictly follow our prompt instructions, which we will set up in a minute as well. Let's also add a memory node. So under nodes let's go to memory, and let's simply add this buffer memory node, and let's attach it to our agent as well. Now finally let's add a tool node. So under add nodes, scroll all
[05:09] the way down to tools, and add the retriever tool, and let's attach this tool to our agent as well. This retriever tool will be responsible for connecting our knowledge base to our AI agent. For the tool name, let's call it business listings, and for the description let's enter use this tool to find information about businesses. Now optionally we can also enable this toggle which will return the source documents in the chat window. Now because this process
[05:40] will be customer facing, I don't want the user to see my data sources, so I'll leave this disabled. Then lastly let's add a retriever. So under vector stores let's add the document store node, and let's attach this to our retriever tool. Great! Now we simply need to select our document store which we haven't created yet, but before we do that I just want to go back to our tool agent and change the system prompt. I'm going to clear this, but instead I'll paste in the following prompt. Let's
[06:12] expand this. You are a helpful assistant for a business listing service called AI Business Directory. Only answer questions related to the business. Your role is to assist the user in finding information related to the businesses like their location, contact information, services, and products. Use the provided tools to retrieve this information. If the answer is not in the context, then simply respond with something like "I couldn't find the information related to that business." Think carefully about your responses and ensure that
[06:42] the information returned for each listing does indeed belong to that listing. Feel free to pause the video and then copy over this prompt or enter something relevant to your business. Now that should be everything we need for our chatbot itself. Let's go and create our document store. So from the dashboard let's go to document stores, let's click on add new, let's call this AI business directory, let's add this, then let's click on this document store, which is currently empty. Now we can start adding data
[07:12] sources to the store. So let's click on add document loader. Within here let's click on Airtable and now we need to provide information about our Airtable database. So let's go ahead and set up Airtable. Go over to Airtable.com and sign into your account. Within the dashboard next to workspaces click on create a workspace. Give it a name like AI business directory and then click on create and let's select start from scratch. Let's rename
[07:42] the space to something like AI business directory and let's press enter. Let's also rename the table by double clicking on this tab and let's call this listings and let's save this as well. Now we can define our column names as well as upload data. To speed things up you'll find a link in the description to a CSV file which you can download for free. Then click on add or import, then click on CSV file, click on browse files and select the file that you
[08:13] just downloaded. Then click on upload one file. Let's select create a new table and let's click next. Then lastly let's click on import and now you should see all of these columns with some sample data. Great! Let's actually delete the table that we created earlier by right clicking on it. Let's delete this table and let's rename this one. So let's call this listings and let's save this. Great! We now have a table with data that we can use. If we go back to Flow wise we first have to set up our credentials.
[08:44] So let's click on this drop down, let's click on create new, let's give it a name like AI business directory and for the access token let's go back to air table, click on your profile, then click on builder hub, then under personal access tokens click on create new token, call it AI business directory. Then let's add a few scopes. First let's add data records read, let's also add schema basis read and let's also add a base and
[09:16] let's add the base that we just created which was AI business directory. Then let's generate a token, let's copy the token and paste it into Flow wise and let's click on add. Let's also select the credential that which is created. Now for the base id we can simply go back to our table and the base id is this first part of the url that starts with app and a whole bunch of characters after that up until the first front slash like so. Then for the table name it's this section
[09:46] that starts with tbl so let's copy that and paste it in here and then the view is this part over here that starts with viw, it's pasted into this field. Cool I'm also going to remove this limit as I want this loader to read all the entries in the database and we're not going to add a text splitter at all as flow wise will automatically create a document for each entry in the database and we can see that 16 chunks were extracted and that is because we have 16 records in our database. Let's
[10:18] click on process to now store these changes. Let's also set up our absurd config so under embeddings because I'm using openai I'll use the openai embeddings function I'll select my credentials for the model name I'm going to use text embedding three small and I'll leave all the other values on the default values then for the vector store I'm going to use pinecone then under credentials let's click on create new then go over to pinecone.io and log into your account or sign up so
[10:50] the first thing we want to do is to get our api key so go to api keys then click on create api key I'll call this ai business directory and let's create this then I'm going to copy this key and paste it into flow wise and for the credential name I'll call this ai business directory as well let's add this let's also select the credential now for the pinecone index let's go back to pinecone and let's create that index under database
[11:21] let's click on create index let's simply call this flow wise then for the dimensions I'll pick on set up by model I'm using text embedding three small so this needs to be consistent with whatever model name you're using over here then I'll click on set configuration and finally we can click on create index so I'll simply copy the index name and add it to flow wise I like giving namespaces to my projects so I'm going to call this ai business directory and I'm going to change the top k
[11:52] to 10 documents this simply means that when the user asks the question the retriever tool will retrieve 10 documents back from the vector store in other words the 10 most relevant businesses to the user's query now finally let's also set up the record manager let's click on postgres then under credentials let's click on create new let's give it a name like ai business directory and let's also go ahead and set up a postgres database to simplify the process of creating a postgres database let's go to superbase.com and
[12:24] sign into your account from the dashboard let's create a new project let's give the project a name like ai business directory let's also set a password let's also set the region and let's click on create new project this will take a few minutes to complete once the project has been created go down to project settings then click on database and under connection parameters we can find all the information needed to complete the setup let's start by copying the username and let's add it to flowwise let's
[12:55] also enter the password that we created earlier and let's click on add and let's select that credential then for the host we can copy that from superbase and add it to flowwise let's do the same thing for the database name like so and let's also copy the port and add it to flowwise as well then the only other setting I want to change is the cleanup method let's change it from none to full this simply means that the record manager will be responsible for preventing duplicate entries in the vector store
[13:25] and delete information when we delete entries from the data source let's save this config let's test it out by clicking on absurd and if everything was set up correctly you should see a message sign that 16 records were added to the vector store and if we go to pinecon we can see that 16 records were indeed absurd to the vector database and if we go to superbase and if we go to database you should see this table with 16 rows great so now that we have a document
[13:57] store we can finally go back to our chat flow let's open up our AI business directory flow and from the document store node let's select our newly created document store let's save this and let's test it out let's say I'm looking for a restaurant near the waterfront in cape town and we instantly get a few responses back like the waterfront seafood spot and if we go to our table and we can see that the waterfront seafood spot is indeed a restaurant near the waterfront in cape
[14:28] town now we can expose this chatbot to our users in many different ways by clicking on this api endpoint button we can embed this chatbot into a website or if we have a python or javascript application we can easily integrate this into our custom applications as well but for simplicity's sake i'm simply going to go to share chatbot let's enable make public and let's copy this url and if we open this url in the browser we will be able to access our AI application from
[14:59] outside of flow wise now we can of course customize the look and feel of this application from within flow wise so for the title i'm going to add AI business directory and i'll also link to an avatar by adding a url to an image that is available online and if you scroll down we can change other things as well like the welcome message the background colors font sizes etc i'm simply going to save these changes and if we go back to our application and refresh
[15:29] it we can see the title and avatar image over here oh and we can also hide this power by flow wise branding so again within flow wise all we have to do is go to this powered by text color and change it to white and of course remember to save your changes and if we refresh this the power by text is basically gone now before we move on to the automation side of this let's simply check if our application is working okay that's great let's also try a complex scenario and i
[16:00] really love this behavior so this is something you can't do with the traditional business directory let's say something like i am planning on visiting cape town tomorrow please recommend a museum that i can visit during the day also suggest a restaurant and hotel near the waterfront let's say in this and this is absolutely amazing we get a museum back and our ai assistant recommended a museum along with a restaurant and a hotel and because
[16:32] this is an assistant with memory we can definitely have follow-up conversations with this assistant as well if you're enjoying this video then please hit the like button subscribe to my channel and share this video now let's have a look at automating updates to our document store at the moment when we make changes to our database so if i add oak and barrel which is a restaurant i'll just enter something like one oak street in johannesburg and i'll just put in some fictitious number
[17:03] and in the description i'll just enter find dining restaurant specializing in steaks and sushi with a rating of 4.5 great now as i showed you in the intro of this video this data will not be available to the chatbot automatically and unfortunately we would and unfortunately we would have to go through this tedious process of manually reprocess the document loader and then manually absurd these chunks into our vector store
[17:33] using this button over here now thankfully flow was thought about this already and what i did was expose a lot of api endpoints that we can consume from outside of flow wise to automate these type of processes let me show you what i mean when you go to flow wise's documentation and under the api reference documentation you can find api endpoints for pretty much every object within flow wise including the document stores here you will find
[18:04] api's for creating document stores listing document stores fetching details of a specific store and what we're interested in is this process loading and chunking operation which is this step over here as well as absurd chunks from document store which is effectively this step over here so to automate this process you can use any workflow automation tool that you want like make.com n8n xi peer etc and i already have plenty
[18:36] of tutorials on my channel for setting up and working with n8n if you do want to run n8n locally then simply open up the command prompt or terminal and enter npx n8n start then click on y to install all the dependencies and after installation is complete you will see a url that you can use to access n8n but for this demo i will be using my cloud instance of n8n instead so let's add a new workflow let's give it a name like ai business
[19:06] directory and let's actually delete this trigger so what we can do in n8n is set up different types of triggers trigger manually is simply used for testing workflows but what we could do instead is use something like an app event and this gives you integration with a lot of external service providers including air table so if i go into air table we could trigger this workflow when something changes within air table for example if a new entry is added to our database table but
[19:36] what i'll do instead is add this on schedule node this simply means that the workflow will run on a set schedule for applications like this business directory i might not want the listing to show up immediately when it's added to the database we could definitely do that using the air table trigger but i personally like to run these sort of updates once a day or once every few hours that gives me a bit of an opportunity to first verify the data to ensure that it's not malicious or junk
[20:08] or to fix typos if i made a mistake otherwise that data including all its mistakes will be instantly added to the vector store and users might actually see that so let's say that i actually want to run this update like every six hours great next i'm going to add a set node we will use the set node to store certain information about our flow wise project first let's create a new field and let's call this store id and for the value
[20:38] let's go back to flow wise and let's actually copy this value over here as that is the unique identifier for this document store and let's paste it into this field i'm also going to add one more value called the flow wise base url now for you that might be localhost 3000 if you are running this locally but since i'm using a cloud instance i'm simply going to copy this url and paste it in here without the front slash so if i test this step
[21:09] i am getting this json schema out containing our store id and our flow wise base url and now we can use these values throughout our project now the first thing we need to do in this process is to fetch the details of our document store we need to do that in order to retrieve each and every one of the document loaders in this document store although we only have one document loader this document store could contain as many document loaders as we want and what we want to do in our
[21:40] automation process is to loop through each of these document loaders and then automatically trigger the preview and process function for each of these in a loop so let's see how that's done to retrieve the details of a document store we can use this get a specific document store api endpoint which looks something like this it's basically the flow wise base url followed by this path and we have to pass an id value which is that store id
[22:11] so we're not going to look at the documentation for each of these api calls but i simply wanted to show you my process of figuring out which apis to call and what is apis need so to call that api within n8n let's add a new node and more specifically the http request node for the method we will set this as a get method again the documentation will tell you what the method type should be for the url i'm going to switch over to expression and click on
[22:42] this button just to expand the view then let's start with the base url and we can see a preview on the right hand side front slash api v1 document store slash store and you might recall that we then need to pass in the document store id as well so let's simply grab that from this previous node and now you should have a url that looks something like this now if we simply test the step as is we will get a failure and more specifically
[23:13] an authorization failure and that is because flow wise is protecting their endpoints through api keys so under authentication let's select generic credential type then under generic auth type select header auth then under header auth select create new credential and for the name let's enter authorization and this must be authorization with a capital a and for the value we need to create an api key
[23:44] within flow wise so back in flow wise go to api keys then click on create key let's call this ai business directory let's add this then let's copy this key and and let's paste it into this field let's save these changes let's click on test step and this time we actually get a response back cool and if we go to the schema view we can see the details about our document store like its name and more importantly all the different loaders and we only have one
[24:15] loader and that's this air table loader but of course if we add more we would see all those loaders pull through as well so i'm just going to clean up this workflow a little bit by renaming this node to get document store details so what we want to do next is simply extract all the document loaders out of this response so that we can use them in our workflow so let's simply add a new node let's add the set node let's rename this to extract document loaders let's rename this and
[24:47] all we have to do to extract the document loaders into this new structure is to simply drag this loader's array into this fields to set we'll keep the name as loaders and we can see that this is of type array and if we run this all this other information has been dropped and we now simply have a very simple structure containing only our loaders which is much easier to work with what we want to do now is loop through each of these document loaders remember we could have multiple
[25:17] doc loaders and for each of these document loaders we want to call the api to execute this preview and process step so let's first loop through these items by adding a new node and more specifically the loop over items node and because the previous node is returning an array of loaders this loop node will automatically loop through these different loaders so for each loader we can delete this node and add a new node and because we're calling an API
[25:47] we'll call the API request node let's rename this to process chunks let's rename this this method should be changed to post then for the URL let's change this to expression again let's open this up and again we need to start with our base URL followed by API v1 document store loaders slash process then for authentication let's change this to generic credential type then for the auth type let's select header auth and let's
[26:19] select our header auth that we created earlier now for this API we need to pass a body so it's an able sent body and as a reminder all of this information is available in the flowwise documentation as well and we only have to pass two parameters first is the store id and for the value we can go to edit fields and grab the store id from over there let's add another parameter and this one is called id and this refers to this document loaders id which we can get from this
[26:52] loop and that's this id field within the loaders array now let's go ahead and test the step now if we go back to our application we can run this entire flow by running test workflow and we can now go to process chunks and from this view we can see all the chunks that were extracted during this process you might recall that we only had 16 chunks within flowwise but we added the 17th record in our database and if i now refresh flowwise
[27:23] we can see that the amount of chunks have changed from 16 to 17 so our API call was indeed successful and we've just automated this process over here now before we move on to the absurd logic i do want to mention that the process of processing the chunks is asynchronous and it can take a bit of time to complete so i actually recommend just adding a wait note into this loop to give this process some time to complete i'll simply make this two seconds so this means that when we run this flow it will wait two seconds before
[27:55] continuing and everything should still be okay now let's have a look at automating the process of observing the data into the vector store so that's the process of clicking this button over here when the loop is complete we'll add another note to this done route over here and we will be calling another HTTP request let's rename this to absurd chunks let's change the method to post then for the URL let's go to expression let's open this up and of course we'll grab the URL
[28:26] from the flowwise base URL over here followed by API v1 document store vector store insert then for authentication let's change this to generic credential type let's change the auth type to header auth and under header auth let's select our header auth like so let's also enable saint body and within the parameters we only have one parameter that we have to pass and that's the store id so within value let's simply go to expression
[28:58] let's expand this and let's grab the store id like so don't worry these errors will all go away once we execute this flow let's test this workflow and we can see that the absurd ran successfully and we can see that 17 records were indeed added and the old records 16 of them were deleted so in theory if we now go back to our chatbot so in theory if we go back to our chatbot we should be able to find this new business listing this oaken barrel listing over here so
[29:29] let's say i'm looking for a sushi restaurant in joannesburg and have a look at that oaken barrel came back as a response so it seems that our workflow is indeed working in fact to test this out i'm going to change the interval to minutes and i'm actually going to change this to every two minutes let's go back to the canvas let's save this workflow and very important change this from inactive to active so this workflow will now run every two minutes let's add a new entry to this
[29:59] table to see if it gets added automatically so i've just added the ai studio which is an ai services company and for the description i'm just saying it's an ai automation agency so let's give it a minute or two to see if anything changes on our document store it's been about two minutes so let's simply refresh this page and let's see what happens and there we have it the amount of chance is changed to 18 and let's start a new conversation with our agent and let's type i'm looking for an ai
[30:30] automation agency and it was able to pick up the ai studio and as you can see our knowledge base is now automatically refreshed on a timed interval now finally i do want to give you a pro tip as well your document store might contain loaders that do not change at all as an example you might have some pdf loader or a csv file and these document loaders never change this is static data if you want to change the pdf document you effectively have to delete this pdf document
[31:01] loader and create a new one so that simply means that it does not make any sense to refresh these pdf documents in your automation process the data is static after all so what you could do in your workflow is add one additional node let's move this up and let's add the filter node like so so now what we can do is actually filter out any document loaders that we don't want to refresh automatically so we can grab the loader
[31:31] id from the left hand side and we could say that we only want to refresh document loaders with an id of air table and you can add additional checks as well so this means when we execute this workflow all the document loaders besides air table will be dropped out of this array this video was a lot of work to create and if you made it this far that i do want to thank you for your time if you want to support my channel then please hit the like button and subscribe and if you would
[32:02] like to learn how to use flowwise or n8n then check out these other videos and series over here i'll see you in the next one bye bye