Flowise Cloud vs Self-Hosting: What You're Missing
Flowise AI (2024) TutorialAging📅 2024-09-18
Deployment Architecture
┌─────────────────────────────────────────────────────────────────────┐
│ DEPLOYMENT PIPELINE │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ LOCAL DEVELOPMENT CI/CD PRODUCTION │
│ ───────────────── ──── ────────── │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Code │ │ Build │ │ Render │ │
│ │ + Test │ ──────► │ + Check │ ───► │ Server │ │
│ │ Locally │ │ Quality │ │ │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │ │ │
│ │ │ │ │
│ git commit auto deploy live app │
│ git push │ │
│ ▼ │
│ ┌───────────┐ │
│ │ USERS │ │
│ └───────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘Deployment Steps
- Core concepts explained
- Step-by-step implementation
- Practical examples
Transcript
[00:00] Flowise's fully managed cloud service has been out for a while now, and today we'll dive into the exclusive features and benefits offered by the service to see if it's worth your heart and money. If you're new to my channel, then welcome. I create videos on building AI-driven solutions using code and no-code platforms. Flowwise AI is without a doubt one of my go-to platforms for building advanced AI solutions, from simple customer-facing chatbots to advanced AI agent teams. There's pretty much nothing
[00:31] you can't build in Flowwise. Flowwise is free and open source, which means you can run the platform on your own machine or self-host it in the cloud using services like Reindeer or Railway, which might set you back about $7 or $8 per month. So why would you consider spending about $35 per month on their managed cloud service then? In this video, you and I will try and find the answer to that question together. Now let's get started. We can access Flowwise Cloud by going to flowwiseai.com, and from here you can
[01:02] click on login or request access, if this is your first time signing up. But instead of doing that, I do want to ask you to use my affiliate link in the description of this video. This video is not sponsored by Flowwise, and by using my affiliate link, you will be supporting my channel, and as an added bonus, you will not be added to the waiting list, and you will receive access to Flowwise Cloud immediately, so please use my link. After clicking on my affiliate link, you can now sign up for Flowwise using one of these
[01:33] providers or your email and password. And after signing in, you should be presented with the Flowwise dashboard. From here, you have all the different features that are available in the open source platform, like the ability to create chat flows, agent flows, access to marketplace, and everything else. But scrolling down in this list, we can also see exclusive features that are only available within the cloud service, like data sets, evaluators, evaluations, logs, files, etc.
[02:05] Now before we dive into any of these exclusive features, I do want to mention one of the most common reasons why you might want to upgrade to the cloud service. Now obviously you can run Flowwise on your own machine for absolutely free, but the big limitation with running it on your own machine is that you are not able to expose your flows to the outside world. So let's say that you are building projects for clients, for instance, a chatbot that is embedded into their websites, you will not be able to embed this chatbot into a website
[02:37] if Flowwise is running on your own machine. In that instance, you will have to deploy Flowwise to the cloud by using services like Render or Railway. And yes, it is very possible to host Flowwise on Render for only $7 per month on the starter plan, and this usually includes a little bit of RAM and CPU capacity. Now it's also worth noting that none of these services include persisted storage, which means that if the service had to be restarted, your database and flows will get lost.
[03:07] So in addition to the $7, you also have to add a disk, which adds about another dollar on top of your costs, and that is only for about 1 GB of space. So this $7 or $8 might seem quite cheap, but if you've ever tried to use these packages in a production environment for clients, you might run into some issues. I've personally had issues with the service went down because of some document loader taking up way too much resources. And you'll notice that if we then wanted to upgrade the package
[03:38] from starter to standard, we are now moving to the $25 to $30 per month range. And now the costs are starting to be very similar to the $35 that the Flowwise cloud service will cost you. And this already includes storage as well, as well as some other benefits like database backups, which is not offered by services like Render or Railway. And of course, we will have a look at some of these exclusive features in this video. The possible downside to self-hosting Flowwise is that you are
[04:09] responsible for keeping your Flowwise instance up to date, and the Flowwise team push regular updates and fixes to the platform on pretty much a daily basis. So the responsibility falls on you to ensure that your instance of Flowwise is up to date. To demonstrate some of the exclusive functionality, I've created this very simple customer support chatbot, and this is simply using a conversation chain with a chat openAI node. And the prompt is quite simple as well. I just entered "You are a friendly assistant called Eve."
[04:40] And of course, we can chat to this assistant, so let's enter something like "Hello," just to test this out, and we get this response back. Great, so far this is nothing new, and you should be familiar with the interface. Now let's have a look at what the differences are. If we go to Settings and Version, we can see our current Flowwise version for this instance, and we can also see the latest version of Flowwise over here. So to update Flowwise, all we have to do is click on Update to Latest, and you can also
[05:11] use the same popup to roll back to a previous version. But let's close this popup, and on the left-hand side, let's go to Account Settings. On this page, we can view our available predictions, as well as our storage usage. Whenever we add files to our document stores, those files will take up space in the file system, and Flowwise gives us up to 1000MB of storage space. And predictions are any API calls to Flowwise. These include any interactions with Flowwise from outside of the platform.
[05:44] So if we have a chat bubble on a website, or if we're using the Flowwise APIs, those will all count towards the predictions. Do keep in mind that if we test our chat pods within Flowwise, those messages do not count towards these predictions. Any messages from outside of Flowwise count. Right, let's have a look at files. Whenever we add files to our document stores, those files will show up in this list. Let's have a look at that. So let's go to Document Stores. I'll create a new document store for my
[06:15] restaurant called Oak and Barrel. I'll then click on Add. Let's open up this document store, and let's add a new document loader. And for this, I'll simply use the Docx file loader, and I'll select my knowledge base from my PC. Now I'm just going to select a text splitter. Let's select the recursive character text splitter. I'll change the chunk size to something like 250, with the chunk overlap of 20. And finally, let's click on Process. And this will load all the contents of the document into this document store.
[06:46] So the document was loaded, and it loaded about 23 chunks. And of course, if we wanted to, we could click on this record and view all the chunks that were extracted from the document. We can see the file that we just uploaded by going to Files. And here we can see that file, and we can see that it's taking up 0.2 megabytes. And of course, if we go to Account Settings, and we can see under Storage that that file is taking up a little bit of space. However, here's one very cool thing. We can go back to Files, and we can clear
[07:18] up space by deleting this file from the file system. And this will not break the document store. So if I go back to Document Stores, you will notice that your document store is still there, and you can still view all the chunks that were uploaded to the document store. So since we really didn't need the file after inserting it into the document store, it made sense just to delete it to clear up space on the file system. How cool is that? Now let's have a look at logs. If you're familiar with using the terminal to debug your Flowize
[07:48] applications, then you'll feel right at home using logs. In fact, this is the same as running Flowize with debug enabled. So just to get some data, I'll click on this dropdown. Let's click on last seven days. And here we can see the terminal output of executing a Flowize chain. Now let's move on to evaluations. Datasets, evaluators, and evaluations are unique to the cloud service, and these are not available in the open source version of Flowize. And quite honestly, these are probably
[08:19] the reason why you might want to consider upgrading to the cloud service. We can use these tools to test and analyze certain aspects of our chat flows. In fact, let's start with datasets. I'll create a new dataset by clicking add new. I'll give my dataset a name, like ocan_battle_support_assistant. And if we wanted to, we could use a CSV file to upload our different tests to this dataset, but I'll simply click on add. We can then click on our dataset. Now let's say we wanted to test the greetings generated by our chat flow.
[08:52] For example, if a user passed in a message of hello, then I'm expecting the LLM to provide a very specific response, or a response containing certain values. So to start, let's click on new item. And for the input, let's just enter hello. This means that when this test is run, the text hello will be passed to the chain, and we're expecting the output to be something like, hello, how can I assist you? So do take note that the anticipated output is actually not used during this process,
[09:23] but it will be helpful when we view the results to remind ourselves as to what type of output we were expecting. Let's click on add. And now that we have some data, we can continue to evaluators. Now evaluators are actually specific tests that we can run on top of our dataset. So as an example, our dataset is going to pass the text hello to the chain. And in the response that is generated from the LLM, we expect there to be certain values, like maybe we're expecting the LLM to
[09:54] include its name, or the name of the rest to run, etc. Or we can also use evaluators to check that the tokens didn't exceed some maximum limit, or that the response time from the service call wasn't too slow. We can also use evaluators to check if the LLM hallucinated an answer as well. So let's create our first evaluator. For this one, we actually just want to check if the response from the LLM contains certain values. So I'll call this oak_ant_barrel_greeting_contains.
[10:26] You can call this whatever you want. For the evaluator type, we have three different options. Evaluate result, which means we will run a test based on the text generated by the LLM. Evaluate metric is something we can use to check the token usage or other attributes like the latency of the service call, or LLM based grading. And this grading method actually uses an LLM to check the response for hallucinations or accuracy. Let's start with text based.
[10:57] And what I want to check is whether the response contains certain values. So I will select contains any, and now I can provide a comma separated list of values. And if any of these values exist in the response, then the test will pass. So I'm expecting the response to contain at least the following words. Hello or lowercase hello. And do take note that these values are case sensitive. I'll also add welcome, hi, maybe an uppercase welcome, etc.
[11:29] So let's add this evaluator. And now finally, we can create an evaluation. So evaluations are responsible for executing tests by combining data sets and evaluators. So let's add a new evaluation. We can give it a name like oaken barrel evaluation. So basically this evaluation is responsible for running tests against my oaken barrel chatbot. So on the data set, I'll select oaken barrel support. And for the chat flow to
[11:59] evaluate, I'll select my chatbot. Let's click on next. Now we can select our evaluators. And we only have one. So I'll select oaken barrel greeting contains. And if we add more evaluators, we could add them to this list as well. We will do that in a minute. Let's click on next. On this screen, we can assign an LLM to greater response as well. But we will get back to that in a minute. For now, let's click on start evaluation. Our test is currently running. And after we refresh this, we can see that the test was
[12:30] successful with a pass rate of 100%. Now in these results, we can view the different executions by clicking on these little arrows. We only have one execution for this test, so we'll see one result over here. We can also navigate to the chat flow by clicking on this button over here. So if we wanted to make changes to the chat flow and rerun the tests, we can quickly navigate to the chat flow, and we can go back to the evaluations quite easily. We can do the same thing for the data sets, of course.
[13:01] So we can make changes to the data set and return back here. To view details about this execution, we can click on this icon to view the results. At the top, we get a few graphs showing the pass rate, the token usage, as well as the API latency. And scrolling down, we can see the input from the data set, the expected output, and under evaluators, we can see all the different evaluators. We only add one, which checked if the actual output contained any of these values.
[13:32] And on the right, we do see the actual output, which was this text over here. And because this text contained hello, our test passed. We can also click on this actual output to see all of this information in a different view. Now let's continue looking at some of the other evaluators. So let's create a new evaluator. Now for this evaluator, I actually want to see how long it takes for our chain to execute. So I'll call this oak and barrel response time. Under evaluator type, instead of
[14:03] selecting text, let's select numeric. Under available evaluators, we can see all the different attributes about this execution. Like the total amount of tokens that were used, the tokens related to the prompt itself, the completion tokens. We have things like the API latency, LLM latency, chat flow latency, et cetera. I'm interested in seeing the API latency, and under the operator, I'll select less than. And I'm expecting this response to come
[14:34] back in less than two seconds, which is 2000 milliseconds. Let's add this evaluator. Let's go back to evaluations. Now there's no way to edit an evaluation, so I'm actually just going to delete this one. Let's create a new evaluation. I'll call it oak and barrel. I'll select the chat flow. Let's click on next. Let's select our evaluators, and this time we can see two evaluators. So I'll select contains and response times. Great. Let's click on next, start evaluation.
[15:05] Let's refresh this. And the pass rate is actually 100%. If we go to the details, we can now see both of the evaluators showing up in the list over here. And if we have a look at the API latency, this was indeed less than 2000 milliseconds. Now let's have a look at the test at files. I'm going to go back to the evaluators. This changes value from 2000 to 1000. Let's save this. Let's go back to evaluations. Let's click on view details, and let's click on rerun evaluation.
[15:36] Let's click on yes, we'll refresh this. Now we can see that the pass rate is only 50%, and if we look at the details, we can see that the second evaluator did fail because it was expecting the API latency to be less than 1000 milliseconds. And in the results, we can see that the actual API latency was 1090 milliseconds. So this was correct. And this is a great way of fine tuning your chains. I'm actually just going to change this back to 2000 milliseconds, like so.
[16:07] And let's actually add our third type of evaluator, and that is the LLM-based grading. Now before we complete any of this, let's just give this evaluator a name, like oak and barrel hallucination check. And what we can do here is output any type of schema that we want, and we can pass a prompt to the LLM. But thankfully, Flow-Wise already did this work for us. So simply click on load from predefined samples, then in the list, select hallucination, click
[16:37] on select prompt, and that's actually all you have to do. What this model will do is check the response from the LLM and then give it a grade between zero and one. One meaning that the response seems to contain hallucinations. Let's simply click on add. Let's go back to our evaluations. I'm going to delete this existing evaluation. Let's create a new evaluation. Let's call it oak and barrel tests. Let's select our data set. Let's select our chat flow.
[17:08] Let's click on next. Let's select our evaluators, just like we did before. Let's click on next. Now for the grading, let's enable this. Now we can select that LLM grading evaluator, and we also have to select our LLM provider, which is open AI. And let's start the evaluation. Right, this test is complete, and it seemed to pass 100%. Let's have a look at the results. So if we scroll all the way to the right, we can see this LLM evaluation section now. And in fact, let's open it up in this
[17:39] view, where we can see that the LLM grade is a score of zero. Zero means there were no hallucinations detected in the response. And we also get some reasoning back from the LLM. Now let's force a hallucination. Let's go back to our chat flow. Let's add something to the system prompt, like always respond with, hey, did you know this guy is actually red and not blue? Let's save this chat flow. Let's go back to our evaluations. Let's click on View Results. This is also very useful, as this
[18:10] dashboard will tell you if any of the underlying chat flows were changed. I'll simply close this popup. Let's rerun this evaluation. Let's view the results. And we can see that one of our evaluators actually failed, and that is the one that checks if the response contains any of these words. And because the actual output did not contain any of those words, this check failed. And if we scroll all the way to the right, we can also see that the LLM evaluation has a score of one, because it's determined that the LLM actually
[18:40] hallucinated the answer or is providing inaccurate information. So I do think evaluations is definitely one of those features that is meant for pro users. So if you are building chat flows for clients and doing this as a business, then these features might be extremely valuable to you. And that almost covers all the exclusive features within Flow Wise Cloud. According to the Flow Wise team, they are also working on a solution to accommodate multiple users within your Flow Wise instance. So if you are running a team, you will be
[19:11] able to allow multiple people to log into your Flow Wise instance, and they can be assigned different roles and collaborate on chat flows. I will cover that feature once it is released, but I just wanted to mention it for now. If you enjoyed this video, then please hit the like button and subscribe to my channel for more Flow Wise content. I'll see you in the next one. Bye bye.