Build a Web App with Response Streaming | LangChain JS Tutorial #3
LangChainVintage📅 2023-05-25
Build Process
┌─────────────────────────────────────────────────────────────────────┐
│ BUILD CHATBOT WORKFLOW │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ PLANNING │
│ ──────── │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Define │───►│ Choose │───►│ Setup │ │
│ │ Scope │ │ Stack │ │ Project │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │
│ ▼ │
│ DEVELOPMENT │
│ ─────────── │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Build │───►│ Add AI │───►│ Test & │ │
│ │ Features │ │ Logic │ │ Debug │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │
│ ▼ │
│ DEPLOYMENT │
│ ────────── │
│ ┌─────────────┐ ┌─────────────┐ │
│ │ Deploy │───►│ Chatbot │ ◄── LIVE! │
│ │ to Cloud │ │ Running │ │
│ └─────────────┘ └─────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘Key Components
- Project architecture
- Core features implementation
- Best practices
Transcript
[00:00] hello and welcome back in the previous videos we had a look at creating a basic chatbot using Lang chain and openai and we also had a look at some of the other key features in Lang chain like prompt templates chains agents and tools and we also had a look at streaming the responses by passing in the streaming property as well as a callback function which will write the response to the terminal as the results are streamed back from openai and we could see this in action by running the node chat demo
[00:32] file in the terminal and we can see the response streaming in the terminal we will move on to some more advanced features in the series but before we move on I first want to create a simple user interface for interacting with our model going forward personally I like learning a new technology in a practical and visual way and adding this user interface will allow us to visualize how a real world application would integrate with Lang chain so let's have a look at what what we'll be building in this video we will create a simple layout
[01:03] which allows the user to enter a prompt and then send it to the langchine model and we can also use this button to clear the current chat history so as an example we can use the same prompt that we used in the node demos and when we click on same chat this will call an API endpoint which we will create that will pass this prompt to Lang chain and the API will then return a streamable response which will stream the output to the screen so we can see that in action by clicking on Saint chat and after a
[01:35] second we can see the result streaming back to our user interface and this creates a fantastic user experience as the user does not have to wait for the full response to be generated before seeing anything on the screen in order to make things as simple as possible for us we will use a framework called nextjs to create our user interface don't worry though we won't spend too much time learning next year is and styling in this video I actually have a dedicated series on this which I recommend
[02:05] checking out if you are familiar with other Frameworks or if you simply want to create an Express app you will still benefit from this video I personally like working with nexjs and find that it makes it very easy to build full stack applications and as a little bonus for those of you who are familiar with Nexia is I will show you how you can create these API endpoints using both the route Handler solution which was introduced in next 13 as well as the older API route
[02:35] solution for both endpoints we will return a streamable response for getting that streaming effect on the front end so let's get started open up a new folder in your IDE I'm using vs code in this video and then open up your terminal and run npx create Dash next Dash app at latest followed by period this will start the next jsb project setup you will then be asked if you want to use typescript and to keep things simple I'll just say no would you like
[03:06] to use es length I'll say no would you like to use Tailwind CSS say yes would you like to use the source directory I'll just say no use the app router we will say yes would you like to customize the default import Alias or just say no and this will now install all the dependencies for our next JS app we can check that our installation was successful by running npm run Dev and we can then open this URL in our browser and in the browser you should see something like this and if you do it
[03:38] means your setup was successful we can now go ahead and remove all of the sample HTML code by opening the app directory and we can then open up the page.js file if you're not familiar with next all the pages of our website will be created within the app directory and the page file contains all the content for this page so I'm going to to remove most of this by deleting everything in between the main tags like so and I'm also going to remove this image import at the top and I'm also going to remove
[04:10] the styling within this class name property so that we're left with a page like this once we save this the web page should update automatically and will now be left with a blank page I also want to remove some of the default styling that is used so we can do that by opening the globals.css file and then deleting everything except these first three lines and now I'll save this file so I'm first going to work on the user interface and afterwards we'll start working on the API endpoints that our
[04:40] front end needs to call I want to change the background color of our page so we can do that by typing in this body tag and within this tag we can now change the background color by providing a property called background color and for background color we can now set the color in many different ways for instance we could type some hash value or just select the color from that drop down list after saving this we can see the background color changing but I'm actually going to use a different way of styling our project which is way simpler
[05:12] than using standard CSS and you might recall that at the start of the project while installing next we said yes to using Palvin CSS Palvin CSS is a CSS framework that makes it very easy for starting web applications and I highly recommend having a look at the Telvin documentation to see several examples of how Telvin can be implemented in your project so instead of using standard CSS like this we will be using Talent Clauses instead so I'm going to remove
[05:43] the background color and I'm going to type and add symbol followed by apply and now we can start adding our talent utility Clauses and for the background color there is only one class that I'm interested in and that's called BG e Gray 800 and you might notice if I remove the 800 for now we are getting a list of possible values from this Auto completion list so in order to get this Auto completion you need to install an
[06:13] extension on vs code and that extension is called telwynth CSS intellisense so after you've installed that you should receive these Auto completions showing up so for the background I will make it gray 800. we do not need to make any more changes to the global CSS file so we can close this now back in our home page we can start creating the structure for our application and since this is not a CSS or HTML course we are going to go through this quite quickly so you are
[06:45] welcome to code along or you can just download the code from my GitHub repo I'll leave a link in the description right so for the main attribute we will add the class Flex in order to align the content then I will set a Max width of of 6xl we can then align the content in the center of the page with MX Auto and I want the content to be centered in the middle of this page so we will call items Center also we will call justify Center as well and I would like this
[07:17] container to have some padding on the top and bottom and I'll do that by specifying a padding of 24. let's also add an H1 tag and this will contain our title for our page which I'll call Lang chain JS demo and I'll add a few icons as well by pressing the Windows key and period and I'll then click on this little bird as well as the chain and this will represent our Lang chain logo let's save this and after saving this we can see our text showing up on the
[07:48] screen but the color is quite dark so we can add styling to our H1 tag by adding the property class name let's add a few styles to this like text Gray 200 and on make the font extra bold the text size will be text 6 XL and I want the text centered and because my screen is zoomed in quite a bit for this recording I'm just going to zoom out on the browser slightly so everything fits within the view then below our title we want to create a form and within this form and
[08:19] within our form we will add an input field which we need to Style by providing the class 9 property and for the Styles We'll add some padding top and bottom of two padding left and right of four we will round the corners with a rounded of medium let's change the background color of the input field to gray 600 we will make the text white and we'll set the width of this field to full and we can now see our input field showing up on the right hand side of our title what we actually want this input field to show up below our title so in
[08:52] order to do that I'm going to wrap our title and the form in a div like so we can then add some styling to this containing Dev to align all these items vertically so we will use flex to align these items and flex column to align these items vertically and I also want to create a gap between these elements with a gap of 12. let's add some more properties to our input field so for this input field we also need to add a property called placeholder text and for
[09:24] this I'll just enter enter prompt in order to access the value of this input field later on in our code we need to give it a name and I will call it prong and then lastly this field needs to be mandatory so I will just call the property required and we now have this input field so next let's also add a button for submitting this prompt so for the button value I'll enter Saint chat let's go ahead install this button by providing a few class names I'll add padding top and bottom of two padding
[09:55] left and right of four this round the corners with rounded medium and will make the text small we'll set a background color of background lime 700 will make the text white and let's save this change and we can now see our button showing up on the screen let's also adds some styling for hovering over this button so we can add a new Clause called hover colon and we'll set the opacity of the button to 80 percent so if we now hover over the button we can
[10:26] see the opacity change the change is instant and I actually wanted to animate so I'll just add one more Clause to this button called transition opacity and if we now save this we can see the button gradually changing color as we hover over it and then lastly I just want to add a type to this button and this button will be of type submit it's also create a button to clear the chat so I'll just copy the original button but I will change the type of this button to type button we will then change the
[10:58] background color to background red 700 and let's change the text in the button to clear chat we can now see our two buttons showing up on the form this sorts out the spacing by wrapping the buttons in a containing Dev and let's give this Dev a few Styles like flicks justify Center and we'll add a gap of four and let's also add some padding top and bottom with py4 great we now need to hook up these buttons to functions
[11:29] whenever they are clicked for the first button it will submit the form and if we don't enter any prompt we will get this message because it is a required field so let's enter any value here and click synth chat you might have noticed that when we clicked on this button the page actually refreshed which is not what we want so let's sort that out on the form we can provide a property called on submit and we can now pause in a Handler function and we can now pause in a Handler function that will get triggered
[12:00] when this form is submitted I will call this function handle chat submit let's go ahead and create this function within this component and just before the return statement we can create a new function called handle chat subnet which takes in the event as a parameter and on event we can call a function called prevent default and this will prevent the browser from refreshing when we click on send chat so let's save this and after saving this you should get an error message like this and this is
[12:32] normal for next 13 and without going into too much detail next 13 renders all of these components on the server by default but when we introduce user interactivity like a form it won't be able to handle that on the server so instead we need to add the following at the top of this file and within quotes we need to specify use client and this will tell next not the render this component on the server but despite will be rendered in the browser instead we can now get the contents of our form by
[13:04] creating a new variable and I'll call it form data form data is equal to new form data this is a built-in JavaScript class and form data will take in the event dot current Target and this will now give us access to the input field in our form we can test that out by writing this to the console and on form data there's a function called get and with ingit we can specify the name of the field that we're trying to access and we called our
[13:35] input field prompt let's go ahead and save this and then let's open up the console in the browser to see if we get that value so in the prompt I'll just type test and then let's click on Saint chat and we can see in the console we are getting the value from our input field great that is enough for this function for now we will get back to it once we've created our API endpoints it's also create create a Handler function for the clear chat button we will call this function handle clear chat which takes in no parameters so
[14:08] before we continue with this function let me first explain what our application is going to do when we click on Saint chat we will invoke our handle chat submit function and this is going to call an API endpoint which we are going to create and it's that endpoint that's going to call all the Lang chain Logic the response from that endpoint will be a stream as you remember when we looked at the demo we could see the response writing out to the screen as it's being generated so land chain will
[14:38] passes the response in chunks and we will then write out the response as and when each of these chunks are received so we need to store that response in some state so let's go ahead and create that state variable in order to use State we need to import use state from react and then let's create our style variable above our Handler functions by calling const square brackets in these square brackets we need to give our variable a name like streamed data and
[15:10] for the second parameter we need to provide the name of the seter function for changing the value of the stream data State variable we will simply call this function set streamed data and this will be equal to use State and within you state we need to provide the default value of the state variable which is a blank string by default so basically after we call our endpoint we will write those chunks as they are being received through the streamed data State variable
[15:40] and we can set the value of this variable using the seter function when we click on clearchat it will invoke this function and all this function is going to do is it's going to call set streamed data and it will set the value back to space and that's it for the handle clear chat function we just need to hook that up to this button so on our button we will specify a property called on click and we will pause our render function to this event so we can't see it yet but when we click on clear chat
[16:11] it will clear the value in our strength title State variable so lastly if we do have values in our streamed data variable in fact for testing sake let's add some value in here I'll just call it test123 and if we do have a value in that variable we want to write it to the screen and we want to Output it just below the form tag so below form I will first provide a JavaScript condition because we only want to render the response if the variable has some data
[16:43] in it so we can do that by saying streamed data which is basically the same as saying not null so if stream data is valid or contains values we want to Output the following and what we want to Output is a containing Dev for our our output I'll also add an H3 heading with a text AI assistant this gives this heading a few Styles like text to XL and text Gray 400 then below the heading we will output a paragraph and within this
[17:15] paragraph We will simply output our streamed data let's give our paragraph content a few Styles like text Gray 200 rounded medium and background gray 700 and some padding all around and now we can see the output of this variable if we go ahead and clear the string so let's set it back to blank that entire section is hidden and that is because of this condition over here right I think that's everything in terms of the front
[17:45] end so let's go ahead and create our API endpoints within nextgi is there's two ways to create API endpoints there are the traditional API routes then there is also a newer solution called route handlers so what I want to do is show you both ways of implementing these apis because there is a benefit to learning both ways and I think the best place to start is to look at the traditional way of creating apis so what we can do is in the root of our project create a new
[18:17] folder called Pages then within Pages create a new folder called API and within API we can create our API route which I'll call chat and within chat we need to create an index.js file and in order to expose this as an API endpoint we can export a default async function called Handler this function takes in two parameters called request and response so the first thing I want to do is to extract The Prompt value from the
[18:49] request body we can do this by typing const in square brackets and this is equal to request dot body we can now copy the this solution that we built in the previous video so we can copy these two Imports and then add those at the top of our API endpoint we don't have to copy the dot EnV code AS next.js offers both in environment variable support so what we need to do is copy all of this code like so and then paste that into
[19:19] our method call so it should look something like this now we can replace this string with a prompt that we extracted from the body so this way it's using the value from this input field instead of some hard-coded value so let's also set up the environment variables by going to the dot EnV file in our previous example let's go ahead and copy this openai API key and then in our next GIS project at the root of the project create a new file called Dot
[19:49] env.local and then paste in your openai API key into this file and then save in order for next year is to pick up these changes we need to you stop the server by pressing Ctrl C then Y and enter and let's simply run our Dev server again this time it should have a view of this environment variable value so now we need to make a few small changes to this code if we had to run this API endpoint the response will simply be written to the terminal as we saw it in the
[20:20] previous video but what we want instead is for these tokens or these chunks of text to be streamed in the response of this service to do this is quite easy all we have to do is replace this code here with the response object which we got from the input of this Handler function on response we can call a method called write and we can then pass in the token as input so as line chain generates these responses it will write each token to the response of the
[20:51] service then after all of this processing has taken place we want to call another method on the response object call response dot end and this will close out the stream and that's really it let's go back to our page file and let's implement this logic let's go ahead and remove this console log and let's call our API endpoint I will create a new variable called response and this is equal to a white Fetch and we want to fetch that endpoint that is located at API slash chat we do not have
[21:24] to specify pages and you do not have to include index either nexture is for automatically scan the Pages directory for this endpoint and in order to call a white we need to convert this function to an asynchronous function and for the fetch API we also need to provide an object where we set the method to post within the body we will call Json Dot stringify and this object will take in the prompt property and to get the value of prompt we will call formdata Dot get
[21:56] and with ingit we need to specify the name of the input field which was prompt then lastly we also have to specify a header's property and four headers we will pass in a property called content type with a value of application slash Json so this code will now call this API endpoint passing in the prompt value in the body this will now call our model and it will write the chunks of text or
[22:27] tokens to the response as we receive it from openai because we are receiving a streamable object and not a simple Json object we need a way of reading the contents of the response so we can create a new reader by calling response body and on the body we will get a function called get reader and this will now allow us to read the stream that is returned by our API so openai will return the response in chunks and for
[22:58] long as we are receiving a response from Lang chain or open AI we want to set the value of our stream data so we can do this by creating a wall statement with a value of true so this will Loop until we exit out of it programmatically we can then get the value from the stream by the structuring done and value from reader dot read oh apologies I need to move this code up into our handle function So within this while loop we'll
[23:30] keep getting values coming back from the body and once done is set to true we want to exit out of this Loop so we can write this condition that says if done is true then break out of this Loop else we want to decode the value that we got back from the stream by calling new text decoder dot decode and value and we are doing this because the value that we get back from the stream is not in a text format or a format that we can use and
[24:03] the decoder will convert this into a string value we can now call our seter function set streamed data to now include this text value so we will take the current state values and in simply a paint the sticks value to the end of it set stream data will give us the previous data as a parameter and what we need to return is the previous data plus this text value so this will keep a painting that takes value to the end of
[24:33] the previous data value let's go ahead and test this out so in the prompt I will just say hello and let's click on Saint chat so after running this I am getting an error message and that is because we haven't installed the Lang chain package yet so back in our terminal let's stop our Dev server and then let's install the line chain package by calling npm install Lang chain and let's give that a few seconds to install after installing land chain let's go ahead and run the dev server
[25:04] again and I'm going to refresh the app and let's try to run this again and after a second or two we can see the response streaming back to our front end and we can click this clear chat button to clear this chat is done so let's now have a look at how we can convert this card that uses the older API route solution in next to the newer route Handler solution offered by next 13. for Route handlers we will not be using the Pages directory but instead we will create a new folder in the app directory
[25:35] called API within API we can create a new folder and this folder name should not conflict with another folder name in the Pages directory so I'm just going to call this endpoint Lang chain then within the land chain folder we need to create a file called route.js and we can create an endpoint in the route file by calling export async function and we can then specify the name of the method that we want to
[26:05] implement and this is a post method this function will receive the request as a parameter and then in the code we can destructure the value of our prompt by calling away request dot Json and lastly we can return a response from this endpoint by calling return new response and we can then pass a value to the response like hello implementing the logic for the route Handler is very different to the logic in the traditional API route file and that is
[26:38] because of how this response object behaves if we flip back to this API solution for a second we can see that we are getting this response object from the parameter list in this function and this response object makes it very easy the stream data to it with this write method unfortunately with Rod handlers it's not that simple we basically first need to create a streamable object and then we will write our chunks to that streamable object and it is that streamable object that we will pass back
[27:09] in this response this will all make sense in a second I promise to keep this code nice and lean because we will be writing quite a fair amount of code I'm going to create a new helper function and I'll call this function run llm chain this function will take our prompt as input and it is this function that will call our Lang chain logic this function therefore also needs to be asynchronous we can then copy the imports from this other API endpoint and
[27:40] we'll just place it at the top of this file and I'll also copy the logic for calling the model like so so let's start work on this function in the route Handler solution we do not have access to this raise object so we need to create something similar by creating a new variable called stream and this is equal to a new transform stream and in order to pass values to the stream we will create another object called writer which is equal to stream dot writable
[28:14] dot get writer so now we have a streamable object that we can return in this function and we have a writer object that we can use to write these chunks to the stream we also did not have to await this response we can just let this call run in the background and as it runs it's going to invoke this function passing in a chunk of text also called a token and we need to write this token to our Stream So to use our stream
[28:45] we first want to make sure that our writer is available and this object retains a promise which we need to await so therefore we need to set this handle llm new token function to an async function once the writer is ready we can then write a token to the stream by calling writer dot right and we would be tamed it just simply pass the token to the stream but this will not work as the readable stream expects us to pass the data to it in a specific format which
[29:17] cannot be string values so we need to encode the data somehow and it's quite easy to do we just have to instantiate a new variable called encoder which calls a built-in class in JavaScript called text encoder so now we can use this encoder object to convert our string or token to a format that the stream understands so we will call encoder dot in code and to be sure that we are passing a string to this I will pass in
[29:49] the token as a string literal then we also want to implement a Handler function that runs when the stream is complete and we will use this to to close the stream on our end and we will use this to close the stream once everything's done this is called handle llm end we will wait until the writer is ready and then we will call writer Dot close so this will now write our tokens to our stream and it is this stream that we need to return at the end of this
[30:21] function and more specifically on the stream object there's a property called readable which will return the readable Stream So I believe we are now done with this helper function and we can now call this function in our post method I will now create a new variable called stream which is equal to run llm chain and we will pass it the prompt value and in the response we will now pass back stream but because stream returns a promise we need to await the call to stream like so
[30:53] now in our front end let's change the end point from API slash chat to API slash Lang chain instead so this will now invoke our route Handler endpoint instead so let's test that out in the prompt I'll enter something like write a poem about the universe let's say in this and as you can see the response is now streaming to our front end personally I don't think there's an issue with either approach the
[31:24] implementation using the older public API routes is very simple compared to the route Handler solution where we need to create a streamable object and return that back as the response but of course the router Handler solution is the newer solution introduced in next.js but it is really up to you to decide which approach you want to follow if you decide to use a different framework than next and maybe you decided to create a simple Express app the logic in the
[31:54] express app will look very similar to this as Express also gives to access to the response object so now we have a very user-friendly way of interacting with our line chain applications going forward please consider subscribing to my channel where I cover all sorts of topics on artificial intelligence and creating full stack applications I'll see you in the next one bye bye