Unlock the Power of Text-to-Speech AI: ElevenLabs Tutorial for Beginners
OpenAIVintage📅 2023-04-15
Tutorial Overview
┌─────────────────────────────────────────────────────────────────────┐
│ LEARNING PATH │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ LEARN │
│ ───── │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Watch │───►│ Understand │───►│ Follow │ │
│ │ Tutorial │ │ Concepts │ │ Along │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │
│ ▼ │
│ PRACTICE │
│ ──────── │
│ ┌─────────────┐ ┌─────────────┐ │
│ │ Try It │───►│ Experiment │ │
│ │ Yourself │ │ & Adapt │ │
│ └─────────────┘ └─────────────┘ │
│ │ │
│ ▼ │
│ MASTER │
│ ────── │
│ ┌─────────────┐ ┌─────────────┐ │
│ │ Build │───►│ Share & │ │
│ │ Your Own │ │ Iterate │ │
│ └─────────────┘ └─────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘Key Takeaways
- Core concepts explained
- Step-by-step implementation
- Practical examples
Transcript
[00:00] introducing the 11 Labs text-to-speech API the 11 Labs text-to-speech model is an AI speech synthesis model that can translate text into a realistic and natural sounding voice this technology has many use cases inaccessibility education language translation gaming and more Leon this sounds good and all but how can I use this API in my own application I'm so glad you asked in this video I will show you how you can use the 11
[00:31] Labs API to add text to speech to your own application we will fetch a list of available voices from 11 labs and then generate audio from a web page welcome to my 11 Labs API tutorial although this example seems simple you will learn how to create a reusable API for interacting with the 11 Labs API in your very own application as for the prerequisites this tutorial is beginner friendly and you only need a very basic understanding of JavaScript to follow along you also
[01:04] need node installed go to nodejs.org and install the LTS version of node you will also need a code editor and in this tutorial I'll be using vs code because we need both a front-end and server-side code I'll be using a framework called nextjs knowledge of react and mixture is is beneficial but it's completely optional in order to follow along so let's begin create a new folder for your project and then open up the command line I'll use the integrated terminal in vs code we
[01:35] can now set up our next JIS application by executing npx create Dash next Dash app at latest because I want to install the project in this directory or just enter period and then enter for typescript I'll say no for eslint I'll say no for television CSS or select yes for Source directory I'll say no and for the experimental app directory I'll choose yes for aliases I'll just press enter this will now install the dependencies for our next JS project
[02:07] let's clean up the project files as well first in the app directory we will be creating our own backend API for communicating with 11 Labs so I'll rename this sample hello folder to something like 11 Labs we can ignore the route file for now but we will come back to it a bit later on in the API section of this tutorial for building our front end we will make changes to the page.js file in this page file we can remove all of these Imports including the font we
[02:39] can also remove everything in this return statement so from the opening main tag to the closing main tag I will simply replace this with a main tag with some text hello world and I'll save this we can now ensure that our next JS project is working by starting the development server we can do this by running the command npm run Dev this will now start our development server at localhost 3000 we should now see our application loading let's clean up the folder structure a bit more let's remove some of these styles from the global CSS
[03:11] file or delete everything except these three lines let's save this we can now start building the front end of our application feel free to jump to the API section of this video if that's all you're interested in the first thing I'd like to change is the font used in our project we can change the font by opening up the the layout.js file then at the top of the file we can import Poppins from the next package fonts and then Google we then need to create a variable for our font so I will call it
[03:42] Poppins I will say that equals to Poppins this likes an object as its input so first we need to specify the weights as an array so I will import the following bytes for our project then we need to specify the subsets as an array and I'll select Latin only we can then add this font to the body tag by adding class names we will set that equal to an object and we will pass in Poppins dot class name after saving this you should see the font changing in the application
[04:12] I would also like to change the background color and add some other styles to our application so in the globals.css file we will add some base styling to the application so I will write at layer base we will then specify the body tag and we'll open apply our Tailwind style sheets by typing add apply we'll add a background of blue 300 we will give the body a height of 100 viewport height and we will align the contents of the body with flicks item
[04:42] Center and justify Center this will move all of the contents in our body page to the center of the page we can now close the global style sheet and then go back to our page file let's remove the hello world text we will then add a new div which will serve as a wrapper for our application let's give this div some styles by adding class name I'll give it a background of white vertical padding of four horizontal padding of four rounded off medium then within this div I'll add a new H3 tag with the text say
[05:14] something we'll give our H3 a few Styles like text to x a font of bold text blue of 800 we'll convert the text to uppercase with a margin bottom of six below the H3 tag will add the input fields for our form so I'll create a new wrapping div or give this Dev a few Styles like margin top and bottom of six I'll align the contents with flicks and flex column for vertical alignment and a
[05:45] gap of 4 between the items I'll add another div that will contain the drop down list for the voices and the label text for the drop down first We'll add a label for the drop down with a text select a voice then below the label We'll add a select element we'll just give it one example option for now so we'll add an option element with the text below this gives the select a few styles by adding class name or give it a background blue of 100 padding top and
[06:17] bottom of 2 padding left and right of four and we'll round the edges with rounded large this gives the div that wraps these two elements a few Styles as well so on this div We'll add class name we'll align everything with flex with a gap of four and we'll align the items Center that's created some spicing between these two elements then below this wrapping div we'll add a text area which is a self-closing element let's give this a few Styles
[06:48] like padding four all around we'll give it a border and they border blue of 100 or around the corners as well with rounded large we'll disable the outline with outline none this also change the placeholder text with placeholder gray 400 and when we focus on the box I want a little drop shadow to appear so I will add a focus within state so this will trigger whenever the user focuses on the
[07:20] box and for this we will add drop shadow of medium so you will notice when I click into this box a little drop shadow appears this also gives the text area some placeholder text so I will add a property called placeholder and here you can add any text next I do want to make this input box a bit bigger so I'll also add properties for columns which will make 50 for rows will make it 10 and after saving that this looks a lot better then
[07:52] in order to have access to the value in the drop down and the value in the text box field will create references through these fields using the use ref hook that comes with react so at the top of the file we will import use ref from react note that when we save this we will get an error message that is because components in next year is our server rendered components by default so if we need to give the user the ability to interact with the page we first need to convert this component to
[08:23] a client component this is real easy all we have to do is at the top of the page in quotes We'll add use client after saving this you shouldn't get any error messages then within the home component we'll create new variables first we'll create a variable called voice ref we will say that equal to use ref then we'll create another variable called text ref which people said equal to use ref we can now create these references by going to the select element and
[08:54] adding a property called ref which we will set equal to voice ref then on the text area we'll also add a ref property and set that equal to text ref we can now use these references to get the values of these fields then below the text area we'll create a button we'll give it some text like generate text to speech we'll give this button a few styles so in class name We'll add padding top and bottom of two padding left and right
[09:25] of four a background blue of 800 will make the text white all around the borders with rounded large and we'll add a hover state so when the user hovers over the button we'll simply set the opacity to 80 percent after saving this we can see the button and when we hover over the button the opacity changes slightly we also need to create an on click event handler for the button so when we click the button we want to call a function called handle generate text
[09:56] to speech Let's go ahead and create this function so just above the return statement I'll just add a comment and then let's create our function or call it handle text to speech it doesn't take any parameters as input and what this function will do is first it will get the value of the selected voice and it will get the value of the text that we've entered it will then call in API which will create in this 11 Labs folder a bit later on and it will pass the value of the voice and the sticks to the
[10:26] API and the API will return back to us the name of the audio file that was generated but we'll get back to that a bit later on so for now when we click the button they simply get the values of these fields and write them to the console so create a variable called selected voice which will set equal to voice ref dot current dot value we'll create another variable called text which will set equal to text ref but current pod value let's write these to
[10:57] the console so I'll write selected voice to the console as well as text let's save this let's enter some text in this box let's open up the console so you can go to your developer tools open up the console and let's click on generate text to speech and we can see our two values pulling through now that we have our user interface completed let's work on the back end logic in order to make use of the 11 Labs API you need to create an account on 11 labs and sign up for one of the packages 11labs offers a very
[11:28] good free package which grants you 10 000 characters per month and for only five dollars a month you can use this model for commercial use and to get up to 30 000 characters and what's very cool is you get access to instant voice cloning which means you can train a model on your own voice and this is something I'll cover in a future video but for now simply select the free package and create your account after logging in you should see a dashboard like this basically at this point you can test out the different models and
[11:58] try out this different sounds so under settings you can select from a range of voices and you can click this little play button to sample the voices if you do what you've always done you'll get what you've always got you also have access to some other settings that you can play with to get some interesting results but in our demo we'll pretty much mimic this interface where we will pull through a list of voices these voices here with this text box and after clicking on generate will get the audio back as an audio file which will store
[12:29] on our local node server but what we need from here is the developer API key to get this click on your profile then open up a profile and on this pop-up you can see the API key simply click on this little button to view the key and then copy the key after copying this key you're welcome to close this browser window so back in our project we will store this key in an environment variable file in the root of the directory create a new file and call it dot EnV dot local we will then create a
[13:01] variable name so I'll call it 11 Labs API key and I'll set that equal to the key that I just copied after doing this you can save the environment variable file in order for the project to see this new variable we have to restart our development server you can install Auto server in the terminal by pressing Ctrl C and then Y and enter let's run the server Again by running npm run Dev we will now go ahead and develop our 11
[13:31] Labs API route in the API folder under the 11 Labs double click on the route file the first thing I want to do is change this from a git to a post method instead we will be passing the selected voice and the message text to this API route so we can get those values by calling const message and voice and we will get that by doing an await on request dot Json this will basically take the request that's coming into this
[14:02] service and then extract our two variables from the request and we will call those values message and voice if I flip back to the 11 Labs website for a second I just want to show you how you can view the available apis for the 11 Labs after logging into your dashboard you can click on resources and then click on API and from here you can see all the available apis and examples on how to call these apis so for text to speech we will be using the first entry
[14:32] year which is a post method this post request takes in two important parameters it will take in the voice ID which is basically the voice that we see in this drop down list and it also takes in your developer API key which is what we stored in the environment variable file then for the body of the service we will pass in the text and optionally we can specify the voice settings adjustability and the similarity boost these are the values that you see under the voice settings section over here but
[15:04] for our demo we will not be setting these but you are more than welcome to play with it in your application the response of this service if it is successful will be a blob containing the generated audio we will take that information and store it as a file on our see so the first thing we need to do is call the 11 Labs API I'm going to write this in a try catch block since the service good file and if we receive any errors I will return a new response from the service where we stringify a
[15:35] new object with only one property and that's error for the value of error I'll return error Dot message I'm going to remove this response to our try block then in our try block we will call the fetch method and we will pass in the API endpoint and for the URL of the endpoint we can actually get the API endpoint in the documentation by clicking on try it out we can just use this defaulted voice ID we can leave the API key blank on this screen and then click on execute
[16:06] after clicking execute we should get a sample curl object and from here we can see the API endpoint for this request I'll just copy this and just note that this value here represents the voice ID e which was defaulted over here so back in our route I'll paste in at URL and we know that this value will be dynamic so I'm going to convert this to a string literal in state and then the voice ID will be a variable so we can use the dollar sign and curly prices and this
[16:36] will set equal to voice which will be the ID it was passed into the request of this API call then the fetch API takes in an object we will specify the method as being post and from the sample URL we can see that we've got two header objects that we need to set so I'll also provide the header object first will be a property called accept with the value of audio slash impact we also need to specify the content type as application slash Json then we also have to provide
[17:09] our API Key by adding a property called x i Dash API Dash key with a value of process dot EnV the this is fetching values from our environment variable file and we called our variable 11 Labs underscore API underscore key then for the body we will call json.stringify within stringify we'll pause in an object we will pause our text by passing in the value message and
[17:40] will also pass in the voice settings which is an object with two values so we will pass in stability as zero and similarity boost of zero as well we are hardcoding these values but if you wanted to feel free to play with these values or as a little side challenge client add these values to the user interface for the user to change Facebook return a promise which we can await and it's assign the result to a variable which I'll call response after the API call we will add an if statement
[18:12] and say if response is not okay so if the service call failed we want to handle this so if something goes wrong we will throw a new error with the text something went wrong so if this was all okay and we received a response in other words the response that we got back from 11 Labs will actually contain the value of an audio file we need to take the contents of that response and then write it to a file this code can seem a bit confusing but just bear with me or feel
[18:43] free to copy it from my GitHub repo so first I'll create a variable called array buffer and I'll set that equal to the response that we got back and on that response there's a function called array buffer after getting the contents of the response we will create a new variable called buffer and say that equal to buffer Dot from array buffer we now have the contents of the response in this variable buffer now we want to write this gibberish to an audio file on
[19:14] our server so I'll just create a new variable called file which will represent the the file name for our audio file this is really all up to you I found a little lag to generate a unique file name using the math class so I will call math dot random then we will call tostring 36 and then only grab the first few characters from this for example the first seven characters now that we have the contents of the audio file in this buffer variable and we've
[19:45] set the file name we now need to write the file to the file system and in order to write falsely the file system we need to import the file system package so at the top of the file I will import FS from the file system package I do want to import one more thing I will import path from the path package this will make it easy for us to specify the folder in which we want to write this file so going back to our code we can now call file system and on file system there's a method called Write file so
[20:17] write file takes in two parameters first is the location where the file should be written to and the second is the content of the file so in order to create this location of the file we will call the path method dot join and I want to write this file to our public folder within public we'll create a new folder called audio and then for the file name I'll use a string literal I will pass in the file name that we created up here dot MP3 and then for the second parameter
[20:48] which is the content of the file we will pass in the buffer value which we created up here then for the Callback function which will run once the file has been written we will pause in an anonymous function and in this function we'll simply console log a message that says fall written successfully so after this has written the file to the file system we want to return a successful response so instead of hello next year is I'll replace this with Json Dot
[21:19] stringify I in stringify will pass an object with the file property and this will be the file name that was just generated so we'll pass in dollar sign curly braces with a variable file dot MP3 right so just to recap when we call this API we'll pause in the message text as well as the voice ID we then call the 11 Labs API passing in the voice ID as well as the message text this will then return the contents of an audio file and
[21:50] we then use the file system method write file to write this contains to a file in the public folder so in public I'm just going to create a folder called audio as well so once we generate the audio we should see the files being written through this audio folder if this was successful we will then return the file name to our application so let's go back to our app so let's go ahead and add the logic to our handle generate text to speech function let's remove these
[22:21] console logs and then add the logic to our own API call I'll wrap this in a try catch block we'll simply write any errors to the console let's add some basic validation so we'll say if text is null or text Dot trim equals space we will show an alert like enter sum text and for return then what this code will do we'll call the fetch API specifically we will call the API in our API route which is called 11 Labs so it's this
[22:53] name over here the fetch API also takes in an object we will pass in the method as post for the headers we will pass in the content type as application Json then for the body we will call Json dot stringify for stringify we'll pass in an object with two values message which is equal to text and voice which is equal to selected voice so phase returns a promise which we need to await and we'll assign the response to a variable called
[23:25] response in order to use the write function we need to convert this handle generate DSS function to an async function after calling the fetch API we will check that the response is okay in fact if the response is not okay we will throw a new error with some text like something went wrong if this was successful we will grab the data by calling a weight response dot Json so this data object if we go back to the route for a second this data object will
[23:57] give us a property called file which contains the name of the file so back in our page file we can actually destructure file from this response value we should be able to test this out now but we've got a small issue when calling this API the selected voice needs to be the voice ID of the voice that we'd like to use in this demo we simply setting the value as Bellar which is not valid so I'm just going to hard code the value for now we can get the sample ID by going to resources to the
[24:29] API Dock and I'll just grab the example that they give in the documentation so instead of passing selected voice for the time being or just passing that voice ID let's give this a spin we will type some text in our box and then click on generate text to speech once this is done we should see a new audio file showing up in the audio folder Let's test this out so I'll click on generate text to speech and after clicking on text to speech we can see this message in the console that says fall written successfully so looking at the route
[25:01] file that's coming from this console log over here also under the public folder and under the audio folder we can see this MP3 file every year I'll click on that and then play it I'm not sure if you heard that but I'll play it again hello world so we now have an audio file with the voice saying hello world great so it seems that our API is working and we are writing the generated audio to our audio directory the next thing we want to do is after generating the audio file we
[25:32] want a little audio element to display on this page and we want the audio to play automatic and this audio element will give us the ability to play the audio again or download the audio file let's work on that next this audio element should only show once an audio file is available so we'll need to use state to keep track of that so back in our page file from react we will also import use State then in the home function or create a new state which will store the name of the audio file
[26:02] that was generated and only when this value is saved should be display the audio controls so to create State you specify a new variable and then these square brackets we will give our state variable a name I'll just call it audio and we then need to create SATA function for setting the value of audio I will call the setter function set audio we'll set that equal to use State and initially the value will be null because it takes some amount of time for this fetch API to return a value so as we're
[26:35] waiting for the 11 Labs API to come back with the audio data and the time that it takes for the file system to write the file we are sort of writing in limbo so I'm also going to create a loading state by calling const loading set loading and I'll set that equal to use state with an initial value of false so what we can do now is above the try block we can set the loading state to True once all of this processing is complete we want to set the loading State back to false So
[27:06] Below the catch block We'll add a new block called finally and in finally we will call the set loading function and we'll set the loading state to false the finally block will EXE cute whether or not our trilogic was successful or not so if we do get a response back we will now set our audio state to the file name so now what I'll do is on the button we will add a property called disabled and if the lighting state is true the button will be disabled and the user will not
[27:37] be able to click the button again while processing is busy if it's false the button will be clickable we can also make it a bit more user friendly by adding a condition around this text where we say that if the loading state is true the more write the text generating please wait else we will write the text generate text to speech let's try that again so I'll type the text hello world and I'll click on generate text to speech so for a brief moment there we would have seen the loading state of the
[28:07] button and we can see a new file in our audio directory so our loading state is working let's add the audio element to our page so below the button I'll add a new JavaScript condition which says that if if the audio element is not initial then we will create the audio HTML element which is a self-closing element the source of this element I'll add a string literal which says we can find this element under the audio folder slash and then the value of the audio
[28:39] state which will be the MP3 file name note that we do not have to specify the public folder here as all static assets are assumed to sit in the public folder let's save this let's generate another file then let's also add a property called controls and save so after saving controls we can now see all of these controls popping up in the element including the play button hello world so clicking this place our audio file hello world there's one more property that I'd like to add and that's called auto play
[29:10] so once this audio element renders to the screen I want it to automatically start playing the audio let's save this and refresh the app this generates some new audio I'll just give it some text like hello there and let's click on generate after clicking on this button and after the file was generated the audio should start playing by itself hello there awesome finally let's remove the hard coding for The Voice ID and let's feature list of available voices from the 11 Labs API so what we need to
[29:40] do is when this page is loaded for the first time we want to call the fetch API on 11 labs to fetch the available voices and then render out those voices in this select drop down if I switch back over to the 11 Labs API page for a second you can see that under the voices section this is get request over here called voices the description says get voices and one thing that's interesting about this specific endpoint is although it states to pass in the developer key in
[30:11] the header it's actually not necessary so because we don't need access to the API key which should only be available on the server side we can actually just call this API from our front end without specifying this API key let me show you how so when the space loads for the first time we want to call a lifecycle hook called use effect which will run once when the page loads and it will fetch the available list of voices from 11 Labs so from react we will call use
[30:42] effect and then just above the return statement we will call the use effect function which takes in a function module as input as well as an array of dependencies if we leave this array initial this use effect hook will only run once when the page loads which is what we want in this function we will call the API on 11 labs to get back the list of voices because the fetch API returns a promise we need to avoid the result so we have two options here either we can call the fetch API and
[31:14] change the then function on top of that or we can await the result like this the only problem is that in order to call the right function the scrolling function needs to be async and we should not convert the use effect function to icing so what I'll do instead is I'll create a new async function called get voices and I will move my logic into this function so for fetch we will call the following URL https API dot 11labs.io
[31:44] V1 slash voices because this is a get request we do not need to specify any other attributes I will assign the result of this fetch to a variable called response if response is not okay we'll throw a new error else or create a variable called data and the right response Dot Json so in this data variable we will now be able to fetch the list of voices in a property called voices I want to store this array of voices in another state variable So
[32:15] Below these states I'll create a new cons called voices with the setter function of set voices we'll set this equal to use State and the default value will just be a blank array so back in our code we will call set voices and we'll make that equal to data dot voices also because this API could fail for a number of reasons I do want to wrap this in a tri-cat block so I'll move all of this logic into my try block and in the catch we'll simply write the message to
[32:46] the console we can now call the get voices function in our use effect so this means that when the page lies For the First Time The use effect recall run which will call the get voices function get voices will go to the 11 Labs API to get the list of available voices if we get the results back we set our new state variable voices to the array that we got back from 11 Labs we can now Dynamic we can now dynamically render out the list of available voices in this select element so let's go back to our
[33:18] select element we can remove this option here and instead We'll add curly braces and we'll map through our voices array which will give us the specific voice that we on and for each voice will return an option tag the text for the option will be voice dot name and on the option element itself we will specify a property called value and this will be equal to voice dot voice ID we also need to specify a key property because we are
[33:48] mapping through an array in react the value of this key property will be voice dot voice ID so after doing this this is fix up one typo back in our use effect this response should be response like that and after saving this we can see a change in our application and we can actually see the different voices pulling through now and if we look at our HTML elements we can see all the names pulling through and the value of these options are the voice IDs great so what we can do now is we can go back to
[34:19] our handle generate TTS function and selected voice one I'll give us the correct voice ID so instead of passing a hard-coded value let's replace that with selected voice so let's try this I'll add some text like Hello World we'll select Rachel and let's click on generate text-to-speech hello world let's change this to the next name Domi hello world and clearly we're getting a different voice let's change it to Josh
[34:52] hello world excellent stuff you are now able to integrate text-to-speech into any of your applications I hope this video was informative if you like this type of content and content on AI in general please consider subscribing to my channel and please like this video it really helps the channel grow and thank you very much for all the support the channel just hit a thousand subscribers and we're getting close to 2 000 subscribers I'm extremely grateful for all the support that you guys have shown and for all the positive comments that
[35:23] I've received I look forward to seeing you in the next video bye bye