Hi, In this tutorial, I am going to teach you how to use Wit.ai through a fun mini project. By the end of this project, you will understand the basics of Natural Language Processing, intent recognition and entity extraction. You will also learn how to use wit.ai for your use cases.
- No technical expertise needed (beginner friendly).
- Basic idea of Python and API calls (Optional).
- Requires Python with version > 3 and streamlit library (Steps to download mentioned in the tutorial).
Ok, so let us consider a real world example to get ourselves excited ❗. How do big companies like Amazon, Flipkart evaluate the customer reviews for a product or automate a considerable amount of their customer care services? This is done with the help of Natural Language Processing (abbreviated as NLP). It is a branch of Artificial Intelligence/ Machine Learning that deals with the interaction between computers and humans through the medium of natural language. This could be text or speech. We ultimately want to derive meaningful insights from the end user’s input.
Now, let us learn about 2 important concepts in NLP which will help us in implementing the project that follows this theory. First, let us understand what intent recognition is. As the name suggests, we expect the computer to infer the intent from user’s input (text or audio) , understand the purpose of the message and thereby take meaningful actions.
Example of this in everyday life: When you ask Siri or google assistant to set a reminder to wake you up in the morning, it doesn’t call your most frequented contact which is totally irrelevant from our intended action. This is because it rightly understands what you wanted it to do and hence, does the same.
Creative Commons License
Formal definition of Intent Classification:
Intent classification is the automated association of text to a specific purpose or goal. In essence, a classifier analyzes pieces of text and categorizes them into intents such as Purchase, Downgrade, Unsubscribe, and Demo Request.
source: google
Now, let us move on to what entity extraction is…. Continuing with the theme of the above voice assistant example, the term 'morning' in your request “Set a reminder to wake me up in the morning” is somewhat vague.
A more meaningful input to the device would be, “Wake me up every day at 7 am”. Now, your assistant wakes you up daily exactly at 7 am because not only did it recognize the intent of the user but it also extracted the entity. In this case, the assistant understands that the value ‘7’ refers to the time.
Note that people who built this algorithm had to define numbers like this to be extracted as 'time' before-hand and trained the AI algorithm accordingly.
Formal definition of Entity Recognition is:
Named-entity recognition is a subtask of information extraction that seeks to locate and classify named entities mentioned in unstructured text into pre-defined categories such as person names, organizations, locations, medical codes, time expressions.
source: google
Streamlit is an open source web framework that lets you build webpages using only Python 🐍. Thereby, removing the need for developers to use HTML, CSS and Js. It uses what is called as widgets/components to construct your website.
Developer Trivia: Streamlit components are essentially Python wrappers of React code (Judges, extra points 😁?).
pip install streamlit
Let us assume our file name to be 'streamlit_basics.py' and that we are in the correct directory.
streamlit run streamlit_basics.py
Streamlit code is very intuitive and thus, you can understand the purpose of each widget by simply comparing the code to the rendered webpage.
For Example: You can place 'Title'/large text on the webpage by passing a string of your choice to the .title() method.
You can learn more about Streamlit here:
https://www.streamlit.io/
🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊 🏊
We will be building a 'Code Editor Assistant' (chatbot) with the help of Python and Wit.ai to help developers build cool software fast (No heavy code,I promise).
Source: Virtual-assistant Illustration by Delesign Graphics (Creative Commons License)
You can interact with the final outcome of this project by visiting the following URL:
https://share.streamlit.io/sharan-babu/fb_wit/Devbot.py
Wit.ai is a Facebook owned open source chatbot framework with advanced natural language processing capabilities. The best part being you won’t have to code to train the NLP model (chatbot) or need prior knowledge of any kind. All you have to do is type and click and your chatbot is ready to be served in your app with the help of a simple API call.
Something that might interest you from a production/business perspective is that Wit.ai provides a generous free tier with up to 240 requests per minute per user and 60 requests per minute per app.
Now, let us build the chatbot from ground up using Wit.ai and I will be explaining the procedure one step at a time.
- Go to the URL https://wit.ai/
-
Click on Continue with Facebook account and fill the necessary details.
-
Your homepage should look something like this. If this is the first time you have logged in, you will also be guided with the help of an interactive tutorial. Nevertheless, you can continue reading through this tutorial to navigate your way through the platform. Now, click on the Create New App button.
-
Select the name for your app and preferred language and click 'Create'.
-
A new project will be created and your screen should look like this:
(Click on image to enlarge it)
-
To the left, you can find a handy menu bar which we will talk about in a minute and at the centre, you can see the main components with which we will be brewing our magic.
-
Now, let us understand how to create an intent and select present entities in the incoming prompt as per our use-case (Code Assistant).
-
In the Utterance text field, type what you think the end user might type. For example: A coder using our web app might type "arrange my list called 'elements' in ascending order". So, this becomes our utterance.
- Select the ‘Choose or add intent’ dropdown button. Now, enter an intent name (a name you are comfortable with) in the text field visible and click the ‘+ Create Intent’ button to the right. In this case, I will be naming my intent ‘arrange_list’.
- Time to select an entity: Select the text ‘elements’. As soon as you do this, you will see a new dropdown menu. Now enter an entity name of your choice in the ‘Entity for elements’ text field and click ‘+ Create Entity’. I will be naming it ‘list_name’. I would recommend you to follow this naming convention if you are making the app along with me.
You can now see that wit.ai has highlighted our entity.
-
Click on the Train and Validate button at the bottom and voila, you just added the first utterance to your chatbot. 👏 👏 👏
-
From what we have done so far, we can realize the role that the extracted intent and entity will be playing in the website's source code. For example, considering the utterance we just added, it is important to know what the user had named his/her 'list' as and what he/she wishes to do with it. With this information, we could return ‘elements = elements.sorted()’ to the user which is the right syntax for sorting elements in ascending order in Python.
-
In the same way, you can add multiple utterances with corresponding intent and entities to the chatbot. The more the better. Add a considerable amount of utterances for the trained model to be well generalized. Only then will it return correct outputs for the user’s input prompt. Try to think from the user’s perspective and add your utterances.
-
Do not forget to hit the 'Train and Validate' button after adding each utterance!!!
-
One useful thing you might notice while working with Wit.ai is that after you add a few examples for a specific intent, Wit automatically tries to classify it.
-
After the end of this project, I would consider adding some other intents like ‘create_list’ , ‘list_reverse’ and ‘list_max’ and the names are self-explanatory as to what they mean.
-
After you are done adding all your utterances to the chatbot, click on the Setting button under the Management section in the Left menu bar.
- Therein lies an important piece of information which is the Server Access token. Copy this as we will be needing it later.
That’s it. Our chatbot/ NLP agent is ready. Now, let us make a simple web interface for our code editor assistant.
If you wish to directly run the final program on your local machine,follow the instructions in the image below:
(OR) follow these steps and code along with me 💪.....
- Install required libraries
pip install streamlit wit streamlit-ace
'wit' is the library to be installed for interacting with the Wit API and 'streamlit-ace' library is to be installed to construct out of the box code-editors on the webpage as implementing one on our own can get quite complicated.
- Create a new python file called Devbot.py and paste the following code:
Read the comments in green to understand the code or follow along for an in-depth explanation of the code.
You can find the above code at the following Github repo link:
https://github.com/Sharan-Babu/FB_wit
- Here,you have one change to be made. At 'line 7' replace my 'access token' with yours which you copied from the settings page for the changes you made to take effect. You can use mine as well.
In-depth explanation of the code:
- We import the streamlit library as 'st' and you can see that this is used multiple times, using which we construct widgets like input box, menu bars and buttons.
- Other libraries to import are Wit and streamlit-ace (out of the box code editor).
- In lines 6-10, we initialize variables for easy readability. Line 8 connects us to the Wit chatbot.
- Lines 12-15 are completely optional. They are parameters of the streamlit-ace component. With this, you can change the supported languages, theme and feel of the editor among many other options.
- The 'output' function from lines 18-23 take the intent and entity extracted from the returned JSON(JavaScript Object Notation) as input and map them to the corresponding outputs. Later, if you decide to add other intents and range of use-cases then all you have to do is write a case/if-block for handling those intents and entities.
- Lines 24-40, we layout the widgets as per our choice on the webpage. As you can see, the streamlit-ace component has multiple changeable parameters like height,keybinding,font-size and tab-size among other things and they are all optional.
- Lines 44-57, we take input from the user with the help of a textfield, pass it to Wit and retrieve the JSON (similar to a dictionary in Python). The returned JSON looks like this:
- Line 48 and line 49 parse through the returned JSON and extract the intent and entity respectively. We wrap the entire block of code in a 'try...except' block to handle errors.
- Save the file. Open the terminal, go to the working directory and run the command:
streamlit run Devbot.py
If you instead want the simplest interface possible, you can use the following code and have a look at the resulting webpage (Compared to the previous code, we have simply replaced the code editor component with a text-field. Nothing else has been done).
That’s it. We are done and now you have a fully functional and useful web app that can help make the lives of devs a little bit easier. Congrats on making your first chatbot with Wit.ai and successfully integrating it with a web application. I would recommend you to make this app more robust or make other useful apps with your newly learnt skills!💃 💃
Scroll to the end to read about my experience with Wit and participating in this competition 👇
Youtube URL for the demo: https://youtu.be/OQjbIcpCgSE
Github repository link: https://github.com/Sharan-Babu/FB_wit
Hosted Website Link: https://share.streamlit.io/sharan-babu/fb_wit/Devbot.py
Please feel free to give your feedback @:
Email: sharanbabu2001@gmail.com
Linkedin: https://www.linkedin.com/in/sharan-babu-39a757197/
It all started with this Facebook Hackathon(2 months ago) as this was the first time I learnt Wit by reading the official documentation and accompanying videos. I experimented with various projects and finally decided on a voice application. Unfortunately, I could not win in that hackathon but learnt a lot about Wit and its features. Later, apart from the pleasant swags I received from Facebook for a successful submission, I was also invited to participate in a Week long Wit.ai JAM session. This was really wonderful for multiple reasons. I got to meet cool people who were using Wit in ways I had never imagined and this was the first time I was part of a professional community of devs 😄. Apart from that, we had daily sessions ranging from Lightning talks to Natural Language Understanding (NLU) which were all very insightful. Later, I was super stoked to learn that I had been selected as one of the 21 regional winners and was on ☁️9 . The tutorial you just read was updated with the feedback I received after the first round in mind. I hope you enjoyed reading it and had something to take away.
From a technical perspective, I like Wit.ai for NLP tasks more than its rivals like Dialogflow or Rasa.
To conclude, I would like to say that Wit has a special place in my heart among other technical skills and am glad that I could contribute to the community. I intend to write more tutorials on Wit.ai and help people learn to use this amazing chatbot framework.
Cheers!