India's diversity can provide a lot of fodder for conversational AI products as India's large base of internet users is generating data that can be used to fine-tune conversational AI solutions.
Currently, Conversational AI is Transactional at Best, Not Interactive
Conversational AI solutions are primarily of two types – Text (chatbots) and Voice (personal assistants) based. Chatbots were at the peak of technology hype cycle the last couple of years, but have not yet been able to deliver on the promise, neither in terms of breadth of reach or depth of skill. The primary reason is that the techniques for effective and efficient human-machine conversations are still in evolutionary stage and the technology industry hyped the first versions of these solutions, significantly underestimating the amount of time and effort it would take to make the machine interactions fluid, smooth, efficient and long-lasting.
The human-machine conversation is composed of two parts:
- Natural language understanding (NLU) — Understanding what the user said
- Natural language generation (NLG) — Formulating a reasonable and on-topic response to the user.
Much of the attention of late has been focused on that first part, but there are many challenges remaining on the generation side, and these tend not to be well suited to machine learning because response generation is not simply a product of collecting and analyzing lots of data. The challenge of maintaining a believable, ongoing, and stateful conversation will require more focus on these NLG and dialog management parts of the problem over the coming years.
Natural language processing is difficult because the tasks regarding linguistics need to include variations in human psychology, cultures and linguistic diversities. Most conversational experiences today are either very broad but shallow (e.g., “What’s the time?” = “The time is 10:00 am”) or very narrow but deep (e.g., a multi-turn conversation in a quiz game). To advance beyond these limited experiences, we will need to get to a world of both wide and deep conversations, which will require the machine’s ability to scale beyond the current technical limitations of recognizing between only a few hundred intents at a time. Another limitation of machine conversations is Personalization - In a natural conversation between two people, each will normally draw on previous experiences with the other converser and will tailor their responses to that person. Computer conversations that don’t do this tend to feel unnatural and even annoying. Addressing this in the long term will require solving challenges such as speaker identification, so that the computer knows who you are and can respond differently to you versus someone else.
Many enterprises and Conversational AI solution developers, in the last couple of years, have been busy developing chatbots of primarily two kinds:
- Enterprise Chatbots — Built to solve enterprise use cases such as customer support, lead generation, etc
- Direct-to-Consumer Bots — Chatbots that reach consumer directly for specific applications
Many of the bots developed were for providing Digital Assistance, Customer Experience, Conversational Commerce, Personalized Advertising and generating Digital Content. However, in most cases, bots are being built for the sake of it without much depth. Most of the enterprise bots currently available are automated versions of FAQs and are unable to hold conversations beyond the few interactive dialogs. Consumer interest will materialize when machine intelligence gets near human intelligence.
Having Said That, The Diversity, Scale and Capabilities Available in India Market Could be Leveraged to Make An Early Mark
India’s internet subscriber base has grown rapidly in the recent to reach close to 500 million subscribers, from 84 Mn in 2012. As per industry reports, India consumed 22% of world’s mobile data between April and June 2018. As per COAI, Indian telcos handle more data traffic than their Chinese and US counterparts, combined. This large and rapidly increasing base of internet users are generating data that can be used to build and fine-tune conversational AI products.
India has 22 official languages, and of its 1.35 Bn population, less than 10% can understand English. Of the current internet subscriber base, majority are non-english speakers. It is estimated that 93% of the next 326 million internet subscribers will be Indian languages first users. This number of Indian language internet users in India is estimated to reach 536 million by 2021 (according to Google-KPMG report). As per Google, 28% of its search queries are already in Indian languages, and this number is growing rapidly. India has the highest number of WhatsApp users in the world and also the highest number of app developers at over 3.2 million (according to Google, India estimates).
Various characteristics of Indian market including large and rapidly growing internet user base, generation of large amounts of digital content and data, multiple languages, bi-lingual conversations, intuitive understanding of multiple languages and colloquial structure leading to relaxed accuracy levels of chatbots/conversational AI solution requirements, large base of software products and app developers make it a conducive market to develop, launch and scale various conversational AI products with multiple functionalities.
The latest NLP technique of Embedded Language Models (ELMo), where-in the NLP algorithm trained on the data set of one language can adapt itself for other (similar, adjacent) languages, without training can work best in Indian scenario. For example, an ELMo based solution trained on Hindi language data set can adapt itself for Marathi and Punjabi as well, thereby scaling faster.
Many Companies are Focusing on India, Both as a Market and Innovation Hub
A significant number of Microsoft’s Cognitive Services’ 300,000 developers globally, are from India. Amazon, which periodically hosts training programmes to create conversation-based applications or ‘skills’ on its Alexa platform, has grown the number of developers from less than 10,000 to 40,000 in just a year of its presence in the country. For Amazon, India is the second biggest market (after US) for Alexa Skills.
Not just global companies, a number of Indian companies have also delved into conversational AI development. According to Tracxn, a startup database, there were over 100 chatbot startups in India as of 2017. While there are a number of companies such as Haptik and Active.ai help develop conversational AI products for their enterprise clients, some are adopting and creating solutions in niche areas such as Niki.ai (digital assistant), Fynd (Fashion apparel chatbot), Myprivatetutor (tutor finder), Lawrato (legal searches), Wysa (mental health tracker) to name a few. Apart from chatbots development, a couple of progressive Indian startups such as Liv.ai (acquired by Flipkart) and Reverie Language Technologies are tackling the Indic languages digitization issue with development of voice engines for Indian languages, with advanced language and acoustic models as compared to global technology giants.
Big brands in India including e-commerce companies, banks, insurance firms, travel companies and entertainment firms have already started implementing chatbots. For now, bots can continue to help us with automated, repetitive, low-level tasks and queries; as cogs in a larger, more complex system. And we did them, and ourselves, a disservice by expecting too much, too soon.
Disclosure: Convergence Catalyst advises Reverie Language Technologies Pvt. Ltd.
An edited version of this Analyst Note has been published as an Op-Ed in Mint on 31st December, 2018.