The Journey of a Natural Language Analytics Company


As a co-founder and CTO of Wizdee, I have to say that during all these years of hard work we had faced a lot of difficult challenges. We were able to win adversity, but most of the times not without a fight. The weapons we used were all gathered while back in academia, where we worked with a wide range of Artificial Intelligence approaches to solve real world problems. Having this experience was crucial to build a product that transforms Business Intelligence in something that anyone can do by themselves.

The First Days: Creating a Product

We started developing Wizdee as a Semantic Search engine for enterprise data. We wanted to make the life of knowledge intensive workers a lot easier, helping them to find information faster and more efficiently. The way to go was building a search engine that would do more than a simple match of keywords in a document. We needed something that could understand the meaning of what was being searched.

When I search for “orders in may”, it sure does not mean that I want all the documents that contain the keywords “orders” and “may”, and Wizdee would have to be able to make that distinction. Moreover, it should be able search across data that populates a myriad of different systems within organizations.

A big part of this data is typically stored in a structured way inside databases ruled by relational data models. The first version of Wizdee was all that. It was able to index data from different database engines, enrich the data model with semantic knowledge gathered from a schema description file, and empower users with a search engine that understood their intent.

Becoming Powerful!

Soon we realised that Semantic Search was not enough. We started showing our brand new search engine to prospective customers, and one after the other was having the same reaction. They could see the potential of the solution, but it was not enough to make the move of replacing their current search solutions. In their minds, their problem was not finding information, their problem was to extract meaning from that same information. And if we were able to create such an intelligent search engine, why wouldn’t we teach it how to crunch that information for them.

They wanted to do analytics, aggregate data and compute metrics, all that using the same search box that we though was already intelligent. We took all that feedback with us and decided that we needed to go a step forward, and so we did. We made use of our background in Natural Language Processing and added an additional layer of computation to our search engine. It gained the ability to cross data from different entities, support for aggregating data by several dimensions, and a bunch of operators to compute the essential metrics.

Our search engine was more powerful than ever, not only it could understand what you were looking for, but also make sense of the data and give you the metrics you needed. You could ask for the “average amount of orders by month” and get that information in a snap, the days of traditional Business Intelligence were over!

The Final Challenge

We were amazed with the things that our search engine was able to do, it was getting more intelligent every day. But the honeymoon was about to end soon, we were having communication problems. Our search engine was confusing things, we were asking for the “average amount of orders by state” and it was grouping orders by the state of the customer, when we wanted them grouped by the state of the store. It was not wrong, but it wasn’t correct either. We were facing reality, Natural Language is ambiguous.

We, humans, deal very well with ambiguity because our brains use a lot of past and contextual information to make sense of what we ear, see, smell, etc. We just needed to teach our search engine that what we say can have several interpretations, and that the correct interpretation depends on a set of things that we can not predict, but can and should anticipate.

The solution we found for this communication problem was to make our search engine generate all the interpretations that exist for the query of the user, and then order those interpretations according to what it thinks is more likely to be the intent of the user. The tricky part is to predict the intent of the user, but our search engine is smart, it learns from examples. So we taught it to order the query interpretations based on examples of queries that make sense to the user, using a set of algorithms that mimic the process of natural selection, the so called Genetic Algorithms.

We were back on track again, we could understand each other, and when things were not clear we just had to tell him the correct interpretation and he would learn from that, not to repeat the same mistake again.

Oh, yes, of course…the User Experience

All was going smooth until we got bored with all that tables, with infinite lines of unappealing numbers. We needed to change the visual of our search engine. Nobody wants to look at tables. We all want charts, colorful and meaningful charts! But, nobody wants to build charts either. No way we would make our users go through a wizard, click on 37 buttons and adjust a bunch of visual components, all that just to see a bar chart of their “sum amount of orders by state by month”.

It was time for our search engine to go back to school again. We needed to teach him how to select the best chart according to the search results and automatically build it from scratch. Again, all we had to give him were examples of good charts for specific types of search results. So, we used another technique that mimics our ability to learn solutions from similar past problems, the Case-based Reasoning approach. We got rid of the boring tables and all we could see were beautiful visualizations appearing effortlessly on the screen.

Our experience as Artificial Intelligence researchers was key to solve the issues we encountered along the way. I’m sure they will continue to play a central role on the time to come. It is not over yet, there is so much we still have to do, and new challenges will surface. We will be working hard, as always, to solve them for you.

Bruno Antunes
CTO at Wizdee


Business Intelligence

No comments yet.

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.