An almost convincing start for Wolfram Alpha

Wolfram Research finally launched Wolfram|Alpha, the much talked about “computational knowledge engine”. It’s unusual description is justified by the fact that we are not dealing with a search engine in the classic sense of the term, nor with the next Google.

At first glance we’re peering at a service that is able to provide objective information about a wide subset of human knowledge. But the idea behind Wolfram|Alpha is much more ambitious, as demonstrated by the first screencast to have been released (narrated by Stephen Wolfram).

Taking advantage of the popular and powerful mathematical software Wolfram Mathematica, this new engine is able to perform calculations on information requested by its users. Therefore there are three components at work: the ability to correctly interpret queries (in English) from the users, the ability to elaborate on the data source so as to reply with coherent results to somewhat complex questions, and finally the not-so-simple task of maintaining an up-to-date and accurate knowledge base for a very wide spectrum of human knowledge.

This is the theory behind – or at least the ultimate aim of – the service. And as such, unlike what has been reported elsewhere, Wolfram|Alpha should immediately be viewed as an addition to what Google already offers (not as a replacement for it).

As expected, the service is extremely good at mathematical calculations. In this case the only limit is the timeout imposed on each query to exclude those that require far too many resources to complete. An example of a calculation that is executed is “integrate e^-x^2”.

For the record, this service is currently under a fair bit of stress because of the initial curiosity of many worldwide, therefore the calculation of the integral above has shown an error message a couple of times (a tribute to HAL 9000):

All sorts of disciplines are represented by the examples on the site, with a particular focus on scientific and engineering ones. The results are elegantly presented and well organized, often illustrated and shown in table form, particularly when the user is asking for a comparison between different data sets. There are also fun tributes, including those to Monty Python and Douglas Adams. But how well does this system work when we step outside of the predefined examples provided by Stephen Wolfram’s team? The results range from exceptionally good to downright disappointing. Let’s see a few examples.

Let’s compare IBM, Google and Sun. As you can see, the results are definitely excellent. The comparison is almost exclusively numeric in nature, but the answer provided by WA jives with what I was hoping for. The same is true if you look for statistical information on a single large company like Telecom Italia.

Since this is a “computational engine” we can even try to perform a few calculations starting with the data that we found to be available in the comparison tables for the companies above. For example, “employees IBM/Google” will show us the ratio over the last few years between the number of IBM and Google employees (IBM currently hires almost 20 times as many people). We can calculate the revenue for each IBM employee by running “(market cap / employees IBM)”, but the engine fails to compare this parameter between companies: “(market cap / employees IBM) / (market cap / employees Google)”, despite the fact that the data for such a calculation is all there in the knowledge base (and as you can see there was already a revenue/employee row in the initial comparison table).

Moving on to something else, we can try to compare two cities like Toronto and Milan, and obtain very useful details. It’s also possible to calculate only certain attributes by running, for example, “population Toronto/Milan” or “distance Toronto Milan”. In the first case the data was updated to 2004, and is perhaps a little outdated now despite the fact that we are talking about demographic data. But the real problem arises when we try to compare the regions of Ontario and Lombardy. We’d expect to see nice geographical, demographical and economical comparisons between the two, yet instead we get nothing. Wolfram|Alpha will report information about little towns like Unionville (NC), but totally ignores a region like Lombardy or a Canadian province like Ontario (with a population of 13 million people). It would seem that this is a big hole in the knowledge of this service. Sure it’s fairly easy to fix, but it’s a symptomatic weakness nevertheless.

If we consider for a moment searches about famous people, we’ll find an excess of zealousness in trying to be concise and objective. In fact, when searching for Barack Obama, the results are limited to his place and date of birth, and stating that he is a head of state. We may be spoiled by Wikipedia, but a photo or a few more details at least were to be expected. For example, at a bare minimum, it could be indicated that we are reading the page of the 44th President of the United States of America. But Wolfram|Alpha reserves this type of treatment for all famous people, including Stephen Wolfram himself.

In the comparison between “Paul Erdos vs Euler” one would expect a nice parallel between these two great, prolific mathematicians (for example, you might expect comparisons on the number of publications, total number of pages, the most significant discoveries, and so on). Instead, the results are limited to a comparison of births and deaths. This is quite disappointing no matter how you look at it. On a side note however, it must be said that WA is quite good at interpreting misspelled names (e.g., Paul Erddsos).

The Natural Language Processing (NLP) capabilities of Wolfram|Alpha are good enough to use the service without encountering any major frustrations, but it doesn’t appear to be a particular revolution or advancement in the field of artificial intelligence either. It is also clear that we are not dealing with a Google-killer nor a Wikipedia-killer, but rather with an innovative new tool that can be used in addition to the existing ones. That said, elsewhere I mentioned that I personally think that this is a step forward for humanity. It may be a small step, but I stand behind that bold statement.

We are at the forefront of a service that will be useful for research and to anyone with a need for correct data as quickly as possible. The knowledge base will have to grow, some data will have to be updated, and the engine will need to permit more freedom in the kind of calculations that are allowed to be performed. There can be little doubt thought that we are witnesses to the birth of something ambitious that has the potential to accelerate the advancement of our civilization.

An Italian translation of this article is available on Stacktrace.it.