Introduction

For a long time there has been a battle in Natural Language Programming between statistically oriented systems that are trained from large quantities against systems as opposed to grammar oriented systems motivated by the modern linguistics associated with Professor Noam Chomsky. Although some modern NLP systems start by parsing sentences, using something like the Stanford Parser.

I've long been interested in the problem of Named Entity Resolution and, in particular, the approach used by DBpedia Spotlight, which collects a large number of possible surface forms (for instance, both "NYPD" and "New York City Police Department" refer to this concept.) Spotlight is different from some other systems in that it resolves entities to concepts, rather than simply marking phrases in the text that correspond to particular roles(for instance, in a sentence like "Frank Boltz was an employee of the NYPD", the pink phrase is a person's name while the second is the name of an organization -- this assignment could be made even if we had no idea what these entities are.)

I like to think of this kind of system as a "magic magic marker", which highlights phrases in text to tag them with either general or specific meanings. A system like spotlight works in two phases, first finding places where the surface form dictionary matches the text, and then determining which interpretations are correct. (For instance, the word "Kate" could be a surface form for the first name of a women having over one of over 1000 names derived from Catherine -- an inspection of the context is necessary to narrow it down to a particular Kate.)

The Microsoft Concept graph is a database of surface forms and possible interpretations. Rather than resolving phrases to specific concepts (say "Pikachu" to Pikachu) it tags phrases with general concepts such as "Pokemon" or "Character". Produced with technology similar to that used to create word embeddings such as word2vec, the concept graph is positioned as a tool useful for understanding short texts such as search queries and tweets.

This chapter is based on my notes of a preliminary investigation of the Microsoft Concept Graph intended to be a rapid evaluation of the product for text analysis applications.

Overview

Let's look at the first 15 lines of the file to get a quick sense of the contents:

(Note that I added the header at the top, the actual file has no header)

The file as a whole has 33,377,320 lines and is sorted by descending score. The label is a label which could be applied to a text phrase, the surface form is the phrase, and the score is a measure of the strength of association between the two. In the top 15 lines we can already see some sense of the diversity of concepts, such as "factor" which represents various properties that an object could have, as well as "state" and "material". Already we can see a few examples where the results are strange such as the concept "free rich company datum" (which seems to represent a property that a company could have) and the issue "stress pain depression sickness" (which seems ill formed and a bit verbose)

Analysis of labels

Already we see one quality that most classifications based on unsupervised machine learning lack: the categories (labels) are meaningfully named, at least mostly. Speaking of labels, the system assigns 5,376,526 different labels. The most commonly assigned labels are here:

The most common labels are short words with broad meanings. The one multi-word label that appears in the top 50, "common search term" turns out to be a strange one, containing search terms that would be used to find pirate software such as "adobe photoshop crack" and "age of empires 3 serial". A look at some of the least used labels shows that there is plenty of room at the bottom:

There are 2,364,966 labels which have only one surface form and, looking at the examples above, these are often gibberish; for practical work, it's clear that a large number of junk records could be removed.

Deep drill into a few labels

Let's take a look at the top 25 "factors":

Note that the "factors" are related to what we could call "predicates" in the RDF world, being attributes that something could have -- it reads like a list of possible independent variables that could affect people. If we look at other labels assigned to age, we get a list of very similar looking concepts:

These concepts form a messy categorization much like a folksonomy, for instance, the distinction between a continuous vs a discrete variable is potentially interesting:

The concept graph picks up this distinction, but looking carefully note that "thickness" isn't necessarily a discrete variable. Although a number of interesting categories exist in the graph, a considerable amount of cleanup would be necessary to create useful classifications.

Precision and Recall Analysis

Let's take a detailed look at a label which ought to have a well-defined list of values, specifically, "chemical elements"

Note that the surface forms fit into a number of categories: (i) chemical elements by name, (ii) chemical elements by abbreviation, (iii) phrases that could stand in for some class of chemical element (ex. "heavy metal", "rare earth"), (iv) isotopes (ex. "7 li"), (v) names of ions (ex. "fluoride"), and (vi) crazy misses (ex. "oxygen carbon gold molybdenum") Many of these are examples of the kind of "near miss" situations that turn up in any kind of classification, particularly when language is involved. If we imagine however, that we're looking for a list of chemical elements, and we're willing to consider abbreviations to be valid surface forms, there turn out to be 70 elements identified by name and 9 with abbreviations and 65 surface forms that are not chemical elements. That gives us precision of 79/144 = 54.8% and recall of 70/118=59.3% for names and 9/118=7.6% for abbreviations

Considering that the concept graph discovered the concept of "chemical element", this is impressive, but if you really need a list of chemical elements you're better off getting them out of DBpedia, where the query

select count(*) {
    ?element dct:subject dbc:Chemical_elements
}

A few sample surface forms

Just to give you a sense of what you will find, I'll show a few examples of what you can get if you look at the labels assigned to surface forms. "Foot" is a good example of polysemy, that is, a word having multiple meanings:

Note that the current version of the Microsoft Concept Graph makes no attempt to distinguish between polysemous concepts, but they are working on this for the next phase. Such a classification isn't as simple as picking "the right choice" because a particular use of the word foot could be as an "area", "extremity", "body part" and "personal weapon" at one time.

The Concept Graph is rich in knowledge about the biomedical domain. We get a nice set of categories for a common drug:

I also often see nice (if repetitive) results when the surface form is a drug catgegory:

(The trouble is, however, that medical applications are going to be held to account for errors, thus a high level of accuracy will be required.)

One area where precision is less important is pop culture, and good results can be had for relatively obscure topics:

Finally, I'll show an example of what I call a "critical error", the kind of small mistake which has an outsized effect:

Note that the system classifies Adolph Hitler as a "good leader" as well as many other things; 19 labels look good (95% precision), but the 1 bad one could deeply offend somebody -- a different situation than the gobbledygook (but not actively harmful) labels we see on so many topics.

Conclusion

The Microsoft Concept Graph is one of many databases that (i) involves the intersection of language and concepts, and (ii) applies to a broad range of human knowledge. In this chapter, I've shown a number of small cross sections which give some idea of what you'll find inside it. It's hard to give a meaningful evaluation of the Concept Graph in general, because quality is a matter of fitness for some specific application. The example where I find 54.8% precision and 59.3% recall for chemical elements is an example of how this kind of categorization is evaluated in a particular domain.