Natural Language Processing hackathon, or don't judge the wine by the shape of the bottle

In April of 2018 I went to a Natural Language Processing hackathon. Organized by Women in Data Science Austin, it took place at Dell, where one of the organizers worked. This was not the kind of hackathon where you hack for the whole weekend straight, crashing on a beanbag to catch a few winks in the breakroom of some hipster startup. No, this was a hackathon with work-life balance. It lasted from 10 am to 3 pm on a Saturday, which is just enough time for you to get deeply enough immersed in a subject to fire up your appetite for it, but not get sick of it. There were no minimal viable products produced, and no prizes, but I got to sink my teeth into the basics of Natural Language Processing.

A data scientist named Becky, who does Natural Language Processing for an Austin company, introduced us to the three cornerstone approaches of NLP -- summarization, topic modeling, and sentiment analysis.

Data scientist Becky talks about topic modeling.

Sentiment analysis quantifies the subjective emotion in a text, e. g. did the majority of reviewers like or didn't like a particular wine? Data scientists don't take into account just the words, but also such nonverbal information as capitalization (a word in all caps is likely to mean the author feels strongly about it), and emoji. Topic modeling finds abstract concepts that occur in a body of texts, a. k. a. corpus. For exaple, if it finds the words milk, meow, and kitten, it might decide one of the topic of this text is cat. If it finds the words bone, bark, and puppy, it might decide one of the topics is dog.

Summarization reduces a text to several key phrases or a representative sentence. Summarization can be extractive or abstractive. Extractive summarization selects a few representative sentences from the text, while abstractive summarization creates a summary of the text.

As an example, Becky gave a phrase: "The Army Corps of Engineers, rushing to meet President Bush's promise to protect New Orleans by the start of the 2006 hurricane season, installed defective flood-control pumps last year despite warnings from its own expert that the equipment would fail during the storm, according to documents obtained by the Associated Press."

Extractive summarization would extract such phrases from it as:

• Army Corps of Engineers
• President Bush
• New Orleans
• defective flood-control pumps

In contrast, abstractive summarization would generate such phrases as:

• government agency
• presidential orders
• defective equipment
• storm preparation
• hurricane Katrina

As many of the hackathon attendees as could fit in the picture.

I can't quite put my finger on it, but it seems that extractive summarization extracts names of specific entities, but not much information as to what happened to those entities or what did they do. But abstractive summarization seems to "understand" what those entities actually represent and what they do, and thereby extracts more "gist" from the paragraph. I could be wrong about it, of course.

According to Becky, extractive summarization is a mostly solved problem by now. TextRank algorithm takes care of it. But abstractive summarization is a very difficult, unsolved problem, though knowledge graphs help.

At the organizers' suggestion, the attendees arranged themselves into three teams, each focusing on one of those three pillars. The organizers brought with them the corpora, a. k. a. texts to be analyzed. Specifically, they brought wine reviews, lots and lots of them. I suppose that's the second best to bringing the actual wine.

Summarizing wine reviews means extracting an "essence" of what the bulk of the reviewers said about a particular wine. It means identifying certain qualities that most reviewers noticed in a given wine. Sentiment analysis meant identifying whether the reviewers thought mostly positively or mostly negatively about the wine.

I ended up in the summarization team. Lead by Randi, who is a data scientist at a big company, we analyzed the wine reviews. By that I mean we called a bunch of functions from pandas, textacy, sumy and other relevant Python packages. The results were mixed. For example, sumy summarized reviews of Moscato in two sentences, but we had no way to tell whether this summarization is good, i.e. whether those were the most representatives sentences from the reviews. It's funny how this is the kind of problem that one has no way of verifying -- at least none that I learned in my 5 hours of NLP bootcamp. Sure, you could read hundreds of reviews and try to get a "feel" whether those sentences were the most representative, but your "feel" would be subjective.

It makes Natural Language Processing feel like black box, and almost like magic -- until you notice that when you ask for 5-sentence summary, the summary includes duplicates for first two sentences. That looks odd, so you take a closer look at the texts and notice that there are duplicate sentences in the document itself. For all its magic, sumy can't figure that out.

Within sumy, you can choose which summarizer to use. First we used LexRank, and it turned out to be very slow. Then we tried another, LuhnSummarizer, and it was much faster, but the results not nearly as accurate. But how would you decide how accurate a summarization is, given that there are no exact criteria for accuracy that I know of? Well, the first summary described mouthfeel and acidity of Moscato. The second included things like the shape and color of the bottle. It left me with the same feeling one often gets interacting with artificial intelligence, that it's both very smart and very stupid at the same time.

Introduction to Natural Language Processing with Women Who Code

In 2016 Women Who Code Austin hosted a series of five presentations on Natural Language Processing. The presenter was our member Diana, who has a Ph.D. in linguistics and has worked in the area of computational linguistics for many years. She did demos of some basic text analysis one can do with the Python Natural Language Toolkit, or in short, NLTK.

She presented all this as a Python notebook. A Python notebook is software that lets you combine text, code, and output of that code on one page. You can run a code snippet right there in the notebook, and the resuls will get updated automatically. So equipped, Diana introduced us to the basics of what computational linguists do. Or if that sounds too ambitious, let's just say she showed some simple things one can do with NLTK.

For example:

• read in the text,
• tokenize,
• tag,
• remove punctuation,
• remove stopwords...
• build a frequency hash table from the rest of words.

The first Austin Women Who Code meeting on natural language processing, with our instructor Diana standing in the center

She introduced such concepts as collocations and bigrams. Bigrams are pairs of words that are next to each other in a text. Collocations are pairs of words that naturally occur in the language together, i. e., a chance of them occuring together is greater than random. An example of a bigram that's not a collocation is Trump's usage of a phrase "Liar Ted" (this was the spring of 2016, the height of the Republican strife for a presidential nomination). If a bigram is not a collocation, but occurs more often than randomly in a text, that can help to identify who the author is / who the speaker is, and some such qualities. It can be a fingerpint of sorts.

The aforementioned tagging is something we do after tokenizing (roughly speakinng, breaking the text up into words). Tagging assigns a 2-letter tag to each word, marking it as a part of speech, such as noun, adverb, etc. "You can use a big list of tags, or a simplified one. Using a simplified list of tags can help with speed of analysis of your corpus," said Diana.

Here Diana noted that tagging words as parts of speech has inherent ambiguity in it -- exactly the kind of thing that makes language and its computational processing so interesting. Here is an example of parts-of-speech ambiguity in a sentence: "They refuse to permit us to obtain the refuse permit". Still, the Python Natural Language Processing Toolkit correctly tags the first "refuse" and "permit" as verb (VBP) and the second instance of each as noun (NN).

This slide shows how NLTK correctly identifies parts of speech in the sentence above. The first instances of "permit" and "refuse" are "VB" -- verbs, whereas the second ones are "NN" -- nouns.

At the second meeting we did all those actions with a corpus of -- wait for it -- Hillary Clinton's emails. Her emails were available for download from the Kaggle site. This was still the spring of 2016, and we did not yet know how sad the implications of those emails will turn out to be, so the choice of the subject wasn't as... emotionally loaded as it would have been just half a year later. And to say "we" did this is an exaggeration, because it was actually Diana that did all the processing and presented the code and the results to us in a Python notebook.

Here was the complete agenda of the meeting:

• Getting data: Hillary Clinton's emails;
• Reading files;
• Using Pandas to create a Dataframe in Python;
• Cleaning data: eliminating punctuation, eliminating stopwords, normalizing data: converting to lower case, tokenizing words
• Visualizing data.

All of this pre-processing of data was done in the Python Natural Language Processing Toolkit (NLTK).

I must say I would have preferred it if Diana had set up this mini-course as a series of exercises for us to do in class and write some code calling NLTK methods ourselves. But if we had done that, we would not have been able to cover even half as much in those four meetups. So I appreciate what Diana did. At least she showed us what kind of beast NLTK is and which fork to eat it with. In the process learned some basic NLP lingo, such as:

• corpus -- a body of text, plural corpora; it's what you process to extract words and do computations with them;
• lexicon -- words and their meanings; example: English dictionary.

However, you need to consider that different fields will have different lexicons. For example: to a financial investor, the first meaning of the word "bull" is someone who is confident about the market, as compared with the common English lexicon, where the first meaning of the word "bull" is an animal. As such, there is a special lexicon for the financial investors, doctors, mechanics, and so on.

• token -- each "entity" that is a part of whatever was split up based on rules. For example, each word is a token when a sentence is "tokenized" into words. Each sentence can also be a token, if you tokenized the sentences out of a paragraph.
• frequency distribution. The frequency distribution method of NLTK counts the frequency of each vocabulary item in the text. It helps identify the most informative words in a corpus.

So overall I got a little familiar with what are the very basics of what natural language scientists do. But somehow, during those four meetings I was still hoping that we'll get past collecting the statistics about words, and get to some mysterious insights about how language works, evolves, and transforms our thoughts, that only computer analysis of language can provide. Of course, my expectations were unrealistically inflated for a set of introductory lessons.

Going back to Hillary Clinton's emails, here is how you would analyze them. This is an "Exploratory Analysis: Getting and Cleaning Data" slide. Here you see the metadata fields that were extracted from the emails. There are quite a few of them.

Python dataframe with the metadata fields extracted from Hilary Clinton's emails.
This slide, "Slicing dataframe to extract subject", shows Python method calls that you would use to extract the email subjects from the dataframe shown in the previous image. Presented in a Python notebook, it alternates code with results of that code. The results can be updated on the fly if you make changes to the code. The MetaDataSubject and MetaDataTo fields contain some familiar names and topics that made the news...

The next slide shows the use of the NLTK method "concordance". It produces a list of the words used in the text, with the passages where they are used. So if you want all occurrences of the word "surprise" in Jane Austen's "Emma", with snippets of context, you can call

emmaText.concordance("surprize")

(Here, emmaText is the variable that holds the text of the Jane Austen's novel "Emma".) From this example you can also see that NLTK has corpora of texts from the Gutenberg project, which is pretty handy.

Concordamce: all the places in Jane Austen 'Emma' where the word 'surprize' is used, obtained by calling a 'concordance' method of NLTK.

Venturing ddeper into natural language processing

The easiest texts to analyze are the news, Diana said. News have very good structure. Sentences tend to be short, and tend to have classical structure: subject, verb, object, etc. Medication instructions are also easy to analyze, since they are required to have readability scores high enough to be suitable for 9-12 year olds. But in literature the sentences are often not conventional and much harder to parse.

At the last meetup we talked a little bit about analyzing texts "for real". And by that I mean a little deeper analysis than just breaking up sentences into parts of speech and gathering statistics about it.

One example where computational linguistics is used is to grade student essays. If you have so many essays that hiring human graders would be cost-prohibitive, natural language processing can help. For example, if an essay is supposed to be on the US Declaration of Independence, the script would check to see if certain words are present in it in a certain way, and will conclude that that student might have a certain level understanding of the topic. (Yes, I know, this raises lots of questions about creativity versus cliche'd, cookie-cutter texts: the latter would be more likely to hit all the points that a grading program is looking for, whereas the former might be difficult for a program to discern. But we didn't cover such questions at the meeting, since it's an uncharted territory.)

We touched upon sentiment analysis, which helps determine how customers feel about an experience they had with a brand or a company. Companies like HomeAway use it to analyze customer reviews of their rental properties. And they discover unexpected things that way. For example, analysis of customer reviews of B&B-type places showed that the greatest predictor of customer satisfaction is whether a house has pots and pans.

Sentiment analysis also shows that, for example, if you try to infer customer satisfaction from the reviews by searching for wait times, you'll get inconsistent results. 15 minutes would be bad for a restaurant, but lightning-fast for an emergency room.

And this is where people try to determine degrees and ways of relatedness or similarity between concepts.

For that, they can use ontologies.

What is Ontology?

A consensus is now established about the definition and the role of an ontology in konwledge engineering: "An ontology is a formal, explicit, specification of a shared conceptualization".

It is used in cognitive modeling.

More about Ontologies

An ontology is a schema (model) describing the types (and possibly some individuals) in a domain, the relationships that may exist between types and individuals, and constraints on the way individuals and properties may be combined.

Here are some examples of ontologies

• Classes: Project, Person, ProjectManager. ProjectManager is a subclass of Person. People and Projects are disjoint.
• Relationships: worksOn, manages. Manages is a sub-property of worksOn.
• Constraints: People work on Projects, not the other way around. Only ProjectManagers can manage Projects.

This simple example enables machine inferences, e.g. if X manages Y, then we can infer that Y is Project, and X is a ProjectManager and therefore a Person.

Onthologies allow people to create trees representing relationships between concepts, like this:

This is an example of a tree that expresses relationships between conecpts in the academia (Student, Employee, Faculty, etc.)

Some people propose ways to neasure the similarity of concepts by some graph metrics, such as the shortest path between two nodes.

Measure of similarity between two concepts in a graph, expressed in terms of a shortest path between two concepts.

More pictures from the Women Who Code Austin meetup series on Natural Language Processing are in my photo gallery.

How come xkcd hasn't addressed this?

I often hear that engineers are literal. This typically includes software developers. I hear this even from people who work with engineers all the time -- you'd think they would take a more nuanced view. But perhaps not many people understand what "literal" means. I'm not even talking about the common, oft-ridiculed use of "literally" to mean "figuratively" -- as when somebody tells you "I literally died laughing" (and they don't have chunks of rotten flesh falling off, so they're not Undead).

Figurative, after all, is the opposite of literal. You can also say that the opposite of literal thinking is metaphorical thinking. Apparently most people don't understand how much certain intellectual activity, such as science and engineering, relies on metaphor. As an example, read this article in Wikipedia about Aspect-oriented programming, and show me even one paragraph in it that does NOT contain a metaphor.

The point is: software design, like any design, models real world problems in some kind of abstract conceptual framework. This is only possible if you think in metaphors.

I really think there should be an xkcd cartoon on this.

Apostrophe carriers shall be assimilated

Apostrophes in names stir lot o' trouble

It can stop you from voting, destroy your dental appointments, make it difficult to rent a car or book a flight, even interfere with your college exams.

More than 50 years into the Information Age, computers are still getting confused by the apostrophe. It's a problem familiar to O'Connors, D'Angelos, N'Dours and D'Artagnans across America.

[...]

"It's standard shortsightedness," he said. "Most programs set a rule for first name and last name. They don't think of foreign-sounding names."

[...]

That's what happened during the Michigan caucus in 2004, when thousands of O'Connors, Al-Husseins, Van Kemps and others who went to the polls didn't have their votes counted.

[...]

All of this confusion has prompted some people to surrender to technology. Iraqi immigrant Lina Alathari was once known as Lina Al-Athari, but dropped the hyphen in America. "There is no pronunciation difference, so I'm fine with it," she said.

My own brand of apostrophe problem

It's not just apostrophes. You don't have to do anything so reckless as inherit a name with an apostrophe in it, to throw computers off whack. We mortals can offend computer's sensibilities by something far more innocuous -- for example, by not possessing a middle name. This has happened to me. I indeed don't have a middle name. It may be unusual in the US, but not back where I'm from. But the computer at Texas Department of Motor Vehicles did not want to believe me. When I got married and changed my last name, I had to update my driver's license. A DMV clerk told me their computer is not letting her leave the middle name field blank. So she said she was going to put in my maiden name in the middle name field. I protested, but she said there was no way around it. Can't argue with a computer.

The funny thing was that a few years before that, when I first got a driver's license in Texas, the DMV computers accepted my middle-name-less-ness with no problem.

A few ArmadilloCons ago I had a conversation about this with an artist named Ctein. As one might imagine, it's not his given name. Whatever name he was born with, many years ago he officially changed it to Ctein. So he now has only one name -- and computers so far have not given him grief about that! Computers must be a lot more liberal in California, where he lives. He was surprised by my account of the stringency of Texas DMV computers.

He wasn't the first one-name-only person I've met. The first one was my roommate in a graduate student dorm. She was from Indonesia. Apparently there is a minority of Indonesians (maybe they belong to a certain ethnicity, I don't know) who go by just one name. She constantly kept running into situations where she was demanded to provide some kind of a surrogate first name. Usually clerks would give up and enter "Ms" as her first name, or sometimes "Fnu" (I guess that stands for "First name undefined").

I bet the digerati who keep saying that the internet would liberate us did not think of this particular way computers would try to force everybody into conformity.

Cute analogies

A few posts ago I was musing about the best ways to introduce scientific ideas in science fiction, to explain them in terms vivid enough to intrigue a reader, instead of slowing down the action. In Sunday's New York Times magazine I found an inspiring example of how it can be done. (I put the particularly memorable sentence in bold.)

Where Protons Will Play

To be sure, the article I'm referring to is a science fact article, not fiction, but I think there's something to be learned from this little perl of explanation. I'm not being sarcastic -- this excerpt seems silly and clever at the same time:

[F]locks of protons will be made to zip around the tunnel in opposite directions at nearly the speed of light. Then they will be forced to crash into each other, with (it is hoped) spectacular results for physics.

Physicists, you see, learn about the subatomic world by smashing things together and then looking at the debris. Imagine a midair collision between two watermelons; it would make quite a mess, but nothing very interesting would result. Suppose, though, you get two protons to collide head-on. If they are moving fast enough, the energy of their collision, converted into mass à la Einstein's E=mc2, will produce a shower of new particles. (It would be as if colliding watermelons splattered into a shower of pineapples, blueberries, mangoes and more exotic fruits.)

World Fantasy Convention: Day 1: Thursday

Thursday was day 1 of the World Fantasy Convention. This year it's in Austin, so I am attending it even though I'm not a big fan of fantasy. Who knows, maybe I could be convinced otherwise. I've heard ome intimidating things about WFC, such as that it's a professional convention, whose goal is to bring writers, agents and editors together to make book deals, and they don't want lowly fans to be cluttering the halls. But they do take the fans' money and sell them memberships, so I guess we are welcome at least in that way.

Thursday night there was only one discussion panel I wanted to go to, so I mostly hung out at the bar and in the ConSuite. This wasn't any different than it is at a usual, "fan-friendly" convention. The pros drink as heartily as us fans. At the ConSuite I ran into a family of very friendly writers -- a woman, her husband, and their 15-year-old son (the latter too had sold SF stories professionally!) -- and had an interesting chat with them. The woman was born and grew up in Portugal (but has lived more than 20 years in the US) is a SF/F writer. She writes in English. She has published a few books. So, it was very encouraging to know that it's not unheard of for a non-native English speaker to successfully write in English. Nabokov wasn't the only one, apparently. The woman said she knew several more SF/F authors in that category.

Authors Leah Bobet and Elizabeth Bear at the World Fantasy Convention 2006. More pictures from WFC can be found on my blog

On the other hand, being a non-native speaker sometimes actually helps to undestand the language, as is evident from a funny fact the guy mentioned. Whenever he calls customer service or some such thing where he's supposed to give his name over the phone, the customer service reps often misinterpret his last name. Especially if the rep sounds like he/she may be a Southerner. His last name is Hoyt, and people (especially Southerners) universally hear it as White. He always has to spell it out very carefully, and even then they are sometimes not convinced it's not White.

As a non-native English speaker, I suspect that spelling versus pronunciation are mapped out differently in my brain than in the brain of a native English speaker. For me there is very little overlap between Hoyt and White, so I would never confuse the two.

So far the World Fantasy Convention seems like any other convention, only more humane. There are no more than 2 discussion panels going on at the same time, and there are breaks for lunch and dinner! This makes convention experience less taxing. Also, people here are a little better looking. :-) (Maybe it's an infusion of literary agents from glamorous places like New York that's raising the looks quotient of the convention. :-)) The difference is rather subtle, though. The people are a little more dressed up than in a typical convention. Instead of Fannish Drab, the color scale runs towards Gothy Black.