Natural Language Processing hackathon, or don't judge the wine by the shape of the bottle

In April of 2018 I went to a Natural Language Processing hackathon. Organized by Women in Data Science Austin, it took place at Dell, where one of the organizers worked. This was not the kind of hackathon where you hack for the whole weekend straight, crashing on a beanbag to catch a few winks in the breakroom of some hipster startup. No, this was a hackathon with work-life balance. It lasted from 10 am to 3 pm on a Saturday, which is just enough time for you to get deeply enough immersed in a subject to fire up your appetite for it, but not get sick of it. There were no minimal viable products produced, and no prizes, but I got to sink my teeth into the basics of Natural Language Processing.

A data scientist named Becky, who does Natural Language Processing for an Austin company, introduced us to the three cornerstone approaches of NLP -- summarization, topic modeling, and sentiment analysis.

Data scientist Becky talks about topic modeling.

Sentiment analysis quantifies the subjective emotion in a text, e. g. did the majority of reviewers like or didn't like a particular wine? Data scientists don't take into account just the words, but also such nonverbal information as capitalization (a word in all caps is likely to mean the author feels strongly about it), and emoji. Topic modeling finds abstract concepts that occur in a body of texts, a. k. a. corpus. For exaple, if it finds the words milk, meow, and kitten, it might decide one of the topic of this text is cat. If it finds the words bone, bark, and puppy, it might decide one of the topics is dog.

Summarization reduces a text to several key phrases or a representative sentence. Summarization can be extractive or abstractive. Extractive summarization selects a few representative sentences from the text, while abstractive summarization creates a summary of the text.

As an example, Becky gave a phrase: "The Army Corps of Engineers, rushing to meet President Bush's promise to protect New Orleans by the start of the 2006 hurricane season, installed defective flood-control pumps last year despite warnings from its own expert that the equipment would fail during the storm, according to documents obtained by the Associated Press."

Extractive summarization would extract such phrases from it as:

• Army Corps of Engineers
• President Bush
• New Orleans
• defective flood-control pumps

In contrast, abstractive summarization would generate such phrases as:

• government agency
• presidential orders
• defective equipment
• storm preparation
• hurricane Katrina

As many of the hackathon attendees as could fit in the picture.

I can't quite put my finger on it, but it seems that extractive summarization extracts names of specific entities, but not much information as to what happened to those entities or what did they do. But abstractive summarization seems to "understand" what those entities actually represent and what they do, and thereby extracts more "gist" from the paragraph. I could be wrong about it, of course.

According to Becky, extractive summarization is a mostly solved problem by now. TextRank algorithm takes care of it. But abstractive summarization is a very difficult, unsolved problem, though knowledge graphs help.

At the organizers' suggestion, the attendees arranged themselves into three teams, each focusing on one of those three pillars. The organizers brought with them the corpora, a. k. a. texts to be analyzed. Specifically, they brought wine reviews, lots and lots of them. I suppose that's the second best to bringing the actual wine.

Summarizing wine reviews means extracting an "essence" of what the bulk of the reviewers said about a particular wine. It means identifying certain qualities that most reviewers noticed in a given wine. Sentiment analysis meant identifying whether the reviewers thought mostly positively or mostly negatively about the wine.

I ended up in the summarization team. Lead by Randi, who is a data scientist at a big company, we analyzed the wine reviews. By that I mean we called a bunch of functions from pandas, textacy, sumy and other relevant Python packages. The results were mixed. For example, sumy summarized reviews of Moscato in two sentences, but we had no way to tell whether this summarization is good, i.e. whether those were the most representatives sentences from the reviews. It's funny how this is the kind of problem that one has no way of verifying -- at least none that I learned in my 5 hours of NLP bootcamp. Sure, you could read hundreds of reviews and try to get a "feel" whether those sentences were the most representative, but your "feel" would be subjective.

It makes Natural Language Processing feel like black box, and almost like magic -- until you notice that when you ask for 5-sentence summary, the summary includes duplicates for first two sentences. That looks odd, so you take a closer look at the texts and notice that there are duplicate sentences in the document itself. For all its magic, sumy can't figure that out.

Within sumy, you can choose which summarizer to use. First we used LexRank, and it turned out to be very slow. Then we tried another, LuhnSummarizer, and it was much faster, but the results not nearly as accurate. But how would you decide how accurate a summarization is, given that there are no exact criteria for accuracy that I know of? Well, the first summary described mouthfeel and acidity of Moscato. The second included things like the shape and color of the bottle. It left me with the same feeling one often gets interacting with artificial intelligence, that it's both very smart and very stupid at the same time.

Introduction to Natural Language Processing with Women Who Code

In 2016 Women Who Code Austin hosted a series of five presentations on Natural Language Processing. The presenter was our member Diana, who has a Ph.D. in linguistics and has worked in the area of computational linguistics for many years. She did demos of some basic text analysis one can do with the Python Natural Language Toolkit, or in short, NLTK.

She presented all this as a Python notebook. A Python notebook is software that lets you combine text, code, and output of that code on one page. You can run a code snippet right there in the notebook, and the resuls will get updated automatically. So equipped, Diana introduced us to the basics of what computational linguists do. Or if that sounds too ambitious, let's just say she showed some simple things one can do with NLTK.

For example:

• read in the text,
• tokenize,
• tag,
• remove punctuation,
• remove stopwords...
• build a frequency hash table from the rest of words.

The first Austin Women Who Code meeting on natural language processing, with our instructor Diana standing in the center

She introduced such concepts as collocations and bigrams. Bigrams are pairs of words that are next to each other in a text. Collocations are pairs of words that naturally occur in the language together, i. e., a chance of them occuring together is greater than random. An example of a bigram that's not a collocation is Trump's usage of a phrase "Liar Ted" (this was the spring of 2016, the height of the Republican strife for a presidential nomination). If a bigram is not a collocation, but occurs more often than randomly in a text, that can help to identify who the author is / who the speaker is, and some such qualities. It can be a fingerpint of sorts.

The aforementioned tagging is something we do after tokenizing (roughly speakinng, breaking the text up into words). Tagging assigns a 2-letter tag to each word, marking it as a part of speech, such as noun, adverb, etc. "You can use a big list of tags, or a simplified one. Using a simplified list of tags can help with speed of analysis of your corpus," said Diana.

Here Diana noted that tagging words as parts of speech has inherent ambiguity in it -- exactly the kind of thing that makes language and its computational processing so interesting. Here is an example of parts-of-speech ambiguity in a sentence: "They refuse to permit us to obtain the refuse permit". Still, the Python Natural Language Processing Toolkit correctly tags the first "refuse" and "permit" as verb (VBP) and the second instance of each as noun (NN).

This slide shows how NLTK correctly identifies parts of speech in the sentence above. The first instances of "permit" and "refuse" are "VB" -- verbs, whereas the second ones are "NN" -- nouns.

At the second meeting we did all those actions with a corpus of -- wait for it -- Hillary Clinton's emails. Her emails were available for download from the Kaggle site. This was still the spring of 2016, and we did not yet know how sad the implications of those emails will turn out to be, so the choice of the subject wasn't as... emotionally loaded as it would have been just half a year later. And to say "we" did this is an exaggeration, because it was actually Diana that did all the processing and presented the code and the results to us in a Python notebook.

Here was the complete agenda of the meeting:

• Getting data: Hillary Clinton's emails;
• Reading files;
• Using Pandas to create a Dataframe in Python;
• Cleaning data: eliminating punctuation, eliminating stopwords, normalizing data: converting to lower case, tokenizing words
• Visualizing data.

All of this pre-processing of data was done in the Python Natural Language Processing Toolkit (NLTK).

I must say I would have preferred it if Diana had set up this mini-course as a series of exercises for us to do in class and write some code calling NLTK methods ourselves. But if we had done that, we would not have been able to cover even half as much in those four meetups. So I appreciate what Diana did. At least she showed us what kind of beast NLTK is and which fork to eat it with. In the process learned some basic NLP lingo, such as:

• corpus -- a body of text, plural corpora; it's what you process to extract words and do computations with them;
• lexicon -- words and their meanings; example: English dictionary.

However, you need to consider that different fields will have different lexicons. For example: to a financial investor, the first meaning of the word "bull" is someone who is confident about the market, as compared with the common English lexicon, where the first meaning of the word "bull" is an animal. As such, there is a special lexicon for the financial investors, doctors, mechanics, and so on.

• token -- each "entity" that is a part of whatever was split up based on rules. For example, each word is a token when a sentence is "tokenized" into words. Each sentence can also be a token, if you tokenized the sentences out of a paragraph.
• frequency distribution. The frequency distribution method of NLTK counts the frequency of each vocabulary item in the text. It helps identify the most informative words in a corpus.

So overall I got a little familiar with what are the very basics of what natural language scientists do. But somehow, during those four meetings I was still hoping that we'll get past collecting the statistics about words, and get to some mysterious insights about how language works, evolves, and transforms our thoughts, that only computer analysis of language can provide. Of course, my expectations were unrealistically inflated for a set of introductory lessons.

Going back to Hillary Clinton's emails, here is how you would analyze them. This is an "Exploratory Analysis: Getting and Cleaning Data" slide. Here you see the metadata fields that were extracted from the emails. There are quite a few of them.

Python dataframe with the metadata fields extracted from Hilary Clinton's emails.
This slide, "Slicing dataframe to extract subject", shows Python method calls that you would use to extract the email subjects from the dataframe shown in the previous image. Presented in a Python notebook, it alternates code with results of that code. The results can be updated on the fly if you make changes to the code. The MetaDataSubject and MetaDataTo fields contain some familiar names and topics that made the news...

The next slide shows the use of the NLTK method "concordance". It produces a list of the words used in the text, with the passages where they are used. So if you want all occurrences of the word "surprise" in Jane Austen's "Emma", with snippets of context, you can call

emmaText.concordance("surprize")

(Here, emmaText is the variable that holds the text of the Jane Austen's novel "Emma".) From this example you can also see that NLTK has corpora of texts from the Gutenberg project, which is pretty handy.

Concordamce: all the places in Jane Austen 'Emma' where the word 'surprize' is used, obtained by calling a 'concordance' method of NLTK.

Venturing ddeper into natural language processing

The easiest texts to analyze are the news, Diana said. News have very good structure. Sentences tend to be short, and tend to have classical structure: subject, verb, object, etc. Medication instructions are also easy to analyze, since they are required to have readability scores high enough to be suitable for 9-12 year olds. But in literature the sentences are often not conventional and much harder to parse.

At the last meetup we talked a little bit about analyzing texts "for real". And by that I mean a little deeper analysis than just breaking up sentences into parts of speech and gathering statistics about it.

One example where computational linguistics is used is to grade student essays. If you have so many essays that hiring human graders would be cost-prohibitive, natural language processing can help. For example, if an essay is supposed to be on the US Declaration of Independence, the script would check to see if certain words are present in it in a certain way, and will conclude that that student might have a certain level understanding of the topic. (Yes, I know, this raises lots of questions about creativity versus cliche'd, cookie-cutter texts: the latter would be more likely to hit all the points that a grading program is looking for, whereas the former might be difficult for a program to discern. But we didn't cover such questions at the meeting, since it's an uncharted territory.)

We touched upon sentiment analysis, which helps determine how customers feel about an experience they had with a brand or a company. Companies like HomeAway use it to analyze customer reviews of their rental properties. And they discover unexpected things that way. For example, analysis of customer reviews of B&B-type places showed that the greatest predictor of customer satisfaction is whether a house has pots and pans.

Sentiment analysis also shows that, for example, if you try to infer customer satisfaction from the reviews by searching for wait times, you'll get inconsistent results. 15 minutes would be bad for a restaurant, but lightning-fast for an emergency room.

And this is where people try to determine degrees and ways of relatedness or similarity between concepts.

For that, they can use ontologies.

What is Ontology?

A consensus is now established about the definition and the role of an ontology in konwledge engineering: "An ontology is a formal, explicit, specification of a shared conceptualization".

It is used in cognitive modeling.

More about Ontologies

An ontology is a schema (model) describing the types (and possibly some individuals) in a domain, the relationships that may exist between types and individuals, and constraints on the way individuals and properties may be combined.

Here are some examples of ontologies

• Classes: Project, Person, ProjectManager. ProjectManager is a subclass of Person. People and Projects are disjoint.
• Relationships: worksOn, manages. Manages is a sub-property of worksOn.
• Constraints: People work on Projects, not the other way around. Only ProjectManagers can manage Projects.

This simple example enables machine inferences, e.g. if X manages Y, then we can infer that Y is Project, and X is a ProjectManager and therefore a Person.

Onthologies allow people to create trees representing relationships between concepts, like this:

This is an example of a tree that expresses relationships between conecpts in the academia (Student, Employee, Faculty, etc.)

Some people propose ways to neasure the similarity of concepts by some graph metrics, such as the shortest path between two nodes.

Measure of similarity between two concepts in a graph, expressed in terms of a shortest path between two concepts.

More pictures from the Women Who Code Austin meetup series on Natural Language Processing are in my photo gallery.

Build Something Awesome… but what?

"Build Something Awesome with OpenStack and the Open Cloud" hackathon could have lived up to its name, if only someone knew what kindof awesome things one could build with OpenStack. Or could explain it to developers. But I'm getting ahead of myself.

Maddy (left), our unofficial team lead and Python expert, and Anna. More pictures from the 2013 OpenStack Hackathon are in my photo gallery.

The September 14th, 2013 OpenStack hackathon was the first hackathon I ever attended. It was organized and sponsored by Rackspace, creator of the OpenStack project. I didn't know much about it, so I assumed that it was just yet another API that lets you build applications. The hackathon event page did not hint at what kinds of applications you could build with it. So I was surprised when it turned out that for the kind of application my team wanted to build, OpenStack kind of… got in the way.

The hackathon started with a 2-hour presentation by Rackspace's developer advocate. He guided us through a tutorial on how to create a DevStack server on Rackspace. DevStack, by the way, he said, is not the same as OpenStack, but the distinction was lost on me. This was by far not the most subtle point that was lost on me.

Left to right: Paige, Jess, Maddy (our unofficial team lead and Python expert), and Christine. More pictures from the 2013 OpenStack Hackathon are in my photo gallery.

After the presentation our team of five, all female developers, rolled up our sleeves to start building the application proposed by one of our members. I investigated the server created during the walkthrough, looking for the directory where Apache keeps HTML files and web scripts. That's where I thought I would place a web application (at the beginning, just a Python script) that we were writing. I saw there was an index.html in the /var/www directory, but its contents were not the one that were displayed when you pointed your browser to this server's root URL. So I went to the presenter and asked why that was. He said, better don't try to use Apache on that devstack server; it's configured in a special way, and if you want to run an ordinary Apache web server, you'd be fighting it all the way. You should create a basic Linux server on Rackspace, not a Devstack server, and install Apache on it. I tried asking him what could we do with this Devstack server, if not write web applications. He said it was mostly for learning. Learning OpenStack. Well, that still didn't answer my question what I could do with OpenStack, but oh well, maybe I should have found out beforehand? It's not like it was any secret that this hackathon was for building things with OpenStack: it was in the name of the hackathon. But I wasn't the only person who went there with assumptions that I could build web applications with it.

Other lessons from this hackathon were more interesting, and came from my attempt to find out what can be accomplished during a hackathon. More about it in the next blog post.

What can go wrong at a hackathon, or Jekyll and Mr. Hyde

The application I worked on at SheHacksATX was called Groove. It is a fertility tracking app that's primarily mobile, but its owner Jennifer wanted a few features to be added to the Groove website. One of the features was an image with clickable parts, each of which produced a different text explanation; another was a rotating image slider with captions for each slide, that would pause when a mouse hovers over it, and resume when the mouse is moved off of it. I took the task of creating the slider, and the other developer in our team took the image. Of course, I didn't create the slider completely on my own: similar Javascript widgets already exist out there on the web. I found one on Github, and adapted it to our application's needs. But even that wasn't trivial. I had to find ways to customize the layout of the captions, and to implement the pause-on-mouseover / resume-on-mouseout functionality. It may sound like a piece of cake to you front-end gurus, but I'm a backend developer after all. I have to spend precious minutes or hours brushing the rust off of such bits of knowledge like how do I make two <div> tags to appear side-by-side, or to climb out of the rabbit hole of nested callbacks in asynchronous function calls.

But programming challenges wasn't what posed a threat to our team's chance of accomplishing our tasks. It was development environment incompatibilities. It turned out that certain technologies are difficult to get working on Windows. I'm looking at you, Jekyll. Jekyll, a website generator, lets you write HTML pages using a certain system of shorthand tags, as opposed to actual HTML tags; Jekyll runs on the server, parses those template files and turns them into HTML.

Well, it turns out that Jekyll on Windows is more like Mr. Hyde. Trying to run it I got an uninformative error, and a cursory Google search made me think that perhaps I need to run it on a higher version of Ruby than the one on my laptop. (Jekyll is written in Ruby.) Jennifer was running Jekyll on her machine with Ruby 2.0, whereas I had Ruby 1.9. So I decided to see if upgrading Ruby fixes the problem. Except there turned out to be no quick way to upgrade Ruby on a Windows machine. On a UNIX-based operating system you would do it with various package managers; on Windows, Google suggested to use this package manager or that, some of which were reported to work on Windows with a few tweaks; just trying to install those package managers caused a cascade of incomprehensible errors. I could see that if I continued down this route, I would spend all day just trying to upgrade Ruby on Windows.

I consulted with Jennifer (Groove founder), and the other developer, and we decided that I won't try to run Jekyll. Indeed, all that Jekyll did in this application was to insert some common HTML sections into HTML pages. I could simply insert those sections myself. So I did. I made a copy of the original file for myself, and replaced Jekyll tags with HTML that was generated from them.

The Groove team at work: Jennifer (left), the founder of the Groove app, and the other developer in our team. More pictures from SheHacksATX are in my photo gallery.

At the end of the day we, the two developers, finished our parts. There came the moment of truth when we each had to push our code to the Groove Github repository. I copy-and-pasted my code changes into the original page (the one with Jekyll tags), and pushed it to Github. My push triggered a build, which Jennifer had set up on Github to run automatically when new code is committed. That's when Mr. Hyde struck again. It said the build failed because the file I committed wasn't in the right encoding. It was already 6 pm in the evening, only 1-1.5 hours until the demos where all the hackathon teams were supposed to demonstrate what they had accomplished, and way too late to figure out why it was complaining about encoding. (I suspected that it was the Windows-style line endings in my file, while Jekyll might have expected Unix-style line endings, but I really don't know.) What to do?

So I committed my code "by proxy", i.e. via the other developer's laptop. She had a Mac, a Unix-based system, and I was pretty sure that a file committed by her would not be corrupted. I emailed her my files, she copied-and-pasted the changes into her file, and pushed it to Github. That's cutting a long story short. The long version of this story involves dancing the Github "dance" two people have to do when the other person's Git branch has fallen one or more commits behind the Github repository. It was also complicated by the fact that the other developer had made her own code changes to the same files. But after just an hour of copy-pasting and white-knuckled Githubbing, we fixed the build, and successfully pushed our changes to the repository. The day was saved, just in time to the pre-demo sushi and cheese snacks.

I have been primarily a Windows developer for a long time, and never understood why someone would need a Mac, except people who find Windows computers too difficult to use -- but surely that's not the case with developers. But this hackathon made me seriously consider that if I want to program outside of Windows stack, I need a Unix-based system. Ruby and Jekyll were just two examples; my earlier failure of getting Python to interoperate with Heroku toolbelt on Windows was another. Yes, I know, I can run UNIX on a virtual machine, but my Windows computer is already too slow even with what few programs I'm running on it. It's food for thought for the future.

But yes. This last part yet again highlights the importance of preparing for a hackathon -- at least setting up development environment for your project ahead of time, and testing it. But that's not possible not knowing what project you'll be working on. Perhaps it would be better if team / project assignments were determined at least a day or two before the hackathon, although that's not practical for every hackathon. I don't know of a good solution that would both preserve spontaneity, and allow people to prepare. Myself, I would err on the side of preparedness, because your day as a developer could be otherwise completely ruined if you waste it to find workarounds for configuration problems instead of coding.

It's a good thing that my years of programming experience enabled me to find those workarounds. Which brings me to my final observation. It's been observed time and time again, and this hackathon was no exception, that many women developers underestimate their skills. Some women who have been programming in the industry for 2+ years had wanted to sign up as beginners (the hackathon had separate pools of beginner coder and experienced coder tickets, so that every team could have a balance of both), but signed up as experienced because the newbie ticket pool drained faster. And that turned out to be the appropriate designation for them. At the end Girls Guild founder Diana remarked: many of you ladies signed up as newbies, but you are not new!

SheHacksATX: a hackathon done right

Hackathons can be a good change of pace for us developers because they force us to experience software development in a different way. At our day jobs we often work on a feature for weeks, because it needs to meet complex requirements that are often subject to many unknowns and last minute changes. But a hackathon can make it possible to implement not just one feature, but an entire prototype, or minimal viable product, in a day. It depends on how the hackathon is run, and just as much on preparation of the teams.

If your team has the right skills, or at least has researched in advance how to implement various moving pieces (for example, OAuth authentication using your chosen programming language), then a hackathon can give you a concentrated dose of accomplishment you often don't get at work. But if you approach a hackathon unprepared, and have to research those technologies as you're trying to use them, you will just spin your wheels. You might waste that day just scratching the surface of technologies you may or may not ever use. (I doubt that most projects started at hackathons ever get worked on again.)

Developers socialize in the morning before the coding starts. Left to right: Kathy, Stephanie, Ruby, Dallas, Nari. More pictures from SheHacksATX are in my photo gallery.

At SheHackATX, a women-only hackathon that took place on April 26, 2014, teams had not been formed upfront. Developers were matched with projects at the beginning of the hackathon. Each project was a woman-run startup (most of them local to Austin) that needed programming help, such as to add some features to their app or website. In the morning, developers wrote down their preferences of apps to work on, and the owners picked programmers they wanted on their team. Hopefully everybody got at least their 2nd or 3rd wish.

Since the teams were formed on the spot, they could not coordinate and learn complementary skills in advance. So it was all the more impressive that half of the teams were able to accomplish the tasks that startup founders wanted in the the time provided (from around 12 to 7 pm of one day). Those startups were Girls Guild, Groove, Yoga Recipe, and Bound Round. In my opinion, they were successful because their founders were able to correctly estimate tasks that might take about a day, and went with small'ish, manageable enhancements for their applications; nothing grandiose.

An example of that would be Girls Guild, a website that matches girls who want to learn hands-on skills (e.g. chocolate making, jewelry making, leather working or photography) via apprenticeship, with "makers", i.e. experts in that area. Both the makers and the apprentices are girls or women. One of the features Girls Guild founders wanted to have added to their app had something to do with being able to delay a payment until another event occurs (the details escape me). They also wanted ability for a maker to schedule interviews with a prospective apprentice through the website, as opposed to a manually emailing back-and-forth. The Girls Guild team succeeded in implementing both features.

Girls Guild and Hearth teams at work. Sitting at the table, left-to-right: Bethany (developer), Diana and Cheyenne (Girls Guild founders). Standing behind them, the Hearth team: Nari and Monisha (developers) and Florence (founder). More pictures from SheHacksATX are in my photo gallery.

Bound Round coders didn't have time to implement a 3D spinning globe (in Javascript, I guess) that the owner wanted, but they implemented a zoomable map that shows different levels of info depending on the level of zoom. Quite impressive for such a short time. Yoga Recipe wanted to integrate their website with Facebook, Twitter, and Instagram; "integration" meant ability for yoga instructors to share the "recipes" (sets of instructions for yoga classes) on social media. They succeeded with Facebook and Twitter. They also implemented ability for yoga teachers to put together playlists for their classes using Spotify.

My team also did well with our application, Groove. More about it in the next blog post, where I'll talk about what we did and how we did it, highlighting the obstacles that a programmer might experience during a hackathon.

Some other startups set their deliverable for this hackathon to be not code, but ideas. The founders of Our Desired Future and Hearth came here with just some ideas for their websites -- some better hashed-out than others -- and asked for technical advice on how to best implement them. Our Desired Future wanted to put together a series of multimedia presentations to tell interactive stories about water usage and water resources of Texas; in case of Hearth, what the owner really needed was to winnow down a number of her eclectic ideas about promoting volunteering through gamification, to something that could serve as minimum viable product. The result of her brainstorming with her team was a landing page for the website. In the process, one of the people on her team concluded that her calling in life was product management. So in these two last cases the product wasn't tangible; the real product was refinement of initial notions into more concrete ideas.

Some hackathons result in demos, some in questions

After finding out that the Devstack server we created during the walkthrough is not suitable for hosting web applications, I created a "plain" Rackspace server, installed Apache on it, and we proceeded to create a barebones web application. One of my goals was to get a clear idea how much progress a group of people could make on an application in a hackathon. The answer is, since we only got started after lunch (the first half of the day was taken up by the Devstack tutorial), and had until 4 pm to go: not much. But we got a little done.

Dana Bauer, Developer and Community Advocate at Rackspace, one of the organizers of the hackathon. She and another Rackspace employee helped us greatly with the registration glitches. At the end she encouraged people to give demos, or, lacking a demo, to stand up and speak about what they did, accomplished, or learned during the hackathon. Though Dana gave people Legos for speaking, very few people (including two from our team) came up to speak. Only one person (Kesten, see below) gave a demo. More pictures from the 2013 OpenStack Hackathon are in my photo gallery.

I am no front-end developer, but nobody else in our team clamored for that role. In a strange coincidence, the other 4 women on our team came either from PyLadies (Python was their language of choice) or from scientific computing background, or the intersection of both. So I assigned the front-end developer role to myself, fully realizing that a web-page hand-coded by me would look rather... homemade. I needed some HTML and CSS templates that would make my creation look at least somewhat professional. Specifically, I needed a web form. I spent a good couple of hours looking for HTML/CSS templates for forms, and most sites were misleading: either they promised free templates, but every template you picked required you to sign up for a fee; some websites promised form templates, but instead of providing the HTML/CSS code they let you create and host a form in their domain, which wasn't what I wanted. After a long search I was able to find a form where I could tease out its HTML/CSS code. Interestingly, throughout my numerous Google searches, Bootstrap didn't show up even once. But when I mentioned my predicament to the people at the hackathon and on Facebook, two of them recommended Bootstrap. Indeed, Bootstrap has form templates. Face-to-face interaction can still be more useful than Google.

The one and only product demo at the hackathon was given by Kesten Broughton, who created a 2D altitude chart as an addon for Google Maps "get directions" service using the Python beta API. His program was called Picycle, and was intended as a service for bicyclists who want to choose an optimal route, either minimizing or maximizing (if they want a good workout) the hills along the route. More pictures from the 2013 OpenStack Hackathon are in my photo gallery.

It took me 3 hours to put together a basic -- extremely basic -- front end to our application. There was no even a question of hooking it up to the backend. There was no time to write even a primitive web service that the front-end could call and display some results. One person in our group, more experienced with Python, finally got Twitter authentication to work in Python. This allowed her to query Twitter API programmatically by making requests from her Python code. Other of our members were just starting out with programming in general (in some cases switching from other professions), so I don't know what they did during that time. Perhaps they were going through Codecademy courses.

To our credit, other teams did not seem to fare better. Only one of the hackathon attendees had even a minimal product at the end to give a demo of, and he admitted he had been working on that product for 2 weeks already. His name was Kesten Broughton, and he created a 2D altitude chart as an addon for Google Maps "get directions" service using the Python beta API. His program was called picycle, and was intended as a service for bicyclists who want to choose an optimal route, either minimizing or maximizing (if they want a good workout) the hills along the route.

The lesson to be learned here is that to participate meaningfully in a hackathon requires quite a bit of preparation. If All Girl Hack Night is going to have a hackathon (I've been threatening to organize one, but always felt woefully unprepared), we would need to prepare in advance. For example, some us will have to be team leads who will have studied the API's of our choice (the APIs will also have to be decided ahead of time), and will be able to guide the teams so they could make tangible progress. That's the main lesson. Lots of advance decisions, planning what to implement, what programming languages and APIs to use, and a critical number of people who are familiar with those technologies and can guide others.

Making sense of criteria for making sense of Javascript frameworks

How do you select the right parts from a talk about how to select the right technology? Especially if you haven't spent a whole lot of time working with any of the development tools that are being compared? Perhaps you just listen to what resonates. And so I did at the All Girl Hack Night presentation on how to select a Javascript framework that's right for you, by Justin Lowery from Cerebral Ideas. Justin compared Angular JS, his team's favorite, with two other popular Javascript frameworks, Backbone and Ember. So while an experienced web developer might be nodding their head sagely at the criteria Justin gave (which I put at the bottom of the page), I just pondered how web development today is still oddly plagued by the same problems as when I did it in 1999.

When you just can't separate presentation from business layer...

Justin Lowery (the presenter).

Back in 1999, when we wrote cutting edge web applications with (ahem) CGI and Perl, separating business logic from presentation seemed a distant dream, which my team at the time quickly gave up on. We wrote spagetti code where HTML was generated in a deeply nested tangle of Perl's ifs and fors. It's 2013, and it sounds like HTML still can't be made completely independent of business logic. And some people say you might as well embrace that, and pick a Javascript framework that lets you hook up Javascript to backend-generated HTML easily. According to Justin, that's one of big advantages of AngularJS.

Those Javascript-hating backend developers

This will let backend developers generate chunks HTML without ever touching Javascript. In Justin's observation, backend developers hate Javascript, but are much more friendly with HTML. Why they hate Javascript more than HTML is anyone's guess. Perhaps it's because backend developers are used to "proper" object-oriented languages, whereas Javascript kind of has objects without actually being object-oriented. It has this strange prototype inheritance, which is not like the "classical" inheritance that backend developers are used to. But I have been a backend developer for a long time, and I'm not averse to Javascript: I've been learning it eagerly in recent months, and I'm fascinated by some of its features like closures.

Freeform HTML versus generating the whole page from Javascript

Then again, there exist Javascript frameworks that not just let, but force you separate presentation from business logic; it's another question if those solutions cause bigger problems than they solve. I am talking specifically about ExtJS. ExtJS expects all the data from the backend to be returned as JSON objects via service calls. I'm not sure you could make it accept custom HTML even if you tried. At least that would be very hard, and it would violate another of framework selection commandments: "Choose a framework designed by someone who thinks like you, otherwise you'd be fighting it all the way". It would be hard to stick any custom HTML into an ExtJS application, because ExtJS does not "hook" into your existing HTML: it generates it all from scratch, from Javascript alone. Some pople don't like that. I suspect our presenter didn't either, because he considers it an advantage if a framework lets you write free-form HTML, like Angular does.

This ties into our next criterion.

Page load time: the "no blank page" criterion

Left to right: All Girl Hack Night members Annine, Yim, Angelina, and Kate at the Angular JS presentation.

Allowing static HTML versus generating it all from Javascript is tied to another criterion: how fast the framework is. Mainly it means page loading time. Justin observed that a perception of loading time is not the same as the loading time itself. This criterion can also be summed up as "No blank page prior to JS parsing". If your application renders some HTML elements before it executes all the Javascript and populates fields with data, some HTML elements are already on the page even though some fields are initially blank. Even if loading the data into those fields takes 2 seconds, the user will still perceive this page as loading faster than a page that stays blank until it generates all the content, though that might take 1 second. In this case Angular obeys "no blank page" rule, whereas ExtJS -- you guessed it -- doesn't.

Opinionatedness: what does it mean?

The most mysterious criterion is probably "opinionatedness" of the framework. Perhaps you have to be an experienced front end developer to fully understand this concept, and I'm not. A Google search did not give me a crystal-clear understanding of what it means. Apparently, opinionated APIs are on the opposite end of a "scale" than REST APIs. Unlike REST, opinionated APIs don't treat URLs as resources. They are more like function calls that URLs: save(collection, key, value) versus http:///collection/{collection}/variable/{key} (this example is from an article Opinionated (RPC) APIs vs RESTful APIs). But from what I read, experts seemed to think that REST APIs are more intuitive than opinionated ones.

Clockwise from the left: Bridget, Patty, Ashley, Maryam, Simi, an IT guy from the company that hosts All Girl Hack Night; Justin Lowery (the presenter), Neelima, unidentified woman, Annine, and Yim.

Opinionatedness of the framework figures into its learning curve. Justin said that Angular JS becomes harder to learn as the API becomes less opinionated. This seemed counterintuitive to me. Why would it be harder to program using RESTful API than one that consists of commands? Googling more about it, I found that AngularJS is called opinionated because it forces you to give your application a certain structure. If so, that can influence the learning curve in two ways.

Again, let's take ExtJS as an example. If you are writing an application that resembles the "Hello World" tutorial, it's easy to use this framework. You follow the steps, and the widgets on the page magically materialize and get populated by the data from the backend. So the learning curve does not seem too steep. But then you find out that a real-world application does not match the confines of the tutorial, and its logic often does not conform to the structure the framework imposes on you. That's when the learning curve becomes steep, as you have to find various backdoors and workarounds around the framework's assumptions. From that perspective, I'd say ExtJS is very opinionated, and this can make it hard to write real-world applications. This framework gives you prefabricated houses instead of individual bricks. But if it gave you just the bricks, a beginner wouldn't know how to stack them into an architecturally sound house.

A meta lesson is that it may be hard to know what concepts are and are not too vague for an overview talk (assuming that overview talks are introductory in nature). All Girl Hack Night has developers from all walks of life, from device driver writers to front-end, and I don't know that many non-front-end developers are familiar with the concept of opinionatedness. I will have to keep that in mind when creating my own presentations.

What kind of philosophy / methodology does the framework use to solve problems?

• How is it architected?
• Is the framework authoritarian or egalitarian?
• Are all problems solved with Javascript?
• Are problems solved with a bigger, more aggressive API?

How does the framework solve common problems of development?

• Templating?
• Data binding?
• Routing?
• Bootstrapping?
• Data management?

How active are the creators in the community?

• Do they have an active forum with many users?
• Do they have stale pull requests or issues list?
• Are they active in the community?

HTML-centric philosophy

• HTML is intuitive for those that are non-Javascripters.
• Strong separation of concerns.
• Event binding is much easier.
• Backend devs that hate JS have limited exposure to it.
• Views are compatible with other frameworks, especially backend technologies.

Programming interviews: recommendations versus experience

Three years ago, when I was looking for a job after 10 years of working at one company, I really wished there were consultants to help software developers to improve their resumes and prepare for technical interviews. How enormously helpful it would be if someone who was a developer once to guide your preparation! Especially if such a person collected many years' and many companies' worth of technical interview questions.

Such a company now exists, and its founder, former Google engineer Gayle McDowell, visited Austin last year to promote her book "Cracking the Coding Interview". In her presentation, based on the book, she advised software developers what to expect at, and how to prepare for technical interviews. I found myself agreeing with Gayle half of the time, and scratching my head the other half. If my experience did not match her recommendations, the reason may have been that her advice was directed to (a) mostly beginner programmers, and (b) those who want to work at big companies. No wonder her presentation was paired with a hiring pitch by Amazon Web Services.

Let us see, then, how my experience of interviewing for software development jobs stacks up to Gayle McDowell's advice.

What happens when "they" pull your resume out of a giant pile Gayle's version Pull resume out of giant stack; Spot-check: company names, positions, projects, schools; Skim bullets to see if you've written real code. My experience I haven't been on the recruiting side of things, but I was hoping that that's how it is -- that a real hiring manager (i.e. someone who knows what to look for in a developer), would look for that, as opposed to for buzzwords from an HR bingo.

Resume length Gayle's recommendation: Your resume should be 1 page only, unless you have more than 10 years of experience. In a 2-page resume, no one will read all the points; they might read 50% at most. The hiring manager will skim your resume and notice only some random job experience points. Some of them won't be your strongest. Wouldn't you want a hiring manager to notice only your strongest experience? If so, cull them and put them on a 1-page resume. My experience: I had many seasoned recruiters tell me that my resume needs to be 2 pages. But then, I have more than 10 years of work experience.

Technical questions Gayle McDowell recomended her company's website, CareerCup.com, for a good list of real interview questions. I checked it and the questions are really good -- they don't test your knowledge of a programming language syntax, but rather test your ability to design, as well as deep understanding of object-orienting programming.

"Don't memorize!" Gayle's recommendation: If an interviewer asked you a question you've already heard before in a different interview, don't just spit it out the answer -- tell them first that you've heard it. It's likely that the interviewer will still ask you to answer the question / solve the problem, but you won't come across as dishonest. Plus, they don't want just to hear you spit out the answer -- they want to know your thought process. Seeing that you've memorized some answer tells them nothing about you -- the question was wasted. It didn't give them any extra information about you. My experience: Perhaps this is reasonable for college students, but if I am an experienced programmer, I might have faced certain types of problems in my programming practice often enough that I know the answer off the top of my head. Such is the question mentioned later in this text, where the answer turns out to be binary search tree. What if at some of my previous jobs I searched for data in large lists? Then I could spit out the answer "BST" before I even heard the end of the question, and it's not because I heard it in another interview, or memorized the answer off of some online preparation site -- it's because I know it from practice.

Push yourself Gayle's advice: When studying a code problem, write the solution from the beginning to end. Don't just stop because you think you understand it. When faced with a whiteboard at an interview, you may discover you don't understand it nearly as well as you thought. Write the code with pen on paper -- it's harder than in an IDE, where intellisense helps you. But you won't have IDE and intellisense on a whiteboard. Using pseudocode is OK temporarily, as long as you tell the interviewer you will fill it out with actual code later, and actually do so. My experience: Agree completely. To make sure you completely understand a problem, you need to write it on a piece on paper from beginning to end. Otherwise it's easy to stumble when solving it on the whiteboard. You are used to the IDE doing so much for you, that when you are faced with a blank sheet of paper, you might start thinking, do you need a keyword "function" to declare a function in C#? Or was that PHP? I use lots of pseudocode when writing code on a piece of paper, and I think if the pseudocode is detailed enough, most interviewers would be fine with that. No one necessarily expects you to remember all the 50+ methods of the generic List<> class when you're coding on the whiteboard.

Data structures Gayle's recommendation: You have to know how to implement, or when to use, the basic data structures: linked lists, stacks, queues, trees, tries, graphs, vectors, heaps, hashtables. Especially hashtables -- they come up often in interviews. My experience: I agree about the importance of hashtables. Many code problems I've been given in interviews required use of a hashtable for best solution. Linked lists were the second most popular data structure -- for example, I was asked to write code to reverse a linked list. There was a time when I was asked to write a program for searching a binary tree. But I was never asked about a stack, queue, trie (whatever that is?), graph, or heap. (As for vectors, they are essentially arrays, and they are used in every coding problem -- you can't write almost any code without it.)

Algorithm Questions Gayle's advice: Study the basics: complex algorithms usually unnecessary. My experience: I agree: I have never been given a problem that required more than a simple algorithm. Well, at one point I was shown the code for merge-sort, and asked what was the complexity of that code (it's O(n log n)), and the interviewer asked me which lines of the code contained the log n part. That was the most complex algorithm I had to deal with at an interview, and even then I didn't have to write code for it, only analyze it.

Decomposing algorithm problems Gayle's advice: To make the problem easier to solve, decompose it in 4 steps: Pattern Matching Simplicity and Generalization Base Case and Build Data Structure Brainstorm My experience: Overall I agree, however, I think we think about decomposition differently.

Example of decomposition Design an algorithm to print subsets of a set: {a, b, c} -> {}, {a}, {b}, {c}, {a, b}, {a, c}, {b, c}, {a, b, c} Gayle's decomposition: You start with empty set: S({}) -> {} Find a solution for a one-element set S({a}) -> {}, {a} For a two-element set, the solution is still obvious: S({a, b}) -> {}, {a}, {b}, {a, b} S({a, b, c}) -> ? You'll notice that there is a pattern: build S(n) -- a subset for an n-element set -- by cloning S(n-1) and adding n to the cloned sets. My comment: This is an example of a problem that can be solved inductively, but there are many problems where induction wouldn't help. Such is, for example, a problem that I was asked at a past interview -- determine if two words are anagrams of one another. It's easy, yes, it's just not inductive. Probably because we are not required to find all anagrams of each word (in which case it would be closely related to the subset problem), only to determine whether one word can be an anagram of the other, i.e. if both words have the exact same set of letters. You don't gain knowledge on how to do it for a 3-letter word if you know how to do it for a 2-letter word, because it's an exact same problem.

Pattern matching as a step decomposing algorithm questions Reverse the order of words in a sentence Write code to reverse the order of words in a sentence, e.g. "dogs are cute" to "cute are dogs". Gayle's way: This problem is similar to reversing the characters in a string: "dogs are cute" -> "etuc era sgod". The answer: reverse full string, then reverse each word. My comment: This solution sounds straight from a joke about mathematicians. How does a mathematician make tea? He/She pours water into a kettle, puts the kettle on the stove, turns on the stove, waits until the water boils, then ours the boiling water over the tea leaves. Now suppose the kettle is already boiling. How do you make the tea then? Mathematicians's answer: turn off the stove, dump the water, and now you've reduced this problem to the one you already know how to solve. I think a simpler way to reverse the word order in the string would be to split the string at spaces, put the resulting words in an array of strings, and reverse that array (many programming languages have built-in functions to reverse arrays. The ones that don't, like C, don't have functions to reverse strings either).

A ransom note from a magazine Design algorithm to figure out if you can build a ransom note (array of strings) from a magazine (array of strings) Gayle's way: What if we used characters instead of strings, and built an array of character frequencies? And generalized to word frequencies? My comment: The suggested approach doesn't seem to simplify much. I don't think it's essentially simpler to do that than to build an array of word frequencies. The key to it in any case is hashtable. Familiarity with the hashtable data structure is pretty much the only thing you need to know to keep track of how many times each word occurs, and therefore if there are enough words to put together the note you want.

Unconventional, but intuitive advice

Some of Gayle's general interviewing advice was not what career advisors typically give, but consistent with my experience. Such as: there is no need to write thank-you cards, because no candidate for an engineering position was ever rejected for not writing a thank-you card. People make a decision whether to hire you pretty soon after the interview - perhaps within a day or so -- and that's much sooner than your thank-you card could reach them. From over a hundred of candidates that she had interviewed at Google, she got only 1 thank-you card, and it did not sway her decision towards that person in any way.

Finally, do not despair if you think you did poorly. Gayle says that you really, truly can't know how well or poorly you did. Seriously, you think you do, but you don't. Candidates are evaluated on the scale of comparison with other candidates, so sometimes you might think you did really poorly, but in the interviewer's eyes you were the best candidate interviewed for this position.

Kids Code Camp, Austin, Texas, April 22, 2012

Teaching programming -- even to adults, but especially to kids -- presents you with a dilemma, all the more acute when you only have 1 hour to interest your students in the subject. You can take a practical approach: let's create a toy program that does something you like: move, bounce, play sounds. Or you can take a more-or-less exploratory approach: write some simple code, tweak various parameters in the program, and see how the computer responds. It is like having a "stick" -- a programming language -- to "poke" the computer with. What does it do when you issue this, that, and the third command to it, even if those commands don't add up to any eventual goal? I, personally, think the first approach would interest more people (especially children) than the second one. But the first one is harder to achieve. It's not easy to design a course where students would write a program simple enough for beginner to accomplish in 1 hour, but meaningful or entertaining enough to make students want to do more of this.

So it's not surprising that the Kids Code Camp -- a day long series of programming classes for kids -- took the second route.

For the first session, Ron Evans (@deadprogram) from Hybrid Group taught kids to write a few basic Ruby programs with KidsRuby. It's a programming environment that lets you easily write a simple program and run it. And by “easy” I mean it eliminates the overhead of creating and saving files, wondering what directory to save them to, how to invoke the compiler, what libraries to include, etc. It lets you run just the "meat" of the program. Not only you don't have to open a file, you don’t even give have to your program a name. Just type the program in the left pane of the KidsRuby window, and press the button "Run". The output will appear in the right pane.

Or... not.

Ron Evans gives out candy to the kids who answer questions right. Eventually he gave candy to all the kids.

The "little" problem with KidsRuby is that if you make a typo -- for example, don't close a string with a double quote (") -- it doesn't print an error message. Your program just won't run. Needless to say, for beginning programmers this makes it one of the, shall we say, least useful programming tools. If your code won't point out your syntax errors, how are you supposed to know what's wrong with it? You’ll probably lose your interest. The only way a kid could sustain interest in this is if there was an adult nearby with knowledge of Ruby, or basic programming -- or if the kid was a precocious genius, armed with programming books and hell-bent to figure out what's wrong.

Turtle.start do
background yellow
forward 30
turnright 90
forward 50
turnright 90
forward 50
turnright 90
forward 50
end


-- draws a square.

Turtle.start do
background yellow
pencolor blue
turnright 45
forward 50
turnright 90
forward 50
turnright 90
forward 50
turnright 90
forward 50
end


-- draws a diamond.

This brings me to another topic. Hell-bent precocious geniuses will eventually figure programming out on their own, but what exactly is the audience to which one should teach? The traditional audience -- kids who are interested in tinkering of any kind, in pressing buttons and typing cryptic commands just to see what they do? Those were the typical kids who studied programming. Or do we also want to teach kids who, like me, would have found such a thing mostly pointless? If, however, someone had shown me what you can make with it -- a finished object, something with a purpose, even if only a toy -- I would have been much more interested. It was via this route that I eventually discovered my interest in programming as an adult.

As I noted above, the purpose-oriented approach is much harder to teach. However, some programming environments lend themselves more easily to this approach than others. One of the sessions at the Kids’ Code Camp explored a graphical programming environment called Turtle. Its commands specify how far a colored dot has to go and in what direction, or how many degrees to turn and in what direction, in order to draw a figure you want. (Sample Turtle code is shown in the sidebar.) So you can fairly quickly learn how to write a program to draw a geometric shape pixel by pixel. I also liked how the presenter explained pixels to kids: put your nose to the screen and see if you see any dots - those are pixels.

Here's the code of What I Did on my Summer Vacation program:

holiday = ask "Which holiday?"
familymember = ask "Family member?"
activity = ask "Activity?"
smallproblem = ask "A small problem?"
puts "It was just " + holiday " "."
puts "I was with my " + familymember + "."
puts "I also went " + activity + "."
puts smallproblem + " but it was OK."
puts "It was a good " + holiday + "."


Writing a program to draw a complete geometric shape may be more than you could learn in 1 hour with Scratch, another children's programming environment that was introduced at Kids’ Code Camp. Scratch doesn't let you manipulate individual pixels like Turtle does. In this environment everything is done graphically -- programming constructs such as loops and variables are colorful interlocking widgets you snap together to “write” a program. It's best suited for creating animations involving a vast choice of cute avatars. It is very handy for explaining such concepts as loops, since you can make the avatar repeat animated motions endlessly. But programming is knowing not just how to create a loop, but how to combine it with other code concepts to achieve what you want. And to do that in Scratch, you'd have to spend more than 1 hour -- especially if you are a kid, especially if you haven't yet learned coordinate system or negative numbers. You have know that in order to tell the avatar where to move. And to get your avatar to move to a precise location, you can only do that with trial and error, by plugging in different numbers and seeing where it goes. Not exactly a riveting activity...

At one point, though, the curriculum of Kids’ Code Camp showed a hint of purposefulness. While teaching Ruby, Ron Evans introduced a "hack your homework" program. It was a ~ 10-line KidsRuby program that asked you "where, when, who, what" types of questions, and plugged your answers into a cookie-cutter essay: "Last summer I went fishing with my cousin", etc. Then it spat out a finished text that probably wasn’t worse than an average student’s homework. :-)

Of programmatic thinking in low-tech situations

These days it's been fashionable to insist that everybody should learn to program. Some argue it's the next kind of literacy (a federal judge who learned how to code could correctly estimate how long it would take to implement a certain function), others say the importance of programming in an ordinary person's life is overblown. There is a vast difference between basic programming knowledge and being a professional software developer. The question is, can this basic literacy have applications in everyday life? Can it improve the life of a non-programmer? I liked this article that demonstrates how knowing how to program benefits even the people who are not programmers. When you have developed a mentality that lets you see many life problems from an engineering perspective, you see how software could help you improve even those life processes that you previously didn't think a computer could solve. It's a matter of thinking algorithmically, of seeing what could be automated. I have a similar problem as the one described in the article. I take lots of pictures of people at conventions and conferences, and there isn't a good way to "connect" photos of strangers with their names (which I forget instantly). However, unlike the casting directors in the article, I have neither assistants with spreadsheets, nor do people parade in front of me one by one like models. I also think that it would be awkward to ask them, after introducing myself, to write their names on a piece of paper, and to pose with it. :-) The best I can do is take pictures of their name tags, but sometimes those are missing, or flipped over to the blank side, or flash bounces off of them in a way that makes them illegible. On the other hand, my tablet allows me to add notes to any picture I take. (Well, I can't add notes to a picture directly, but I can save it to Evernote with a note attached.) So perhaps I'll just have to take two pictures of everyone -- one with the real camera, for quality, and the other with the tablet, for documentation. If a situation is not structured, there isn't much room for a programmatic solution to a problem.

My drafts are a continuum

I don't know if I finished the first draft of my novel. At the beginning of last year I said I was going to, but I don't know. That's because I'm no longer sure what should count as a first draft. The zeroth draft was clear. I had a beginning and an ending, but the middle lacked some chunks. Now all the parts are in place, and in the right order. Finding the right order was nontrivial. Maybe normal writers write a story in a linear fashion, as it develops in time, but that's not me. I write chunks and scenes, and wrack my brain about how to make them fit together. Which one should come before another one? The story can be told in any of the million ways.

I can't help but see stories from a software engineering perspective, not something that develops in time, but as a modular structure that occupies (mental) space. The logical way the modules fit together may not necessarily be the temporal way.

I finally worked out the order in which to string the scenes together. But I also determined I need more scenes with certain characters, because they are more instrumental to the story than I thought before. So can I really say I have Draft 1, or is still Draft 0.5? Or perhaps novel drafts should not be versioned like software releases?

I also have several short (or "short") stories I am working on, all but the newest one completed, and in various states of rewriting. One of them underwent five rewrites -- five completely different variations on the same theme with the same characters, converging on the same finale -- and this is still not the end. I need to figure out how to make the final version shorter. Might I be overthinking this just a tad?

That's the state of my writing for 2011. As far as my programming goes (and I can only speak of spare time projects, not anything I do at work), I'll spare the gentle reader an account of all false starts and stops on my coding endeavors throughout the year. It suffices to say that for the last month I've been trying to get an application I wrote to work on a Windows-hosted website. It is supposed to automate certain tasks on which I spend a lot of time, so it is important to me. To make a long story short, I found out that there are certain incompatibilities between the development environment on my laptop, and my Windows website environment, that can make it impossible to host that application. There are a couple of ways to deal with this, and I don't know which is more viable. It can also be said that when it comes to web hosting, I have champagne tastes on beer budget.

How come xkcd hasn't addressed this?

I often hear that engineers are literal. This typically includes software developers. I hear this even from people who work with engineers all the time -- you'd think they would take a more nuanced view. But perhaps not many people understand what "literal" means. I'm not even talking about the common, oft-ridiculed use of "literally" to mean "figuratively" -- as when somebody tells you "I literally died laughing" (and they don't have chunks of rotten flesh falling off, so they're not Undead).

Figurative, after all, is the opposite of literal. You can also say that the opposite of literal thinking is metaphorical thinking. Apparently most people don't understand how much certain intellectual activity, such as science and engineering, relies on metaphor. As an example, read this article in Wikipedia about Aspect-oriented programming, and show me even one paragraph in it that does NOT contain a metaphor.

The point is: software design, like any design, models real world problems in some kind of abstract conceptual framework. This is only possible if you think in metaphors.

I really think there should be an xkcd cartoon on this.

Girls hack Arduino

In all my programming life I never tried communicating in code with real-world objects. But now there is something to make it easy (or at least doable): Arduino. Sharon Cichelli demonstrated it at the All Girl Hack Night in October of 2011. Arduino is a microcontroller that lets you program blinky LED lights, or things that make buzzing alarm sounds, or all sorts of other electronic. It also accepts input from buttons or sensors, or anything that translates a physical action into electric pulse. You can plug things into one of its pins (there are 13 of them, if I recall), and have it "read" electric pulses from those pins, or "write" to them.

"Read" and "write" probably shouldn't be in quotes, because they are the same programming concepts as reading and writing to/from any input/output device. So, to switch on an LED light that's plugged into pin 13, you would digitalWrite(13, HIGH). As with all things digital, the function digitalWrite only recognizes on/off, or in this case, HIGH/LOW values. HIGH means the device plugged into the pin is on, LOW -- off. Simple, right? These basics allow one to sit down and program a primitive blinky light in minutes. The programming language is fairly high level. Its function names, as one can see, resemble human language -- this isn't assembly, where you push bits around with three-letter commands. It enables you to quickly do simple demos, like Sharon Cichelli did. Just as you digitalWrite to a pin to turn a device on, you digitalRead from a pin to recognize, say, a button press. Again, you pass to it the number of a pin into which the button is plugged in.

You write the code in an IDE, and with one mouse click upload it to an Arduino connected to your computer via USB.

Sharon Cichelli demonstrates Arduino programming.

Arduino programming seems encouragingly simple at first, but as with everything, once you get deeper into it, you discover complexities. In fact, one's lack of electronics background might put brakes on the progress. Having been told that a LED light is plugged into a pin number 13, you (and by you, I mean me) might notice that the light actually has two wires, both plugged into tiny slots. Why are there two wires, and which of them is # 13, you ask. Oh, it's because the light needs to be grounded. The other wire is plugged into the grounding pin, you can ignore it for the purposes of your program. Yes, grounding is something you should have remembered from physics classes, but didn't. Then you digitalWrite to it, and it still doesn't light up -- that's because something is wrong with the hardware, and the circuit does not close as expected. Maybe the light isn't plugged in firmly enough. So, there are two dimensions to debugging Arduino code: a programmatic and an electrical dimension. As if catching just the software bugs wasn't tricky in itself!

Sharon couldn't help but demonstrate troubleshooting techniques, as things malfunctioned. Pins "floated". Still we had enough time to progress to a more advanced part of the lesson: Pulse Width Modulation. If you can only send "on" or "off" signals to a LED, how would you dim the lights, or produce nice, slow fade on / fade off effects? The fading or increase in brightness is caused by flickering very fast between on and off positions. If the light spends more time in the "off" position than "on", it appears to fade; if in on position, it grows brighter. If we graph HIGH and LOW signals versus time, where every HIGH signal is a vertical bar, we'll see that the "width" of the HIGH signal decreases (increases) over time. So that, simply put, is Pulse Width Modulation. Sharon pair-programmed with someone (sitting another person down at her laptop's keyboard, and dictating code to her) to implement the slow fade.

Besides blinking and buzzing things, you can connect Arduino to vibrating things. Erm. I mean, like a mat under a cat food bowl. A vibrating mat would scare off a cat that is afraid of vibration. This was the idea one of the girls had. How do you make a cat lose weight, if you also have a normal-weight cat, and each time one sees the other eating, she will eat too? The goal is to make the fat cat to eat less frequently than the normal-weight cat. So, this girl got an idea to put a vibrating mat under the cats' food bowl, so that when the cat steps on it, her weight would send an input to Arduino, which would trigger the vibration, scaring the cat off. The smaller cat's weight would not trigger this response. And the fat cat's weight would only do it at certain times but not others, otherwise the pudgy feline would never eat.

And if you don't have problematic cats, there is still a wide variety of applications for Halloween costumes, such as blinky eyeballs like Sharon made for her own Halloween costume.

Am I the only one to want this? Linking to any part of an HTML page

Here is a pipe dream. I would like HTML to have ability to link to any paragraph in any web page. I would like to make it so that when the user clicks the link, the page would open with the paragraph of interest at the top of the browser window.

I only know how to do it if some section of a web page has an <a name="xxxxxx"> in it. And if the page I'm linking to isn't created by me, I have no control over placement of <a name>'s in it. But is it possible to link to any element of a page? If not with HTML, then with Javascript or CSS? Can one write a Javascript that, when a user clicks on a link, will bring up the desired element of the page that has loaded? I kind of doubt it, because once a page has loaded in a browser, the scripts of the previous page stop executing, correct?

And I'm pretty sure there is no way to do this if the desired paragraph is not in any uniquely-identifiable HMTL element -- for example, if it only separated from other paragraphs with <br/>

If there is really no way to do this, I wonder if such functionality was never considered by They Who Design HTML? Has nobody ever wanted it? I wish for it when an explanation or example of something I'm writing about is buried deep in some other web page, and I know that my reader, if he/she clicks on the link, is not likely to find it quickly, or to spend much time searching for it.

I suppose one can write a third-party application that would let you mark up snippets of web pages and save them; then you could link to the paragraph on the third party website. It could provide a link to the original article, for those interested in the context. But that's an additional layer of complexity. There may even have been such a service, called iLighter; it allowed you to highlight web pages and save those highlighted snippets on its servers. I know I installed it in one of my web browsers a couple of years ago, but didn't do much with it. I had app fatigue even then (too many new web applications to try), which only got worse since. And iLighter doesn't seem to be around anymore -- I guess not too many people found it useful.

I know, I could write a Javascript that would pop up just the paragraph I want to link to, when the user's mouse hovers over the link to the article. So the user could get just the quote, but read the article for context if they'd like. Of course, this is inefficient, because the quote may change some time in the future, and somehow I would have to become aware of the change (which won't happen unless I monitor that web page every day), and modify my script. So, it's not a real solution.

If only there was a little girls hack night...

My not-yet-6-year-old daughter isn't very curious about what I'm doing or where I'm going when I'm not with her. So it was unusual for her to show interest in my going to All Girls Hack Night. As I said before, All Girls Hack Night is a monthly meeting of female developers, where they get together to work on their code projects and talk, with a brief presentation on a programming topic thrown in. I told Erika I was going to a meeting with women programmers. I didn't expect her to make anything of that bit of information, let alone to take interest in it. But she did. "So you are a programmer?" she exclaimed. "Yes," I said. "I didn't know it! How do you act like a programmer?" she asked. I thought this was probably her way of asking "what do programmers do?" I said, "you write code on a computer", feeling uncertain, as usual, how much background information to supply. How do you describe succintly, without taxing a 5-year-old's attention span, what is code and how does one write it? She didn't ask me to explain, but had other questions. "Who are those ladies?" I said I could name some names, but she doesn't know them. Of course, that was even less helpful. Then she told me several times that she wanted to come along. I hated to tell her that this is for grown-ups only. I feel like I should do something about this interest of hers. I know there are some programming environments and languages for children, but they are aimed for older children, at least nine years of age. (And probably for a good reason -- most programming concepts are too abstract for a 6-year-old to understand.)

The previous All Girls' Hack Night in December was sparsely attended (yes, holidays), but this one picked up again, with one woman doing a short presentation on CodeIgniter, a PHP application framework. Then a representative of our sponsor, TabbedOut (that provided the meeting space) talking about her company and mobile software development it does. The rest of the time we were supposed to be working on our projects, but as in all coworking situations, we mostly chatted.

The talk of web development (in which many of these women specialize) inevitably brings up the topic of cross-browser compatibility, or lack thereof. Many web developers would like nothing better than for people to stop using IE 6, so that they would never have to code special hacks just to get a page to display in IE 6. But sometimes even convincing your own family to drop it can be very hard, especially if a family member is older, not too tech-savvy, and stubborn. One woman weaned her mother from IE 6.0 by installing Firefox and changing its icon to IE. Her mother, who was't very observant, noticed it only half a year later. She got angry, but the daughter asked, when was the last time you got a virus? True, mother admitted, she hasn't got a virus in a long time (that coincided with the time she had been using Firefox), but that's only because she was "being safe on the internet". The daughter didn't buy it, pointing out that she used to have to wipe out the hard drive on mom's computer every 3 months to get rid of malware. But you can't very well explain this to a person who, seeing a printer that advertised itself as wireless, demanded to know why the daughter plugged it into the wall. After all, it's wireless!

First All Girl Hack Night

My post on the first All Girl Hack Night meetup is now on GeekAustin.org . On 9/28/2010 female programmers of Austin got together to socialize and work on their code projects.

More pictures from the meetup can be found in my photo gallery.

Inspiring stories from Texas Linux Fest

At the Texas Linux Fest I was particularly intrigued by Amber Graner, who went from a complete Linux newbie, to becoming a Ubuntu User Magazine contributor in one year. She spread a message that everyone, no matter how non-technical they are, can become proficient in Ubuntu. As they progress, they can participate in user forums, educating less advanced users. If I'm not mistaken, she said she recently accepted a position as Ubuntu User Magazine editor, though I haven't been able to find a confirmation of that on the net. Whatever her position is, the magazine is now her day job.

My old acquaintance Janet S., who I first met at the Atheist Community of Austin, was at the Texas Linux Fest too. She is now involved in writing FLOSS manuals -- technical documentation for open source software. And what do you know -- a couple of weeks since TXLF she started a job doing developer documentation for Mozilla.

Amber Graner gives a presentation "A Year NTEU* (pronounced In-to) Ubuntu and the Open Source Community". More pictures from Texas Linux Fest are in my photo gallery.

Those stories are inspiring. While I am not by all means closed off to a possibility of writing proprietary software (if you are a prospective employer Googling me, please be assured of that :-)), I've been thinking there are serious advantages to working in open source. It opens prospects in self-marketing you won't get in closed-source. Quite simply, if you're writing proprietary software, it's very hard to prove to prospective employers that you've done worthwhile, creative things. They have only your word to take for it. And even if you were permitted to show pieces of your code to outsiders (extremely unlikely), they still won't understand the context your applications worked in, the needs they addressed, or technical challenges they conquered.

But if you are writing open source software, depending on how widely used it is, chances are that someone on your prospective employer's team may be familiar with it. Also, its open nature allows you to blog about your work, and answer questions about it in user forums, building your online visibility and reputation.

At the nonprofit I'm currently working for we use Drupal and the LAMP stack -- Linux, PHP, MySQL and Apache -- to build our web application. At the very least this will give me opportunity to openly blog about the problems we are facing, and the ways we are solving them. I'm too busy actually building the application to blog about it yet, especially we as we are trying several approaches and it's not clear which of them will "stick". But I'll have to start blogging about it soon. Drupal is open-sourced, with a huge community seeking answers to their own problems -- thus, lots of social capital to be gained. I'm hoping it may become useful for me one day if, as I suspect, this nonprofit does not secure funding, and I'll have to continue to search for jobs.

Pictures from Texas Linux Fest are in my photo gallery.