“It’s a big universe Jerry, and some things in it are talking monkey work”
Gabriel, The Prophecy
During the 2009 H1N1 flu pandemic I was working as a research assistant in a computational epidemiology lab. Early on in the epidemic data sets were scarce, but the CDC was publishing what they had online in HTML tables. The dozen or so assistants in my lab were tasked with hand-scraping all of the CDC data and compiling it into CSV files for analysis. Our task was tedious, but required focus and attention, and so it kept me from even listening to a lecture while I copied and pasted and removed markup.
I put up with it the first time we had to collect a small sample. When the professors decided we’d be aggregating the entire data set, and potentially others, I wrote a script to do my part. After finding out what I was up to, they pitted me against my cohort in a race to complete our current data set. It was a modern twist on the steel drivers’ ballad, only I was running the steam engine, and racing a whole team of John Henrys.
Within 15 minutes I had a python script and my data set. In 20 minutes, the whole lab’s data scraping for the week was done. Totaling it all up, what would have been 100 man-hours or more was finished in half an hour. The machine had won.
Unlike John Henry’s story, though, no one died trying to beat my machine. No one even lost their job. Instead, everyone got to spend the reclaimed time on the important stuff – constructing and validating models of the disease.
That was a powerful experience, and it drove home a lesson that I’ve read about a dozen times since in one form or another: There’s so much interesting stuff in the world that sits firmly in the domain of talking monkeys. There’s so much time lost to work that could be better accomplished by transistors.
Embrace your talking monkey work. Leave the transistor work to the transistors.
Automate all the things.