The Capitol 10,000 is a total friggin’ blast. Anyone with a remote interest in running should participate. Rarely do you get to run up the middle of the Congress bridge straight to the capitol through an unstoppable sea of humanity. On your way to the finish line you’ll pass spectators offering beer, bacon, and donuts.
Before you can catch your breath, they make the results available online. I’ve been looking for an opportunity to jump back into Python, so naturally I scraped the HTML and ran it through a script so people can play with the data.
Here’s how it went down.
Results are posted to mychiptime.com. They have some client-side JavaScript query their backend with your specified parameters. With a little help from Chrome’s developer console, I found their URL scheme. Use wget to download the data. It’s HTML that gets inserted directly into the page with a $(“#blah”).html(response).
The HTML returned by their API is truly hideous. A mere 20k rows cost 16MB… because they’re chock full of <b>, <font>, and “OnMouseOver”. After pulling out the relevant data, the file size is about 800KB.
Here’s the python script used to generate tab delimited .csv files from the HTML input. BeautifulSoup can take a few minutes to parse the larger files.
from bs4 import BeautifulSoup
#produce tab delimited CSV files from large html files
for year in range(2008, 2013):
print 'processing ' + str(year)
file = str(year) + '.html'
f = open(file, 'r');
html = f.read()
soup = BeautifulSoup(html)
table = soup.find("table")
#first row with column titles
rows = table.findAll('tr')
i = 0
#write data to csv file
outfile = str(year) + '.csv'
out = open(outfile, 'w')
this_row = ''
for row in rows:
cols = row.findAll('td')
for col in cols:
b = col.find('b')
text = str(b.string)
this_row += text
this_row += ' '
if i % 2 == 0:
this_row += '\n'
out.write(this_row)
this_row = ''
i += 1
out.close();
f.close()
Playing with the data
Building a simple scatterplot was way harder than it should have been. Google docs’ spreadsheet crashes when I try to build a chart. The Python library matplotlib chokes on the rows where Age is “None”. Wolfram Alpha rejects it, probably for the same reason. LibreOffice’s Spreadsheet just made my laptop really hot. Octave is inscrutable. But Plot for OSX did the job.
2012 Cap 10k
I’d like to see someone with MATLAB skills come up with some more advanced plots or divine some insight from the data. The columns available for 2012 are:
Name
Division Place
Gun Time
Chip Time
Overall Place
Age
Zip
Gen Place
Total Pace
Total Div
Total Gend
Tot AG
Here’s a zip archive of all the tab delimited CSV files 2008-2012.
I loved this book. The narrative happened to be about a tech company and I’m sure that helped draw me in, but at its core this is just a great story. There are a handful of criticisms aired in this HN thread focused around the technical accuracy of products and software lineage. Those things don’t really matter. Isaacson was telling a multi-faceted story and managed to weave in the stuff that really mattered. As far as I can tell, he imparts the exact impression that he intends to.
A useful metric of a book’s efficacy is whether or not it’s convincing. This book was. Whether or not the portrayal of Jobs was 100% accurate isn’t all that important. Isaacson sold it. It worked.
Night by Elie Wiesel
English teachers like to throw this in to the curriculum to give kids a little holocaust exposure. It’s the trump card of bleakness. Great AP test essay material. Here’s my attempt:
The Times quote on the cover describes Wiesel’s story as “A slim volume of terrifying power.” But what gives the book its power is his vivid portrayal of powerlessness.
But seriously. By the end you’ll be a little pissed off. Looking for justice? Look no further than the Wikipedia article on Adolf Eichmann.
Shortly after the execution, Eichmann’s body was cremated in a specially designed furnace, and a stretcher on tracks was used to place the body into it. The next morning, June 1, his ashes were scattered at sea over the Mediterranean, beyond the territorial waters of Israel by an Israeli Navy patrol boat. This was to ensure that there could be no future memorial and that no country would serve as his final resting place.[58]
Look at that cheesy-ass cover. Under normal circumstances I would not have read an ostensibly dopey self-help book. But, a few people I trust all confirm that the method delivers on its promises. Besides, an effective placebo is still effective so I didn’t have a ton to lose.
I’ve been using Omnifocus for about 3 weeks now. It’s a nice organization system, and it works to a point. Capturing tasks certainly feels good. I know exactly what my obligations are and what I need to do to advance my progress. But these lists don’t solve the problem of being the only person working on a substantial software project. Knowing exactly what you need to do next doesn’t offer much solace when the task list is 50 items long and there’s nobody to delegate to.
That said, I enjoy the method because it has helped me manage the smaller projects I’m involved in. That’s mostly because it’s a formalization of common sense: Be organized. Make lists. Figure out what you need to physically do next. Keep your calendar clean.
At some points it reads like an advertisement for Allen’s consulting business, but he’s gotta eat so I won’t hold that against him. It’s a little bit difficult to give a full evaluation of the method at this point, so expect another report in a few months. All I can say now is that it isn’t hurting anything.
It’s a popular mantra: We’re becoming slaves to technology. Many people, most above 30, say that instead of living in the real world, we’re glued to our devices. But this notion is a knee-jerk reaction that ignores a powerful trend.
Since humanoids showed up, they have developed technology to makes their lives easier. From the rock-wielding apes in A Space Odyssey to the passengers on steam-powered locomotives, technology has consistently given us the ability to make unpleasant tasks more efficiently. The Internet and personal computing comprise a microtrend rolled up inside this larger epochal arc. We are not “slaves to technology”. We are living in an age where technology frees us from unnecessary errands faster than ever before. To deny this progress is counterproductive.
In the late 90s search engines started to really get good for general research. I can’t remember the last time I opened a physical encyclopedia. It’s not surprising considering the amount of work that would entail. I can’t fathom driving to a library, finding the reference section, looking for the correct volume, and thumbing through pages for a specific topic. It would be completely nonsensical to do this instead of typing the word into my browser’s URL bar.
Fastforward to the early 2000s when the pace really starts to pick up. Napster obliterated the record store. The last piece of physical music media I paid for was Smashmouth’s Astrolounge on casette tape. Today all ~30,000 of my songs live on Amazon’s cloud. I really can’t believe people still have CDs littering their homes.
5 years ago Netflix started to put a dent in movies and television. Watching a good TV series commercial free years after its debut is a wonderful experience. Streaming a film within 45 seconds of learning it existed is even better. When cast in this light, there were only three good things about Blockbuster:
You ran into people you knew in your neighborhood.
For a while you got to play Pokemon Snap on that N64 they had in there.
I can’t remember the third thing, but I will put it here if I do.
Anyway, recognizing this unstoppable march of information technology isn’t rocket surgery. Friedman would likely consider the aforementioned examples as legitimate “flattening”. You can’t stop it, so don’t stress out about letting go of old inefficiencies.
I’m free from going to the post office, free from going to the library, free from going to the record store, and free from going to blockbuster. I have a finite amount of time on this earth and I don’t intend to spend it en route.
Whovians may find the Cybermen to be a better metaphor.
You will become upgraded!
Why is this rant in a post of book reviews?
I came home on New Year’s Day to find a Kindle Touch in my mailbox. I then downloaded one thousand books for it. Check bookstores and libraries off the to-flatten list. They’re toast.
Here’s what I actually read.
Decoded by Shawn “Jay-Z” Carter
This book did two worthwhile things: 1) Makes an effective case for rap as a legitimate art form. 2) Establishes Jay-Z as a cohesive personality. To be honest I didn’t need a ton of convincing on either of these points. I was a fan long before this book was written. But I would recommend it to people on the fence about rap.
A People’s History of the United States by Howard Zinn
If I’m trying to read as many books as possible, this was a poor choice. At 900 pages, it took me a while. Regardless of politics, this book is important. Contemporary histories do emphasize hero figures rather than the experiences of the common folk. Summary: things have been pretty terrible for most people for a long time. Zinn explores a weird utopian fantasy at the end, but it’s still a great read. And as a bonus, check out this 2005 Daily Show interview with Zinn. I think Jon likes him, but keeps his distance because of the whole communism thing.
The Zombie Survival Guide: Complete Protection From the Living Dead
This book shows up on Amazon as “also purchased” by people who bought Robopocalypse and Ready Player One. Unfortunately, this one didn’t do it for me. The satirically preachy style just wasn’t as funny as critics make it out to be. It’s worth noting that I only read the first 2/3. I didn’t see it getting any better.
The Innovator’s Cookbook edited by Steven Johnson
This is an enjoyable collection of essays. A few feel like much business literature in that they express very obvious ideas with invented jargon. But on the whole I like the book because it generally confirms what I already believe: creative people and creative spaces are absolutely vital to developing vibrant cities and successful companies. Save yourself the time and read the best article on the rise of the creative class.
Meet Winston. He’s traveled closer to outer space than you ever will.
On December 31st I launched a near space weather balloon with the help of some friends. We emulated methodologies used in similar projects that successfully launched balloons to 80,000 ft. and recovered them. We captured some decent video.
Design
Hardware Comrade Perry handled nearly every element of physical design. While we talked about the project for a semester, everything came together in the 48 hours preceding the launch. We originally envisioned a styrofoam cooler with a lexan pane in it for the camera to see through. We discarded this idea for a more compact package. The final design looked like this:
It’s a styofoam box tied to a parachute and a balloon.
Inside we have
Motorola Defy running our tracking app on CyanogenMod 7
A Sony Cybershot DSC-W55 digital camera
Re-usable anaerobic heating pad from CVS
Software There were two software components: An Android app for the smartphone and a node.js web app for storing and displaying coordinates sent from the phone. Originally we planned to use the phone for everything– pictures, video, and tracking. But, while building the app, I learned that the Android API is ill-suited for our use case. Specifically, taking pictures and recording video from a service rather than an activity is pretty difficult. The workaround? Just throw a digital camera in there. Perry took this in stride; I added this requirement the day of the launch.
The web app was far more fun to build. The node.js app just waits for post requests containing JSON-encoded coordinates. When it receives one, the server fires a socket.io event in the browser. The client-side JavaScript then plots the position on a Google map and connects it to the previous position with a red line.
On GitHub we have the source for the web app and the source for Android client. A lot of it was written the day of the launch, but it worked. As the balloon floated over Round Rock, we saw its path drawn in real time.
Bill of Materials
Motorola Defy (borrowed from @kmobs)
Sony Cybershot (clandestinely borrowed)
Foam Insulation (borrowed)
Metallic tape (borrowed)
Big model rocket parachute ($30)
Helium cylinder ($70)
Latex weather balloon ($26)
Android Mascot ($3)
Prepaid sim card for phone ($10)
Wrench ($5)
Total project cost: $144
Launch
The day of the launch was hectic. We hadn’t quite finished assembling everything into the box. After fitting everything in there and charging our devices, we headed out to get the helium. After picking up the cylinders, we realized we needed a wrench to open the valve on the tank, so we picked one up at HEB. On the drive over we verified that the smartphone got connectivity through the box. It turned out that the styrofoam insulation was coated in metallic foil. It was effectively a Faraday cage. If you look at the map above you can actually see when we placed the phone inside the box while en route to the park. Before launch, I peeled off half of the strips, then walked around a soccer field to make sure the server was getting a signal.
Once everything was good to go, we filled the balloon. about 5 people stood in circle around it for 15 minutes, trying to prevent the wind from whipping it into the ground and releasing about $70 of helium. It was difficult to tell when the balloon was full. It was supposed to have an 8 ft diameter, so we ballparked it using Perry’s arms.
Finally, we let it go:
Seeing the balloon float away so quickly mesmerized everyone for a bit. I was brought back to earth when someone asked if we were getting a signal. I looked at the web app on my phone. We were!
We drove back to Perry’s house to play the waiting game. The balloon took a meandering path.
The flight started in Old Setter’s Park, denoted by that green patch. (Points before that are from the drive there.) The last contact we had was east of Pflugerville, about 9 miles from where we launched it. This was expected behavior. At some point the balloon was supposed to rise above the range of the cell towers and go into a communication blackout. Unfortunately, the blackout never ended. We drove back to Perry’s house and waited, and waited, and waited.
Despite a delicious four course meal whipped up by Perry’s mom, I was a little upset at this point. Everyone was looking forward to recovering the balloon, but all we could do was speculate on why we couldn’t get a signal. We wrote off the project as a fun learning experience. But three days later I woke up to the best voicemail I’ve ever received.
Nick, I’m calling from Giddings, Texas. A little small town between Austin and Houston. I was feeding my cows this morning and found a box with a parachute attached to it…
Even if we could control where the balloon had landed, would couldn’t have chosen a better destination than the land of Mr. Evan Gonzales. He picked up our payload (still in once piece) and drove into town to meet us. But the hospitality didn’t end there. He showed us the way to a fantastic local eatery, the City Meat Market.
Post Mortem
While we thought the balloon was lost forever, we considered these potential causes of failure. Battery Trouble
The phone battery charged far slower than I would expect. Originally I thought there was a problem with the USB cable when the device kept connecting/disconnecting rapidly while I was debugging. But the same issue seemed to surface when charging the phone with another cable. The USB connector may have been damaged, preventing an adequate charge. Landed in dead zone
This is the T-Mobile prepaid data coverage map for the area the balloon likely would have landed in. Those beige areas have no data whatsoever. The balloon could have landed in a field, futilely trying to phone home until its battery died.
We now know that this is the most likely cause of failure. When I returned Evan’s call, it was dropped once. The coverage out there is pretty spotty.
Insufficient Lift
It’s possible that we underfilled the balloon and it stopped rising at some altitude. It may have risen out of cellular range and then stuck there, drifting for far longer than the expected two hours across a vast distance.
We now know that this is probably not the case. The video shows that it got pretty high up there, although definitely not to the 80,000 ft. seen in similar projects. If the digital camera had a bigger memory card we probably would have gotten video at a higher altitude that others have seen.
Future Considerations
If we were going to do this again in a month, I would do the following:
Put a bigger memory card in the camera. A 2 gigabyte memory stick was only enough for an hour and a half of low quality video. We could have gotten our hands on something better, but we ran out of time. This limitation kept us from recording the apex and the descent, two of the coolest parts.
Log data locally on the phone instead of sending it to the server and discarding it. This is an obvious design oversight that left us with a huge hole in our data. The mobile application should function like a plane’s black box.
Send coordinates to the server using SMS as well as Internet. Even if the phone can’t get data, it still may be able to send text messages. The Twilio API would make this integration with the web app seamless.
Use a more durable camera. This GoPro would be ideal.
Use a redundant, more durable GPS device. This hiking locator was used by some UCSD folks.
Use a foam box with no metallic foil. This likely affected the phone’s ability to transmit data.
I began 2011 with a bold new year’s resolution: Consume more books than movies. I failed miserably. Movies are just too easy to watch. But I still found time to read some great (and some not so great) books:
The World is Flat by Thomas Friedman
I put off reading this for a long time. Friedman sheds some light on globalization for us laymen, but the value comes from the specific examples he gives about real companies. It’s full of “that’s interesting” moments.
Breakfast of Champions by Kurt Vonnegut
“The story isn’t as important as the writing.”
–Jodi Z, right now
My girlfriend made this remark when I told her I couldn’t remember why I enjoyed Breakfast so much. The joy of Vonnegut comes from the short passages that happen to be particularly poignant.
The Big Switch by Nicholas Carr
I recommend this book to any layperson who mentions “the cloud”. Carr walks the reader through the history of electricity as an analog to computing. It’s a compelling read, but it wasn’t enough to get me to take a job at a certain San Francisco PaaS company.
The Book of Sand and Shakespeare’s memory by Jorge Luis Borges
Lots of fun little inception tales in here. Borges must have been a writer for Arrested Development because I could read this thing again and it would still be just as good. Fun fact: In the edition of the book I read, Borges gives a shoutout to UT’s Perry-Castañeda Library where I checked out the book itself.
Hitchhiker’s Guide to the Galaxy by Douglas Adams
You’ve already read this so I won’t discuss it. I will tell you that one of my friends used to say “That’s so 42, dude.” in high school. Bullies were invented to sniff out phrases like that.
A Scanner Darkly by Philip K. Dick
I wish I had never seen the film, because I kept picturing Keanu Reaves’ rotoscoped face. The narrative was great, but the druggie lingo started to wear on me by the end.
Little Brother by Cory Doctorow
This was a ton of fun. I immediately gave it to my little brother as a sneaky way to teach him about public key cryptography, RFID, and sticking it to the man.
Machine of Death by various authors
I’m not sure if “concept book” is a real term. If not, consider it coined. The concept of this anthology is that there is a machine that gives you a short vague description of how you will die. About 20 different authors give their takes on the idea in the form of short stories. It’s a good read because many of the authors are pretty clever.
The Restaurant at the End of the Universe by Douglas Adams
The first original was funny, so I really had no choice about reading this one. The obnoxiously cheerful computer and the paranoid android are the number one reason to read these books.
Making the World Work Better by various authors
This book was given to every single IBM employee. It’s three accounts of IBM’s rise, near-fall, and resurrection. I love Watson’s original motto: “Think”. It’s so basic, such a far cry from the bullshit corporate mission statements of the last few decades. But the best part of the entire book is the paragraph where the author hand-wavingly explains that IBM is totally not liable for that Holocausty stuff and basically solved Apartheid. No big deal.
The Wave by Todd Strasser
The target demographic for this book is likely preteens, but I picked it up because the wikipedia article was intriguing. Poor decision. It’s an embellishment of a true story in which a high school teacher demonstrates how easy it is for a population to get swept up in Nazism.
White Noise by Don DeLillo
Probably the most “important” book I read all year. I’m reminded of it whenever I hear the reverberations of advertising ringing in my ears. “This program brought to you with limited commercial interruption by AT&T.”
Robopocalypse by Daniel H. Wilson
This book is exactly what it sounds like. The narrative is told through journal entries detailing many different subplots that eventually converge. I used to find frequent context switching kind of annoying, but now it’s somehow more appealing to my web-addicted brain. The survival section could have been a lot longer. I’m not really convinced that the humans could overcome their new robot overlords in 50 pages. It could have been drawn out and split into several volumes. Can’t wait for the movie.
Ready Player One by Ernest Cline
This was so easy to finish because it’s a thriller-for-nerds. Cline knows how to relate to his target demographic. At one point the protagonist is inserted into War Games and has to “play” as Matthew Broderick’s character. It’s good to know that there’s a substantial number of people who also aspired to be like him.
I’ll be reading a lot more in 2012, as per my new year’s resolution. A kindle showed up in the mailbox today so I picked up some books– 1000 of them.
