Cap 10k Race Data

What's the rush?

The Capitol 10,000 is a total friggin’ blast. Anyone with a remote interest in running should participate. Rarely do you get to run up the middle of the Congress bridge straight to the capitol through an unstoppable sea of humanity. On your way to the finish line you’ll pass spectators offering beer, bacon, and donuts.

Before you can catch your breath, they make the results available online. I’ve been looking for an opportunity to jump back into Python, so naturally I scraped the HTML and ran it through a script so people can play with the data.

Here’s how it went down.

Results are posted to mychiptime.com. They have some client-side JavaScript query their backend with your specified parameters. With a little help from Chrome’s developer console, I found their URL scheme. Use wget to download the data. It’s HTML that gets inserted directly into the page with a $(“#blah”).html(response).

wget http://www.mychiptime.com/searchResultGen.php?eID=3526&show=all"

The HTML returned by their API is truly hideous. A mere 20k rows cost 16MB… because they’re chock full of <b>, <font>, and “OnMouseOver”. After pulling out the relevant data, the file size is about 800KB.

Here’s the python script used to generate tab delimited .csv files from the HTML input. BeautifulSoup can take a few minutes to parse the larger files.

from bs4 import BeautifulSoup
#produce tab delimited CSV files from large html files
for year in range(2008, 2013):
	print 'processing ' + str(year)
	file = str(year) + '.html'
	f = open(file, 'r');

	html = f.read()
	soup = BeautifulSoup(html)
	table = soup.find("table")	

	#first row with column titles
	rows = table.findAll('tr')

	i = 0
	#write data to csv file
	outfile = str(year) + '.csv'
	out = open(outfile, 'w')
	this_row = ''
	for row in rows:
		cols = row.findAll('td')
		for col in cols:
			b = col.find('b')

			text = str(b.string)
			this_row += text
			this_row += '	'

		if i % 2 == 0:
			this_row += '\n'
			out.write(this_row)
			this_row = ''
		i += 1

	out.close();
	f.close()

Playing with the data

Building a simple scatterplot was way harder than it should have been. Google docs’ spreadsheet crashes when I try to build a chart. The Python library matplotlib chokes on the rows where Age is “None”. Wolfram Alpha rejects it, probably for the same reason. LibreOffice’s Spreadsheet just made my laptop really hot. Octave is inscrutable. But Plot for OSX did the job.

2012 Cap 10k

I’d like to see someone with MATLAB skills come up with some more advanced plots or divine some insight from the data. The columns available for 2012 are:

  • Name
  • Division Place
  • Gun Time
  • Chip Time
  • Overall Place
  • Age
  • Zip
  • Gen Place
  • Total Pace
  • Total Div
  • Total Gend
  • Tot AG

Here’s a zip archive of all the tab delimited CSV files 2008-2012.

Book Thoughts: February 2012

Steve Jobs by Walter Isaacson

I loved this book. The narrative happened to be about a tech company and I’m sure that helped draw me in, but at its core this is just a great story. There are a handful of criticisms aired in this HN thread focused around the technical accuracy of products and software lineage. Those things don’t really matter. Isaacson was telling a multi-faceted story and managed to weave in the stuff that really mattered. As far as I can tell, he imparts the exact impression that he intends to.

A useful metric of a book’s efficacy is whether or not it’s convincing. This book was. Whether or not the portrayal of Jobs was 100% accurate isn’t all that important. Isaacson sold it. It worked.

 

Night by Elie Wiesel

English teachers like to throw this in to the curriculum to give kids a little holocaust exposure. It’s the trump card of bleakness. Great AP test essay material. Here’s my attempt:

The Times quote on the cover describes Wiesel’s story as “A slim volume of terrifying power.” But what gives the book its power is his vivid portrayal of powerlessness.

But seriously. By the end you’ll be a little pissed off. Looking for justice? Look no further than the Wikipedia article on Adolf Eichmann.

Shortly after the execution, Eichmann’s body was cremated in a specially designed furnace, and a stretcher on tracks was used to place the body into it. The next morning, June 1, his ashes were scattered at sea over the Mediterranean, beyond the territorial waters of Israel by an Israeli Navy patrol boat. This was to ensure that there could be no future memorial and that no country would serve as his final resting place.[58]

This is likely the best instance of “People don’t forget!

 

 

Getting Things Done by David Allen

Look at that cheesy-ass cover. Under normal circumstances I would not have read an ostensibly dopey self-help book. But, a few people I trust all confirm that the method delivers on its promises. Besides, an effective placebo is still effective so I didn’t have a ton to lose.

I’ve been using Omnifocus for about 3 weeks now. It’s a nice organization system, and it works to a point. Capturing tasks certainly feels good. I know exactly what my obligations are and what I need to do to advance my progress. But these lists don’t solve the problem of being the only person working on a substantial software project. Knowing exactly what you need to do next doesn’t offer much solace when the task list is 50 items long and there’s nobody to delegate to.

That said, I enjoy the method because it has helped me manage the smaller projects I’m involved in. That’s mostly because it’s a formalization of common sense: Be organized. Make lists. Figure out what you need to physically do next. Keep your calendar clean.

At some points it reads like an advertisement for Allen’s consulting business, but he’s gotta eat so I won’t hold that against him. It’s a little bit difficult to give a full evaluation of the method at this point, so expect another report in a few months. All I can say now is that it isn’t hurting anything.

JavaScript Hoedown #3

JavaScript Design Patterns

  • Analogs to gang of four
  • Can do more than avoid anti-patterns

CreativeJS

  • Great aggregation of JS art

Sublime Package Control

  • A million great packages for ST2
  • jsformatting, jslint, prefixr, bracket highlighting, html snippets
  • ⌘⇧P to see the menu

jCanvas

  • Abstract away canvas API with easier functions.
  • See it in action at bromansion.com

Twitter Bootstrap

  • Key to faking it.

Backbone 0.9 Update

  • Shuffle syntax
  • .get() method totally exists.

Blogs

Book Thoughts: January 2012

It’s a popular mantra: We’re becoming slaves to technology. Many people, most above 30, say that instead of living in the real world, we’re glued to our devices. But this notion is a knee-jerk reaction that ignores a powerful trend.

Since humanoids showed up, they have developed technology to makes their lives easier. From the rock-wielding apes in A Space Odyssey to the passengers on steam-powered locomotives, technology has consistently given us the ability to make unpleasant tasks more efficiently. The Internet and personal computing comprise a microtrend rolled up inside this larger epochal arc. We are not “slaves to technology”. We are living in an age where technology frees us from unnecessary errands faster than ever before. To deny this progress is counterproductive.

In the late 90s search engines started to really get good for general research. I can’t remember the last time I opened a physical encyclopedia. It’s not surprising considering the amount of work that would entail. I can’t fathom driving to a library, finding the reference section, looking for the correct volume, and thumbing through pages for a specific topic. It would be completely nonsensical to do this instead of typing the word into my browser’s URL bar.

Fastforward to the early 2000s when the pace really starts to pick up. Napster obliterated the record store. The last piece of physical music media I paid for was Smashmouth’s Astrolounge on casette tape. Today all ~30,000 of my songs live on Amazon’s cloud. I really can’t believe people still have CDs littering their homes.

5 years ago Netflix started to put a dent in movies and television. Watching a good TV series commercial free years after its debut is a wonderful experience. Streaming a film within 45 seconds of learning it existed is even better. When cast in this light, there were only three good things about Blockbuster:

  1. You ran into people you knew in your neighborhood.
  2. For a while you got to play Pokemon Snap on that N64 they had in there.
  3. I can’t remember the third thing, but I will put it here if I do.

Anyway, recognizing this unstoppable march of information technology isn’t rocket surgery. Friedman would likely consider the aforementioned examples as legitimate “flattening”. You can’t stop it, so don’t stress out about letting go of old inefficiencies.

I’m free from going to the post office, free from going to the library, free from going to the record store, and free from going to blockbuster. I have a finite amount of time on this earth and I don’t intend to spend it en route.

Whovians may find the Cybermen to be a better metaphor.

You will become upgraded!

Why is this rant in a post of book reviews?

I came home on New Year’s Day to find a Kindle Touch in my mailbox. I then downloaded one thousand books for it.  Check bookstores and libraries off the to-flatten list. They’re toast.

Here’s what I actually read.

Decoded by Shawn “Jay-Z” Carter

This book did two worthwhile things: 1) Makes an effective case for rap as a legitimate art form. 2) Establishes Jay-Z as a cohesive personality. To be honest I didn’t need a ton of convincing on either of these points. I was a fan long before this book was written. But I would recommend it to people on the fence about rap.

A People’s History of the United States by Howard Zinn

If I’m trying to read as many books as possible, this was a poor choice. At 900 pages, it took me a while. Regardless of politics, this book is important. Contemporary histories do emphasize hero figures rather than the experiences of the common folk. Summary: things have been pretty terrible for most people for a long time. Zinn explores a weird utopian fantasy at the end, but it’s still a great read. And as a bonus, check out this 2005 Daily Show interview with Zinn. I think Jon likes him, but keeps his distance because of the whole communism thing.

The Zombie Survival Guide: Complete Protection From the Living Dead 

This book shows up on Amazon as “also purchased” by people who bought Robopocalypse and Ready Player One. Unfortunately, this one didn’t do it for me. The satirically preachy style just wasn’t as funny as critics make it out to be. It’s worth noting that I only read the first 2/3. I didn’t see it getting any better.

The Innovator’s Cookbook edited by Steven Johnson

This is an enjoyable collection of essays. A few feel like much business literature in that they express very obvious ideas with invented jargon. But on the whole I like the book because it generally confirms what I already believe: creative people and creative spaces are absolutely vital to developing vibrant cities and successful companies. Save yourself the time and read the best article on the rise of the creative class.

The Great Near Space Balloon Misadventure

Meet Winston. He’s traveled closer to outer space than you ever will.

On December 31st I launched a near space weather balloon with the help of some friends. We emulated methodologies used in similar projects that successfully launched balloons to 80,000 ft. and recovered them. We captured some decent video.

Design

Hardware
Comrade Perry handled nearly every element of physical design. While we talked about the project for a semester, everything came together in the 48 hours preceding the launch. We originally envisioned a styrofoam cooler with a lexan pane in it for the camera to see through. We discarded this idea for a more compact package.  The final design looked like this:

It’s a styofoam box tied to a parachute and a balloon.Near Space Balloon Picture

Inside we have

  • Motorola Defy running our tracking app on CyanogenMod 7
  • A Sony Cybershot DSC-W55 digital camera
  • Re-usable anaerobic heating pad from CVS

Software
There were two software components: An Android app for the smartphone and a node.js web app for storing and displaying coordinates sent from the phone. Originally we planned to use the phone for everything– pictures, video, and tracking. But, while building the app, I learned that the Android API is ill-suited for our use case. Specifically, taking pictures and recording video from a service rather than an activity is pretty difficult. The workaround? Just throw a digital camera in there. Perry took this in stride; I added this requirement the day of the launch.

The web app was far more fun to build. The node.js app just waits for post requests containing JSON-encoded coordinates. When it receives one, the server fires a socket.io event in the browser. The client-side JavaScript then plots the position on a Google map and connects it to the previous position with a red line.

On GitHub we have the source for the web app and the source for Android client. A lot of it was written the day of the launch, but it worked. As the balloon floated over Round Rock, we saw its path drawn in real time.

Bill of Materials

  • Motorola Defy (borrowed from @kmobs)
  • Sony Cybershot (clandestinely borrowed)
  • Foam Insulation (borrowed)
  • Metallic tape (borrowed)
  • Big model rocket parachute ($30)
  • Helium cylinder ($70)
  • Latex weather balloon ($26)
  • Android Mascot ($3)
  • Prepaid sim card for phone ($10)
  • Wrench ($5)
  • Total project cost: $144

Launch

The day of the launch was hectic. We hadn’t quite finished assembling everything into the box. After fitting everything in there and charging our devices, we headed out to get the helium. After picking up the cylinders, we realized we needed a wrench to open the valve on the tank, so we picked one up at HEB. On the drive over we verified that the smartphone got connectivity through the box. It turned out that the styrofoam insulation was coated in metallic foil. It was effectively a Faraday cage. If you look at the map above you can actually see when we placed the phone inside the box while en route to the park. Before launch, I peeled off half of the strips, then walked around a soccer field to make sure the server was getting a signal.

Once everything was good to go, we filled the balloon. about 5 people stood in circle around it for 15 minutes, trying to prevent the wind from whipping it into the ground and releasing about $70 of helium. It was difficult to tell when the balloon was full. It was supposed to have an 8 ft diameter, so we ballparked it using Perry’s arms.

Finally, we let it go:

Seeing the balloon float away so quickly mesmerized everyone for a bit. I was brought back to earth when someone asked if we were getting a signal. I looked at the web app on my phone. We were!

We drove back to Perry’s house to play the waiting game. The balloon took a meandering path.

The flight started in Old Setter’s Park, denoted by that green patch. (Points before that are from the drive there.) The last contact we had was east of Pflugerville, about 9 miles from where we launched it. This was expected behavior. At some point the balloon was supposed to rise above the range of the cell towers and go into a communication blackout. Unfortunately, the blackout never ended. We drove back to Perry’s house and waited, and waited, and waited.

Despite a delicious four course meal whipped up by Perry’s mom, I was a little upset at this point. Everyone was looking forward to recovering the balloon, but all we could do was speculate on why we couldn’t get a signal. We wrote off the project as a fun learning experience. But three days later I woke up to the best voicemail I’ve ever received.

Nick, I’m calling from Giddings, Texas. A little small town between Austin and Houston. I was feeding my cows this morning and found a box with a parachute attached to it…

Even if we could control where the balloon had landed, would couldn’t have chosen a better destination than the land of Mr. Evan Gonzales. He picked up our payload (still in once piece) and drove into town to meet us. But the hospitality didn’t end there. He showed us the way to a fantastic local eatery, the City Meat Market.

Brisket and a side for $7!

Post Mortem

While we thought the balloon was lost forever, we considered these potential causes of failure.
Battery Trouble
The phone battery charged far slower than I would expect. Originally I thought there was a problem with the USB cable when the device kept connecting/disconnecting rapidly while I was debugging. But the same issue seemed to surface when charging the phone with another cable. The USB connector may have been damaged, preventing an adequate charge.
Landed in dead zone
This is the T-Mobile prepaid data coverage map for the area the balloon likely would have landed in. Those beige areas have no data whatsoever. The balloon could have landed in a field, futilely trying to phone home until its battery died.

We now know that this is the most likely cause of failure. When I returned Evan’s call, it was dropped once. The coverage out there is pretty spotty.

Insufficient Lift
It’s possible that we underfilled the balloon and it stopped rising at some altitude. It may have risen out of cellular range and then stuck there, drifting for far longer than the expected two hours across a vast distance.

We now know that this is probably not the case. The video shows that it got pretty high up there, although definitely not to the 80,000 ft. seen in similar projects. If the digital camera had a bigger memory card we probably would have gotten video at a higher altitude that others have seen.

Future Considerations

If we were going to do this again in a month, I would do the following:

  • Put a bigger memory card in the camera. A 2 gigabyte memory stick was only enough for an hour and a half of low quality video. We could have gotten our hands on something better, but we ran out of time. This limitation kept us from recording the apex and the descent, two of the coolest parts.
  • Log data locally on the phone instead of sending it to the server and discarding it. This is an obvious design oversight that left us with a huge hole in our data. The mobile application should function like a plane’s black box.
  • Send coordinates to the server using SMS as well as Internet. Even if the phone can’t get data, it still may be able to send text messages. The Twilio API would make this integration with the web app seamless.
  • Use a more durable camera. This GoPro would be ideal.
  • Use a redundant, more durable GPS device. This hiking locator was used by some UCSD folks.
  • Use a foam box with no metallic foil. This likely affected the phone’s ability to transmit data.

Related Reading

Thanks

The best part of this project was that good people came out of the woodwork to help me play with a giant balloon.

  • James Cameron- Cooler and Lexan
  • Perry Helmer- Parachute, Physical Design
  • Megan Haske- Launch assistance
  • Jodi Zik- Winston Android Doll
  • Keyan Mobli- Motorola Defy
  • Evan Gonzales- Recovery