We need to archive feeds

As of 2015, Facebook has overtaken Google as the main source of traffic to news sites[1]. The average Facebook user spends 40 minutes a day on the site[2]. If we fail to archive news feeds we will lose important sociological information.

The Internet Archive does a great job of archiving the web, among other things like VHS tapes. They’re located here in The Richmond and you can go visit them on Fridays during lunch[3]. I’m concerned that while they are doing a great job of getting the data, they’re missing really crucial metadata, i.e. feed presentation.

The Wayback Machine is necessary but not sufficient for a full representation of the state of the web as it is experienced by real people. If you really want to understand the zeitgeist of today, January 17, 2017, you wouldn’t want to only see the front page of the NYT. You’d want to experience the web mediated through individuals’ news feeds. I imagine a good number of researchers would like to sample feeds in the years leading up to the 2016 presidential election.

I see there is a chrome extension that can be used to archive a feed, but I haven’t seen anyone scraping feeds at scale over time. Imagine a sort of opt-in system where volunteers give archivists read-access to their social media accounts. With a few hundred opt-ins you could get a decent sample of all the different cultural/subcultural bubbles.

The obvious candidate to solve this problem is Facebook itself. They might already have a feed time machine internally! Contributing a few hundred feeds (and appropriate developer time) to the Internet Archive would be a great move and a PR win.

[1] http://www.adweek.com/socialtimes/facebook-is-now-the-top-referral-source-for-digital-publishers/625300

[2] http://time.com/3950525/facebook-news-feed-algorithm/

[3] If you live in San Francisco you should go do this. You will have fun and meet some interesting people.

Shot on a Nexus 5X

Google has advertised the Nexus 5X around San Francisco, claiming a high quality camera. Here’s what the photos look like in practice.

Conclusions

  • Good for close-ups of people and dogs.
  • Good for landmarks in bright light.
  • Okay for landmarks in low light.
  • Bad for far away objects.
  • Bad in low light for candid shots of people. They move too much.
  • Okay for landscapes; best on clear days.

The Joker vs. Quantitative Easing

In The Dark Knight (2008), the Joker destroys a large pile of cash by setting it on fire. This demonstrates the Joker’s commitment to nihilism but raises a much more interesting question: What is the deflationary effect on Gotham’s economy?

To answer that we need to know if the money was actually going to be spent.

It absolutely was.

Before the events of the film, newly elected District Attorney, Harvey Dent, has spearheaded an anti-money laundering campaign but has failed to halt the business of the mob which has access to a few crooked banks under the control of Chinese financier, Lau.

Batman helps his cop friend Gordon identify Lau’s banks via drug purchases with marked bills. Lau’s banks are performing two types of services. The first is exciting: money laundering. The illicit local deposits in Gotham are “layered” by being sent to Lau’s company in Hong Kong and then “integrated” by being paid back to Gotham gangsters. The mob spends this money on the ostensibly legitimate parts of their business like real estate, capital goods (trucks, cement), and payroll which is spent on consumer goods (cannoli).

The other type of service the banks perform is boring: commercial banking. The mob’s deposits are on the books as cash reserves and count toward reserve requirements held against business loans, car loans, and mortgages for the general public. Easy credit helps businesses grow and is great for consumer spending.

Lau’s corrupt cops tip him off to Gordon’s raid and he explains to his ethnically diverse cadre of gangsters that he has moved the deposits to a secure location that is “not a bank”. This is bad for the general public. If the mob is unbanked, there’s slightly less credit available for the people of Gotham.

Remember that the film is set in 2008 which happens to be the beginning of the financial crisis. Gotham needs all the cash it can get.

Later on, the Joker kidnaps Lau and forces him to give up the non-bank location of the money. We see Lau tied up in a chair atop a mountain of cash. Assuming a billion dollars in hundreds is about two pallets, the total value of cash mountain is on the order of 10 billion dollars. Note that Batman had already forced Lau to withdraw the money from the banks so he shares some culpability for the deflation, at least in the short term [1].

Is this a meaningful reduction of the money supply?

WSJ says:

From December 2008 to March 2010, the Fed bought $1.7 trillion of Treasurys and mortgage-backed securities [2]

You can sort of see that here in the dark blue line. Notice the vertical axis is in trillions of dollars [3].

quantitativeeasing

So $1.7T over 15 months is $113B dollars a month. The Joker burned less than a tenth of that in the warehouse which isn’t terrible.

There’s also fallout from dismantling the mob that’s hard to quantify. A nontrivial number of henchmen, goons, and thugs will be out of the street if Batman gets his way. These people will have a hard time transitioning to the legitimate economy which systematically practices de jure discrimination against anyone with a criminal record.

But at the same time, the removing the presence of the mob in Gotham lowers costs of doing business for everyone else. If the Taco Bell franchise has to pay protection, it can’t hire a guy to keep the drive-through open on weekends. All told, the mob is probably a net negative for the local economy.

Okay, so burning a few billion dollars isn’t enough to cause serious monetary damage. In fact, it pales in comparison to the non-monetary effects of the Joker’s campaign. Terrorism hurts growth in 2 ways, says the IMF [4].

  1. Direct damage is physical and logistical. A) Businesses are literally exploded and can’t operate. Remember the hospital destroyed by the Joker. B) Decreased productivity lowers output (the National Guard shuts down all bridges and tunnels)
  2. Indirect damage is psychological. Decreased consumer confidence lowers consumption. The assassination of the police commissioner isn’t getting anyone into Gotham department stores.

In summary, Batman’s decision to take out the Joker has a sound economic rationale. He’s firmly on the side of growth. See if you can spot the omission in Gordon’s closing remarks.

he’s a silent guardian, a watchful
protector… a dark knight.

What he didn’t say is that he’s a protector of markets. Which isn’t surprising given Bruce Wayne’s nominal day job.

[1] Had the total amount been seized by the police under asset forfeiture laws, the cops would get to keep up to 60% for new toys while the remainder goes to the New York state treasury. https://www.ij.org/asset-forfeiture-report-new-york

[2] http://blogs.wsj.com/economics/2010/11/03/qa-on-qe2-what-a-fed-move-would-mean/

[3] http://www.economist.com/blogs/graphicdetail/2014/10/daily-chart-21

[4] http://www.imf.org/external/pubs/ft/wp/2005/wp0560.pdf

Additional reading

Aaron Swartz highlights political and philosophical dilemmas appearing in the film.
http://www.aaronsw.com/weblog/tdk

Screenplay
http://www.pages.drexel.edu/~ina22/splaylib/Screenplay-Dark_Knight.HTM

An argument for Lyft

There’s a lot of rhetoric swirling around the Lyft debate concerning an evil taxi lobby and macro concerns on Austin’s congestion. I want to dissect the larger discussion apply a sober focus to the most emotionally charged aspect: Safety.

The anti-ridesharing crowd employs a false syllogism.

  • Taxi licenses increase public safety.
  • Lyft drivers are unlicensed.
  • Legalizing Lyft would decrease public safety.

There’s an implicit assumption in this argument: That licensed taxis are the only form of private transportation on the road now. This is utterly false. Anecdotally, about 20% of the cabs I take are illegal and supposedly dangerous.

The dance is a familiar one if you’ve ever spent time downtown. It’s 2:30 am. You’re on San Jacinto or Congress, looking for that illuminated Yellow Cab dome. After 10 minutes with no luck, Joey McRando pulls up in a black SUV and offers a ride. You take seven seconds to size him up and decide that you could choke him to death if it really hits the fan. There’s some awkward bartering and you’re on your way.

Ridesharing apps alleviate the sketchiness of this fairly common situation. Even if their driver requirements are less stringent than taxi licenses, the Amazon-style rating system with stars and reviews is a far better guarantor of safety than me ascertaining the fighting ability of a random person.

Policy decisions should be underpinned by sober assessments of conditions on the ground. There’s a clear pragmatic way to keep me from hopping into random cars in the middle of the night.

Legalize Lyft.

 

Update 7/7/2014:
Two or three weeks after this post was published, Lyft went from a flaky don’t-hold-your-breath operation to a reliable way to get around Austin. It’s still not legal, but the Lyft Death Star is fully operational. Yeah, buddy.

Update 1/17/2017:
Both Uber and Lyft left Austin in May 2016 after a ballot measure overturning a driver fingerprinting requirement failed. There are several replacement apps but they cost more and none of them seem to work as well. I have been living in San Francisco for years now. This one is on y’all.

Major: Computer Engineering

I recently received a degree in Computer Engineering from the University of Texas. It was a long time coming.

Here’s a rundown of every course I took at UT. This may be helpful if you’re considering attending UT or trying to pick your major.

E E 306 INTRODUCTION TO COMPUTING
302 and 306 are the very first courses designed to get you interested in either electrical or computer engineering. This course confirmed what I already knew: I like computers. The curriculum employs a bottom-up approach designed to take away the magic of computing. If you’re lucky, you’ll get the venerable Yale Patt.

E E 302 INTRO ELECTRICAL ENGINEERING
306’s dual course is a light introduction to circuits. You’ll learn KVL loops and Thevenin equivalences. In a perfect world this would be the only required EE course for CE majors.

M 408D SEQ, SERIES, AND MULTIVAR CALC
Multivariable calculus is a solid math course. I’m generally pretty bad at mathematics, but I appreciated this one. It picks up where AP BC calculus left off.

C C 304C INTRO TO THE NEW TESTAMENT
I’m thankful my family never forced me to go to church, but I did miss out on some culturally important narratives and biblical themes. You’d be surprised at how many little turns of phrase come from Jesus quotes. The course was worthwhile.

M 427K ADV CALCULUS FOR APPLICATNS I
So differential equations are the foundation for several types of engineering. I appreciate the idea, but I had a hell of a time grokking them. The math department has some terrible teachers. Use MyEdu before taking a lower division math course.

PHY 103N LABORATORY FOR PHY 303L
This lab course is attached to PHY 303L. It’s an exercise in tedium and divining meaning from unclear an unclear instruction manual. The lab TA was a dick until the last day when he handed out the course evaluations. Nobody was fooled.

E E 312 INTRODUCTION TO PROGRAMMING
This course gets you started with C and a little tiny bit of C++ in the last week or so. There’s a good amount of pointer tricks and doing clever tricks with the stack. It’s stuff you wouldn’t normally do, but it’s good for demonstrating an understanding of the mechanics of activation records. Remember that scene from The Matrix Reloaded where Smith creates a bunch of clones of himself? The professor played that clip to explain recursion. “Neo is the base case,” he said.

PHY 303L ENGINEERING PHYSICS II
The single biggest mistake I made in my academic career was not testing out of this course with the Electricity and Magnetism AP test. The physics professors are notoriously bad teachers. If you take it upon yourself to learn physics on your own in high school, you will likely be better off.

E E 411 CIRCUIT THEORY
The professor for this course flew in from California every Monday and Wednesday to teach it. Most lectures were accompanied by a story about Bob Widlar‘s drunken antics. This course is probably unnecessary for Software Engineers.

E E 316 DIGITAL LOGIC DESIGN
Logic Design is an important course. It’s a light introduction to things like muxes, adders, and VHDL. I would recommend it for software engineers because it forces you to think about playing with bits in clever ways.

E E 322C DATA STRUCTURES
This course is tied for first place with Algorithms. Students call it “Java”, but it’s much more. You’ll implement linked lists, hash tables, and all kinds of trees.

M 325K DISCRETE MATHEMATICS
This is the one math course I actually enjoyed. The teacher was a lecturer, not a professor of any sort. But she was possibly the best teacher I had at UT. The course is full of proofs, but they’re totally manageable. For any question they throw at you, you’ll only have a few tools in your toolbox so you won’t get stuck for long.

E E 319K INTRO TO EMBEDDED SYSTEMS
This course is definitely the most fun. You basically hack away on microcontrollers. The lab section is a series of miniprojects that you show off to a lab TA. When I took it they were using the Freescale 9s12DP512, but they’ve since switched to an ARM architecture of some sort.

E E 333T ENGINEERING COMMUNICATION-W
This course involves several papers and a powerpoint presentation. You’ll learn that many people are terrible writers and even worse speakers. This course is for them. It should be an easy A, but the groups are paired randomly, so you’ll have to have some team cohesion. And that’s probably part of the lesson.

E E 313 LINEAR SYSTEMS AND SIGNALS
You could call this course “Applied differential equations”. Not my cup of tea. But if you like analog stuff this may be for you. Dr. Hall owns an audio company and demos the tech at some point.

E E 438 ELECTRONIC CIRCUITS I
This is the only course I didn’t finish. Halfway through the semester the ECE department released a new course catalog with a more specialized track for Computer Engineers. I used a Q-drop (penalty-free mechanism for quitting), but not before completing most of the circuit design labs.

E E 360C ALGORITHMS
This is the big papa of software courses. Learn them, love them.

E E 351K PROBABILITY & RANDOM PROCESSES
This is known as one of the more a difficult courses in the curriculum, but it definitely should be in the curriculum. Part of the issue for my class was that the professor (Vikalo) was teaching it for the first time and he was a master of the subject. The best math professors can teach it to you like you’re five. That said, I’ll probably end up using some of the Bayesian stuff eventually.

M 340L MATRICES AND MATRIX CALCULATNS
Dr. Arlo Schurle is likely the best math professor at UT and he makes Linear Algebra a cakewalk.

ARH 339M AMERICAN ART SINCE 1960
This is a good course if you like art or think you might like art, but I wouldn’t want to make it my major. Be prepared to memorize the year, artist, and media for about 250-300 works of art.

UGS 302 CIV SOCTY IN POSTCOLONL STATES
This one was a gem. The course examined the influence of Tocquevillian associations in the Middle East and North Africa. The best part: I took it the semester before the Arab Spring. Nobody saw it coming.

E E 445L MICROPROCESSOR APPLICS AND ORG
This top-tier course picks up where 319k left off with more advanced mini-projects on the 9s12. For the final project, you design a PCB and get it fabricated. My project was a handheld pong console. I’ll never forget debugging the system with a multimeter and finding a dead wire in the middle of that fucking ribbon cable. These two microcontroller courses are almost completely architected by Valvano.

E E 364D INTRO TO ENGINEERING DESIGN
364D constitutes part one of “senior design”. You form your group and select your project from a list of projects provided by industry sponsors. This stage is all about process and planning. You write a series of very similar papers and design plans, but don’t really build anything. The single most important lesson from this class is that you should only take on group members that you trust 100%. If you’re on the fence about somebody, just say no. That temporary pain you feel from rejecting someone is well worth the year of agony brought on by an incompetent team member.

M 348 SCI COMPUTATION IN NUM ANLY
This course was such a letdown. I had hoped to learn MATLAB or R, but instead we implemented numerical methods in C++. Even worse, the programming component was actually very small. The vast majority of the time was spent working traditional math problems. But, it is kind of cool to learn the algorithms used by your graphing calculator.

E E 372N TELECOMMUNICATION NETWORKS
The professor, Scott Nettles, is a good teacher and a chill dude. He takes you through the basics of networking from bits on a wire up through BGP and IPv6. The course is a mixed undergraduate/graduate course, but undergrads are spared from the x-kernel labs.

E E 155 ELECTRICAL/COMPUTER ENGR SMNR
I needed an extra hour to get the requisite number of hours required by the 2010-2012 catalogue. This seminar features a new guest speaker faculty member from the ECE department every week. Professors spoke about their research into metamaterials, automatic program transformations, medical devices, and autonomous vehicles to name a few.

GOV 312L AMERICAN GOVERNMENT (Credit by Exam)
I avoided government entirely at UT by claiming AP test credit and taking US government from ACC online. I did this because I wanted to have more time to focus on my other technical courses. I regret it a little bit. My roommate fulfilled 312L with a course on Mexican government that sounded extremely interesting.

E E 360F INTRO TO SOFTWARE ENGINEERING
Nearly everyone blows off this course which is a huge shame. They will pay for it later. The textbook is Mythical Man Month, and there are a variety of papers assigned as reading. The content itself is reasonably interesting, but Perry manages to suck the life out of it in lecture.

E E 464K SENIOR DESIGN PROJECT
The second semester of senior design is more substantial in that you actually build the product. But for the most part, it’s more of the same: long hours spent editing papers with five authors. The key to success here is real time collaborative editing using Google Docs while talking over Skype.

R M 357E INTRO TO RISK MANAGEMENT
I took a rare chance to get a formal background in finance. This course covers many different types of insurance, something that most people never get any exposure to. The professor I had, Patricia Arnold, takes attendance (?!), but is seriously passionate about insurance which helps drive lectures about a traditionally dry subject matter.

E S 377 INTERDISCIPLNRY ENTREPRENRSHP
1 Semester Startup is positioned to be the killer app for UT. I was fortunate enough to get DebateTab into the very first class. Josh Baer and Bob Metcalfe are solid resources who really want to help. If you take this class. the limiting agent will be yourself and your team’s commitment to school. It’s hard to convince someone to hack when he’s struggling with a math course. You also can’t be upset when your cofounders who are in the country on student visas get hired by Microsoft and Salesforce.

E E 360T SOFTWARE TESTING
This class is important, but it won’t make you a great test writer. The goal is to cover as many types of testing as possible from the Ammann & Offutt text. It begins with JUnit basics and proceeds to survey graph coverage, logic coverage, input space partitioning, and syntax-based testing. The homework assignments with the JPF model checker were some of the strangest, but most interesting assignments I’ve had.

E E 461L SOFTWARE ENGR AND DESIGN
This is the new software lab that’s supposed to prepare you for the real world with JUnit, bash, svn, ant, UML diagrams, and Hoare logic. The course falls short because it’s disorganized and no substitute for teaching yourself. But, it’s better than nothing if you’ve never used anything listed in the previous sentence.

E E 360P CONCURRENT AND DISTRIBUTED SYS
This is where Vijay Garg really shines. His teaching style involves setting up a problem, asking students to solve it, and then pulling them forward through the history of solutions. The key here is that he doesn’t linger so long that you lose focus. After taking this course you will be a much better programmer. Concurrency is one of those topics that you just don’t pick up from “Learn PHP in 24 hours”.

A week before graduation I learned that I would be awarded a double major (in electrical engineering) which was surprising because I intended to only major in computer engineering. Apparently you can’t major solely in CE.

Courses I should have taken

If I had an extra year I would definitely take the following courses. I probably could have found space for them if I’d planned better or worked harder, but hindsight’s 20/20.

E E 460N COMPUTER ARCHITECTURE
This is considered the most difficult and the most time consuming course in ECE. But it may be the most instructive of all. Chiou, who teaches it, says:

Prof. Patt, who is one of the world’s leading computer architects, designed the class.  It covers about 80% of what I studied in a graduate computer architecture classes at MIT (6.823.)

The quote says a lot because all the descriptions of MIT coursework I’ve read make it sound twice as rigorous as anything I took at UT. The problem sets from 2011 are available on Patt’s site.

EE445M EMBEDDED AND REAL TIME SYSTEMS LAB
445M is the most involved embedded systems course you can take as an undergraduate, mainly because you write your own operating system. From the course description:  implementation of multitasking, synchronization, protection, and paging; operating systems

E E 125S INTERNSHIP IN ELEC & COMP ENGR
You can claim one hour of credit for having an internship related to your major. My two years at IBM would have qualified, but I was too lazy to get the paperwork signed. It would have been a hassle because my manager was in Raleigh.

A smarter or more motivated person could easily finish this program in less than 4 years. I don’t regret taking my time, though. College is worth savoring, especially at UT.

Why not CS?

I avoided Computer Science because most of the descriptions I read made the discipline sound heavy on math and theory, and lighter on actual hacking. Formal proofs are difficult and boring for me, so I chose engineering. In retrospect, I’m not sure that this was the right choice because there’s a good amount of freedom within a degree plan.

If you major in computer engineering, you’ll constantly be referred to as a CS or EE major. I found this to be irksome because I made a conscious choice to get an engineering degree, but I also don’t care for circuits. But you can’t really let this get to you because the degrees are functionally equivalent in the workplace. This is justified because neither department is stupid. The CS department is well aware that you need to learn some assembly and logic design and the ECE department ensures that you know how to analyze big-O complexity of an algorithm. You’ll be in good shape either way. And if you know that you just want to build webapps, you might consider CS because you’ll have a better chance of meeting web hackers. I can count the number I met in ECE on two hands.

Aftermath

During my senior year I interviewed at four software companies and received four job offers. I started work at Indeed in June. If you want one of the more engaging programming jobs, you can’t just write code for class. You can either A) be really really smart, or B) be someone who spends a lot of time building cool stuff. Go to Hacker Lounge to meet people who are both.