We need to archive feeds

As of 2015, Facebook has overtaken Google as the main source of traffic to news sites[1]. The average Facebook user spends 40 minutes a day on the site[2]. If we fail to archive news feeds we will lose important sociological information.

The Internet Archive does a great job of archiving the web, among other things like VHS tapes. They’re located here in The Richmond and you can go visit them on Fridays during lunch[3]. I’m concerned that while they are doing a great job of getting the data, they’re missing really crucial metadata, i.e. feed presentation.

The Wayback Machine is necessary but not sufficient for a full representation of the state of the web as it is experienced by real people. If you really want to understand the zeitgeist of today, January 17, 2017, you wouldn’t want to only see the front page of the NYT. You’d want to experience the web mediated through news feeds of real people. I imagine a good number of researchers would like to sample feeds in the years leading up to the 2016 presidential election.

I see there is a chrome extension that can be used to archive a feed, but I haven’t seen anyone scraping feeds at scale over time. Imagine a sort of opt-in system where volunteers give archivists read-access to their social media accounts. With a few hundred opt-ins you could get a decent sample of all the different cultural/subcultural bubbles.

The obvious candidate to solve this problem is Facebook itself. They might already have a feed time machine internally! Contributing a few hundred feeds (and appropriate developer time) to the Internet Archive would be a great move and a PR win.


[1] http://www.adweek.com/socialtimes/facebook-is-now-the-top-referral-source-for-digital-publishers/625300


[3] If you live in San Francisco you should go do this. You will have fun and meet some interesting people. Also, I wish private tech companies would have explicitly open lunches. It could work really well in SF. “Hmm, should I have lunch at Indeed or Slack today?”

Blacker Mirrors

The question of whether or not we’re living in a simulation has cropped up in public fora recently. The thrust of the argument is that future societies are going to love computers and love simulating stuff.

a technologically mature “posthuman” civilization would have enormous computing power; if even a tiny percentage of them were to run “ancestor simulations”… the total number of simulated ancestors, or “Sims”, in the universe … would greatly exceed the total number of actual ancestors. [1]

So if we’re in a simulation it’s most likely being run by a far-future posthuman civilization. That’s totally fine by me. I’m not going to lose any simulated sleep over it.

But consider a similar, but very low probability scenario: near-future simulation.

Suppose that Moore’s law makes computing power more abundant over the next hundred years (or some pre-post-human time-frame), but simulating a universe is still extremely resource intensive. Simulations run on custom-built hardware and are only within reach for major governments and a few hedge funds. The number of simulated realities is low enough (<100) that there is active monitoring. The administrators aren’t interested in individuals, but the behavior of societies at-large. They want to win wars, craft policy, and beat the market. Like an applied version of Foundation’s psychohistory. This scenario is a little depressing, but doesn’t bother me too much more than the far-future hypothesis. It’s depressing, but not invasive.

The much more concerning scenario is that our simulation is owned not by BlackRock, but by a really well-capitalized Viacom. Our simulation is scoured by editors for juicy tidbits that are edited into compelling reality TV segments, a la The Truman Show. In this scenario, the most embarrassing internal monologues of every person on earth are played over laugh tracks for millions of fans in the base reality.

I am less concerned about free will and determinism than I am about being laughed at by someone higher up in the simulation chain.

[1] https://en.wikipedia.org/wiki/Simulation_hypothesis#Bostrom.27s_trilemma:_.22the_simulation_argument.22
note: Why is there a photo of Nick Bostrom on that page? What value does that add?

Shot on a Nexus 5X

Google has advertised the Nexus 5X around San Francisco, claiming a high quality camera. Here’s what the photos look like in practice.


  • Good for close-ups of people and dogs.
  • Good for landmarks in bright light.
  • Okay for landmarks in low light.
  • Bad for far away objects.
  • Bad in low light for candid shots of people. They move too much.
  • Okay for landscapes; best on clear days.

The Joker vs. Quantitative Easing

In The Dark Knight (2008), the Joker destroys a large pile of cash by setting it on fire. This demonstrates the Joker’s commitment to nihilism but raises a much more interesting question: What is the deflationary effect on Gotham’s economy?

To answer that we need to know if the money was actually going to be spent.

It absolutely was.

Before the events of the film, newly elected District Attorney, Harvey Dent, has spearheaded an anti-money laundering campaign but has failed to halt the business of the mob which has access to a few crooked banks under the control of Chinese financier, Lau.

Batman helps his cop friend Gordon identify Lau’s banks via drug purchases with marked bills. Lau’s banks are performing two types of services. The first is exciting: money laundering. The illicit local deposits in Gotham are “layered” by being sent to Lau’s company in Hong Kong and then “integrated” by being paid back to Gotham gangsters. The mob spends this money on the ostensibly legitimate parts of their business like real estate, capital goods (trucks, cement), and payroll which is spent on consumer goods (cannoli).

The other type of service the banks perform is boring: commercial banking. The mob’s deposits are on the books as cash reserves and count toward reserve requirements held against business loans, car loans, and mortgages for the general public. Easy credit helps businesses grow and is great for consumer spending.

Lau’s corrupt cops tip him off to Gordon’s raid and he explains to his ethnically diverse cadre of gangsters that he has moved the deposits to a secure location that is “not a bank”. This is bad for the general public. If the mob is unbanked, there’s slightly less credit available for the people of Gotham.

Remember that the film is set in 2008 which happens to be the beginning of the financial crisis. Gotham needs all the cash it can get.

Later on, the Joker kidnaps Lau and forces him to give up the non-bank location of the money. We see Lau tied up in a chair atop a mountain of cash. Assuming a billion dollars in hundreds is about two pallets, the total value of cash mountain is on the order of 10 billion dollars. Note that Batman had already forced Lau to withdraw the money from the banks so he shares some culpability for the deflation, at least in the short term [1].

Is this a meaningful reduction of the money supply?

WSJ says:

From December 2008 to March 2010, the Fed bought $1.7 trillion of Treasurys and mortgage-backed securities [2]

You can sort of see that here in the dark blue line. Notice the vertical axis is in trillions of dollars [3].


So $1.7T over 15 months is $113B dollars a month. The Joker burned less than a tenth of that in the warehouse which isn’t terrible.

There’s also fallout from dismantling the mob that’s hard to quantify. A nontrivial number of henchmen, goons, and thugs will be out of the street if Batman gets his way. These people will have a hard time transitioning to the legitimate economy which systematically practices de jure discrimination against anyone with a criminal record.

But at the same time, the removing the presence of the mob in Gotham lowers costs of doing business for everyone else. If the Taco Bell franchise has to pay protection, it can’t hire a guy to keep the drive-through open on weekends. All told, the mob is probably a net negative for the local economy.

Okay, so burning a few billion dollars isn’t enough to cause serious monetary damage. In fact, it pales in comparison to the non-monetary effects of the Joker’s campaign. Terrorism hurts growth in 2 ways, says the IMF [4].

  1. Direct damage is physical and logistical. A) Businesses are literally exploded and can’t operate. Remember the hospital destroyed by the Joker. B) Decreased productivity lowers output (the National Guard shuts down all bridges and tunnels)
  2. Indirect damage is psychological. Decreased consumer confidence lowers consumption. The assassination of the police commissioner isn’t getting anyone into Gotham department stores.

In summary, Batman’s decision to take out the Joker has a sound economic rationale. He’s firmly on the side of growth. See if you can spot the omission in Gordon’s closing remarks.

he’s a silent guardian, a watchful
protector… a dark knight.

What he didn’t say is that he’s a protector of markets. Which isn’t surprising given Bruce Wayne’s nominal day job.


[1] Had the total amount been seized by the police under asset forfeiture laws, the cops would get to keep up to 60% for new toys while the remainder goes to the New York state treasury. https://www.ij.org/asset-forfeiture-report-new-york

[2] http://blogs.wsj.com/economics/2010/11/03/qa-on-qe2-what-a-fed-move-would-mean/

[3] http://www.economist.com/blogs/graphicdetail/2014/10/daily-chart-21

[4] http://www.imf.org/external/pubs/ft/wp/2005/wp0560.pdf

Additional reading

Aaron Swartz highlights political and philosophical dilemmas appearing in the film.


An argument for Lyft

There’s a lot of rhetoric swirling around the Lyft debate concerning an evil taxi lobby and macro concerns on Austin’s congestion. I want to dissect the larger discussion apply a sober focus to the most emotionally charged aspect: Safety.

The anti-ridesharing crowd employs a false syllogism.

  • Taxi licenses increase public safety.
  • Lyft drivers are unlicensed.
  • Legalizing Lyft would decrease public safety.

There’s an implicit assumption in this argument: That licensed taxis are the only form of private transportation on the road now. This is utterly false. Anecdotally, about 20% of the cabs I take are illegal and supposedly dangerous.

The dance is a familiar one if you’ve ever spent time downtown. It’s 2:30 am. You’re on San Jacinto or Congress, looking for that illuminated Yellow Cab dome. After 10 minutes with no luck, Joey McRando pulls up in a black SUV and offers a ride. You take seven seconds to size him up and decide that you could choke him to death if it really hits the fan. There’s some awkward bartering and you’re on your way.

Ridesharing apps alleviate the sketchiness of this fairly common situation. Even if their driver requirements are less stringent than taxi licenses, the Amazon-style rating system with stars and reviews is a far better guarantor of safety than me ascertaining the fighting ability of a random person.

Policy decisions should be underpinned by sober assessments of conditions on the ground. There’s a clear pragmatic way to keep me from hopping into random cars in the middle of the night.

Legalize Lyft.


Update 7/7/2014:
Two or three weeks after this post was published, Lyft went from a flaky don’t-hold-your-breath operation to a reliable way to get around Austin. It’s still not legal, but the Lyft Death Star is fully operational. Yeah, buddy.

Update 1/17/2017:
Both Uber and Lyft left Austin in May 2016 after a ballot measure overturning a driver fingerprinting requirement failed. There are several replacement apps but they cost more and none of them seem to work as well. I have been living in San Francisco for years now. This one is on y’all.