So 19 months into the Open Government Directive, we seem to have a backlash. The government has spent millions of dollars collecting, organizing, and cataloging its data to make it more available to the public. An unprecedented effort. Some of this data is frivolous, some of it is valuable, but I think we can all agree that more transparency is always — always — a good thing.
Not so, says Ellen Miller of the Sunlight Foundation, one of the leading advocates for government transparency. On Tuesday at the Gov 2.0 Summit, she made it clear that transparency wasn't enough. She also wants accuracy, relevance and quality in the data. Instead, Sunlight found $1.3 trillion in inaccuracies on USAspending.gov. She's also got some choice words for data.gov and other Open Government initiatives. The keynote was a remarkable turn: the administration was completely eviscerated by one of its closest allies. Today, I read that Fast Company's Austin Carr is similarly disillusioned by this week's announcement of Challenge.gov. I think it's safe to say there will be more pieces like this in the next few months.
I love quality data as much as the next person, but this is a perfect example of treating government as a vending machine, and it's poisonous. In 19 months, citizens have access to more data than they ever had before. In some agencies, it takes an average of 43 months to get a new project off the ground. The fact that the US government is even attempting this is amazing.
As I mentioned last year, this is exactly how to prevent innovation in government. If you want change, you have to tolerate imperfection and risk. If every program manager thinks they'll end up on the front page of the Washington Post or get dressed down onstage at Gov 2.0, nothing will change.
Now, some of the data is ridiculous, yes. But now we know it's ridiculous. Before some of this data was public, nobody knew it existed. The government employees who worked with it probably assumed it was valid. One of the main reasons to release the data is to permit public scrutiny and that's exactly what we got. Ironically, Sunlight's $1.3 trillion discovery is an example of the process working, not a failure. This isn't a case of Greek-style institutionalized malfeasance. You're just seeing how hard it is for one of the largest, most complicated organizations on the planet to keep its records straight. I'm not surprised by it at all.
Sunlight has, I think, dangerously conflated transparency for reform. You get transparency first, and that compels reform. That's the whole point. You don't ask for perfection right out of the gate, it's unreasonable. Red Hat's CEO, Jim Whitehurst, is fond of saying "if everyone walked around naked, we'd all go to the gym more often." So the government's naked, and it's gross. The solution isn't to tell the government that it's gross. It knows it's gross. The solution is a long, difficult, complicated, and unpleasant series of reforms that produce better quality data. That requires patience, diligence, perseverance. From both sides.
[Update: Tom Lee of Sunlight responded to this, and I've responded in turn. We're all still friends.]