13. “To find the latest version of a regulation online, a person
should first check the table of contents in the most
current Kansas Register, then the Index to Regulations in
the most current Kansas Register, then the current
K.A.R. Supplement, then the Kansas Administrative
Regulations. If the regulation is found at any of these
sequential steps, stop and consider that version the most
recent.”
I put laws on the internet. LAWS ON THE INTERNET. It’s 2013. But the government does this badly, and I try to do it better.
For example, this is a law from the website of the U.S. state of Alabama. I’m not hiding anything, no parts of the website have been erased. This is everything on the page. No menu. No title. Nowhere does it even say this is a law. This is *terrible*.
A law from the state of Louisiana. Again, this is everything. Nothing’s been removed. This is very bad. The stuff of the World Wide Web in 1993.
This is a law from my state, the state of Virginia. It’s our open meetings law. No navigation, context, title—just this. At least there are links to the other laws that are mentioned. I’m not showing you the worst sites—this is just what law websites look like. But there’s good data here, if we can identify it.
Look: the law’s number, title, mentions of other law numbers within the law. And structural information—what laws come before and after.
And more—important words and phrases. This is good. We can get real meaning from this. There’s metadata here, not metadata as we’d want it, but we can extract it, if we try. If we extract it, then we can make it more understandable by people and by software.
My software, The State Decoded, scrapes that data to show the same law like this. This is based on feeding my software a bulk file of the state of Virginia’s laws and waiting 10 minutes. It’s all automatic. The State Decoded is free, open source. You can get it at statedecoded.com. Its development is funded by the Knight Foundation. It’s being used by individuals, groups, businesses, and governments all around the United States to take the laws from bad websites and put them on much better websites.
It’s developed in the open on GitHub. Every line of code, every bug, every planned feature—it’s all available for anybody to view, fix, add to, duplicate, etc.
After 18 months of development, it’s almost finished.
Laws are the central data source for important open government data. Without laws, many of these data sources cannot be connected. In the United States, as in many countries, courts do not publish opinions about legislation. They publish opinions about *laws*. You can’t connect bills to court opinions without going through laws. That is true for many important datasets—they depend on having bulk data about laws. In the open data ecosystem, laws are very important. If this interests you, please come to my workshop at 1:00 this afternoon, where I will teach you how laws work and how to get useful information out of your laws. You will learn how legislation *really* becomes a law. It’s not how you think.
Why am *I* putting the government’s laws on the internet? Why am I scraping this data off of terrible web pages? Why isn’t the government doing this *right*? My job should not exist, and the Knight Foundation shouldn’t have to pay me to write this software.
Government needs to step up. This is their job. Or, at least, it’s their job to make it easy for us to do this for them. In a time when governments don’t have much money, they can’t afford multi-billion-dollar software contracts. They need our help, and we *want* to help.
This is an quote from the the state of Kansas’s website for their regulations. I’m going to read it out loud, even though it’s up on the screen, because I want us all to appreciate how dumb that it is. Why is this so hard? Because they have put a lot of data online, but they don’t know how to connect it. That’s because it’s not in a database, it’s not XML or JSON. It’s a big pile of HTML.
Government needs to give us their raw data. Hackers, journalists, businesses, and other governments will use that data and make it better for *everybody*. Look at weather data. The U.S. government opened up their weather data computer system to the public in the 1970s, and now we have a huge economy built on weather data. Or the global positioning system. In the 1990s, the U.S. government let the public use the military satellite navigation system, and we have an enormous economy built on GPS data, ranging from shipping navigation systems to Foursquare. Government gave us access to data that they were going to generate anyway, and the private sector turned that data into something much more valuable. We need more success stories like this. We need more data.
In March, President Obama issued this executive order, ordering every United States government agency to provide an inventory of all of their data sets, online or off, as a JSON file on their website. It has to be a file named “data.json,” like “whitehouse.gov/data.json.” They must do this by November. This executive order also makes CSV, JSON, and XML the only acceptable file formats for bulk data from now on. No more PDFs. No more Excel files. Data has to be machine-readable. Every country needs this.
Governments should install The State Decoded. It’s free! They should contribute to its development, because that will cost much less than paying the vendors who write the software that they use now. There are so many programs like this—Linux, Apache, MySQL, etc.—and so many specialized programs that you and I have never heard of, but that are no less important. When government makes software, or pays companies to write software for them, they should release it into the public domain, put it on GitHub, and accept pull requests. The White House has started doing this, experimentally, and I’m hopeful that they’ll keep doing it, and that more governments will do it, too.
Last October, World Wide Web creator Tim Berners-Lee established the Open Data Institute in the United Kingdom. They work with government to help them publish their data. They help government to do things right. This October, with the support of the White House and the Knight Foundation, I’m replicating the Open Data Institute in the United States. Many governments want and need help with open data, and I think we need to help them.
This is an important job. It is not enough to complain about what government doesn’t do. Local, provincial, and national government need help, and those who want help should receive it. This, I think, is how our relationship with government should work right now.
Argentina needs an Open Data Institute. All of the countries of South America need Open Data Institutes, organizations run by people who have the knowledge and the passion to help government do this important work. You must start these organizations. I will help. Vamos. Gracias.
