Barack Obama Directs All Federal Agencies to Have an API
blog.apievangelist.com
blog.apievangelist.com
"... The question was APIs or downloads...
Personally, I believe [data] is one of the best
ways for citizens to keep track of their government
(local to federal) ... APIs tend to deliver what
their “owners” want them to do. Raw data means
everyone has an opportunity to check each other’s
work. Of course, raw data can be manipulated as
well, but it is harder to obscure."
- http://spatiallyadjusted.com/2012/04/03/sharing-data-downloa...I couldn't agree more. APIs are great, but are not the key to open government, for two reasons:
1. They don't provide simple and easy access for non technical individuals into raw information.
APIs shouldn't exist for querying historical datasets if the dataset is not already available in a static format. Release the data, then build an API if there is demand (or the private sector doesn't do it, better, for you).
2. Historical data access is poorly served by APIs.
There is no such thing as a good 'general use' API[1]. API's are appropriate for specific service based transactions that involve some level of processing. Examples:
* VAT/GST number validation
* Road closure notifications
* Identity services
3. Bonus reason: government agencies suck at building APIs.They're not good at determining what is genuinely high value to end users, they tend to prefer visible projects that can justify budget increases, over genuinely useful, but less easily communicated ones (cf. the US national highway system and pork barrel politics), and there is an entire industry of enterprise companies heavily invested in keeping it this way.
TL;DR Release the data, let users build the APIs. Everyone wins.
Bootnotes:
[1] I lie. That's exactly what publishing raw data at stable URLS on a website achieves.
Consider that an API might provide access to "raw" data directly.
I'd ask not for data over APIs but for more universally useful properties such as stability, currency, consistency, etc.
It's not, necessarily. An API provides a [very] limited view into the data. A view selected by the publisher. That's part of the problem.
> I'd ask not for data over APIs but for more universally useful properties such as stability, currency, consistency, etc.
Amen. I'd add: clear license and usage conditions, simple and concise metadata, and frequent updates.
Not sure whether that's likely, he was just pointing out that API vs data are not mutually exclusive.
Instead, a good API makes government into a platform for free (and paid) services to be built to deliver that data in innovative ways. Examples of this are starting to appear in places like Chicago which has opened up a lot of data access – for things including transit (bus tracking, etc.) and a lot more. Giving hackers platforms to innovate will definitely yield better results than just throwing gobs of data at the general public. (Never mind that not all raw data is created equal or that raw data also requires savvy people to distill.
It also means that it's potentially going to be easier for one unit of government to interact with (or at least query) another. That may be big as well.
Whenever you put the word 'government' and 'innovation' into the same sentence, you need to check your working. Government is, by and large, terrible at technical innovation. And innovation by mandate will, predictable, be a non-starter.
That said, there are areas where it is appropriate - see my earlier examples (tranit bus feeds are another one) - where the relevant government agency has found a niche that (a) only they can provide the data, and (b) it is a very easily defined problem that they're solving.
Finally, I really take issue with this statement:
> "giving hackers platforms to innovate will definitely yield better results than just throwing gobs of data at the general public."
Utter rubbish. The "general public" is not stupid. The general public includes huge numbers of people with the domain knowledge and wherewithal to analyse the data. To claim that open and transparent government is better served through an elite technical class, instead of directly to individuals is simply false.
And even then: the progress that SpaceX has made in the last few years is an almost text-book proof of my point. NASA provided SpaceX with a wealth of knowledge and experience that has been very poorly utilized (dollar for dollar) over the last 20 years. Within a very short time-frame we've suddenly seen innovation in the space sector like we haven't seen since the Apollo programme.
As for the military sector - given that maintaining a monopoly over the application of military force is (fundamentally) the primary function of government, it's not surprising that this is the one area of government that has understood it's place for a very long time: issuing contracts with stated operational aims and leaving the private sector to provide the innovation. Yes, the system is flawed - the F-35 programme is a bit of a disaster - but even then it can be at least partially blamed on government agencies interfering with the procurement process.
And yes, we've seen tremendous innovation from SpaceX - but where would they be without that knowledge and experience from NASA? Just because NASA has been turned into a bureaucratic mess with funding that bounces around doesn't mean they haven't created a lot of value and new knowledge, most of which would never have been funded by the private sector. Government work isn't about having good returns on money spent, which strikes me as both a blessing and a curse.
I wonder how many current day innovations have sprung from the initial work of places like the DOE National Labs, and how many might in the future, for all that it's a government program.
Refer (3) in my original top post. This is something they would prove remarkably adept at achieving.
> ...but where would they be without that knowledge and experience from NASA?
Exactly! A thousand times so! Releasing their knowledge and experience that only a government funded entity could have amassed (c.f. raw data), has sling-shot a private enterprise capable of rapid innovation. This is precisely why I think that releasing the raw data is the most beneficial outcome.
Only recently have the govies started thinking about how their data can be useful to the general public. In the past everything had been stovepiped and guarded with peoples' live(lihood)s. Hopefully this makes them think about data integrity throughout the life of that data.
Think about how data used to be provided to the public before. A bunch of government folks had to collect data and make sense of it themselves and put it together in a report destined for congress. It's waaaaay different to just provide that raw data to the public. Rather, I think what we'll see is more sanitized data sets, after they've been internally analyzed and vetted (probably multiple times). Not exactly transparent.
But I hope one day, after many iterations of API building, we'll get to a point where the data truly is transparent.
The problem is that they can't anticipate how their data is most useful. Release and let the users decide.
> But I hope one day, after many iterations of API building, we'll get to a point where the data truly is transparent.
So, we should wait many years while, predictably, large budgets are blown on faltering "government innovation" projects, when we could just have the data and use it today?
This just doesn't make sense!
If a president could have a meaningful impact on this sort of thing, it would be in setting a high bar for the quality of information released by agencies. Any sort of requirement of this kind is completely absent from the announcement.
So rather than being about transparency as it's being touted, the announcement is a celebration of high tech obfuscation. Soon the same sort of insulting, opaque, useless information spouted by officials in press conferences will be available via HTTP. This is at best a neutral day for democracy.
While this might not set a "high bar for the quality of information," the President's effort shows a level of commitment to both technological streamlining of government agencies and to transparency and is, at the very least, a step in the right direction.
If the President of the United States wants to be impressive -- and I don't know that APIs are where one would start -- he can do better than make announcements.
For example: http://ideascale.com/userimages/sub-1/736312/ConOpsFinal.pdf is a document (Dec 2009) documenting the vision for Data.gov, and http://www.whitehouse.gov/sites/default/files/omb/assets/mem... is a memoranda from 2009 that first started requiring the online sharing of major datasets.
And for those cases, the agency can still justify why the CSV is better. There are always ways to get around the rules, see Section 508 rules for handicapped users.
>If a president could have a meaningful impact on this sort of thing, it would be in setting a high bar for the quality of information released by agencies. Any sort of requirement of this kind is completely absent from the announcement.
This is a step in the right direction, once the data is more accessible, the "users" (developers) can request for better data. The /DigitalStrategy page requirement is really good in my opinion and will make things simpler instead of the mishmash of sites buried in menus and behind authentication walls.
>So rather than being about transparency as it's being touted, the announcement is a celebration of high tech obfuscation. Soon the same sort of insulting, opaque, useless information spouted by officials in press conferences will be available via HTTP. This is at best a neutral day for democracy.
Whoever said this was about transparency in government? It's not about transparency per se.This is more about making information easier to access and find. In many cases agencies already have APIs, Webservices, data dumps, but they're really buried. How is making them more visible neutral?
How useful this is depends on how clear the data is, how well they document things, how sane their document formats are -- in other words, it depends on things that are much harder to mandate than just "have an API". I'll predict in advance that most of the APIs here will be pretty half-assed.
So I guess I agree with you but would even take it further.
Of course, if that API is Stuxnet or Flame-based....
If we had to wait for higher-level, coordinating standards first, progress might never come.
[1]: http://sunlightfoundation.com/blog/2012/06/01/bulk-access-de...
3 months to get a "machine-readable" status report on implementing an API?
Then, complete the implementation in 12 months?
If it takes 3 months for an agency to get a status report up, how long will it take them to implement said API? Government work, sheesh....
Remember that this has to accommodate the agency in the worst situation. The alternative would be to give separate deadlines for each agency in the memo, but that may be hard to determine from the outside (after all, "what are you going to do, how, and when?" is the first thing he's asking for) and you still have to wait for the last one to have your "Hey, everyone, there's data! Have at!" press conference.
This is basically a conversation about project scope and management. Assuming this is more complex simply because it's a government agency is flawed thinking, though. I can certainly perceive complexity due to expectations or maybe the participants themselves (I've worked with some fed agencies), but that should have zero bearing on addressing the project itself.
But in reference to some of the specifics of how this is different (sensitive data, system load, system attacks, worst-case-scenario) -- this is begging for an iterative project approach as opposed to a waterfall basis. In spite of the fact that these aren't unique issues to government agencies, the context of addressing those issues might be (I have some idea of the technical capacities of our government.) I can accept there may be a learning curve in this project, but believing this project yields unique problems that haven't been dealt with elsewhere -- respectfully, that's just wrong.
If this is all very new to these federal agencies, that simply suggests a smaller initial scope that can grow after multiple iterations. Teams can learn about requirements, refine their APIs, shore up their operation, etc. Not to over-simplify, but this is basic project management.
Again, the context of the government's perview may be different than little-startup-dot-com, but the process of getting from A to B isn't really all that different.
2 weeks for the director of each agency to delegate someone to be responsible for this. 2 weeks for said responsible person to figure out what an API is. 6 weeks for them to go around to everybody in the agency asking "are you doing any APIs yet?". 3 weeks to take the feedback and turn it into a semi-coherent report.
If anything, I think 3 months is optimistic.
- hire an entire department's worth of people
The worst approach I can imagine is to start this project by hiring new, dedicated people. This project only succeeds by incorporating it into the very fabric of an agency's operation. Remember, new work must go on even after this API is in place. Hiring a dedicated group that somehow has to reverse-out everything the agency does going forward for purposes of external API access is a recipe for failure. Separate teams will already be at odds with each other; better to leverage the existing teams, as their the ones in best position to understand the context of how external access to their operations should function. - wait for them to devise an API for your agency
I'm not sure if this refers to a lack of understanding the requirements, or to a lack of competency on the part of the team itself, but the presumed outcome is late delivery. If it's complex requirements, that simply suggests a basic, iterative project where scope is managed tightly and the duration is rather short (allows the team to learn as the project moves along.) If the suggestion is competency on the part of the team, hiring/contracting a few competent individuals to align with team leaders on the project has worked well for past projects. In either case, proper management of the project approach can address issues of timeline. - code it
This goes to team capability, but also to an understanding about integration. Again, solvable problem based on the team capacity. If this suggests a unique code stack that's not already available elsewhere, I'd need to understand the justification. Project management 101. - provision server space and servers
This implies the technical operations of our agency are fully-loaded, or can't address this in a reasonable timeframe, or purchasing requires some inordinate amount of lead time, or some other unknown reason. If the suggestion is that dependencies exist that threaten the timeline, those dependencies should be mitigated. Again, project management 101. - deploy
This is a physical step of pushing bits live, so automation tends to make this a quick step. In my project, we thought about deployment actions when we determined how to devise our API. At this point, deployment is an operational aspect -- not a how-do-we-do-this function. This project doesn't proceed without understanding this step.Disclaimer: I've actually done this type of project work for significant operations of size and complexity. I've also worked with a few federal agencies, so I have some familiarity with the lay of the land.
It's really not that daunting of a project. I would respectfully suggest that you re-consider your own presumptions, as there are a lot of ways to address this type of work.
What will this bring? Well, the US govt has X agencies. The result of this decree will be that, within 12 months, all of the public will get to enjoy the thrills of having X incompatible web API's, one unique one per agency.
What data are you looking for that isn't available?
The only thing I can think of that's difficult to get is some law enforcement, military and "national security" related information, but APIs aren't going to change that.
I don't have a ton of experience with it (mostly USGS and NOAA), but the Department of Commerce and the Department of Interior both make a ton of data available.
Decision makers are often excited about technology but don't really get the ground level experience. They want to do all the things...on a roadmap...with milestones. Mobile has to be involved in some way.
The issue is far more complicated than the comments I see in here are giving credit for. Don't get me wrong, there's going to be delay as the PHBs get themselves wrapped around what an API even is, but they'll have the directive routed to their CIOs before that, and they will understand the requirement, and how impossible it is.
The biggest issue is that the data isn't really owned by the government entities. I mean, the data is theirs, but it's locked up in their vendor provided tools, and/or their custom, built-by-vendor products. If they're using Oracle AquaLogic (or whatever it is now) to host the majority of their portal content, they're dependent on Oracle to either come in and show them how to implement the feature (which is a significant service dollar cost) or they're going to have to wait until Oracle builds the ability for API exposure into the product if it doesn't exist yet.
If they've got custom-built portals, they'll need to consult with the vendors who wrote them or maintain them now and get them to add that in. That means that they'll have to modify the contract originally bid for the project, which is going to eat up a couple months of the timeline alone. Then they'll have to figure out what sort of things actually make it into the API, how to segment sensitive data reliably, get it through ISSO testing, etc. It's almost impossible for a project of any significance.
On top of that, they'll have to do it with a budget they don't have, and with resources allocated elsewhere. The only way the government really gets anything done is by committing large amounts of resources to it in an uninterrupted fashion. They don't have the capacity to be agile, and to some extent, that's by design.
I assume that this will lead to some discussion on API standards, as multiple agencies simultaneously realize the implications.
I imagine after 15 years they may have a chance at this, but I would caution those of you who have never worked in huge government IT shops to take this with a grain of salt. The situation is so bad in many places that Congress has been passing laws making it illegal for the federal systems not to behave in a certain way. And still things are broken. We passed the point of desperation many years ago.
Big IT in general is broken, and government IT is the most dysfunctional of any IT on the planet. I remain hopeful that this executive order can accomplish something, but I'm not holding my breath on it. Hopeful is one thing. Excited like this guy is? Not at all. Maybe in another 15 years. Maybe.
Meh, they're evil, it's probably SOAP and you have to discover it.
edit: Sorry guys! Didn't realize a tongue-in-cheek comment about the NSA have an API was so super-serious! It's like Oprah in here giving out downvotes anymore. "It's a free downvotes for you and you and you."
312 Pay No Attention To The ProxyIn the case of Amazon, this was achieved by CEO fiat, and strongly tied to employee evaluation. (To the point where employees in groups that failed to do so would have been evaluated right out of the company.) I wonder if POTUS has this kind of power over the federal bureaucracy.
Also, I would wonder if this is to be done securely.
;)
yada yada ... barak-obama-directs-all-federal-agencies-to-have-an-api/
yada yada ... barak-obama-directs-all-federal-agencies-to-have-an-api/index.php
With regards searches, it depends on what you search for. HN_Search is not always obvious, and it doesn't index minute by minute. You may have done the search before the engine caught up.
Thus a mandate that "all agencies should do x" with tech, before reforms in the acquisition arena create severe problems down the road, and only really benefit the contractors who make them.
By the time these Apis really see the light of day, we will be complaining about them.
air_force.launch({"f22": 3, "b2": 4})
Bulk data.
Agree with polemic.
All part of politicians (particularly Democrats) trying to look like they have a clue. Give up already for heavens sake and get back to managing the deficit.