API Discovery: Can we do better?
blog.apis.guru
blog.apis.guru
JSON-LD provides a vocabulary (schema) for linking various documents together, so you can for example link a blog post document directly to its author. Hydra then provides a vocabulary that lets you describe how to fetch that author document, what fields are required to create a new author object, and how to update the blog post to instead link to that new author. In this case, both "author" and "blog post" can be their respective schema.org documents.
[1]: http://www.hydra-cg.com/ [2]: http://json-ld.org/
I'm seeing people using GraphQL[1] more these days, which also solves the problem of API schema discovery because it's one endpoint with built-in introspection[2].
> people using GraphQL[1] more these days
I fully agree. That's why we also maintain a list of GraphQL APIs: https://github.com/APIs-guru/graphql-apis
> solves the problem of API schema discovery because it's one endpoint with built-in introspection
You're absolutely right. We plan to do a few interesting projects around GraphQL. Subscribe to our blog to not miss announcements :)
I now have wrapped a couple of API's in swagger and using swagger-ui to create beautiful interactive api docs for users.
Moving forward I'm going to wrap most of our legacy systems around a swagger spec'ed API and discontinue any alternative access to these systems as time goes by.
I'm the author, and I definitely know about Swagger/OpenAPI. By coincidence, I'm maintainer of the collection of 250+ Swagger specs for public APIs: https://github.com/APIs-guru/openapi-directory
But I have learned the hard truth over last two years: API catalogs aren't scalable solutions for API discovery. That's why I'm pushing this.
P.S. it is pretty easy to generate Schema.org type based on Swagger/OpenAPI spec.
And here is github repo: https://github.com/APIs-guru/openapi-directory
Ah cool, thanks for the link and effort. Will have a look at it as time allow.
> It is pretty easy to generate Schema.org type based on Swagger/OpenAPI spec.
Do you have any example or gist of a snipped of code to accomplish this?
> API catalogs aren't scalable solutions for API discovery. That's why I'm pushing this.
But does the API catalog not enable auto discovery? Agreeing you maintaining a list is not scalable.
Minor critique but the word "discovery" particularly in terms of service oriented is heavily overloaded. It took me a little while to understand what the article meant by discovery and I'm still not entirely sure I do.
Do you mean this discovery: https://en.wikipedia.org/wiki/Service_discovery or do you mean a discovery more in the human sense (ie a consumer search directory listing API services)?
One of our goals is to mine information about APIs, rather than relying on user input (hence the "catalog"). We still have a long way to go, though. In Web API land, many things follow a power-law distribution: for a few well-known APIs, a lot of information can be found (https://apiharmony-open.mybluemix.net/apis/instagram). For many other APIs, information is sparse and hard to mine.
Adding Schema.org (or similar) annotations would help. But it may be as hard to convince providers to do this, as it is to convince them to write OpenAPI specifications (or similar).
When you can't use the government make a conglomerate of large companies with key hold on industries.
One of the bigger challenges was finding appropriate APIs to use. The idea being that it had to make sense as a realtime datapoint. Current temperature in California is one example.
Part of the solution I tried for the end user was to add a searchable database of APIs, but I never found many that had that kind of realtime datapoint. The majority of use cases for APIs seems to be CRUD apps, authentication, unchanging datasets, or really specific once off data transforms.
I feel like that's a bummer, as I really love the idea of being able to build up a dashboard of random data points about the world that interest me. The number of astronauts currently in space, the temperature somewhere I like to visit, the traffic density on specific roads. All retrievable individually from various apps, but not in one scifi intelligence center style place.
For anyone interested, the site was https://apiblocks.com
The `api.txt` can simply be a text-based listing of links to API descriptions for all APIs exposed by that site publicly. Most API description formats already allow for the API name, category, endpoints, etc. so no need to reinvent the wheel there. A good idea would be to add the `<link rel="api" href="api.txt" />` tag on the webpage too. This would make discovery and cataloging much easier.
Some API cataloging sites have come up with interesting ideas. For example, there is a service https://sdks.io/ that can automatically generate SDKs right out of the API descriptions that they have crawled. It is powered by https://apimatic.io/
Essentially, there can be a Link HTTP header to the "Entry-Point" document for the API, which further links to the vocabulary that describes the resources understood by the system and the actions you can perform on them.
Link: <http://api.example.com/doc/>; rel="http://www.w3.org/ns/hydra/core#apiDocumentation"
One time, while interviewing a candidate, I picked a random buzzword on his resume (which happened to be API) and asked him to talk about it for a bit. He immediately launched into a mini-rant about all these crazy details with APIs that seemed to be foregone conclusions. Do people actually agonize over whether the API speaks XML or JSON? Or fret over the lack of a console? Are we talking about the same thing? You have an endpoint, you send it a request, you get back some data. WOW SO HARD!
Usually each service provides a client package for whatever language you're using, but sometimes its from a third-party, and they're always in different levels of maintained-ness. In Ruby, there hasn't been a single overwhelming HTTP client, just several that passed in and out of popularity, so depending on when the initial release of the gems you're using were, you'll end up with 3-6 different HTTP client implementations, each with their own way to mock requests for your tests.
On top of the client libraries, you have to spend a not-insignificant amount of time reading the service's documentation, to see how they prefer you do auth, what methods to call on the library, what the payload needs to look like, what response codes to expect, what exceptions may be raised... It is a non-trivial amount of work for each additional service. And then you get to stay on top of API updates for the dozen services you integrate with, and hope they have an email list or RSS feed so that you have some warning before they completely break your integration.
The promise of schema.org (and JSONLD, and Hydra, as I mentioned in another comment), is that if we could all just stop NIH every damn API we write because "those look too complicated, how hard can it be to just return some JSON?", then people like me, who need to integrate with several APIs, wouldn't have to waste so much time on figuring out how your special snowflake actually works.
> I find it strange that people regard APIs as their own
> thing. I have never "searched the web" for "email APIs",
> in fact that whole sentence just sounds amateurish.
> WTF is an "API console"? Is that not curl?
> Do people actually agonize over whether the API speaks
> XML or JSON?
Part of the reason it is weird to see people talking about "APIs" now is because API is such a core, fundamental concept in programming that to use the term in its present context (as in, a web API which you issue commands via HTTP) does a disservice to all the other APIs out there. When people on HN talk about API considerations they seem to omit the 95% of other use cases there are for APIs outside of the realm of startups and webapps.
For example, bash could be considered an API to the Linux System Base, and bash commands use Linux's syscall API to achieve the user's ends. Syscalls are issued commands over an ABI usually handled by libc, and return data in the format prescribed by the function signature.
My TV presents an API that is accessible over USB. Granted, I am pretty sure that port is only for use by service techs, but if I wanted to I could probably hook up some electronics and figure out the programming interface. Commands are issued through the USB bus and data will (hopefully) come back in some format.
My MIDI keyboard offers an API through buttons, a little LCD panel, and MIDI-over-USB. I issue it commands through button presses on its control panel, or through MIDI or OSC protocols, and it also returns data (either a human-readable response on a LCD screen or more data over the USB wire). In both those cases I'll have to decode that data to figure out whether my command was accepted and/or what to do next.
Finally, a popular web-based CRM solution offers an event-lifecycle and data-tracking API that I have to interface with at work. I issue it commands over HTTPS and it responds in XML-over-HTTP, which like JSON, is fairly easy to parse in any number of different ways, and in any case its responses are only ever in XML so once that was written it was done with.
This is why this "API" dialogue is so strange. It is reductive and if overarching solutions are going to be prescribed than they need to take into account the wider and entire world that concept lives in. I don't see how many of the prescriptions offered help the use cases outside the HN crowd -- in fact I would argue that the author needs to go back and comb existing literature outside of web programming, because the engineering and computer science worlds have been dealing with exactly these problems for decades.
The "APIs" talked about here refer to a very specific type of product for a very specific audience; there is a bigger world out there and people seem to forget that.
> What do you do if you know that you need an API for a thing, but you don't know who offers that, or who a good one is? Nothing wrong with searching for "email APIs", "payments API", "maps API", etc. - I don't know why you think that's "amateurish".
OP is about making it even easier to access that documentation.
> Search for "[product name] API doc/faq/whatever" or search for code examples of people interfacing with it?
I have experience with Open Graph http://ogp.me/ and oEmbed http://oembed.com/ but something about humans having to manually categorize a site has never sat right with me. We should be able to grok structure from sites by how other sites refer to them, similar to what Google does. This may be related to open source search, I don't know.
However, I'm not sure if this goes far enough. There exist some formats for describing what APIs can do in natural language, though I'm not convinced that they're really that useful [1][2].
[0] http://apisjson.org [1] http://alps.io [2] http://restdesc.org
Not having to read documentation to understand whether the "table" property in the JSON document means "coffee table" or "tabular data" is going to save developers worldwide umpteen man-years. Still, most API developers continue creating snowflakes with proprietary designs, protocols and vocabularies.
JSON-LD is a large step in the right direction by making it easy to share vocabularies. Hydra[2] takes the next logical step by standardizing the protocol as well, so no matter what your API does, it has one standardized way to express it.
[1]: http://json-ld.org/ [2]: http://www.hydra-cg.com/
Because--oh I don't know--APIs aren't restaurants?
Is this really a problem, or is this a tool to help lazy blub developers be even lazier and blubbier?
If you care enough about an API to worry about what things it does via the docs--and why aren't you using a client lib anyways?--then you care too much to be satisfied with a simple sketch of the API via Google search results.