The Future of APIs: APIs aren’t the endgame they won't stay forever
blog.goodapi.co
blog.goodapi.co
Shirky's "The Semantic Web, Syllogism, and Worldview" http://www.shirky.com/writings/herecomeseverybody/semantic_s...
Doctorow's "Metacrap: Putting the torch to seven straw-men of the meta-utopia" http://www.well.com/~doctorow/metacrap.htm
Doctorow talks about problems with metadata, but these problems might apply equally to APIs and the API vocabulary discussed in the article. Specifically:
2.1 People lie 2.2 People are lazy 2.3 People are stupid 2.4 Mission: Impossible -- know thyself 2.5 Schemas aren't neutral 2.6 Metrics influence results 2.7 There's more than one way to describe something
The fundamental problems are that (1) getting people to agree on things is a surprisingly difficult and political problem that can never be solved once and for all, and (2) people have incentives to lie. If you invent a generalized way to look up any weather forcasting API, somebody is going to realize that they can make money gaming the system somehow. PayPal is really in the business of fraud detection, and Google is in the business of fighting against blackhat SEO (and click fraud).
So take your automated API discovery utopia, and explain to me what happens when blackhats try to game the system and pollute your vocabulary for profit. Tell me what will happen when 6 vendors implement an API vocabulary, but none of them quite agree on the corner cases. This is the hard part.
WSDL didn't fail because it frequently tried to describe SOAP connectivity (and everyone knows that SOAP is obviously bad for all things), it failed because it was still people writing the APIs and the descriptions of those APIs. And since people aren't perfect, humans had to intervene to find and fix the bugs to properly communicate with a WSDL defined API.
Until AI gets good enough, or we adopt a specific definition to meet all use cases (should be fun to watch), such attempts are going to keep failing. Because it's humans hiding in the box of the turk, and will be for the forseeable future, and computers are still pretty terrible at communicating with humans.
Overwhelmingly in practice people wrote wsdls and xsds by hand, incorrectly, and treated them as secrets instead of publishing them along with the API endpoints. There was other "enterprise" BS during the XML craze too. In one case I even had to work around bugs in an IBM XML firewall, or its configuration. It was a nightmare[1].
- IsBooleanPropertyIncludedBoolean
- Massive, incredibly slow to build libraries
- Check everything in case it was null
- Never really sure what was wrong when it went wrong
Also they'd sometimes break and you'd have to hand edit the wsdl's to get them to work again (at least salesforce used to break theirs every now and again, the wsdl would be incompatible with the .Net tool because of certain character not escaping properly).
Credit to Matt Levine: https://www.bloomberg.com/view/articles/2016-01-19/banks-oil...
We haven't crossed that valley yet, mostly because we don't produce code that way either (flow instead of code). If we allowed that style of data flow, and a means of functional side-effect-free-validation waves hands vaguely, interop between different systems becomes an eventuality of the modeling.
The question is mostly... is the cpu-burn to detect the model that yields good results (ML-ish), cheaper than the human burn to write it on paper and agree. Present day we are ignoring the answer to that cost question for a variety of reasons...
Perhaps this is just a function that it's a marketing post designed to simultaneously appeal to different audiences while explaining the problem to decision makers that are unfamiliar with what problem they're trying to solve. I sympathize, but as a designer acutely familiar with problems around API discovery, the first half was an extremely cringey read.
Anyway, you quoted all the right sources (save for Tim Berners-Lee's Semantic Web and Giant Global Graph), and I wish you much luck, but I think you're aware that this was tried before [1][2], where much less human interaction and intervention was required, and it nonetheless faltered. "Complexity" was a scapegoat at the time, and I think that's an unsatisfactory, almost too convenient of an answer, so how do you avoid that same fate?
[1] https://en.wikipedia.org/wiki/Web_Services_Discovery#Univers... [2] https://en.wikipedia.org/wiki/Web_Services_Description_Langu...
Nothing in the article is new in the concept, but maybe™ the time is now right. Frankly, what the part I'm concerned about isn't the semantics sharing at runtime, but it's the de-coupled, declarative approach in writing the clients.
With hypermedia, we've failed at the gates of client development. The devs tend to tight-couple their code with APIs, ignoring the consequences. If there won't be an incentive on client's side, then nothing from the article will matter.
JSON-LD looks like a reimplementation of something that was already done by XML, XML Schemas and WSDL. If several technologies that were designed to be semantic failed at automating network API integration, why would you think a sub-format for JSON will succeed? What does it do differently?
To really solve the problem of service integration we need to rethink our approach to "services" altogether.
One solution that I see would involve a global registry of semantic symbols (e.g. "temperature", "location", "time") and a constraint solver. So yeah, distributed Prolog on the global scale. Systems would exchange constraints until they reach a mutually agreeable solution or fail. Then the derivation tree would be used to generate a suitable protocol. While I think this is possible, I don't think there is any real interest in stuff like this right now.
This is not a bullet proof concept. There can be infinite loops and it can have really bad performance in some cases. But at least it's something that would be able to do simple corrections and lookups automatically. (E.g. you need temperature in Celsius, but the server stores it Fahrenheit. This is not something that should force you to write code, because we know what temperature means and there are globally available conversions.)
...
Here is something related: https://fenix.tecnico.ulisboa.pt/downloadFile/395145629051/e...
Please, please... Write a blog post and tell us what you think. I want to hear it.
API providers are typically businesses or other actors whose interest is to lock the API clients nto their service. What would they gain by making their API interoperable with their competitors?
That is only viable for newcomers into the type of service and only works if they clone the API of an established player, not improve on it and standardize it.
They don't have to make it interoperable per se. It'd be enough to use some terms from a shared vocabulary (user, account, address) and then have some business-specific terms.
This way the business can use an existing library that knows how to handle user profiles. It's not that the full client has to be generic, a UI component that knows how to present a portion of a dictionary is enough.
I guess the availability (technical possibility of building) of interoperable API does not mean that it will be used in every case. It would be used in cases where there is an economical benefit and vice versa.
Reduced differentiation between underlying services drives them from product world into commodity world. Lower margins and stronger competition at that level certainly benefit the big players. Services in question, maybe not so much.
I think most developers who has had the displeasure of doing integrations with third parties want to make that process as smooth as possible for whomever is to integrate with their system.
Because it also makes it easier to use those API's, and for people to adopt theirs.
So you will find the better services adopting this first, because they can compete on having a better service.
Do they? I work for a company that does a fair share of M2M integrations, yet we almost never talk to the other organization, we merely use their APIs. I don't know what's typical, but our case certainly isn't rare.
The fundamental difficulty with APIs is that they force clients and servers to use the same domain model. Humans are necessary because only humans have the intelligence to reconcile differences in domain models.
The idea of inventing some kind of discovery language is just deferring the difficult work of reconciliation. Computers will need to be as good at induction as the human brain before it will be possible to eliminate humans from the process.
The Web grew strong because sites were interlinked. REST APIs are supposed to be generalized Web and yet they are missing the links! What Went Wrong?
I'm not sure REST APIs are supposed to be a generalized Web. In my view, REST APIs are simply a loose protocol on top of HTTP that allow computers running human-written code to communicate with one another. In general, there are several human-driven processes that must occur before REST APIs have any value. Example human involvement: Is this REST API valuable to my business? How much does this REST API cost to use? Does their domain model match ours enough to extract value from integrating with their REST API?
We could adopt a highly rigid language describing what it is that an API provides and what purposes that data is useful for, but that's restrictive and very brittle, especially against Silicon Valley's favorite activity of disrupting established ways of doing things. Like going from proofs in ZFC to proofs in first-order logic, we can make it more stable but only at the cost of losing lots of power and expressiveness.
If you can use a common vocabulary to access multiple APIs, that requires that all APIs implement the same feature set. Which means getting the API sources to agree on the features to implement, and how to describe them, and stop them from adding any features on that the others don't have. But of course, they'll all be motivated to add their own features, to distinguish themselves from their competition.
And once a API consumer is using a feature that other API producers don't support, then the consumer is locked into that producer, and the whole shared vocabulary is for naught. And of course the API consumers will be looking for additional features, because those translate into features that they can offer to their customers.
Basically, this requires API producers to work together to hobble their ability to meet their customers' needs, all to make it easier for their customers to drop them for a competing endpoint. So it looks like a net negative for everybody.
Usually the seller defines the API, but where the buyer is more powerful, the buyer sometimes does. See, for example, General Motors' purchasing system for suppliers. WalMart has something similar. There, the seller must adapt to the buyer's system.
There are a few systems where there are interchange standards good enough to allow new parties to communicate as peers without a new implementation. ARINC does this for the aviation industry.
We have yet to develop systems where both sides enter into communication and figure out how to talk. This is needed. XML schemas were supposed to help with that, but nobody used them that way.
- SOAP has service discovery and interface definitions.
- SOA has service discovery and interface definitions.
Some of these are like over 20 years old. They also included many other features. I would not describe this as being "the future".
It's still weird to me that people see APIs as a "thing", worthy of attention from business types and other nontechnical money-men, And how is it that a "good API" requires client modules for all these different languages? Do people not know how to make HTTP requests anymore or something?
(On the client note: "But it makes integration easier" or "I'm lazy, screw your docs, gimme [gems|pips|npms]" is the usual response I hear, though, given that each of these calls represents an external dependency requirement, maybe its good that they aren't as easy to use as the rest of your language?)
code reuse is why these clients exist, if you are a developer I can promise you that you are not issuing tcp requests by hand and instead prefer to use some standard library (curl) to handle that for you.
I look at a service like a plate of hors d'oeuvres across the room on a fancy table.
A client module (aka SDK) is the silver platter with the hors d'oeuvres neatly lined up for your selection, brought to you by the handsome waiter who is the only other person in the room who speaks your language.
One could walk to the table, and hardcore partygoers do, but lazy, entitled sacks like me prefer the delights to be hand-delivered. ;)
That is a complex question. For example, our company's ERP is a niche product specific to the printing industry. It has a SOAP API, and a WSDL so large that it will crush any SOAP library that is not doing code generation. We are one of only a handful of their customers who are NOT using their pre-generated SDKs because we do everything in Python and Elixir, and they only provide SDKs for Java, PHP, and .NET.
They rely upon the SDKs to provide API documentation, rather than the WSDL. The CRUD endpoints are consistent and work well - until they don't, and then you have to guess why a call failed due to some undocumented business rule. The process endpoints are a nightmare. Every call is documented as follows: Name, number of parameters. That's it. Parameters are all called IN0 through IN9. There is no documentation specify what the parameters are, which ones are required, what the data types are, or acceptable values/ranges. We have to dig through the SDK code (usually PHP) and make guesses.
To make matters worse, the state-side developers didn't write and don't maintain the API. It's offshored, and any question takes a couple weeks to get answered.
I realize that none of this is SOAP/WSDL's fault. But the fact that the vendor can get away with this shows how many shops really truly can't handle simple HTTP requests, and are hamstrung unless the vendor supplies an SDK. Otherwise the vendor could never get away with this.
As an aside: We ended up writing a REST microservice to translate JSON calls to SOAP for CRUD. (This was also necessary because we discovered that if we did not rate-limit the SOAP calls, the ERP choked). This dramatically simplified our integration code. But to this day, through we have repeatedly requested it, we cannot get full documentation on the process calls.
It is a scary thought but if we want to remove humans in this process we shouldn't even be able to understand the communication.
Thoughts?
Howdy,
This thread is making me consider dusting off a compiler that I wrote for a language that I created for designing APIs. That’s because I strongly agree that lack of versioning in many client/server architectures makes it difficult for devs to evolve their codebases. So, in this language I designed, the versioning of changes is a core concept.
When a server offers an API which can potentially deal with different types of clients, or with clients that need stability, then versioning is a must to have a chance for a sane codebase. Versioning allows the natural evolution of the API, while maintaining compatibility with existing clients.
Out of curiosity, if I were to bring the codebase up to date (C++), and make it downloadable, installable, and usable for free/open-source, maybe for Linux and Windows, would anyone be interested in contributing to a kickstarted for that?
regards,
Vlad
Think of layers in a convolutional neural network, for example. Each layer of neural units provides information to the next layer, but fixing the output of the higher layers limits the trainability and ultimate accuracy of the trained network. In order to maximize fitness, full backpropagation (or similar) is needed, with all layers being trained.
What's needed for self-negotiated API's is a generalization of the CNN model (or similar) into a variable-length serial communication format. Humans would define a fitness function either explicitly or implicitly by interacting with the system, and the self-negotiating API system would use some many-parameter optimization algorithm to alter both the Server and Client(s) and maximize the total fitness.
Since written code will only ever do a defined set of operations most web Apis are fine being written in RPC style.
The only time I've advocated a RESTful approach is when there was a lot of public data being exposed and human developers may well have explored the data.
When an AI can navigate an interpret data that it hasn't seen before then things will get interesting
Regarding APIs that understand each other, it might very well be too much in the direction of AGI.
That means that if we want to solve this we have to look at research that allow AIs to understand each other.
* Imitation learning
* Language grounding
The latter is also known more abstractly as the symbol grounding problem (https://www.wikiwand.com/en/Symbol_grounding_problem) and led to many debates in history. A collection of APIs seems useful, have them interact with each other - by getting the human out of the loop - might be a lofty but unattainable goal.
It seems like protocol buffers was made to solve a lot of the problems brought up here. GraphQL types and schemas seem to go a long way towards this as well.