Hacker News Reader

I disagree, those are all technical issues that are typically blocked by socio-technical and organizational problems.

Retrofitting an API is a context specific problem, with few things that generalize but the above aren't the hard parts, just the complex ones.

I am going to point you to a Public Policy book here, look at page 32 and their concept of 'wicked problems', which applies to tech but doesn't suffer the problems with the consulting industry co-opting as much as in the tech world.

https://library.oapen.org/bitstream/id/f3358d25-56c0-47f2-90...

Obviously most companies won't accept the Amazon style API edicts that require Externalizable interfaces and force a product mindset and an outside in view.

Obviously an API is still a "Cognitively complex problem" as defined by the above.

But what makes most API projects fail is the fact that aspects arise that result in ‘wickedness’ A.K.A intractability.

API gateways, WAFs, RESTful libs with exponential backoff with jitter....etc... all exist and work well and almost any system that you have a single DB you are concerned about killing is possible to just add a anti-corruption layer if you don't have too much code debt, complicated centralized orchestration etc....

But these general, but difficult tasks like adding an API almost never fail due to tech reasons, but due to politics, poor communication, focusing on tech and not users, unrealistic timelines, and turf battles.

You are correct that "APIs require thorough documentation or they're useless" but more importantly they need enforceable standards on contract ownership, communication.

The direction of control, stability, audience, and a dozen other factors effect the tradeoffs there as to what is appropriate.

Typically that control is dictated purely by politics and not focused on outcomes.

That is often what pushes these complex problems into failed initiatives, and obviously this is far more complicated. Often orgs can't deal with the very real uncertainty in these efforts and waste lots of project time on high effort, low value tasks like producing gantt charts that are so beyond the planning horizon that they can only result in bad outcomes at best.

The point being is that unless you are on the frontier of knowledge and technical capabilities, it is almost never the tech that causes these efforts to fail.

When I read OP’s list of wants for an API, I don’t read it so literally.

I read it as an example of the complexity required to implement something like an API.

I agree that it’s also a people problem but I disagree that many of these failed initiatives aren’t due to bad tech as well. A complex initiative requires picking the right tech that gives you a good benefit/cost ratio. Knowledge of tech plays hand in hand with the political aspects and IMO many orgs lack both completely.

> A complex initiative requires picking the right tech that gives you a good benefit/cost ratio.

That choice should never be a one way door, you don't control your customers or the future, it is the frame problem.

The idea is to be able to iterate and adapt, not get things perfect from the start.

It is fine and expected that one will use past experience as a starting point, but you always have to reflect and check your assumptions along the way.

It is a complicated topic ruled by nuances and contexts, but if you cannot pivot away from your initial assumptions over time, that lets you know you are leaking implemention details.

While time is limited and abstractions have a real cost, you need to apply them where they make sense.

Some, like just breaking code onto two files in the same directory are fairly low cost, others are much more expensive.

For several years I was jumping in to save failed cloud migrations.

Whenever tech was the 'blocker' it was because of a myth that there was only one way to do things.

But lets say you are writing machine code for Apple silicon? Do you really think those registers are concrete? They aren't. They are a facade hiding 100s behind a legacy interface.

Vendor mitigation important and often neglected and also results in people producing balls of mud.

But there are a lot of distributed monoliths pretending to be micro services out there. And we had decades of people producing fragile enterprise service buses that were built because people wanted to future proof their systems.

The sizing of components and balancing integration and disintegration drivers is incredibly hard, you will never get it right.

You have to leave options open, no matter if that is through abstractions or keeping biz logic centralized to assist in a easy rewrite of context boundaries that arise from scale or changing needs.

Obviously choices have benefits and costs, but for most needs you can keep options open, if not it is probably best to reevaluate the reason for choosing a solution.

Obviously there are predatory vendors like Oracle that base their entire income on captive customers, but that relates to the vendor mitigation above.

I think the US federal government de-risking guide is a good overview of that topic. Microsoft is a strong driver for this BTW.

https://guides.18f.gov/derisking-government-tech/

For anyone who wants to read more about what the Amazon experience was like, I highly recommend Steve Yegge's platform rant at https://gist.github.com/chitchcock/1281611.

It’s depressing how far Google seems to have fallen since then.

lol, I can already hear the responses of plenty of managers/management I worked with to these points.

> APIs need to be well-designed or the client may need to make multiple API calls when one should suffice. Or the API could be confusing and people will call it wrong or fail to use it at all.

That's fine. It's a first iteration we can change it later if people complain. Lets get something out there and iterate.

> APIs require authentication and authorization.

Lets not worry about authorization/permissions. It's just one key or whatever account they use to log in with now.

> APIs need handle data securely or you'll leak data you shouldn't, or allow modification you shouldn't.

You're saying you don't know how to do that?

> APIs need to be rate-limited or sloppy clients will hammer your API.

Either "Lets worry about that later when it happens" or "Here is the first github link from a google result for 'api rate limit open source free'. Lets use that"

> APIs require thorough documentation or they're useless.

Either "We need to have the API first, docs, clis, etc could come later after we have gauged the usage or had asks for them. We can handhold the customers asking for them for now." or "here is a github project that autogenerates docs, SKDs and clis from an OpenAPI spec"

> APIs need good error messages or users who are getting started will not know why their calls are failing.

You're saying you don't know how to do that?

"It doesn't have to be perfect. We have customers who are asking for it and to win them we need to implement something then we can work with them to improve it. Otherwise we will lose them"™

Yep, helps a lot when your management is in the domain.

Phased implementation isn’t the worst idea, they just have to be committed to ending the experiment or seeing it through.

Then again, these answers are valid. Your fancy API documentation will only be skimmed, nobody is going to notice the elaborate and elegant URI design or the discussion around whether to use PUT or PATCH, and the super-secure token mechanism you come up with will be written on a sheet of paper in the office.

Most projects are far too fluid in their shape to warrant a proper design up-front anyway.

They’re really not. That approach is how you get “why do I need this account ID to look it up in the API but the only way I can get it is by using the UI? Why can I update this object but not see its current state?” and other complaints that users very reasonably have about a thing I worked on.

Don’t get me wrong, I love to obsess over that kind of detail, but complaints like these usually aren’t a deal breaker, and yet it’s reasonable to count those complaining users as successful conversions regardless.

Recently I was working on a 12-year old project where the product people kept asking for "quick wins" and "leave the refactorings and in-depth analysis and difficult problems for later". When will they get to the "later", if 12 years was not enough? I eventually left.

Of course, YMMV. Every company and manager is different.

In every organization I've been in, "later" meant "never".

And there's an implied risk that once you have added an API for customers to exfiltrate data or integrate with their own system (or, worse, IMHO: a competitor's tool) then they'll never return to your site again

I'm cognizant that if it truly is a make-or-break for the deal, there may not be any choice, but along with all the risks you cited is an underlying obsolescence one

Every single one of these is true of a website, but we have far more websites than APIs.

If there's one thing APIs suffer from more, it's "social cost to make changes". A concurrent vN+1 largely resolves that though, unless your API consuming ecosystem is large enough to be worth investing vastly more resources into.

Is "let's just make a website?" a common thing people say? It sounds like the issue isn't with "make an API", it's with "just".