Why We Chose API First Development at Our Startup | Hacker News Reader

Why We Chose API First Development at Our Startup

blog.pop.co

97 pointsjqueryin73 comments

I find the biggest win about API first development is it forces discipline upon you. I find I can get most of the benefits if I code in a disciplined fashion. It's usually not easy though.

In many ways, I find API first development most useful in languages such as python, which lacks the idea of an interface. Similarly, I find interfaces not much more than yet another tool that forces discipline upon you.

It seems like as we develop these abstractions which increasingly force more discipline upon us we forget that classes and functions are supposed to be abstractions themselves. We are supposed to be relying on the interface of those classes and functions, not the actual implementations, even if the class itself doesn't use the interface keyword.

One of the oft maligned issues of c++ is the .h file. I rather like them though. They're a constant reminder that "this is all you should know". It becomes obvious when you add cruft onto it because you don't have that cruft obscured by hundreds of lines of implementation. It's much more difficult to notice it in languages where the implementation of a class or module is present alongside its interface.

Yep I agree. APIs are often much easier to test as well. Proper testing is part of being disciplined.

In the past 6 months, at the urging of my friend Dave (sup!), who suggested this style of development, it's all I've done, even for small personal projects. I sort of take it further, eschewing frameworks somewhat, relying on libraries around actual "Business Logic" to deliver stuff as needed to outside clients.

It's been an eye opener. Sometimes it requires a little more upfront thought, but I end up with a much more maintainable project at the end of it. TDD is your new best friend if you go down this road, and the best part is: it's easier to do test driven development when things are nicely separated like this!

It's basically SRP writ large, and something tells me it's another one of those "ideas" that the grey-beards learned a long time ago that we are only re-learning now. Although REST does kick the crap out of SOAP...

I agree on all fronts here. I should note that we support all modern RESTful request methods for CRUD operations: GET, POST, PUT, and DELETE. Internally it simplifies our routing as each corresponds very well to the type of action we're performing.

Two problems:

1) Search engine indexing is broken with client-side rendering

2) Page takes longer to load, and so you will want some server-side rendering to take place (even if to seed your initial models, so they don't have to call back immediately).

You posted the same two problems in two different areas of the thread. As mentioned here (https://news.ycombinator.com/item?id=6763780), your first assertion is incorrect. Page loads do take longer, but you also win being able to support multiple frontends (consumer facing web, backend management web, mobile apps, desktop apps, other services, etc) much easier than if you had to maintain server-side rendering as well as an API. After the first page load, however, you're transferring far less data and will be loading pages much quicker. Sure, if you have a mostly static site, it's not worthwile, but if your app is highly personalized, this is a great direction to go.

Nobody said doing an API-based design requires you to (purely) client-side render. Lots of people are consuming APIs and rendering (initially) on the server, then consuming the same or similar APIs to do updates on the client. Best of both worlds -- this approach has neither downside you mention (even though the downsides are not as bad as you suggest even if you do pure client-side rendering).

>Nobody said doing an API-based design requires you to (purely) client-side render.

And if you go down that route, you pay for the increased deployment and maintenance complexity as well as the RTT to call the API service and consume the response as well as the CPU cost to render the page. There's no free lunch here. You can have both the API and rendering service on the same box, but now it's looking less and less clean and tidy, and more and more like a traditional web-app.

It's a tradeoff (isn't everything) but I don't think the cost is cleanliness/tidiness.

Your client (could be a web app consuming the api from the server) accesses the api like anyone else. You've created an api that others can build on - and you know that, because you're doing it yourself.

More to maintain. More cpu processing. Still a cleaner architecture though (compared to, say, having to maintain an external api for others to use while having a web app that talks directly to the db).

If this is a strain on deployment and maintenance I would take a close look at those two processes in your organization. We maintain 18+ services (5 people) and having our front-end web apps call APIs only (never direct db calls) has not hit performance, deployment or maintenance issues. In fact, it's helped in all those cases, especially the last two (for us, YMMV yadda yadda).

It's an opportunity to improve performance as the abstraction gives you a caching boundary which is appropriate to the application.

1) Search engine indexing is broken with client-side rendering

There is no rule stating that an API must be consumed client-side, and obviously a great many can't be (depending on how auth is set up, etc). For projects where SEO matters but where an API will also be useful (or if you've just gotten into the habit of building user-facing bits around an API) you can render everything server-side, or first loads server-side and subsequent loads client-side. Airbnb's Rendr (https://github.com/airbnb/rendr) is neat for this, although there are a million other ways to do it.

If you combine this approach with something like CouchDB, your API can basically just be your DB interface and you can focus on building the user-facing bits.

EDIT: Sorry, everyone wrote the same thing at the same time :)

>There is no rule stating that an API must be consumed client-side, and obviously a great many can't be

I wrote several responses to this but here's another one, if the server is serving rendered HTML to the client, how is that API-first development?

I'm not disputing the merits of this approach, I'm just wondering what we're arguing for here.

it's very possible to write code that can return responses in HTML or as JSON, XML etc depending on what the client asks for. Doing this allows you to concentrate on the API first but you can still add the presentation later.

We might be arguing about the meaning of words, but an API can be a lot of things. (I assume you know the things I'm about to say, and I don't mean any of it to come across as condescending--I'm legitimately trying to explain what people are arguing for when they talk about API-first development.)

All that an API needs to be (by definition) is a specification for how two pieces of software are meant to communicate. So yeah, technically the SQL interface to your database is an API--voila, API-first. But usually when people say API-first they are talking about an architecture that promotes reusability by considering that their web application might not be its only eventual consumer. That looks different depending on what you're building; if you're building Wikipedia "API-first" could mean that you're building a public API and then building a website around it (which you may still choose to render server-side for the reasons stated previously). If you're building an online game maybe "API-first" means you intend to have mobile clients consuming the API once you finish the web client. If you're building a business application, maybe "API first" is keeping your options open regarding future third-party integrations. In every case the controller logic has been encapsulated and there's an interface that could potentially be accessed by multiple consumers. (Specifically, in the case of a REST API it's an interface that can be accessed via HTTP, which is handy because then anything that speaks HTTP can communicate with your application.)

>how is that API-first development

We have several clients, the web site is one of the clients. The iOS app and the set-top-box are also clients. Figuring out the service boundary of your queries and events is part of the API development.

You develop a backend--a piece of software, running on your servers--with an API as its interface. Then you develop a frontend--another piece of software, running on your servers--that consumes that API, and has HTML as its interface. These are two separate projects.

1.) I'm using Prerender.io which targets bots and renders the page for them using PhantomJS - it's really neat! The only drawback is the response time will increase a bit for the bots, which may affect SEO.

Is there any reason you couldn't actually pre-render the pages and then serve cached versions to the bots? That almost feels like cheating though, as far as SEO goes.

You should be pre-rendering them, since a long wait to first byte negatively impacts SEO. Google suggests this https://developers.google.com/webmasters/ajax-crawling/docs/.... I recently launched http://crawlspa.com to offer this as a service.

I recently launched a service which addresses (1) [http://crawlspa.com]. An example page usually rendered with javascript: http://crawlspa.com/?_escaped_fragment_=dashboard.

I did this when I first started writing REST APIs. It felt natural and the "right" way to do it. However, it didn't scale well.

And I'm not talking about website scalability with traffic.

It eventually made development more difficult than it was worth. For example, doing things like enabling/disabling an object is a whole API call rather than a simple UPDATE statement. Either you have to set up an API endpoint to handle toggling the status of that object, or you have to send a whole (sparse) request just to flip a flag. Stuff like that can be tedious.

I love REST APIs and I loved what they have done for the web, but I've changed my mind over the last few months that this separation of design is always necessary.

The other mistake that I've seen people make is to try to build their application completely on top of a public API. Sometimes you want to do things in your application that, either for security or scaling reasons, you don't want someone consuming the public API to be able to do (bulk operations, administrative functions, cross-account updates, etc).

You either need private API functions or you need to allow actions that aren't triggered by an API call.

I didn't discuss this in the article, but I should note we do indeed have a fully private API that functions independent of the other and is utilized by our administrative management tools.

Our public API is a front end for our other services with auth added on and better guarantees against breaking. We don't depend on it directly for anything public facing and I wouldn't suggest anyone completely rely the same API they offer partners/openly to avoid getting into a tough spot with managing changes.

You're 100% correct that it's not the absolute quickest or easiest method for implementing what would be deemed a quick change. It is, however, a great way to reduce risk. It's my personal opinion that something even as simple as an UPDATE statement comes at a great cost on your production environment. It should be tested all the way up the chain of environments, backups should be issued on your database, etc, etc. It's really not easy unless you're whole-heartedly okay with the potential ramifications of "cowboy coding" so to speak. I myself have been bitten by performing live query UPDATEs/DELETEs very early on in my career, and API first development is designed to protect people like myself from bad things happening.

Yes, separating this logic is tedious, but it's core to the concept and it scales just fine.

If every time you want to enable/disable an object you simply write an SQL update query, if you ever want to add any other requirement when an enable/disable occurs (example: you want to record the reason why an object is enabled/disabled) you've now cornered yourself into a situation where you need to identify all the code where you've written the SQL update query.

A RESTful API makes a lot of sense when dealing with plain data, but starts making less sense when you're attempting handle full fledged use cases.

However, your core API logic should be separated from whatever network layer exposure you're giving your API (and ideally should handle endpoint negotiation for you)

A thin layer that maps methods to api requests should take care of this issue and be simple to write. This is exactly what database drivers and service sdk libs do, abstracting connection details into an easily consumable interface.

REST APIs are nice, right up to the point you need a transaction... three things need to happen in the correct order, and all succeed or are rolled back. If the API is too granular, then you end up pushing this logic to the client, and that turns into a disaster.

I'd personally rather go back and extract the smallest possible functional API, rather than build a massive API upfront. You don't really know where the transaction boundaries are as you are developing, and it is easy to expose an API that not really possible.

The way we handle this (we have 18 internal services) is that processes requiring transactions are formalized into a endpoint/resource. The actual transaction becomes tangible (e.g. you can cancel them by DELETE-ing on the resource) instead of just being documented. Only logic the client needs is the data to kick off the process. Async or long-running processes get a little tricker, but polling or webhook callbacks work fine for us.

A transaction should be the lowest level by definition e.g. it's not an API if you are just proxying CRUD on your db tables. Agree on smallest API possible (otherwise you will die in maintenance hell after you pick up a few clients) - but your API should be the only point of contact for your site and other clients.

Some interesting thoughts on doing financial transactions using a REST API here: http://jpos.org/blog/2013/11/w55y/

You missed another benefit towards API first development: it makes it much easier to get a head start on mobile development as you can reuse many of your endpoints for common operations.

You bring up a great point. I didn't cover mobile development specifically as we hadn't had a use case for a mobile app yet as the site is full responsive. I opted for more of a blanketed statement in the section on "Developer Liberation" as to how the API covers all client bases equally well.

I love this development and it's something I fight to try to get places that I work to adopt. Building your service through the eyes of a client gives so much clarity about both the what and the how. It pairs really nicely with TDD as well. You also end up getting an API for free out of it and have an easy transition to Service Oriented Architecture if you so choose.

I thought all modern web/mobile apps were architected this way? Is this novel information?

How do you handle authentication and sessions? Just curious on the approach you took.

We use CORS, which unfortunately is more of a headache than it's worth and lacks IE <10 support. Our backend proxies and handles session management via a secure, hashed cookie helper library which maintains expirys among other things. We take advantage of shared cookies on a global cookie domain . There's some ugly handling for IE10 regarding CORS and how it handles cookies which I won't get into. Authentication state is managed via the session, which we actually query our backend for on page requests. It's a bit of overhead to have to always be querying for the user/auth object to check login status, but it's easily cacheable (and bustable).

This approach is very attractive to developers because it makes things so neat and tidy and perfect, but there are two problems that you quickly run into:

1) Search engines can't index your page properly. If you rely on this to drive traffic, you'll have a problem

2) Your page takes a longer load (even more noticeable on mobile), so you will inevitably move some work back to server, whether to render some or all parts of the page, or seed your initial models with current values so that they don't have to immediately call back.

1. Not really true nowadays. Google can trawl Javascript sites. Other search engines may be able to as well.

2. Initial page load will be slightly higher (and subsequent pages will be much faster) but I can't imagine it'll be too noticeable. People get away with a lot - even on mobile - loading numerous JS libraries, tonnes of CSS, etc. As long as you go easy in other regards, this shouldn't be a problem.

There's no reason that API driven design has to involve client-side rendering. You can always consume your API on the backend and render on the server.

>There's no reason that API driven design has to involve client-side rendering. You can always consume your API on the backend and render on the server.

I'll give you that I suppose, though I understand it as a lightweight server, and heavier client. On the other hand, splitting up your components into API and page-rendering servers (in addition to your database, caching, messaging servers and your reverse proxies and load-balancers) complicates your deployment and maintenance and you pay for that - so it isn't quite the panacea and now you really have to decide whether it makes sense to go down that route, that early.

It's not really any more or less complicated than doing it on the client-side, especially if you use something like Her (https://github.com/remiprev/her). It's definitely more complicated than just using a database though - whichever way you do it. It's the other advantages that it brings that makes it potentially worth doing.

MVC manages it and is possibly the most common pattern in web dev.

1. Not really true nowadays. Google can trawl Javascript sites. Other search engines may be able to as well.

People have been saying this for a long time, but as of 6 months ago my experience didn't support it (Javascript sites get indexed, but not ranked as well as traditional sites). I haven't tried recently--does anyone have up-to-date information about the effect of client-rendered sites on SEO?

There's a guide here: https://developers.google.com/webmasters/ajax-crawling

If you use a client-side MVC framework, e.g. Ember, it should work out of the box.

I don't think it will, ideally you need to recognise _escaped_fragment_ requests at the server level and respond appropriately.

It would depend how you're serving the static pages. How did you do a fair comparison of a non-ajax/ajax site?

I should note that our logged in application utilizes the API whereas our frontend site is primarily static content for purposes of page speed and to lighten server load.

>whereas our frontend site is primarily static content for purposes of page speed and to lighten server load.

Yeah, that's absolutely the right way to do it. So it's less "API-first", and more "API-first-unless-we-want-it-fast". =)

A more accurate depiction for us is a separation of our public facing website, which is more for marketing, from our backend, logged in customer website, which is what I'd consider our actual application. A few endpoints carry over such as login, logout, forgot password, registration, and contact/support.

Are there some go to standards for API design that are followed?

I use the following sources as standards:

- Vinay Sahni's Best Practices for Designing a Pragmatic RESTful API: http://www.vinaysahni.com/best-practices-for-a-pragmatic-res...

- HN discussion on Vinay's post: https://news.ycombinator.com/item?id=5819231

- Twitter's Error Codes & Responses: https://dev.twitter.com/docs/error-codes-responses

Thanks these are great!

If I can throw my own in: http://john-sheehan.com/blog/dont-build-the-best-rest-api-bu...

Apigee's free ebook is pretty nice. http://pages.apigee.com/web-api-design-ebook.html

http://jsonapi.org/ is an emerging standard.

http://dev.billysbilling.com/blog/How-to-make-your-API-bette... is a great blog post on API design.

Coauthor of JSON API here, expect to hear lots of neat stuff soon. ;)

A new site was just launched to help with this: apicommons.org

My general advice is if it's internal, build what feels right to your team. If it's external, research the APIs you like or are popular (Twilio, Stripe, etc.) and don't be clever.

"Designing REST + JSON APIs" by Les Hazlewood is a good starting point:

http://www.stormpath.com/blog/designing-rest-json-apis

A "How We Developed POP.co Following the API Driven Development Approach" would find a lot of people quite interested too, I think.

Perfect timing!! I was looking for it. Do you know any opensource or free tool that can generate the commented code + API Documentation required, from the API spec?

Only found: http://apiary.io/ but it doesn't generate any code.

This merely sounds like they're adhering to the buzzword of the year 2008 - SOA. I'm I wrong? (Not that there's anything wrong in that, just nothing new)

It's a lighter weight form of the same idea. SOA had a lot of great benefits for its time. This revision of those ideas also has a lot of merit.

From what I've seen in enterprise java, SOA tends to just refer to internal services at a very low level of granularity - basically seems to be OOP with classes described as services. (This may well be a mangling of a good concept.)

I wonder if API first approach is suitable for all type of app. Is there anyone taking this approach for a CMS site, a eCommerce site or a stock trading site?

Great initiative, I personally prefer to refer to this as "Eat your own sh*t".

How do you handle authentication?

Vaguely answered here: https://news.ycombinator.com/item?id=6763221

I'll just leave this here:

CLIFMO: CLI's First, Maybe Only

life will be better. trust me. it's a beautiful thing.

(Also a big fan of RDD or README Driven Development. I didn't coin that term but I was doing it long before I finally heard somebody coin it. RDD and CLIFMO FTW. I have only a love/hate relationship for TDD, in comparison -- it's a mixed bag, advantages and disadvantages, awesome and terrible, depending -- but only pure love for RDD and CLIFMO.)

You, Sir, are a gentleman and a scholar. I'm a believer and I'll be using the CLIFMO and RDD acronyms from now on!

CLI = Command Line Interface? Can you expand upon this a bit more? I can't find any other references to this term.

Our entire software deployment system across all of our servers is built on systems built on systems built on a command-line tool I wrote in my first month.

It's really useful because anything that you can do from our deployment panel you can do by hand by logging into the server (and actually, even more stuff which isn't exposed).

The one thing I haven't gotten around to doing is revamping the tool to properly support JSON output for all commands. It might require a new logging module to handle errors though…

nice! that sounds like CLIFMO to me. yeah it's amazing how much easier it makes so many things. development-wise it yields so many freebies that pay off later.

Reply on news.ycombinator.com