Don't build a general-purpose API to power your own front end
max.engineer
max.engineer
Long story short, if you are working on a personal project, please, consider the most dumb setup. With the vast options of super polished modern frameworks, it'll take you pretty far. A few more weekends and I'll prob be ready to go prod.
Edit: remembered something fun. I have a page that requires to poll the backend and then take some action. So I thought a bit of Ajax is fine. I opened the corresponding HTML file and started typing:
<script>
jQuery.ajax(...)
<meta http-equiv="refresh" content="5">This is not limited to personal projects. I can’t recall more than a single project I’ve worked on during the last decade where front-end code was really useful. Some cool stuff, ok, but never worth the pain.
I acknowledge it can be useful. For some real time project. Not for your crud-for-a-living.
I want to rewrite the GCP console with HTML, using their api and not one damn line of JS.
Some of these problems are because of frontend cruft (perhaps with the goal of nudging users towards APIs) to be sure. But plenty of them are due to the reality that the cloud providers' backends often do not, ironically, work very well at scale.
IT's such a joy to spin up servers and work in that UX compared to GCP and AWS !! Scaleway had a good one a few year back, but now every page feels like a "marketing flyer" that screams BUY-ME all the time :/
On a professional level, the requirements may mostly be for a basic CRUD app but it will have a few requirements for interactive or real-time features that are not possible with a static html crud app. The client is not going to want to hear your hacker news spiel about how JavaScript has ruined the internet. Now you’ve got to embed mini SPAs into your static html, and you’ve increased the complexity past what it would have been had you just used a SPA in the first place.
If you really need reactive global state i'd prefer to use MobX.
Your state is on the DB, those libs handle fetching and caching it.
Honorable mention, haven’t used it myself but there is RTK Query too, which is a library like those, but based on Redux, could be easier to debug.
For small projects where you want to prop drill is enough, fine. But we are talking about very small projects. Anything bigger and there is a significant drawback to that approach.
I do that too. A friend "stole" that idea from the tour of go, I really like it. Just go generate before building to make add static files, go build, done. 1 binary, it just works.
People usually think that it is either a modern SPA or jQuery spaghetti. Those are two extremes. If you put 10% the effort you were putting into building your SPA + API key into organizing better a "modern traditional" stack.... it can be wonderful.
But I knew doing it in Rails or Django would've been faster, but I haven't got enough experience with either.
Let's say I want to create my app in Rails (or django). I don't need it to be pixel perfect, but I do want some flexibility around the UI. ActiveRecod is fine. Can I just plug this into the admin and customize as I see fit? I know Django has a built-in Admin, does Rails?
FWIW, I have used Django (without the admin) and tried learning Rails several years ago but it was too overwhelming for me. Rack middleware, the ruby syntax, the conventions—all of it seemed over my head. But if I can learn redux, react, etc, I feel like I can do Rails too.
Do you have any recommendations on resources? I want to learn how to make something that looks really good, is convention-based, lets me plug in my custom code when needed, and lets me prototype fast as hell.
Hope you enjoy it!
Regarding the Django admin (in rails you have ActiveAdmin[1]) think of it just as a glorified database explorer. It is an internal tool for developers, product managers and maybe for your support team. It is in no way thought to be used by end users. Every attempt I've seen to use it as such was a catastrophic failure.
With Django, if you know plain HTML and CSS, with the tools I've mentioned in the comment you're responding to, you can build almost anything... For example, let's say you need a highly interactive client side table.... you can always just attach a Vue or a React component for it by using Unpoly compilers [2].
I'd say this stack is less useful the more your app needs to work fully offline... but if you don't have that constraint... I cannot think of anything that can't be built faster and safer. Just an example: Authentication is something very risky to do your self, and has ton of corner cases. In Django just plug django-allauth, configure a few settings and done! You have a rock solid battle tested well documented authentication system, which otherwise would take you months or years to get right (both featuer, and security wise). Check django-packages too [3].
And regarding learning resources, the official documentation is awesome. You have also popular books such as Two Scoops Of Django among others. And almost all video learning platforms have quite decent courses.
Like how do we efficiently receive, insert and query a gazillion rows of data. How can I use HTTP headers to my advantage to minimize data transfer in an a dynamic app. What is the best way to push data to multiple clients. How should I organize a user permissions system with users, roles, groups and inheritance and then connect that with resources dynamically (resources can be added and removed) but in the most efficient way (preferably with one query) or other complex problems that you can typically be faced with.
The implementation is then just a detail, because the problem is already solved in your head (or whiteboard), thus the language becomes irrelevant. Just pick a language where you, the programmer, can be as efficient as possible to transform the solution to code.
That is why I mostly code PHP, I don't get distracted with language nonsense, instead I focus on solutions to problems I solve my head and then type it down.
Code should be minimal but yet readable and understandable thus elegant, condensed to what you trying to solve, not the other extra fluff around it.
This also have the consequence that many of my solutions never gets typed down to code if there is no practical need for it at the moment, solving problems with your mind only can be as much as satisfactory as to coding. Remember we are engineers, we solve problems, programming languages are just tools we use, the problem and the solution will still remain the same regardless.
It may be too heavy or whatever but I easily build spa, pwa, electron from one place.
I forced myself not to look for the new shiny thing unil I really get to a point where Quasar and Vue are not enough.
I am an amateur dev and all users will have evergreen browsers and fast connections so plenty of concerns are moot.
What ends up happening though is that you have to build that general purpose API for mobile apps regardless, and right after that developers start using it to render their web app components. Rinse and repeat.
This is easier said than done, however, as you basically need to reinvent parts of HTML but worse (ex. attaching event handlers to UI elements, etc.)
Or, use GraphQL. Or a BFF for each client.
Why didn't you just come out and say "Use Hotwire!" or "Use LiveView!" ;)
What's even more frustrating is this is the rationale for moving to something like graphQL. We have engineers advocating for it because then it's "just one request per page" and it doesn't click that the framework we are using is pushing us into a less than favorable API design.
isn't this the point of graphql? You can write your front-end with easily generated typed responses.
It might not be faster than rails/html, but it's a hell of a lot easier to reuse, understand, secure, and audit.
Its possible, sure, but thats not the point. The point is the convention drives people to make entity-based endpoints and thats what I cannot stand. People are running the rails generator to build an entity and then when that doesnt scale we have to reconfigure the API. Overall not a fan of rails and the conventions it promotes.
To each their own. No framework will have the right conventions to see a complex project through 100% with zero customization or deviation. I'm still fond of Rails though I haven't used it in a while. Nothing I've encountered since -- Node, Sprint Boot, Scala+TwitterServer, various Kotlin libs/frameworks -- has been as productive or appealing for me as Rails. You don't know what you've got until it's gone, as they say.
The overhead of GraphQL vs Rest on a backend is around 30lines of code in my experience (developed 5 different production Graphql servers, including a very large one). Routes become resolvers, but that’s mostly the only change.
But in the end you get a much nicer API, GraphQL codegen, type safety, auto generated docs, etc...
Try out Basecamp sometime - watch the routes, they're all well bound to resources, yet it feels very much like an app. The progenitors of Rails are still doing things the old way and they're pretty good at it.
Let's just say it did not go well. Despite best intentions of being API-first with a SPA front-end, when you have a data-heavy and query-heavy application, it is absolutely the wrong choice ten times out of ten.
It leads to (not so) hilarious situations where the older, server-side rendered version of your app that uses jQuery absolutely demolishes your new hotness in performance. Try explaining that one with a straight face.
If you end up in this position I consider it a design failure - most data-heavy apps are probably better off with a command/query API.
Not really, it's a very valid solution when you have multiple clients (Web, Mobile, TV, etc) and multiple API calls in the backend. Netflix had a similar solution back in the day (not sure they still do) to solve the issue of clients calling multiple APIs.[1] Which isn't much different from calling multiple endpoints of the same API. When you have one client and know everyone else isn't going to use your API, it might be overkill. But if you have a very granulated API for general purpose (like, say, Twitter) but want your Web or Mobile clients to have a very narrow and specialized one, it makes sense.
I wouldn't consider this solution a failure, just the outcrop of a highly distributed backend system.
[1] - https://www.nginx.com/blog/building-microservices-using-an-a...
They are heavier applications not simple websites, and you pay for that in performance and often UX latency. The modern web feels slower than it did in 2010 in many situations. Latency and lazy developers not implementing affordances for when things are loading or the page is changing. Your app is probably slow for everyoelse, you need loading spinners! Especially if you are hijacking the browser navigation.
I personally don't think the performance trade off is worth it. And it drastically increases front-end complexity in terms of state managment, which you now have to also manage in the client. The complexity is just moved further away from the server and productivity is gained on the visual part but lost everywhere else.
You could argue, well, fix the API performance right?
That would be a valid suggestion. However, if the API is general purpose, what performance level is acceptable? Should it be able to perform tasks as fast as the average API consumer would expect, or should it be fast enough to serve the UI as well?
It creates a requirements problem in my opinion. If you were to have an API team, is this really on them, or is it on you for trying to use it in a way it wasn't necessarily built for?
This disconnect is why I don't believe a UI should ever be written against a generic API that is data or query heavy. It would involve too much coordination to get it right, which removes the advantage of having components and teams separated in this way (which is often done nowadays).
I've built and used APIs for full systems in all 3x: RPC, REST, GraphQL as both a creator and consumer and as far as I'm concerned everything else is dead
The one thing I disagree with is dismissing the idea "But we can reuse this API for the mobile app too!"
Depending on how you organization is structured, it can be common for the mobile app team to end up in need of APIs that aren't being delivered promptly.
Should that happen, having a web front-end that's entirely powered by APIs can level the playing field enormously - the website can no longer "cheat" and not bother with an API, which means the mobile team will get everything they need.
I truly start to think that organizations with silos in what you can code are just wrong and prone to this over engineering thing.
Just make your backend the shared space between your teams. There is no reason a qualified front end or mobile programmer could not at least write the controller for the endpoint they need.
That is well-served by GraphQL mutations and queries.
BFF pattern can be more approachable and reduce client code, however, and that's a plus.
Best. Pattern. Ever.
Edit: spelling
I don't like this concept of sending the page structure as JSON.
It would require all parts of the UI to just be stuck on "loading" while the back-end essentially the entire page.
If you decide to turn this into "ok well make it /page/a/component/3" then we're back to square one on the whole idea.
You application API is churny, specific and tuned for certain screens and user interfaces.
Your general data API is, well, general, rate limited, concerned with limiting the ability of that expressive power to damage your system, etc.
https://intercoolerjs.org/2016/01/18/rescuing-rest.html
One you get over the hump of splitting your data and app APIs, the next step is to realize that your application API can be a hypermedia API rather than a dumb JSON API, and you are off to the races:
In practice this ends up building out a reasonable approximation of what a "public" api would be. Eg, your WidgetList component forces out a /widgets endpoint, which might get re-used by some other widget too. That's fine. The point is you're still working UI-first and making the minimum viable backend.
With lots of components on a page, you might end up making multiple calls for the same information. That's also fine. You can optimize that later if it becomes a real problem.
Locally, it might all go super fast, but as soon as you deploy it, that dashboard calling 30 endpoints is going to feel insanely slow just waiting for the network to become free for use.
If there's a lot of redundancy in the requests, you can solve that in the request layer. Each request from a component doesn't need to become a separate network request. But if it's all unrelated data... maybe you have an IA problem.
(edit) What could go wrong? https://www.youtube.com/watch?v=y8OnoxKotPQ
Erm, even with that solution you still have to consider the changes that impact this schema. These problems didn't go away, they simply became masked differently.
Ended up working out okay, and the idea of each pages gets a service endpoint was a bit weird at first, but really gave it the flexibility to avoid needing to go touching the legacy layer very often.
On the backend, for performance we have two choices:
1. Over-fetch. It's relatively cheap doing that from a cache. Most queries don't use too many different variations.
2. Optimize what data we fetch based on the node's children in the GraphQL query received. I don't think people do this often enough but... GraphQL gives you the full query as an AST, so you can use that at runtime to know what your node's children will be rendering before they're hit. Because we enforce #1, we don't have to build some general-purpose query-builder a simple check of "Are you X named query?" is good enough. Internally it looks kind of like JSON RPC.
Am I doing it wrong? It seems to give you the best of both worlds because GraphQL APIs have great tooling and documentation behind them, and if you cut out all the 'general' purpose aspects in production you can be fairly efficient (e.g. our GraphQL API is about 5% slower than the JSON API it replaced).
The big win with GraphQL comes from the client-side tooling. You can have a React page with 100 components that each request their own data, and have it automatically batched up into a single request. Your page no longer has to predict what data the components will need. When a junior front end dev tries to reuse a component from page A in page B, the page B query will automatically be updated to fetch the data required for that component.
For everything else it is needless complexity and an additional failure mode. The tooling and operational aspects aren’t as mature as a conventional HTTP API (yet?) so I’ve found it to be a higher cost for limited gain.
I wish we could standardise on something that feels more 'native', have RESTful (the tangent w.r.t. Fielding, as it's actually practiced) JSON responses , JSONSchema or OpenAPI even, as popular widely implemented IETF RFCs.
Or the same with GQL, but with corresponding changes to HTTP to make it make more sense. Have a response be a type consisting of nullable success data and error data perhaps, or more layered status codes as rigid as they are but allowing what GQL wants to express by 'here is a successful response that contains errors'.
GQL has little choice, it can either call the whole thing an HTTP error, or, to be honest it probably does the right thing, it can call it a success and describe the error in the response data. In an hierarchical model of your choice, that's genuinely an HTTP-level success.
In the sort of respecifying I'm describing, you could expand status codes to describe 'mixed' results, or 'upstream error's, or whatever. I don't really have great suggestions ready because my only point is that I think the status quo is bad.
I think that's only true for relatively small internal apps. I work on an internal app with a few hundred thousand lines of JS, and there are way too many different permutations of different types of pages to write an API catered to each use case. GraphQL has been fantastic.
I'm sure there are others with the opposite experience but I can vouch both for the desire and expectation of devs to build general-purpose APIs when not needed, and the ongoing pain resulting from that year after year.
This. I have spent countless hours pulling my hair trying to understand backend business logic only to see part of being implemented in the front end.
What's more, the supposed generic backend API makes quite a lot of assumptions about the front end orchestration, so the API can be used only in conjunction with the front end it is serving.
Now not only is your backend API not reusable but also the business logic is brain split between the front end and backend.
> I wish. It’s pretty hard to measure these kinds of things in our industry. Who’s gonna maintain 2 architectures for the same software for 3 years, and compare productivity between them? All I got is a mixed bag of personal experiences. Feels inductively justifiable.
Oh wow this is sort of fascinating. Maybe we could crowd source experiments like this. Like maintain some random open source app with two different backend structures for X years and blog about it, share battle scars.
Did I miss the whole point?
I'll go even further on one point: "Imagine if you could just send it the whole “page” worth of JSON" => why do you even send it as a separate endpoint? Embed it in your page! It works great with a SPA: the first load use embedded JSON, and next partial data refreshes hit endpoints returning JSON with the same format.
Imagine trying to make sense of:
merge(pages.a.section.main.data, pages.a.section.secondary.data)
What does that even mean? It’s going to get hard to prep abstract data.
I'd say, it can make sense. But one big issue is how it muddies the line of where certain frontend decisions should be made. For example; if you have a person's profile picture, and you want to round it, should that be a property on the JSON structure? E.g.
{ "profilePicture": { "url": "https://mycoolpicture.com/pic.jpg", "rounded": true } }
Or a property of how the frontend transforms that structure into HTML? In other words, its an opaque, intrinsic property of the "profilePicture" "JSON-component", every profile picture is rounded, because that's just how profile pictures are.
Until you hit the one page that wants one that isn't. No problem, maybe a generic "picture" "JSON-component". Or you hit the page that needs a squircle. Ok maybe we do need a "border" field, and why not just make its value the same as the CSS `border` property and oh my god are we just reinventing HTML?
There's no right answer to this question, and it will appear in literally every single component you build. Maybe its text weight: How generic should it be? Very-generic: "title". Kinda-generic: "heavy". Or literally just CSS.
What do these styles mean when faced with user browser preferences, such as dark mode or accessibility systems? By the API contract, "cardTitle" means "roboto, 16px font, #0000ff font, whatever"; but not always, right? Does the frontend disobey the contract? Do you send these preferences to the backend and let it handle it? Does the contract allow for lee-way in how its responses are interpreted? How much lee-way?
Lets say I want a sidebar. That sidebar has six items, so we list them. But what happens when the site is displayed on a phone? Well, for the sake of just providing an answer to continue this line of thinking: the designers want it to become a bottom bar. No problem, except, uh, the API says it should be a sidebar and we're not really communicating how big the screen is to the API are we? Well, maybe we are, sure, we should have considered that during the v1. Is the server really the best source of truth on how many items a 976px wide screen can hold? Four maybe I guess? Maybe the frontend just sees `sidebar`, interprets it to "understand" "implicitly" that it means "bottom bar" on small screens, and do the best it can, and now the entire point of this exercise is out the window and we're not obeying the API.
Another problem is during visual redesigns. This may demand the recreation of your entire API surface; you just doubled the work of a visual redesign. If the data model were generic, your backend team may not have even had to have been consulted.
Another is in interaction. Button which opens a new tab to google.com; easy, no problem. But in every reincarnation of this idea, I've never seen a strong argument for how to handle even a basic form, with a button to submit the input to another API endpoint. Do you have something like
{ "form": [ { "formTextField": { "id": "firstName" } } ... { "formSubmitButton": { "endpoint": "/createUser", "verb": "POST", "arguments": { "id": "firstName" } ... } } ] }
How do you handle the response from /createUser? What if there's an error? What if one page that calls /createUser needs to display the error in a snackbar, but another page needs to display it in-line with the button? Do you just... not do that? Just throw every error into a snackbar? Ok, are you capable of updating the local cache with the response from createUser, such that another call to the page-rendering API isn't necessary?
This doesn't feel as good to me as the rest of this line of thinking, which is already not great. It feels like "we had this really cool idea to render our entire frontend in JSON, oh crap we need to support interaction patterns other than queries, uh, how do we shoehorn that into our cool idea, ok, hold my beer this is cute."
Look; there's a reason why data and views are separate. This idea isn't new. I hope you realize that, but I'm afraid you don't; our industry does tend to revolve in cycles, but this is one that shouldn't come back.
There are really specific use-cases for something like this which are actually powerful, from my point of view. The example from our app is, best put: imagine a google search results page. No interaction except simple hrefs. Maybe you have a ton of "Link" { "title" "subtitle" "body" } cards, maybe a "MapResult": { "latitude" "longitude" }, basically displaying a list of cards which may have different content. The frontend needs to know how to render each card kind, but the specifics of how its rendered are kept presentation independent; nothing is asserted about structure, just content and relationships between the content. It can work for that. It still has problems, especially as the business wants more and more things shoved into this model that really wasn't built for it, but it can work.
I know its a meme, or the buzzword of the day, or whatever, but: GraphQL is actually really good at solving the problems you think this solves. It doesn't do it for free. You have to think about N+1s and composite queries and performance around those and designing a good schema. But, from the perspective of just the API language you're talking, its actually pretty good. And if you're really having performance problems on a page, you can always amp up your API caching, or introduce page-specific queries which collate data on the backend into a small number of database queries, or whatever.
You can decide what to leave to frontend on a case-by-case basis. I would say unless people can choose if a picture is rounded and it's saved in database, this is valid to leave up to frontend.
> Maybe its text weight: How generic should it be? Very-generic: "title". Kinda-generic: "heavy". Or literally just CSS.
If text has sections that are formatted differently, backend could provide content for those sections separately. Styles are up to frontend.
> Does the contract allow for lee-way in how its responses are interpreted? How much lee-way?
Structure and content on the backend, presentation on the front-end.
> designers want it to become a bottom bar. No problem, except, uh, the API says it should be a sidebar
Call it something a little more encompassing, like "secondaryNavBar".
> you just doubled the work of a visual redesign. If the data model were generic, your backend team may not have even had to have been consulted.
There's a fair criticism in here, however the work is not doubled. 1. You can make some redesigns without changing field names, then slowly align JSON keys. The damage from this concession is contained in each page individually. That said, updating the JSON keys as you go should not be much more work than the redesign itself. 2. Should we be optimizing for rare redesign when instead we could optimize for ongoing maintenance? 3. How likely is it that redesign doesn't need anything new from backend anyway? 4. How likely is it that redesign won't introduce new N+1 problems?
> Do you have something like { "form": [ { "formTextField": { "id": "firstName" } } ... { "formSubmitButton": { "endpoint": "/createUser", "verb": "POST", "arguments": { "id": "firstName" } ... } } ] }
To render the form, you don't need to send the form to the frontend. Frontend could render the form by itself. You just just send it any pre-filled values. Form's action/method are also okay to include.
> What if one page that calls /createUser needs to display the error in a snackbar, but another page needs to display it in-line with the button? Do you just... not do that? Just throw every error into a snackbar?
Just because you mostly render full pages, doesn't mean it's illegal to have endpoints that give you snippets of data in response. Your createUser endpoint can respond with just a list of errors. It wouldn't be bad for mutating endpoints to do that, the benefits of this approach still remain.
I've addressed the points you've made, but seems as though they allowed you to build up a strawman, and the rest of the comment is knocking down that strawman. That's not what's being proposed here. It's okay, maybe you are arguing against specifically what you have at your work. I could help adjust your framing if you'd like, let me know.
I would defend myself by saying that it feels like the opposite is also true. I built up a strawman and struck it down, but the recommendations in the original blog post, and here, are defined in extremely abstract terms, then defended with "you can do it however you want, there's no rules" such that every team who implements it will do it differently, and inevitably most will spend years landing on ten bad ways of doing it, never reaching the nirvana of what was promised in the abstraction.
I do like the idea; but in a limited capacity, and I'd caution teams from planning an entire application around it. I'd stick to fragments of pages which are highly data-driven, and be more cautious around layout and structure.
1. All structure and content goes to backend. 2. Find good names for size-adaptive sections. 3. Responses to write requests can be anything.
I think 1 and 3 are covered in the article, but elaborating on them is probably out of its scope. And 2 is more of a general programming advice.