Go in Production – Lessons Learned
tdom.dev
tdom.dev
Frameworks like Echo and Gin eschew the universal middleware signature `func (http.Handler) http.Handler` for their custom built ones, and as a result cannot tap into the ecosystem of middlewares that target Go's net/http. Between a stdlib package and a third party package, which do you think is more stable?
You are not constrained to net/http's router. Chi is a good example of a third party router that implements features not in net/http but still conforms to the `func(http.Handler) http.Handler` interface, and as a result middlewares that work for net/http will also work for Chi.
I think you're overstating the cost of using "proprietary" middlewares. I'm most familiar with Gin; it is trivial to wrap a "universal" middleware in a Gin middleware (gin.WrapH). It is not trivial to make a universal middleware that does the equivalent of gin.Context#AbortWithError.
I don't fully disagree either - obviously `gin.WrapH` everywhere is noise, but so is e.g. `chi.URLParam(req, "abc")` compared to `c.Param("abc")` in Gin, and Gin's parameters are much cheaper for a middleware to tweak. (I choose chi here because it's one I'm familiar with that tries hard to keep the standard Handler signature at the expense of interfaces to its own features.)
A lot of problems would go away if the standard signature was `func (Context, ResponseWriter, Request) error`. Sometimes a minimal universal option is nice because minimalism is also a virtue; but also sometimes it's just missing necessary features.
I helped many teams move from Java to Go, and came across this so many times. Everyone wanted to find the Spring equivalent for Go (or Django, Ruby on Rails, etc) when it doesn’t actually exist. That’s by design.
No ill feelings toward the developers of Echo, or other similar frameworks (Go Micro specifically). However, touting their framework as the way to build APIs in Go is disingenuous and leads to software that is difficult to maintain.
Just my two cents.
I'm not sure how it's "by design". Go, by itself, doesn't give much more than e.g. Java gives with its Servlet base libs, etc. And yet Java has Spring, and some popular framework could very well emerge as the "THE" framework for Go.
Whether that framework would conform to the stdlib interfaces / middleware is another question. It might not, if it's compelling enough (there are lots of popular Java non-servlet API conforming frameworks, e.g. Play, Vert.x, etc).
It's only because of fragmentation (and small still market, with lots of NIH) that one hasn't emerged as such, not some special Go design. Many/most Go-ers like to keep it simple, so they don't adopt any big framework lib (after all, if they didn't keep it simple, they wouldn't be using Go).
For a very good perspective on this check out this excellent article by the creator of Stanza language responding to his friend's request of not creating any more new languages but just libraries:
https://jaxenter.com/stop-designing-languages-write-librarie...
Coming from WSGI, I found go's middlwares familiar and positively pleasant.
I too ended up on Chi as the router that worked well without getting in the way.
Yeah, I'm not much into Go development but I've talked to a few devs who built real web apps with it and they rarely use any libraries, but do use "interface style" libs like Gorilla Mux.
Here's a snippet from my podcast where I talked to Jon Calhoun on using mostly the standard library and about 15k lines of code to build a video course platform in Go: https://runninginproduction.com/podcast/42-creating-a-video-...
Gorilla is perhaps the most maximalist Go web library "system", and its mux is by far the most complex of common routers. Using Gorilla for a site is about as far as being outside the Go stdlib as you can be before reimplementing core protocol handling.
Youll find that utilizing http Round Tripper along with the Handler interface in the http library will make middleware easy.
Youll eventually dig into filepath and path methods for extracting path parameters from url paths.
And logging and recovery will be a concept youll need to extend outside of just http and into the rest of your application.
I totally advise new comers to use a web framework, but in the long run, you probably wont want too.
The standard library is very low level. Want sessions? DIY. Want user auth? DIY. Want CSRF protection? DIY. The list goes on.
It feels like a waste of time implementing these "solved problems" from scratch, but the biggest problem is how easy it is to introduce security vulnerabilities when implementing from scratch, or forgetting to do so.
It’s nice to learn concepts from first principals by using the standard library. But once I know how these things work, I’d rather rely on someone else’s battle tested code and best practices.
Yes, you can add in separate libraries to solve these specific problems, but they are less likely to compose as well as they would in a framework. On top of this, each time you pull in a new library you have to spend time evaluating it. When I use a framework I don't have to think.
This isn't even a new idea. Many Ruby on Rails plugins are actually Rack plugins (even to the point of Rails itself being implemented as a collection of Rack middleware). Rack is the interface that defines how a request is to handled, similar to the Go stdlib interface.
It's definitely true that idiomatic Go tends towards copying being better than dependencies, but the standard interfaces make it much easier to use and swap tried and tested dependencies because they all share the same interface.
Also, here's a list of out-of-the-box library implementations for all the features you mentioned:
Sessions: https://github.com/gorilla/sessions
CSRF: https://github.com/gorilla/csrf
User auth: https://github.com/qor/auth
That's not the sentiment expressed here. To implement these things from scratch is not the only alternative to using a framework.
I also advice any individuals coming from Django or friends to use a web framework.
But after some time spent with the stdlib, and understanding how some of those implementations work, it gets to the point where Id rather not read another set of documentation, learn a new mental model, and deal with bugs. This all comes with a framework.
After awhile you realize that the stdlib provides most of what you need, and that writing more vanilla Go can be simpler then learning a full framework.
I will admit, most my work in Go revolves around internal services, and dont deal with web technologies such as CSRF and CORS. So I do acknowledge my opinion here is leaned toward those use cases.
What you say is true, it's easy to do with stdlib, but it's neither enjoyable not readable. A router lib and a validation lib are realistically still needed.
The same applies for the sql package, it's basically unusable without at least something like sqlx.
This Go enlightment seems to only touch those that don't realize that programming to interfaces was already a thing back in Objective-C and WebObject days, or using Smalltalk categories (later formalized as traits in Pharo).
Also that languages like Java and .NET also have a similar Http server on their standard library since Java 6 (2006) and .NET 2.0 (2002).
The reason we don't use them beyond toy examples, is that they don't scale when things start getting hard and something like IIS, ngix or similar is called into action.
- workload distribution - security and authentication - role management via some form of directory services like LDAP or AD - running tasks in background in response for certain events - mapping into various kinds of databases and information sources - handling caching - have a way to manage reusable components of html/css/js + respective backend code
Yep one can make use of libraries to achieve all of that, but then most likely they don't compose in an easy way, nor make an eco-system.
Hmm? You seem to be mixing two things here. Nobody is suggesting not using NGINX/Caddy etc. The comment was about being able to go very far with just the standard library.
This leads to go web frameworks being sort of semi-hard mountainous turds of code generation. You’ll want to stick to stdlib after sifting through them and deciding they aren’t worth it.
The rest though I do not miss at all. Middleware patching attributes directly into the request object? Python's sad excuse for incremental typing? Ignoring the database bits of Django I struggle to find anything that I don't enjoy doing more in Go. I was a HUGE fan of Django REST Framework and assorted boilerplate for a long time, but I am so extremely glad to be off that ride. Yeah, it lets you do really cool, complex things succinctly and cleverly. The problem is that it lets you do really cool, complex things succinctly and cleverly. The cleverness becomes the enemy. I now have learned to appreciate code that is utterly stupid, obvious, nearly braindead. Every codepath is screaming at you. That is Go in a nutshell. if err != nil { ... } ad nauseam. Sounds terrible... but it kind of isn't.
There are some things I would not use Go for. Game development is one of those things. Web servers, though? If you are going to be doing serious work in production environments, Go is absolutely among the best choices.
I am still, however, looking forward to trying Rust more and more. My initial impressions with Rocket.rs have been lukewarm. (One thing that is bothersome but not quite a deal breaker is compile times. A lot of crates make the experience bad almost immediately.)
I thought RoundTripper was a purely client interface - am I missing something? Servers instead have the base Handler, net.Listeners, ConnState, BaseContext, ConnContext etc.
There's a well-trodden path of developers who were trained in PHP/Rails/JS/Django/etc starting on Go. Their first question is always "what framework should I use?" (you can see this being asked at least once a week on r/golang). They then go down that path, and find (what they think to be) Go to be clunky and boring and hard to think about. There's a minority that decide that what Go needs is another framework, because the one they tried obviously isn't working for them.
Then there's a split, and some people write articles like this one, and probably move on to Rust or whatever language is next for them. The others start understanding. They refactor their code to not use that logging library because they start to understand the understated power of the stdlib's logging functions. They refactor to get rid of the ORM that's become a problem. Slow realisation dawns and they finally get rid of the framework and go back to writing http.Handlers. This process usually takes a couple of years (well, it did for me anyway).
And then our reborn Gopher goes to the forums to spread the light: "all you need is the standard library! You don't need a framework!", and is met by derision and misunderstanding. The Go community gets a reputation for being unjustifiably anti-frameworks and a bit weird about ORMs too. Eventually our hero shrugs their shoulders and lets the newbies find their own path, contenting themselves with upvoting those who understand.
There has to be a better way to communicate your point.
I wrote a whole logging library, and a "better" database access library. I even put them up on Github and asked for feedback in the Go google group. Mostly I got told "you don't need either of these", which I ignored. Until I finally came to understand.
It's a well-trodden path. I'm not the first to walk it, and I see others starting on it now.
I rarely find myself straying far from the standard library. If there's something that takes a bit more code but doesn't bring a dependency then that's what I'll do.
I definitely agree that there comes a day where you just see the light and realise that simple is good, the standard library is good, and that you don't need most of the stuff a framework gives you. But it gets really hard convincing others that Go is a great language because it's not Rust. Don't know what there's so much hate for it in this community. I've never fell in love with a language like this before. It's just incredibly productive for me and fits my thinking perfectly.
Frameworks need to die.
I found it pretty useful, the tip about sqlx, echo, and a quick docker file example were useful reminders, the rest ive seen elsewhere but yea.
I get what your saying but I found this useful.
1. You probably do NOT need a framework
Use the default Go HTTP libraries. For the other functionality that you'll need, use libraries. If you throw in with frameworks like Labstack Echo, you're forever coupled to the incredibly specific and one-note behavior of the framework you choose. The dependencies you choose should be light, with their most attractive aspect being the interfaces they provide. I point most strongly to go-kit as an EXCELLENT set of libraries for writing HTTP services. Their Log package is a small example of what I look for in quality libraries.
2. You NEED a good code structure
You need to be structuring your code, yes. But you should NOT be leaning on the conventional directory structure to give meaning. You will need to read more of other peoples code than you will need to write your own code, and other people will not be following your code structure. Much more useful is to get good tools and practices reading code. Go, unlike other languages (e.g. C#), is meant to be readable and understandable without a heavy IDE there to help resolve elaborate indirection and overloaded imports. If you have the code on your disk and can use Grep, you'll do fine. If you're using an editor which supports language servers like Gopls, you'll be able to fly through the codebases.
3. Pick a DB driver wisely, and the wisest choice for SQL driver is database/sql
SQLx is not database/sql, which is a real problem when most all the ecosystem is built around database/sql. The only real pain point people have with database/sql is the scanning, which is why other people have built libraries to help with this: https://github.com/kisielk/sqlstruct
Probably just use database/sql.
4. Docker
Most all of this advice is fine.
Basically, I recommend sticking to the more lightweight and more standard implementations, as they have the abstractions you'll need to the long haul. Though if you're just trying to get a thing going ASAP and it has to be Go, do what you gotta do with the code that catches you're fancy.
Any examples of good go web projects to look at? All of the ones I've found have fallen into either "directory dump" or "I'm an mvc but not really".
If this wasn't on the front page of HN I'd have stopped reading here, but it is, so I didn't, then I regretted it.
FWIW, I, who is usually writing JVM apps in Java/Kotlin occasionally Scala, had an app to write that that extensively used K8s APIs, and I found that the best client API was (naturally, considering the makeup of the K8s ecosystem) the Go one, so I wrote the app in Go.
I had to overcome some common Go issues (use a map[t]interface{} when you need "is x in a list of unique elements", if you need more advanced set operations on custom types, you're going to need to generate code or hard code each implementation, because generics are for suckers, allegedly. Hit some issues around channels and blocking reads and buffering, but worked around them easily enough using pointers and locks where channels weren't appropriate)
But aside from the limitations of Golang, over all it was a rather straight-forward experience. Stuff worked, tools worked, Intellij IDEA integration worked, Go's module system worked (DIAF $GOPATH), code ran, no worries.
Will say though, Go logging realllllly needs some work from the community. An SLF4J equivalent at least. I used logrus as my logging library as it gave me the easiest route to making my logs go to Kafka in Logstash format, but it has some insane defaults - like the log event timestamp defaulting to "seconds since start-up".
Oh, and be very careful with named returns. Naming a return, but then assigning nothing to it, will compile. Which is, to say the least, rather contrary to how most of Go works.
This, I believe, comes down to Go having sane default values i.e. if you name a return of type string and assign nothing to it, you'd get "", which is a valid string value and something you can check against.
But I agree it should at least be a lint warning.
The other one is http status codes are http status codes. As in the http request was done correctly, but the application code wasn't. More specifically, http layer was executed successfully, but the application layer was not.
No, this pattern is not the right one for REST. Don't call it REST, call it command/action RPC. The 90s called - they want their architecture back.
It's pretty infuriating actually. On a similar note I have to use certain command line tools provided by a third party vendor that exits 0 on failure, and writes something to STDERR (on success it exits 0 and writes something to STDOUT). The Unix conventions evolved over decades because they were consistent and useful.
But there's an rfc and that's what defines correct usage.
I used to use HTTP status codes in this way because I understood it was the correct REST way of doing things.
However, one day, a sysadmin contacted me to tell me that we had broken a release because our API was returning a 404. Actually it was a problem in the checking script that was checking for data that was no longer in the DB.
By making application codes equal to HTTP status codes, we had removed any way to distinguish between fatal errors and API results.
There was no server-side error. The requested resource was unable to be located because it no longer existed. You should return a 404 here, and not just for non-HTML API clients.
Application-side fatal errors are in the 500 block. So yes, you absolutely can distinguish this case.
I should have mentioned that the API was behind a reverse proxy. This leads to the following questions:
Which requested resource is missing? The API itself or the item requested from the API? How does a client distinguish between these?
If the API itself was not found that’s one of 502, 503 or 504.
200 responses with an error property is an antipattern precisely because of things like reverse proxies. They have no way to unpack and interpret every developers pet error format. They _do_ understand http status codes and can act accordingly, like retrying requests, not caching responses, etc as appropriate.
What if the configuration changes and the API path/URL is no longer in the reverse proxy config? What if the reverse proxy is dynamically configured and our application didn't register itself properly?
They _do_ understand http status codes and can act accordingly, like retrying requests, not caching responses, etc as appropriate.
Of course applications should return HTTP codes when appropriate e.g 500. The principle is that using HTTP codes for application specific information e.g item not found in DB, is a blunt instrument.
They have no way to unpack and interpret every developers pet error format.
I would argue that they don't need to. The conversation is between client and API at higher level than HTTP. HTTP codes are great for information such as "its broken", "please authenticate", "its busy". But HTTP codes are not so useful for things such as "item not in db", "parameter x is missing", "unsupported API version" because this information is nothing to do with HTTP.
2. You can return a proper HTTP status and a JSON response body detailing what happened.
GraphQL defines the format of the reponse in case of errors[1]
GraphQL doesn't use HTTP status codes to communicate out-of-the-ordinary conditions. You can expect to always get HTTP status 200[1]
The data response is a mirror of the query you sent in with the data present[2]
How to query for data is explicitely laid out[3]
How to send in parameters is explicitely laid out[4]
[1] https://graphql.org/learn/serving-over-http/ [2] https://graphql.org/learn/ [3] https://graphql.org/learn/queries/#fields [4] https://graphql.org/learn/queries/#variables
We do have nice wrappers around the request to raise a proper named exception regardless so it doesn’t matter.
Edit: actually you made me think a little bit more about this, if you can make your mutations idempotent, spurious retried POST requests shouldn't be a problem at all. However "delete the last record" is not an idempotent operation by definition but also one you wouldn't use in the real world - usually you delete by ID.
Edit 2: it's easy to make the server reject mutations sent via GET.
For graphql specifically, if you allow GET-ing the GraphQL endpoint (which usually isn't the case by default), it's trivial to ensure only queries go through that method.
how to cache data (POST is not cacheable) -> You can use GET requests and GET requests are cacheable.
how to auth data (anyone has access to everything) -> Authentication or authorization? What do you mean with anyone has access to everything?
how to... -> yes?
GET requests are a crutch added to GraphQL precisely because of limitation of POST requests.
And the backend still has to normalise the GET request, and possibly peek inside it to make sure that it is the same as some previous request.
> how to auth data (anyone has access to everything) -> Authentication or authorization? What do you mean with anyone has access to everything?
Your schema is a single endpoint with all the fields you need exposed. Oh, but a person X with access Y might not have access to fields A, B, C, and D.
Too bad, these fields can appear at any level of the hierarchy in the request, deal with it.
> how to... -> yes?
A GraphQL query is ad-hoc. It can have unbounded complexity and unbounded recursion. Ooops, now you have to build complexity analysers and things to figure out recursion levels.
A GraphQL service usually collects data from several external services and/or a database (or even several databases). But remember, a GraphQL query is both ad-hoc and with potential unbounded complexity. Oh, suddenly we have to think how much data and at what time to we retrieve, how do we get the data without retrieving too much, and without hammering the external services and the database with thousands of extra requests.
That's just from the top of my head.
Ans so you end up with piles of additional solutions of various quality and availability on top of GraphQL servers and clients: caching, persisted queries etc. etc.
How are GET requests a crutch? If anything GraphQL is completely agnostic to which HTTP method you use to access it. You don't even have to run GraphQL over HTTP, it can work over MQTT, NATS, telnet...
> And the backend still has to normalise the GET request, and possibly peek inside it to make sure that it is the same as some previous request.
Which is what any caching proxy must do anyway?
> Your schema is a single endpoint with all the fields you need exposed. Oh, but a person X with access Y might not have access to fields A, B, C, and D.
In your GraphQL implementation you can just deny fulfilling requests that contain fields person X doesn't have access to. This problem is not limited to GraphQL, it's a generic authorization problem.
> A GraphQL query is ad-hoc. It can have unbounded complexity and unbounded recursion. Ooops, now you have to build complexity analysers and things to figure out recursion levels.
You don't have to build a complexity analyzer or figure out recursion levels, there are already tools that do that for you. But you can go another way and just create a list of approved queries.
> A GraphQL service usually collects data from several external services and/or a database (or even several databases)
Usually? That's just speculation. And that's entirely on the implementation of that service, it has nothing to do with GraphQL spec/technology itself.
They were not in the original spec IIRC. URL's are limited in legth (it's not in the spec, but most clients have a limit) etc.
> Which is what any caching proxy must do anyway?
Nope. A caching proxy can benefit from HTTP Cache Headers [1]. But cache headers don't work well with GraphQL's GET requests, and don't work at all with the default, which is POST.
> This problem is not limited to GraphQL, it's a generic authorization problem.
GraphQL makes it significantly more complex though. Because your requests are ad-hoc.
> You don't have to build a complexity analyzer or figure out recursion levels, there are already tools that do that for you.
Indeed. By adding more and more complexity. And no, tools only solve a part of the problem. Simply a dataloader on a server doesn't entirely solve the N+1 problem.
> But you can go another way and just create a list of approved queries.
Turning it into REST with none of the benefits of REST.
> Usually? That's just speculation. And that's entirely on the implementation of that service
It's not speculation. That's the main use case for GraphQL. But even if you just slap it on top of a single database, you still have the problem of ad-hoc queries hammering your database.
They were not in spec because the spec doesn't say anything over which medium it should be transported. In fact the spec [1] only mentions the word HTTP 5 times: 4 times in example data and one time discussing implementation details when sending data over HTTP. GraphQL can't be faulted for the limits of the transport over which it is used.
> Nope. A caching proxy can benefit from HTTP Cache Headers [1]. But cache headers don't work well with GraphQL's GET requests, and don't work at all with the default, which is POST.
How do cache headers not work well with GraphQL GET requests? That is entirely up to the server that implements the API. If that server doesn't implement caching well, that's not GraphQL's fault.
> It's not speculation. That's the main use case for GraphQL. But even if you just slap it on top of a single database, you still have the problem of ad-hoc queries hammering your database.
The main use case of GraphQL is any two things that want to exchange data with each other. Merging data from multiple data sources as its main use case is simply not true. The ability of GraphQL to merge different data sources is one of its abilities but it's not intrinsic to GraphQL.
> Turning it into REST with none of the benefits of REST.
And what exactly are those benefits? I'm here defending GraphQL yet none of the downsides of REST are being taken into account. GraphQL brings structure where there was none, that alone is a significant reason to choose GraphQL to structure your API.
> N+1 problem
There are tools like Postgraphile that solve this. It converts your GraphQL query into one efficient database query.
> ad-hoc queries hammering your database
And what prevents anyone from hammering a REST API? GraphQL doesn't release the developer from implementing sane constraints - something that has to happen with any API implementation and not specific to GraphQL.
If not the spec, then original documentation. GET is a late add-on.
> How do cache headers not work well with GraphQL GET requests?
In REST:
- a resource is uniquely identified by it's URI
- when the server sends back cache headers, any client in between (any proxies, the browser, any http clients in any programming language etc.) can and will use these cache headers to cache the request
In GraphQL GET:
- http://myapi/graphql?query={user{id,name}} and http://myapi/graphql?query={user{name,id}} are two different requests
- it gets worse for more complex queries, especially if they are dynamically constructed on the client
- each of those is viewed as a separate query with separate caching
- cache normalisation and query normalisation are a thing in the graphql world (and non-existent in REST) because of that.
That's a yet another layer of complexity that you have to deal with
> And what exactly are those benefits? I'm here defending GraphQL yet none of the downsides of REST are being taken into account.
I wish anyone was willing to discuss the downsides of GraphQL. Bashing REST is the norm, but GraphQL is the holy grail that accepts no criticism.
Benefits of REST over GraphQL, off the top of my head:
- it's HTTP, plain and simple. So everything HTTP has to offer is directly available in REST. See this HTTP decision diagram [1]
- caching doesn't require you to normalise and unpack every single request and response just to figure out if something is cached
- You know your requests, so you can provide optimised queries, resolution strategies, necessary calls to external services as required by the call
> There are tools like Postgraphile that solve this. It converts your GraphQL query into one efficient database query.
I'd love to see that proven for any sufficiently complex and large database.
> And what prevents anyone from hammering a REST API?
No ad-hoc queries prevents anyone from hammering a REST API that you can specifically tune to the specific request and data you need.
GraphQL requires significantly more care especially if you're not running it on just one database. And even then, oops, joins: https://news.ycombinator.com/item?id=25014918
And we're back to requiring the graphql server to be able to limit recursion depth, query complexity, etc. etc.
[1] https://github.com/for-GET/http-decision-diagram/tree/master...
Instead, if you build on the standard library, you can compose your application from there- a good muxer, some standard middleware that are generic http.Handler's, a session library, etc.
Or, and bear with me here, you can outsource those decisions to other folks, and just write your business logic.
* go-chi/chi
* rs/zerolog
* html/template
* spf13/viper
* raw SQL
My conclusion from both experiences is that Go is a tool like any other, with its strengths and weaknesses, and with particular idiosyncrasies that make it particularly important to properly architect the application. It can punish you pretty hard[1] if you don't follow proper patterns, or the architecture you decided.
For instance, I think the issues were more pronounced with the project using Mongo precisely because of the flexibility Mongo offers, which allows to delay deciding on schemas, and thus means that the architecture itself can change constantly.
[1]: Such as making coupling and code repetition to balloon pretty fast.
No need to use Echo..
https://github.com/kyleconroy/sqlc
SQLx is good for simple read queries, but iirc for write operations you still need to map things manually and reads with JOINs are a bit tricky. Gorm might be good for simple CRUD applications, but it's magic has a performance cost.
That has a bad smell
In go, if you write code like the following:
conn, err := db.Connect()
defer conn.Close()
Yes, like most of go, it's manual and painful and poorly thought out, but most people do follow these patterns, so for the most part go code will safely unwind from a panic.