How to ship production-grade Go
oreilly.com
oreilly.com
1. Don’t wrap errors, log.
Errors have two purposes - control flow for your application, and information for the developer and operator. If you are already logging the flow of your program, there is no need to wrap an error and create a call stack - you already have the call stack. If you cannot follow the control flow from your logs, you need to improve your logging, because you will also need to debug production problems that do not produce a literal error object.
And if you want more causality than you can get out of your logging, consider upgrading from logging to tracing. I contribute to http://opentracing.io so naturally that’s the tracing API I would suggest you look at.
2. Do not do anything with panics except recover from them or exit.
If you have a panic that you cannot recover from (and in general, that should be all of them), then it means your application is in a unknowable state. While Go is not as unsafe as C, any and all state in your program could be bad, and it is completely unclear what is still working and what is not. Nothing you do at this point is safe. For the love of god, do not hang your application and prevent it from exiting by trying to send a Slack message. Your goal should be to restart your program from a fresh clean state as quickly as possible, not to tie it up on the way out. Let the program exit and have an external monitoring program, whose state is NOT corrupted because its in a separate process, do all of the triage and reporting.
An error is logged and propogated, then logged and propogated, then logged and propogated, then pretty soon your logs are 50x copies of themselves and the flood of logging sweeps you away.
You either (a) know what you to do about an error and log it or (b) you don't know what to do and propogate it.
An error should be logged eventually, but it shouldn't be logged 20 times.
You run a process that exits with a non-zero code. Is that an error that should be logged? Or just informational? (E.g. `diff`.)
You request an HTTP resource that doesn't exist. Loggable error? Or normal result?
Only the caller knows.
I'm not entirely opposed to Dave Cheney's errors package but in general I think it's trying to be too clever and novel. If anybody wants to unwrap the errors you return, they have to basically buy into that package and its ecosystem. It's strongly coupled to itself, and that's a bad design choice IMO.
We've been fine-tuning error logging and handling for decades. We ought not reinvent the wheel solely because new language constructs are available. My opinion is that structured data and not "errors with behaviors" wins every time, especially when you're building distributed systems and you are going to need to slice/dice your logs when you investigate problems.
The process is "panicking", but it is still reliable enough to post a message to Slack? Even writing a log entry might be too much, depending on what exactly the failure was.
Well.. most likely yes. If I invoke panic, all state is exactly as it was just prior to that statement. And even if not, it's not gonna "launch the missiles" if that slack post or log-write fails now, is it?
> Even writing a log entry might be too much
"Too much", how? Either it succeeds in which case it'll help you, or it won't which has the same result as not attempting it in the first place: no log entry..
Meanwhile, the crazy code that panicked is still running it's other goroutines - remember, you called recover! - so maybe now the webserver still has an open port and is allowing your users to access whatever strange state is left inside... gonzo things really can happen if you let a corrupted program stay on rather than shutting it down immediately. Even if nothing bad is happening, you still are out of commission for that entire period.
The point is, murphy's law always comes into play. So if we're talking about production best practices, consider that "most likely fine" means "definitely not fine at scale over time". Just make sure that whatever you're doing during shutdown can't block.
I'm seriously wondering if you ran into any trouble with recovering panics in production, because that would imply all Java, C#, Python, JS and Ruby server code in production which is happily catching and logging exceptions in the main request handler is constantly running into corrupted state.
At every job I've worked at it was generally discouraged to handle MemoryError exceptions in python and OutOfMemoryError exceptions in Java. It's safer to assume there is nothing you can do on such occasions, because sometimes (usually the worst possible time) there isn't.
And as for 2., I recall a hilarious incident with an email alert turning a deployment into a spambot, so... :)
In statically-typed languages like Java, you should be able to infer, mostly, what your program is doing at runtime, and if it's doing something you're expected, you raise an exception/error so that you stop processing whatever you're processing.
That's an unreasonable viewpoint. Whether you understand what a piece of code does at runtime is down to the particulars of the code far more than it depends on the language you choose, and it's nothing but code smell to put try/catch everywhere because the code isn't doing what it's supposed to be doing. At best we might agree that it's difficult to maintain a high level of code quality in a large Python project, but try/catch everywhere won't improve things.
There are generally two good places to put try/catch. The first is to put try/catch around a small piece of code that is expected to throw an exception, like parseInt(), and handle the exception right there. The second is to put try/catch around a piece of work which is large enough to make a decision about even when you don't know why it failed—things like top-level HTTP request handlers in web servers, or individual work items in batch processors where you can decide to skip the item and move on to the next. In either of these cases, you're using exceptions to alter control flow.
That's the same way it works in Python, Java, C++, C#, and so many others. And Go uses the same principle, too, even though Go uses error returns. Which is why I am still confused about your original comment—since Go does not have exceptions. Go literally just has panic() and return.
You should always assume there may be runtime errors regardless of whether you have some sort of static type safety or not.
There are some languages that can guarantee no runtime errors, but they make significant sacrifices in terms of features [1].
Perhaps if your exceptions are type-checked. AFAIK Java is the only language that does that.
I find that "Use exceptions, but don't use them for flow control" always comes from Java folks.
Everyone else either doesn't use exceptions, or embraces them.
Because it's so easy to expose http/https in golang, I see a fair amount of code that takes the easy path of handling GETS and not doing much else. No support for HEAD, or compression, proper mime types in Content-type, or lacking Cache-Control headers, Etags, 304 responses, etc. Go can do all these, of course, but I see them missing often.
I've seen, for example, blogs, that (probably not on purpose) defeat browser caching. Not great for performance, or your egress bill.
I quite like it because you can access the net/http constructs (ResponseWriter, Request, etc) if you really need them, but in most cases it removes a lot of boilerplate.
A bare bones http/s server is so easy to spin up with nothing other than the stdlib, people who are used to using full stack frameworks forget about all the stuff it did for them behind the scenes.
That's not really a Go-specific complaint, that's a complaint about a loooot of implementations of web stuff.
Edit: Compression is less important than it sounds, too; for security reasons, you shouldn't compress dynamic content going over HTTPS, so nginx doing the compression only helps for HTTP, and it's only a handful of lines of code to implement it in your Go layer too so it's not a huge win. If nginx did fix a lot of those problems without having to think about it, it would be helpful, but it doesn't.
I'm normally the sort of guy who puts everything behind HAProxy, I may still do that for convenience (certs in the same place etc) but imagine moving to a point where I put the Go programs behind, but in TCP mode.
* Enable compression, if desired.
* Redirect HTTP requests to HTTPS.
* Add headers for security best practices like OCSP stapling, DH params, HTTP STS, HPKP, etc.
Those are really things best handled in the edge and not by your application. Your proxy is probably going to do a better job of these things than you are and the config will be simpler. Not to mention if you have to update any of those settings you can do a hot-reload of your proxy's config -- your Go app has to be recompiled. And oh yeah, the moment you have more than one Go app sitting behind your host you need a proxy anyway.
I think this is a dangerous way of thinking. Writing more tests does not necessarily improve code quality or decrease the risk of defects. A focus on coverage can lead to bloated tests which end up taking more time to maintain than the original code.
https://dev.to/gotime/42-race-detection-firmware-and-product...
They have a bright future even if this article does not contain the most accurate advice.
Useless, ad ridden article, frankly.
Well, the article is specifically about Go instead of generalized universal advice so she put concrete syntax examples using the Go language (vs "any code" in any language). There are examples to wrap errors, report panics, and using structured logs. The specificity of real code snippets is useful.
>Useless, ad ridden article, frankly.
I'm guessing this is the real root of your complaint. In my view, the ads are tucked away to the side, not obnoxious, and are simply related to other O'Reilly offerings.
The actual programming advice in the article looks like it has value and I think you're being unfairly harsh.
And to mirror a sibling comment, on mobile the ads are interspersed every couple of paragraphs, making it very hard to follow the flow of the article.
Unfairly harsh? Clickbait headline, generic advice (with just enough Go examples to "justify" the title), aggressive styling to hide ads in the flow of the rest of the article... I disagree. I'm perhaps not being harsh enough.
I can count at least three inline O'Reilly ads, with adblock enabled.