Instrumentation in Go
gbws.io
gbws.io
We have similar needs at Sqreen but for security monitoring and protection reasons: we need to dynamically instrument functions at run time, while not asking any code modification to our users. To do so, we instead leverage the Go compiler to perform compile-time instrumentation that inserts hooks anywhere interesting. You can read more about this approach at https://blog.sqreen.com/dynamic-instrumentation-go/
With this option, the compiler invokes every toolchain binary (compile, asm, link, etc.) through the provided program. So you can basically write a proxy program intercepting calls to `compile` to do source-code instrumentation, exactly like you would with go generate, but now automatically done during compilation on every package.
The problem is there are a lot of off-the-shelf observability frameworks for Go that require the pattern in the article: you emit a value to a hook function at the point of production. This sucks at any reasonable scale because you are now required to just take a global lock (or write a channel, which is the same thing) every time you record a value. This is a significant contributing reason why Go servers fall apart when given access to more than a handful of CPU cores.
You mentioned an observation methods, but essentially they are absolutely the same as hooks, just inverted (with a bit less overhead on branching and hook call). E.g. your example with bytesReceived counter can be implemented with atomic operation and further export on demand by some other goroutine.
I usually refuse to use atomic increments for services because it scales very poorly. Even a mutex-protected increment scales much better than atomic increment.
Anyway how will you collect that counters inside your component (to be observed later on demand)?
The go implementation of OT makes extensive use of the Context [1] object to support tracing, logging, and metrics. You write once, and then the user only needs to decide which exporter to use, and possibly filters to apply.
That said, it is still beta, and will be until Q4.
Am I understand it right, that you suggest instead of doing it in a generic way (any tracing method could be plugged in this way), to stick to one particular library?
And since this is a standard, it means that when a user includes a bunch of disparate libraries/remote calls in his app, he can define his instrumentation wishes once for everything during app initialization. The libraries can even cooperate since they use the same underlying API and concepts for instrumentation.
I agree that this uppercase words can be called as somewhat purposes of the tracing API. I believe also that I was talking exactly about the same approach in the article. But I can't get how sticking to one particular library will help here? In other words, I think it will just greatly reduce the usage options.
What Im saying is that you can leave the generic hooks in the library (as it described in the article and gtrace) and then let the user choose which library to use inside what hooks.
With hooks, you get nothing unless you hook, which means you have to write hooks for everything you're interested in (and also come up with a way to export the data), and so the user must touch code all over the place.
For example, if `Client` is defined as an interface, we can have two concrete implementations. One that holds business logic – `NetClient` – and one that holds logging/developer logic – `VerboseClient`. As we inject dependencies, we can compose a client like this: `NewVerboseClient(NewClient())`. This way you may even test you logging logic by composing it with different implementations of clients (one that always fails, for example) to test the different kinds of logs.
But it doesn't work well in Go by several reasons.
For example, you usually define interface not in the package where its implemented, but where its needed. Thus you must provide such wrappers in every place where some (subset) of the `Client`'s methods are used.
Another reason is that such thing rejects an ability to use your struct as is, without interfaces (and thus sometimes without heap allocation for your struct).
And, finally, if you want to adopt such approach in proprietary/internal software, where interfaces usually change more often than in libraries, you will change code in N places instead of 1 (in best case); where N is number of instrumentation methods you want to provide "as feature".
Rob Pike, one of the 3 original Go architects, has stated in the past he doesn’t like syntax highlighting [1].
> Syntax highlighting is juvenile. When I was a child, I was taught arithmetic using colored rods (http://en.wikipedia.org/wiki/Cuisenaire_rods). I grew up and today I use monochromatic numerals. — Rob Pike
Maybe Sergey Kamardin, the author of this article, is following Rob Pike’s philosophy. That being said, the website has references to a Go library called Chroma [2] which purpose is to colorize source code. The library borrows ideas from a popular JavaScript library called “highlight.js” [3] and that is why you can also see “highlight” CSS classes. The website downloads this CSS file [4] which contains a couple of instructions to make the code look the way it looks.
The official Go Blog has popularized this type of design [5] and many Go programmers has adopted it in their own websites.
[1] https://groups.google.com/d/msg/golang-nuts/hJHCAaiL0so/kG3B...
[2] https://github.com/alecthomas/chroma
If you don't need a variable to be colored to know it's a variable, then why do you color them?
A counter argument would be that you have this huge visual cortex in your brain which can use cues like this to help you quickly find what you're looking for.
A counter argument to that would be that you should know your code well enough to not need to rely on anything like that.
When I started programming, I had notepad and that was it. I didn't need colors. I use colors now, though. What's that say? I have no idea.
that's not a counter argument to that. nobody in the world no matter how brilliant knows giant complicated codebases inside out. and the best way to parse those is to use colors to aid your brain in differentiation. otherwise its all one garbled mess.
When I was a child I had a plush gopher.
No need for benchmarcks to know that a developper is less productive and more error prone without semantic highlighting.
I programmed for a full decade with Notepad alone. I don't think colored keywords are important, though I use syntax highlighting today.
At the same time, it might slow down other tasks, which can be harder to identify. I was just changing some html template, and I made a dumb "extra quote"" error because of highlighting. It probably took me more than a minute, which would offset a great number of small speedups.
I guess the conclusive proof won't be easy to find.
But did you know there's a Go font?
The Go Tour has no syntax highlighting, but the design on the pages are fit for the purpose of learning the language: https://tour.golang.org/welcome/1
The typical Go-presentations (http://go-talks.appspot.com/github.com/SatishTalim/slides/sa...) tend to be very short and readable, but you'll find www-pages/videos about Go having both syntax highlighting and not, good design and not. Good Go code tend to be short. Many gophers prefer syntax highlighting, so there's no need to force-adopt "opinions" when not strictly necessary or ideal. Alot of Go is very community and stdlib-driven though.
How to do it yourself: https://blog.joshsoftware.com/2014/03/10/how-do-i-create-a-p...
I assume it depends on your current solution, but for a new project, I find that setting up instrumentation and pushing traces to Zipkin or Jaeger is very straightforward, at least on the JVM. Out of the box I can see the overhead on top of DB queries and plain HTTP calls. And if some bit of internal logic is particularly time consuming, I can add a custom span in two lines of code and figure out what's going on. There are of course other ways to achieve the same thing, but the experience feels nicer.