Go for System Administrators | Hacker News Reader

Go for System Administrators

blog.lusis.org

174 pointspolvi61 comments

Open articleView on HN

A long post that gives two arguments near the end in favor of Go over their old language of choice (Python):

1. No dependency issues.

2. Compiles with no issues on different platforms.

And one sort of argument in between:

3. Some of the tools that they use were written in Go.

While (1) is true for deployment, dependency handing is not a solved problem in Go. Since import statements do not allow you to specify versions, you have to keep track of which commits of a particular project you trust. I am pretty sure that this problem will be solved, but it currently isn't.

(3) is perhaps an argument in favor of being able to read Go, but not to switch to Go. By those standards, everyone on Unix should be writing C ;).

One thing that annoyed me about both Python and Rubyis that the way things in the standard library worked changed from release to release in subtle, incompatible ways. I remember getting burned by a change in how iterating over characters in a string worked in Ruby in the 1.8.x versions. Sorry-- I don't remember the details, it's been a few years. The point is that if you want to run in on a different PC than you developed the code on, or upgrade, these are buried landmines which are going to blow your leg off sooner or later. If you're working on an open source project, often the first sign you get that your Python code doesn't work on older or newer versions is a user reporting a mysterious RuntimeException.

virtualenv can sort of solve a lot of these issues, if you have complete control of all computers running your software, and if you can find a Python version that works for all the packages you need. But for a lot of open source projects, these assumptions may not be true. And then you have the issue of stagnation. It sucks to be stuck on Python 2.4.x in 2013. But how do you make a business case for a potentially very risky upgrade? This is the real reason why initiatives like Python 3.x and Perl 6.x have been DOA, in my opinion.

> One thing that annoyed me about both Python and Rubyis that the way things in the standard library worked changed from release to release in subtle, incompatible ways.

I got bitten by one such implementation detail changing in between releases in python.

It had something to do with the pickle machinery, like the 'set' builtin changing from using __reduce__ to __reduce_ex__ or vice versa, or something like that, and all the pickles that were serialized in 2.4 were deserializing incorrectly in 2.6. I can't remember the details now :\

I think with Python and Ruby the assumption is that the typical code base is either in a state of permanent change or dead and thus the advantages of improved code / improved API outweigh the advantages of backwards compatibility. It's a tradeoff, there is no universally right solution there.

If you want to write code which is meant to last for decades unchanged you want ISO C or C++, backwards compatibility is a primary virtue there. My ANSI C89 code from 20 years ago still builds and works as expected when compiled as ISO C11 with the newest compilers.

Of course there are also languages between these extremes.

His reference to a lack of dependency issues is a runtime aspect: A Python program requires the right Python interpreter and all the right Python modules to be set up correctly on the target system. For Go, you can build a single binary in a different place where you control the packages you use, and deployment is as simple as using scp.

I think you realised that, but versioning imports in Go is at exactly the same state as versioning imports in Python. I can't write an import statement in Python that describes what version I need, nor can I in Go.

You can't specify it when you import it, but pip/virtualenv solve that issue for the vast majority of use cases.

If you're including external tools, there's a bunch of tools that do it for Go too, such as rx (http://godoc.org/kylelemons.net/go/rx).

They are not standard. setuptools, cabal, Maven, et al. are. I have two Go libraries, and I do (and cannot) specify its version dependencies in a standardized manner.

Maven isn't standardized. There are still a lot of projects that use Ant + Grail or Gradle. There's also OSGI, which is sort of yet another way to manage dependencies in Java.

Maven doesn't prevent versioning issues from cropping up, either. One common example is where dependency A pulls in library C 1.0, but dependency B pulls in library C 2.0. They often conflict, and then your software fails at runtime (fun fun!) There are a bunch of hadoop bugs where exactly this happened, although I'm too lazy to look up the JIRAs right now.

I feel like if you want a stable version of something, you should use yum, apt-get, or your package manager of choice. I don't think the programming language should try to do the package manager's job. It never seems to end well.

So far, I haven't seen actual harm caused by a lack of library version numbers, and there seems to have been a lot of good done (people using and testing latest versions, green master policy, etc).

Sure, there are a number of similar tools for Go. Over the next couple years, I suspect the community will settle on a "standard." I think it's the lack of a standard that the OP is bemoaning. Go just isn't old enough for that yet.

His reference to a lack of dependency issues is a runtime aspect: A Python program requires the right Python interpreter and all the right Python modules to be set up correctly on the target system.

I know, that's why I said that it is true for deployment. It's an advantage of compilers that compile in packages/modules statically into the binary. E.g. Haskell also has this advantage.

The downside, of course, is that if there is a security vulnerability in some package, you have to recompile and redeploy all your binaries, rather than replacing one dynamic library.

Whether it is an advantage depends on the usage scenario.

Or Haskell or Java or Ruby. But these languages have standard (or nearly standard) tools that allow you to specify versioned dependencies (Cabal, Maven, Gem). Go puts package management more or less in the language, but doesn't do version management.

So, your comparison here is not fair. You should compare Go's package management with Python plus setuptools/easy_install, Java plus Maven, Haskell plus Cabal, etc.

  > So, your comparison here is not fair. You should compare 
  > Go's package management with Python plus 
  > setuptools/easy_install, Java plus Maven, Haskell plus 
  > Cabal, etc.

That makes Go look even worse, doesn't it?

Not from a devops viewpoint. Just deploying a new binary is way simpler than dealing with pip/easy_install/gems/cpan/etc.... and all the headaches those involve.

Hell even the stated "just update the library you link to" advantage of compiled programs tends to be overstated as you end up with some software that works and some that doesn't. So statically linking in dependencies and library dependencies by my mind is less of a issue now.

Hell even the stated "just update the library you link to" advantage of compiled programs tends to be overstated as you end up with some software that works and some that doesn't.

If you don't break ABI compatibility it will work. A security fix very rarely changes the ABI.

True, but I have seen it happen with things that end in racle and start with o.

"rather than replacing one dynamic library"

Library and version management at Integration (and before test) simplifies the question "what version are you on?" There is one answer, which for many environments may be useful.

What's stopping someone making a 'Python linker' that packages all the dependencies a Python program needs into a single file? Wouldn't that solve the dependency problem just as well? Is it simply that someone has done this for Go, but not for Python? That seems like a hell of a big switch to make just for the sake of one tool.

http://www.pyinstaller.org/

So instead of a 5 MB Go binary ("5 MB? Ye gods! Unacceptable!"), you get a 200 MB Python distribution, because you need the correct Python version plus all the correct libraries.

Since import statements do not allow you to specify versions, you have to keep track of which commits of a particular project you trust. I am pretty sure that this problem will be solved, but it currently isn't.

Go forces everybody to use green master policy. Every library is expected to have green master and stable API. If you need to change API of your Go library, you must create new repository.

Thats sounds alot like the policy in the TeX world, where forks of libraries with breaking changes need a new name. Quite frankly, it's even harder to follow than the occasional breakage at version changes, especially when you try to find out what the "modern version" of what you are using is.

That's not entirely true. Go allows you to tag with the Go version you are aiming for and imports will use that. You do not have to keep your master clean. If you clone explicitly into your GOPATH then you can develop on any branch you like. Similarly, if you use git for version control and so does your library then you can always submodule your dep into your GOPATH and submodules allow you to specific the exact commit you wish to use.

Yes, you do have to keep your master clean. The version tagging thing is only intended to be used when you start depending on a new language feature not available in older versions of Go. For example, if your new code needs something that's only in Go 1.1, you could create a 1.0 tag right before making that change. But people using new versions of Go will get whatever is in master, not whatever is at that tag.

Yes, you can always clone explicitly into a path, but then we're not talking about the module system any more, are we? We're just talking about you copying around files locally. It's misleading to state this like it's the recommended or usual way to do things.

Yeah it's much better than Python but not good enough also. We should still talk about a better and safer module system in Go community. People do their own versioning by forking repos - this is pain in the ass. We need a good module system with versions; like NPM.

My impression is that static linking is undervalued by developers who have not been responsible for the production deployment of binaries to larger (hundreds or hundreds of thousands of systems). When debugging production systems at that scale you want to minimize the number of moving parts given the problem is so hard to begin with.

ABI compatibility just postulates that the bytes entering and leaving the interface will be marshalled the same way. However, the semantics of the function must by definition always change when you change the implementation.

You are arguing that these semantic changes are sometimes inconsequential, like for security updates. Based on my personal experience I think a more conservative view is appropriate and any changes to the dependencies of a program should be tested before large scale deployment.

While I'm too tired to read this thru thoroughly, I had to skim it and upvote it. This was actually my first thought when go was introduced. I don't see it taking over the web world, or the gaming world or anything of that sort, but I do see it being the next tool for multithreaded applications related to system work.

Go may never "hit it big" with the mainstream but it very well could find it's place in administration. That's what I hope to use it for as I get more proficient.

It was built around software engineering principles (rather than experimental language constructs). The attention to small details (imports via git/bzr/hg, gofmt, go run, etc) deeply resonates with system admins (or at least myself and the OP).

That's my impression too. The lone sysadmin who has to hack something up and moves on to other work likes Go; real developers who work in teams and have to maintain their codebase a month down the road stay the hell away from it.

My coworkers whom aren't gurus can read and edit Python code. Go? I doubt it...

How is go so much more complex? The fact that struct types can be passed either by reference or by value?

It's not go. It's the fact I work with admins and analysts that are not formally trained developers. Python and PowerShell are their tools.

real developers who work in teams and have to maintain their codebase a month down the road stay the hell away from it.

As a fake developer, our team has had little issue with code maintenance "a month down the road". I'm not sure what "real" developers" are using but their platitudes must be an effective resource.

Real Programmers wrote in machine code. Not FORTRAN. Not RATFOR. Not, even, assembly language. Machine Code. Raw, unadorned, inscrutable hexadecimal numbers. Directly.

- The Story of Mel.

Real Users hate Real Programmers - FreeBSD fortune

This is timely as I was just fiddling around with some CLI and subprocess communication in Go as I learn the language. Is there anything like envoy for Go? While interacting with processes wasn't too tough (once I found the bufio package -- doh!) there's still a lot of boilerplate involved. That's partially unavoidable given the static nature of the language, but it would be nice to have a library that streamlined some common patterns.

I think what Go's subprocess stuff needs most is better examples. There aren't a lot of examples out there yet. For example, I saw one guy on StackOverflow use a complicated Reader thread to print a subprocess' output to stdout, when all he had to do was set Command#Stdout to os.stdout.

I hope that I'm not assuming too much here (maybe you really did need bufio for your code), but in a lot of cases you don't need it. Like if you want to supply stdin to a process, just use Command#Stdin = strings.NewReader("what I want to send to stdin"). For getting the command output, it's often enough to just call Command#CombinedOutput and cast to a string-- no bufio required unless you're doing something fancy.

I wrote some utility code that might or might not be helpful: https://github.com/cmccabe/test2/blob/master/subprocess.go

I definitely agree about the examples. There's really nothing out there except what you might run into on SO.

In my case I wanted to programmatically communicate with an interactive command line app, so I ended up using StdinPipe and StdoutPipe along with a bufio.Writer and bufio.Scanner. This allowed me to easily send strings and read the lines that came back, which worked reasonably well. I agree that this would be overkill for simpler cases, in which your approach makes sense.

Do you use os/exec [1] package or directly start processes with os.StartProcess?

[1] http://golang.org/pkg/os/exec/

I used exec

So, where should I Go to get started learning Go as a sysadmin?

http://tour.golang.org/

If you understand that they seem to be written as a learning exercises by the respective authors, I found that there's a good introduction to the nuts and bolts to be had browsing attempts at creating userspace tools in go: eg. https://bitbucket.org/telesto/goblin/ or https://github.com/manveru/goblin .. it seems to be a popular first project :)

I think Mark Summerfield's book is quite a good introduction:

http://www.qtrac.eu/gobook.html

Many of his examples touch upon sysadmin work, such as the MP3 playlist conversion, implementing concurrent grep, and coverage of common file formats (XML, JSON, plain text) and archive formats. It's also a good introduction to Go. At times it drinks the kool-aid a bit to much, it sometimes takes quite mundane constructs as an example how stellar Go is (let's be honest, Go is quite a boring language, but boring can be good at times).

sysadmin here; I've been enjoying An Introduction to Programming in Go by Caleb Doxsey[1]. Short, sweet, and right to the point without being terse.

There is nothing in this book that will directly relate to using go as an automation language, but I'd rather learn the fundamentals before learning specialized examples.

[1]: http://www.golang-book.com/

https://gobyexample.com/ is also a great resource

Anyone else able to build this? I ran into a brick wall:

    $ ./build
    # github.com/coreos/etcd/store
    src/github.com/coreos/etcd/store/store.go:633: method s.checkNode is not an expression, must be called
    # github.com/ccding/go-config-reader/config
    src/github.com/ccding/go-config-reader/config/config.go:35: undefined: bufio.NewScanner
    # github.com/coreos/go-raft
    src/github.com/coreos/go-raft/log.go:241: function ends without a return statement

Looks like the code is using Go 1.1 features and you have an older install.

didn't read the article, but the headline could benefit from s/Go/Golang/

Interesting Post.

Compiling in production? so this is the lang to "go" in case you're such kind of sysadmin which allows a compiler in production.

And have I to assume, it's the lang to "go", in case you're the kind of sysadmin which does not package things using the platform standards?

Is "go" be the solution to "administer" backups, monitoring, deployment, staging, network devices, provisioning, on a heterogeneous and scalable environment ?

For me, the "Zen" as sysadmin, is to impose as few toolchains and dependencies external to the _app_ to support, as possible. Going to "go" for me is as say that "java" is the lang to go if you're a sysadmin.

Fact 1) Sysadmin should be polyglot. For her needed tasks, and for the apps to support.

Fact 2) The right tool for the job. There is many many sub-jobs to administer a system properly (including security), there is no "one tool".

Fact 3) The good sysadmin is invisible, and her job goes unnoticed. Even if it's not the same to support a single app in a few servers, than to administer a multinational company internal and external networks, I still think that adding "go" to any system planning, just to "administer it", is crazy.

Fact 4) How many good sysadmins are good in "go"? you know, people moves, teams change, requirements change... is this choice really "helping" on the task of "administer" the infrastructure of the company in the long term?

Anyway, I don't say it will not work. There are many environments, and different people. It may work. I simply think this is not a "general purpose" advise to follow.

Congrats if it works in the case mentioned. Anything that always just works, is great, even if not advisable for other environments, teams or companies.

> For me, the "Zen" as sysadmin, is to impose as few toolchains and dependencies external to the _app_ to support, as possible. Going to "go" for me is as say that "java" is the lang to go if you're a sysadmin.

That seems to be the main advantage of Go that the author mentions. The Go compiler builds statically linked binaries.

Once built, the single file can be copied and run on any common system, without installing the compiler/interpreter, packages or libraries onto that system.

While I like Python, running Python scripts on servers requires considerable effort for deployment. You have to ensure that the right versions of the right packages are installed (which is even more difficult when it's someone else's server). Whereas with Go "deployment" would be just a file copy.

A Java tool could be just a file copy too.

If we have to support a Go app, and we are a Go team, maybe it's the right tool for the job, even if we have to reinvent the wheel with configuration management of the systems, backups, auditing, monitoring, provisioning, and "app to really support" integration or control. etc.

Else, it has more inconveniences than advantages, imho.

Maybe I'm just ignorant on this language (a big possibility) and there is already tools in Go for PXE install some hundred of different servers, to define their O.S. configuration, to have recovery plans, provisioning and backups, to monitor stuff as the packets per second on a switch port, etc, etc, etc.

If it covers such typical needs and it's free of surprises and issues... as said... great.

Note: I'm not saying python is "the solution" neither. My networks typically need to control more than one scripting/programming language and more than one DSL. And I think that there is no "language" which will free the team of writing custom tools, unless the "sysadmin" level is limited to google-it->copy->paste->makes-ping?->go-to-the-bar (else repeat).

You're arguing against a strawman. The author didn't say "Throw out all your old tools, sysadmins, and rewrite them in Go"

OK, have to agree on this.

But he is saying: ignore the team, ensure your job position increasing the bus factor, ignore security updates best practices. Among other things.

That's not what I'm saying at all. In fact I was pretty clear about that fact.

What I said was that I, personally, am going to be writing more personal tooling in Go. I was pretty clear that our team is NOT switching off Python. It's in the last section entitled "This isn't a switching story"

Sorry, you're totally right. I think I over-reacted and distorted the original article with my personal opinion.

Even I'm thinking to dedicate some spare time to explore Go thanks to some articles I've seen in the late days.

Next time I will try to be more in line with what is being said.

Thanks for reading, understanding and clarifying.

You can't install a java tool-- any java tool-- just by a file copy. At minimum, you need to install the proper version of the JVM (different software may need different versions)-- that's at least 50 MB and maybe 100 these days. Then you need a directory with all the jars which the tool depends on-- figure another 50-100 MB or so for that. Finally, you need a shell script to set up the CLASSPATH, set up any JVM arguments you might need (increased heap size, anyone?) and actually locate the binaries.

In Go, you just copy the binary to the machine and run it.

About statically linked binaries:

1) I've only loved them, to recover failing disks and broken systems (i.e. a statically linked ddrescue)

2) I think it goes against the general Linux architecture (i.e. it multiplies the number of needed upgrades on a vulnerability/bug announcement, this is, you need to upgrade each "Go" tool used by the "sysadmin", if there is 2, 2 upgrades, if there is 134, 134 upgrades, on each server).

3) I'm aware it's a solution to certain issues (i.e. dependency problems), when proper solutions are not available (paranoid component control, local mirrors of all the stack, CI, staging environments, etc)

Compiling in production? so this is the lang to "go" in case you're such kind of sysadmin which allows a compiler in production.

I don't think the article was suggesting that, it's suggesting compiling once before deployment for a given architecture, including all dependencies, as opposed to trying to install dependencies as packages. That is one of the points in Go's favour, as it requires you to produce one binary with static linking, which you can then use anywhere, as opposed to a script which requires dependencies to be installed with it. Of course there are ways round this for scripts (I've packaged entire ruby stacks into apps before for example), and there are downsides (updates to dependencies require a rebuild of the binary).

That's quite separate from how you actually get the binary into production, for which you could use your preferred packaging tools, or a deployment tool like ansible.

Fact 2) The right tool for the job. There is many many sub-jobs to administer a system properly (including security), there is no "one tool".

I don't think he suggested anywhere that Go was the only language that could be used for system administration, just that it would be his preferred language for writing tools from now on. You're reading a bit too much into his statements.

PS Please don't abuse quotemarks like that!

I see those points where wrong. Others don't. (?)

If quotes are a reason to get down-voted, I will be more careful. Sorry.

I doubt anyone downvoted for minor points of style, more for the aggressive dismissal. I've given you an upvote as don't really feel it deserves the mob-handed moderation, even if it was a bit inciteful.

I think this will very much come down to how you see the sysadmin role - in some orgs they play a very close support role and help debug apps etc (polyglot required), in others they might provide more abstract services for developers and never touch app code, only instrument and manage it on the process level (perhaps more where this writer is coming from). So perhaps the wide gamut which sysadmins cover is to blame for your disagreement?

Sure my opinion is affected by my personal experience or career. Anyway, I should have been more careful and respectful in the way to express myself.

I don't have anything against Go in particular. And probably my opinion was biased by bad experiences not related directly to Go.

Thanks to take the time to explain your view point and make me see.

Reply on news.ycombinator.com