Network Programming with Go | Hacker News Reader

Network Programming with Go

jan.newmarch.name

100 pointsbasil27 comments

I find it funny how he skips an introduction to Go saying there are many such introductions already available but explains networks and the internet in great detail!

I really like this book, but what I found most difficult when implementing a website is the authentication part.

How password should be encrypted? How session id should be generated and passed to the user? How session information should be saved? (cookie? file? database? memory?)

Also, I'd be interested in https, from theory to implementation.

And also caching, that is an important topic.

I understand that your comment is about this book, but I'll go ahead and answer your questions with links to packages (plus I get a chance to plug a few of my own creation :). If you want to learn how they work, reading docs and code is the best exercise. It's not a substitute for the proper book, though, but may turn out to be useful.

How password should be encrypted?

Choose your package:

    go get github.com/dchest/scrypt
    go get code.google.com/p/go.crypto/bcrypt
    go get code.google.com/p/go.crypto/pbkdf2

How session id should be generated and passed to the user?

If you mean how you can authenticate logged in users, then you can use this package:

    go get github.com/dchest/authcookie

How session information should be saved? (cookie? file? database? memory?)

Up to you, but generally it's better to avoid cookies for storing anything other than IDs. There are a few packages for "sessions" like in PHP, but I don't remember their names.

One more important question to add:

How do I protect against XSRF attacks?

    go get code.google.com/p/xsrftoken

You can read documentation for these packages here: http://go.pkgdoc.org/

As for HTTPS, it's a complicated thing. Pick any good book on it, then apply your knowledge to the standard package crypto/tls (http://golang.org/pkg/crypto/tls/).

Wow, Thank you very much!

https is quite straightforwards in Go; the most direct means [1] essentially adds two parameters to http.ListenAndServe to supply paths to your cert and key file. The more flexible indirect ones involve either providing a TLS config [2] or using the crypto/tls package to create a TLS listener [3]. It comes down to which abstraction level works for your configuration.

Due to Go's pervasive use of io.Writer / io.Reader / net.Conn interfaces, once you have the problem of loading keys sorted, everything else is the same as writing a plain old TCP/IP listener. Configuring TLS will always be a pain -- decades, committees and broken implementations have seen to that.

As for actually verifying passwords and session integrity, Go's crypto packages are also pretty straightforwards and composable. The crypto/hmac [4] package is easily combined with various implementations of hash.Hash. In exchange for this simplicity, though, Go gives you plenty of opportunity to emulate LinkedIn and blow your own foot off, forgetting salt and common sense. There are plenty of higher level packages that offer to do it for you. (Blow your foot off, or hash your passwords -- I recommend reading implementations to decide which. :))

Also of interest when dealing with writing webapps in Go, the database/sql [5] package is an important abstraction. Since it is a more recent package, it often doesn't show up in google searches for sqlite, postgres, etc. This can give an impression that Go's database support is fragmented.

1 -- http://golang.org/pkg/net/http/#ListenAndServeTLS

2 -- http://golang.org/pkg/net/http/#Transport

3 -- http://golang.org/pkg/crypto/tls/#Listener

4 -- http://golang.org/pkg/crypto/hmac

5 -- http://golang.org/pkg/database/sql

Thank you very much for the answers!

Aside from being interesting and a new thing to learn, is Go used extensively in the professional world?

For network programming I've often used Python + Twisted + ZMQ, what are the most commonly used alternatives?

Go is very new language, so I doubt its currently being used much outside Google.

Answering your second question, almost every good programming language has some good networking libraries. Choice of language usually depends on what you are building and what are your requirements.

Here's a list of a few organizations that are using Go: http://go-lang.cat-v.org/organizations-using-go

I'm developing a DNS hosting service (http://slickdns.com) and am using Django to do the bulk of the development, but I've also written a little utility web server in Go as it needs to be high performance but also lightweight and Go fits the bill perfectly. Updates are also a breeze as it's just a single static binary, compared to Django/Python where I have to make sure that any 3rd party dependency packages are installed first.

Last week I also developed a little random password generating web server with a web service API in Go. It's live at http://random-password-please.com/, and the source is at https://github.com/jbarham/random-password-please. IMHO it's a good illustration of how easy it is to write network servers in Go as it makes use of goroutines and channels and it all fits together very easily and naturally.

You might find that the blocking style of Go networking code is far less mind bending than using twisted.

Yes, and the beautiful thing about it, is that the blocking calls you make are actually implemented as evented IO. This is great when you have a large number of goroutines; it keeps it very fast!

All I/O is evented I/O. It's just that with modern I/O requirements it's too expensive to use a whole thread to wait for the event.

Wait, what? Do you have any references for that? My impression was that Go creates new threads for new Goroutines when the current ones are blocked. Is that not how it works?

The goroutine is a user-thread, Go can create as many threads as there are cores and schedules user-threads inside these threads without the programmer managing that, you just create goroutines. When a goroutine "blocks" it schedules out from the thread and another can replace it.

http://golang.org/doc/effective_go.html#goroutines

The Go runtime creates new threads to run goroutines when a thread is blocked making a syscall. The net package avoids having too many threads blocking on syscalls by using epoll/kqueue etc. for socket I/O.

I use Go very heavily for "duct tape" in production, combining processes like machine vision, VNC clients, and video transcoding into a uniform internal API. It has been conceptually a lot easier to work with than my native Python since it offers better (for me) abstractions and a very manageable concurrency model.

Using Go in a professional setting really isn't a technical challenge. The biggest concern is if you have to grow your team, you're either going to have to teach Go (which isn't that hard -- I've seen two different people come from dynamic languages and pick the basics up in a week) or hire people who know Go (often already happily employed).

To further mangle Charles V: "I speak Spanish to God, German to my horse, and Go to connections that might make me wait."

Is there any way you can control the number of native threads in go ? I wrote a small go example that used net/smtp to bombard myself with emails. Just a loop that fired off a goroutine, which each sent an email.

It was quite disturbing that firing off several thousand goroutines resulted in several thousand native threads - resulting the system trashing. If I inserted a very small pause between creating the go routines, this did not occur and just 1 or 2 native threads were created. I tried the runtime.GOMAXPROCS, to no awail. Is there any way of controlling this kind of behavior ?

It's likely this is related to DNS lookups. Doing a lookup on a modern operating system requires calling in to some C library. The call in to the C code means that the thread can no longer be used to schedule goroutines, the runtime doesn't know how long the C call will take so it creates a new thread to keep running goroutines. For calls in to C that are very quick this can lead to a lot of thread creation.

You can do one of:

1. limit thread creation by using your operating systems limiting features. eg ulimit

2. rebuild the standard lib with DISABLE_CGO=1 to avoid the calls in to C if that's possible in your environment.

3. batch all your dns lookups in to a single goroutine.

4. make changes to the operating system so provide async dns lookups.

Thanks, I'll do some testing with these points later on. Though it does sound like go should batch up gethostbyname/getaddrinfo C calls on one or a few threads. Having DNS lookups be the cause of thousands of threads to be created, up till the point where go routines fails is rather ridiculous for a language that's supposed to be good at concurrency imho.

Batching the C calls means that you all lookups are delayed by other lookups. This isn't very good for concurrency especially since it's reliant on network traffic. If you had a few lookups in the queue, you could end up having goroutines waiting a few seconds or for lots of lookups this could balloon out to minutes for a lookup. This is the trade off. In general most programs aren't making thousands of concurrent DNS lookups.

i just use http://www.unlocktheinbox.com/dnslookup/mx/ or some other online dns lookup tools.

Ever notice how all Go tutorials are just about Web servers somehow?

It seems like when trading C for Go I'll be trading a few more lines of code for a bunch of black magic and a new syntax. Not sure if want.

It's easy to forget that the black magic that runs C is commonly confused with just being an operating system.

I'm not much of a Go programmer and do 99% of my work in C but I've played with it a bit and there are places where the abstractions and facilities provided in it are so nice they make it worth it.

The whole goroutines and channels mechanism is fascinating. I've seen and used coroutines in C that in a way achieve what goroutines do but the simplicity and the combination with channels is really great.

Comparing a co-routine based network server in C and Go would be the way to go to see the real strength of Go, IMNHO.

Reply on news.ycombinator.com