Is Go fast enough for large computations? | Hacker News Reader

Is Go fast enough for large computations?

9 points99990000099919 comments

Let's say I basically want to generate a series of random numbers billions of times.

Would Go be a good choice for this? I keep trying to learn Rust, but it's very difficult for me.

> Let's say I basically want to generate a series of random numbers billions of times.

That's a bit vague, if you only need pseudo random numbers consider a xorshift based approach, e.g., https://pkg.go.dev/github.com/db47h/rand64/xorshift. Xorshift is fast and simple.

> Would Go be a good choice for this? I keep trying to learn Rust, but it's very difficult for me.

If you need fast but don't want to write Rust or C(++), Go is a good choice for this. It will likely also have a good amount of examples and existing code available for most common problems.

Let's do some napkin math. A modern cpu runs at 4 GHz so it runs 4 billion cycles per second. Generating a random number takes, if we are being very pessimistic, 100 cycles. So we can generate 40 million random numbers per second. There's some overhead for seeding the random number generator too, but it's amortized over the large number of random numbers generated. You want a billion number so it takes 25 seconds. If Go is ten times slower (it isn't) than good C code, the job still takes only 250 seconds.

So would you argue it's not worth it at the end of the day to switch to a faster language. I have a pretty quick solution in Go, but it's not scaling well.

Original solution was in Python, which is much slower.

Yeah. Unless your Go code truly suck you won't see that much improvement.

Link a gist of your code

we've replaced production python and perl systems with Go. It is faster because the concurrency story and speed of Go is faster. On average, replacing services at scale (as in, billions of networked requests per day), we saw that Go reduced our fleet footprint by 10-20x. In the best case, we went from 70+ nodes running Twisted Python to 1 node running Go, so we bumped it to 3 nodes in case it failed.

Was it hard to convince your team to switch to Go ?

What other languages did you consider?

Do you see a positive future for Golang devs?

Rust appears to of picked up a lot of its momentum. I vastly prefer Golang.

We were building an smtp server and needed a good concurrency story. We tried for two weeks with gevent and got frustrated (monkey patching was misbehaving on low level network stuff). Someone on our team got the server mostly working over the weekend in Go with no prior Go experience. We made the case to the powers that be and were greenlit to finish the smtp server in Go. Go quickly became our default language at the company, undoing the recent decree that we were to be a Python shop.

There was some early talks about going to Java instead, but Go had too much momentum and folks didn't want AbstractFactoryFactory over-abstraction and craved the more simple Go approach having just experienced the pain of Python Twisted (which was aptly named).

As for the future of Go, all signs point to continued increases in adoption. I think it hits the sweet spot for backend, networked services given the languages I've tried. It is easier to learn than Rust, has a better concurrency story than Python and Perl while out performing them by an order of magnitude, it is easy to navigate unlike Ruby, and is simpler than what standard Java devs push for and uses less memory. And the deployment story is great; just a single executable.

Thanks for the answer!

I built a small project in Go and I found it to be a very fun experience.

The syntax is very easy compared to Rust.

Any tips for finding a higher level (200k+) role with Go ?

If the problem you're trying to solve is literally generating randomness, you might look into custom hardware. I think some companies have devices to measure subtle variations in the output of tiny laser beams, as one example of how pure randomness can be quickly and continuously found.

Is that really faster than the kernel’s CSPRNG?

If you are not doing AI with custom Go code or low lvl close to hardware programming -> No. Every other instance -> Yes.

I mean, it's very hard to do and you have to build the GPU shader/processing pipeline yourself, but it's not impossible. [1] Gorgonia implemented most of the necessary libraries for tensor math for their alphago implementation.

[1] https://github.com/gorgonia/agogo

You will most likely be IO bounded.

I don’t follow. Why do you think so?

Unless you write your numbers to /dev/null, your program will wait a bit for the storage, the network, or the next program.

const int randomnumber = 4;

// It was picked with a dice roll.

This is why I hn

   man od

Reply on news.ycombinator.com