Hacker News Reader

Comment by krakensden | Hacker News Reader

From the documentation: > So if you have 1M elements as input to a plinq function, you will end up 1M goroutines.

> Theoretically, there is not much wrong with that.

Understatement in the finest tradition of man pages and RFCs.

I wrote that statement thinking about the experiments people are making using millions of goroutines and still end up low cpu utilization when goroutines mainly depend on IO. Go has a neat scheduling for that. Please correct me if I am wrong.

pretty much any task/job/thread scheduler on the planet is going to pay some absurd overhead costs to schedule and run a million tasks compared with a reasonable number (like 10, or 100), without a measurable improvement in performance. a very smart scheduler can minimize these costs but they will still be there; the state has to be stored and updated.

Does it really sound alright to you? I mean it's not the end of the world or anything, but it's adding extra overhead which simply isn't needed when you're not gaining anything from it in this case.

I assume people want to use golang for the speed advantages of servicing many connections at once. If each connection is spawning hundreds of goroutines it's not going to work so well.

I gotta admit I never seen the problem that way, now I can clearly see I am wrong. Probably I need to limit number of goroutines spawned with a configurable default value. Thanks for pointing out Ryan!

Here's a gist I created a couple months ago that kind of explains the performance characteristics of goroutine per loop vs. partitioning with goroutines

https://gist.github.com/nathanwdavis/6290428

Basically, for big lists a goroutine per loop is very slow in comparison.

Just for reference, a single goroutine has a 8 kb stack size. Having one million of these things in memory at once would require 8 Gb of memory. Obviously, the scheduler kicks in, but 8 kb chunks of memory still has to be assigned and freed for every routine started and stopped.

Not intended as a criticism- no one writes every possible feature immediately! It was me expressing enjoyment at the dry way you documented this implementation detail.

Reply on news.ycombinator.com