What do you think about goroutines and channels?
Even JS can be used to implement such concepts via the use of generators[1].
With the added convenience that shared mutability is pretty much nonexistent.
https://github.com/clojure/core.async/blob/master/src/main/c...
That's not something that could be done with golang as far as I know.
> "fundamentally it's functionality other languages can provide via library support"
You were saying that CSP can be added as a library, citing Clojure's core.async.
All I was saying was that the way in which core.async was implemented doesn't feel like a great example of a 'library' in the sense that most people would understand in the context of a discussion about Golang.
Golang is a static, compiled-to-machine-code language without macros (in the LISP or C sense) or homoiconicity. The reason core.async can be implemented as a library in Clojure is that it has these things.
If you're talking about adding CSP to a language just by adding a library and without having to get into the internals of the language, core.async isn't a good example.
Again, happy to be corrected.
I've also linked to js-csp, a JS library obviously not implemented using macros.
I can also find other examples of implementation as libraries, but I have no experience with them:
- Scala: https://github.com/rssh/scala-gopher
Maybe things have changed since 2013, but I feel like this is a fundamental limitation of running on the JVM vs what Go can provide in its runtime.
Edit: Also, it appears to be much easier to simply "run out" of Clojure coroutines than Go goroutines, but perhaps that's also changed. Anyways, my point is that by core.async operating as a macro you still can't overcome limitations of the underlying runtime, whereas Go's runtime was purposely-built to support goroutines.
File I/O or everything else treated as syscall by Go runtime might turn you program into 10k-os-threads-monster. Scheduler will be creating new OS threads to replace those locked on syscalls until thread limit is reached and whole program crashes. Only way to prevent it is to restrict your syscall layer into fixed-size goroutine pool.
I had an interesting case recently - my app serves some data from tons of files laying in NAS, accessing it by NFS mount and one day NAS hunged completely, every I/O call to it was lasting forever. Even 'ls /mount-point-of-nas' was just doing nothing forever until Ctrl-C. In my case I've applied poweroff-poweron cycle to NAS, and everything went right in minutes, just as NAS booted. And after it I wondered, what if my server was written in Go, instead of Erlang...
And, BTW, you can never be sure, that underlying libraries of your code are safe to use.
Anyway, there's a reason the phrase "tacked-on" has such negative connotations.
Implementing goroutines and channels requires language and runtime support for green threads that are n:m multiplexed on top of native threads. It can not be implemented as a library in most languages, at least not efficiently. Any language with thread support can set up threads and put a concurrent queue between them, but that's hardly the same thing.
Languages such as Go, Erlang and Haskell do this. Interestingly, early versions of Rust had green threads (iirc) but later migrated to using native threads only.
C and C++ are a bit different because you need to resort to assembly and know details about the target arch to do stack switching but that's acceptable.
Having event handlers run in separate threads is very much supported in both Qt and Gtk (they can't be UI event handlers in quite a few cases, but network events and file reading in separate threads scheduled by the central event loop like in Go is not a problem).
I will say that it's much better organised and with much less caveats in Go.
And a point of personal frustration : both Qt and Gtk support promises through the event loop, Go does not. I find that a much more natural way to work with threads.
It's also cooperative. An infinite loop will effectively kill it (a single infinite loop will kill it before Go 1.2 I believe, but now you need enough of them). More importantly, there's a number of simultaneous syscalls that will kill a go program.
I like about go that it's moving the OS into the application. The thing is, Go's OS is not a very good one. It doesn't have the basic isolation that OSes provide. I hope it will improve.
No, it isn't.
> like every other scheduler on the planet
This is not true either. In fact it doesn't make sense. Schedulers are not single threaded event loops. The scheduler (any scheduler) is entered in various scenarios. Sometimes voluntarily, sometimes not. Sometimes the scheduler code can run concurrently, sometimes not. Sometimes the scheduler code can run in parallel, sometimes not.
The Go scheduler is both concurrent and parallel.
> It runs, sequentially, in different threads which makes the situation confusing
I don't know what this statement means. The Go scheduler certainly runs on different threads. So what.
> It's also cooperative.
Actually it's not purely cooperative, it does voluntary preemption, very similar to voluntary preemption in the Linux kernel. The check happens in every function prolog.
> More importantly, there's a number of simultaneous syscalls that will kill a go program.
There's self-imposed user-configurable limit that defaults to 10000 threads for running system calls. The limit has nothing to do with the Go scheduler, it can be set arbitrarily high with no penalty.
> It doesn't have the basic isolation that OSes provide.
The most basic isolation provided by operating systems is virtual memory. Go is a shared-memory execution environment, so this doesn't apply. What other "basic isolation" is provided by operating systems that's missing from Go?
It means that some of the data structures the scheduler examines on every run are shared data, with locking. That makes it effectively single threaded, even if it technically runs on different CPUs (at different times). Put another way it means that it'll never run faster than a single-threaded scheduler would.
> Actually it's not purely cooperative, it does voluntary preemption, very similar to voluntary preemption in the Linux kernel.
You mean preemption inside the linux kernel, in some kernel-space threads ? Because it sounds very different to preemption of applications.
> The check happens in every function prolog.
So it's cooperative. The standard that is normally used is simple : does "for {}" crash("block" if you prefer) some part of the system ? On the linux scheduler the answer is no. In Erlang the answer is no. On the Go scheduler, the answer is yes.
On the linux scheduler with proper ulimits it's bloody hard to crash the system, for instance, forkbombs, memory bombs, ... won't do it. I hope we'll get a language where you can do that too (and the JVM comes quite close to this ideal, some JVMs actually have it even).
This is true for every scheduler, not only for the Go scheduler.
> That makes it effectively single threaded
It would limit parallelism to one, if there was a single lock. This used to be the case, but now the locking is more finely grained. But this only matters if there's lock-contention anyway, which is not the case for current Go programs.
> single-threaded scheduler
Again, no such thing as a single-threaded scheduler. Even when talking about systems with a big kernel lock, or with a global scheduler lock. The scheduler is not "single-threaded" or any other term like that because the scheduler is not a thread, it's not an independent thing, it only runs in the context of many other things.
> Put another way it means that it'll never run faster than a single-threaded scheduler would.
As mentioned already, this is not strictly trye for Go, but this matters more for thread schedulers in kernels, less so for the Go scheduler, mostly because the number of threads executing Go code is very restrictive, maximum 32 threads at the moment. It's very likely that this situation might change. For example, the SPARC64 port that I am doing supports 512-way machines, so I'd need to increase the GOMAXPROCS limit. Then maybe we'd have more lock contention (I doubt it).
It's true that the scheduler will probably not scale this well, and will need improvement, but it's unlikely it will be because of lock contention.
> You mean preemption inside the linux kernel, in some kernel-space threads?
Yes, voluntary preemption inside the Linux kernel, not preemption of user-space threads. The Linux kernel can run a mode (common and useful on servers) where it might yield only at well defined points. The name is a misnomer, this is not really preemption, but it's not cooperative scheduling either. It's something in the middle and it's a very useful mode of operation, nothing wrong with it.
> So it's cooperative.
It has the good parts of both cooperative and preemptive scheduling, but yes, it's certainly cooperative.
> The standard that is normally used is simple : does "for {}" crash("block" if you prefer) some part of the system?
Not with GOMAXPROCS > 1, which is now the default on multi-way machines (all machines).
> On the Go scheduler, the answer is yes.
Only sometimes. This is fixable while still preserving voluntary preemption, since the voluntary preemption-check is so cheap that you can do it on backward branches if you really need it. This wasn't done since this wasn't a big problem in practice, even with the old GOMAXPROCS=1 default, but there's room for improvement.
> On the linux scheduler with proper ulimits it's bloody hard to crash the system, for instance, forkbombs, memory bombs, ... won't do it. I hope we'll get a language where you can do that too.
I don't understand the analogy. It is not clear what "crash" means here, and it is not clear how it would apply to a runtime environment. All that stuff, forkbombs, etc, means that you can configure the system so arbitrary code can't affect the system in those particular ways.
But for a language runtime you don't have arbitrary code usually, you control all the code. So I don't understand how any of these would apply.
Coming back to the scheduler. There's always room for improvement. Until relatively recently, the Go scheduler barely scaled past two threads! (although not because of lock contention). Now it scales really well to (at least) 32 threads. There are still improvements to be made, and I am sure they will be made. I was just addressing the "single-thread" issue.
> It has the good parts of both cooperative and preemptive scheduling, but yes, it's certainly cooperative.
For me, the best part of cooperative scheduling is that you can work entirely without locking shared data structures, because you get "transactions" for free. This means it's rather difficult to get data races, deadlocks, etc. Go's scheduler certainly does not give you that, trades it for spreading work over different cpus.
So it has problems:
1) necessity of locking, using IPC mechanisms, ... (like preemptive schedulers, and let's face facts here : channels aren't enough in real world apps)
2) everything gets blocked by large calculations (like cooperative schedulers)
3) more generally, easy to crash by misbehaving thread due to unrestricted access shared resources (not just cpu) (like cooperative schedulers)
And advantages:
1) Actually uses multiple cpu's/cores/... (like preemptive schedulers)
2) integrated event loop that scales (like cooperative schedulers)
If you want to see a programming language with a "scheduler" that doesn't have the bad parts of cooperative schedulers, check out Erlang. If you attempt to crash erlang with infinite loops, bad memory allocation, ... (on a properly configured system) that just won't work, the offending threads/"goroutines" crash leaving the rest of your program running fine. The offending threads will restart if you configure them to do so (which is really easy).
The same can be achieved, with much more work, on the JVM, or, also with much more work, with python's "multiprocessing" library, part of the standard library.
> > On the linux scheduler with proper ulimits it's bloody hard to crash the system, for instance, forkbombs, memory bombs, ... won't do it. I hope we'll get a language where you can do that too. > I don't understand the analogy. It is not clear what "crash" means here, and it is not clear how it would apply to a runtime environment. All that stuff, forkbombs, etc, means that you can configure the system so arbitrary code can't affect the system in those particular ways.
Crash means that the system/"program" doesn't respond (in a useful manner) anymore.
[0]: http://libmill.org/
"Libmill is intended for writing single-threaded applications." http://libmill.org/documentation.html#multiprocessing
Erlang at least is consistent on this -- it has that hammer well tuned and isn't afraid to pound nails with it.
But OTP isn't really possible either because Go lacks links and monitors.
Plus not being asynchronous and no distribution (wouldn't want to go over the network with sync channels anyway)...
From a personal standpoint, it's missing a lot of the features I like, namely ADTs, list comprehensions, folds, maps, etc. But that's just my personal style, not something wrong with Go. Go programs don't necessarily always look pretty but you can usually understand them after a minimal amount of study because the language is so simple.
Yes, CSP[0] is a very interesting concept. But it's not something unique to the go language.
[0] https://en.wikipedia.org/wiki/Communicating_sequential_proce...
Except when it is not. Message passing style of concurrency is just a dual of the classical blocking concurrency with critical sections, mutexes, monitors and conditional variables. An actor is a dual of a critical section. Actor's mailbox is a dual of a mutex. Sending/receiving messages is a dual of wait/notify. With any complex CSP program you can have all the same problems: race conditions, starvation, deadlocks (livelocks) etc.
Queues ("channels") are a good way to limit complexity of threaded code by treating each process as an agent. Besides some syntactical sugar Go doesn't really support this better than most other languages with threading, like Java.
Go has a weird thing going on where channels are sometimes used as a kind-of-replacement for iterators, which is error-prone since the "obvious" way to do it doesn't allow the consumer to stop the generator without a side channel. This can lead to buggy code that leaks goroutines, since goroutines can not be garbage collected.
One of the best ways to reduce the complexity of threaded code is immutability - preventing race conditions by making sure a structure never changes while being read. Curiously, Go has no way to mark an object as immutable, and does nothing to detect or prevent objects being unsafely accessed from different threads.
The magic of channels comes with select{}. Considering them to be only threadsafe queues is really missing out.
Supporting select{} in other languages is possible, but difficult and rare.
Zero length channels are a key part of coordinating concurrent threads in Go.