Go channels, goroutines and GC available in Nim
forum.nim-lang.org
forum.nim-lang.org
The basic algorithm is Deferred Reference Counting with cycle detection.
I'm sitting here feeling very impressed. Deferred reference counting has very good semantics for games. Even better, you can control the cycle detection part separately and run that part at an advantageous time. (Though it's probably better to just let GC do its thing, unless you really know what you're doing.)
I am currently writing a multiplayer game server in golang, by making sure almost everything is allocated on the stack, and heap sizes are small. This gives me an efficient, nearly pauseless server. However, something like Nim could give me even more flexibility.
Not if you need to be thread-safe.
> Even better, you can control the cycle detection part separately and run that part at an advantageous time.
How would that work with multithreading? (Assuming you had a thread-safe GC, which Nim's isn't.)
I'm interested in reading more about it, because Nim is doing a lot of experimental stuff and it's always interesting to look at its designs.
(Edited to remove speculation about how well it will perform before reading about it.)
Note that shared, lockable heaps need not be heavyweight structures. It is entirely possible to imagine a shared hash table with one heap per bucket and fine-grained locking, for example. Collections for such small heaps can be fast because the number of roots is limited, and (depending on what invariants you guarantee), you can even forgo stack scanning for most collections or limit the number of stack frames that need to be traversed.
At the programming language level, this then mostly involves maintaining mutual exclusion (in Eiffel, the necessary semantics are attached to how "separate" types are handled) and having the optimizer get rid of unnecessary copies.
Care to offer an explanation why?
Ref counting is not superior or inferior to explicit, restrictive ownership semantics. Those are simply different trade-offs.
Nim might be strictly inferior for writing a heavily multi-threaded web browser because of its memory management approach but that does not mean the approach is generally inferior.
Seems to me that Nim aims to be a "casual", rapid development / friendly (Python-like) language. Ownership semantics like in Rust do not fit there.
For different use cases Rust-style ownership semantics (when the performance of GC and runtime interoperability become an issue), or Azul-style pauseless GC (when you're willing to trade throughput for latency), or shared-nothing architectures (when you need them for legacy reasons like JavaScript, or want a simple programming model like Erlang) can work great.
Why? Just performance or is there a design reason also?
It's a price that a lot of people are willing to pay, because of how badly they want what they're paying for.
Nim and Rust have different goals, different tradeoffs, and overlapping target audiences. Having both is good. Making them fight is bad.
Since it's 2015, it seems fair to point out when a new language offers something neat, but in a way that isn't safe.
Of course it does. If you turn off expensive runtime checks, you'll get SIGSEGVs. That doesn't happen in Rust, because it is semantically impossible to dereference NULL.
Note: these unsafe features have two purposes. One is to interface with C/C++ code. The other is to be able to write close-to-the-metal code in Nim rather than in C (where you wouldn't gain any safety by using C, but lose the expressiveness of Nim). This is, for example, how Nim's GC is itself written in Nim.
None of the unsafe features are necessary for high-level programming, i.e. unless you actually want to operate that close to the metal.
First of all, Nim's backend does not target the C standard; it targets a number of "approved" C compilers, which makes it (1) a bit easier to avoid undefined behavior, because these C compilers know that they may be used as backends by high-level languages and provide efficient means to disable some of the undefined behavior and (2) allows Nim to emit optimizations specifically for them. For example, Nim knows that gcc understands case ranges in its switch statement and can optimize for that. See compiler/extccomp.nim for some more examples. Nim also makes some additional assumptions about how data is represented that are not defined in the C standard, but are true for all target architectures (or can be made true with the appropriate compiler configurations).
Second, regarding the specific cases of undefined behavior:
1. That shift widths aren't subject to overflow checks is an oversight; most shifts are by a constant factor, anyway, so they can be checked at compile time with no additional overhead. Nim does not do signed shifts (unless you escape to C), so they are not an issue.
2. Integer overflow is actually checked, but expensive; there's an existing pull request for the compiler to generate code that leverages the existing clang/gcc builtins to avoid the overhead, but that hasn't been merged yet; -fno-strict-overflow/-ftrapv/-fwrapv can also be used for clang/gcc to suppress the undefined behavior (depending on what you want) and one of them may be enabled by default in the absence of checks.
3. Nils are not currently being checked, but they will be. There's already a nilcheck pragma, but that isn't fully implemented and also not available through a command line option. This will be fixed. Until then, you can use gcc (where -O2 implies -fisolate-erroneous-paths-dereference) or use --passC:-fsanitize=null for clang to fix the issue.
(work in progress)
Because it isn't true.
"in my opinion I don't see it any less safe than languages that don't have automatic memory management"
Strawman ... the comment was about scripting languages.
"and/or languages like Rust"
Then it would be unwise to pay any attention to your opinion.
Did you not see the other person who just said that?
> Strawman ... the comment was about scripting languages.
I do not have a clue what you are trying to say...
With regards to memory safety, it is not. https://news.ycombinator.com/item?id=9050999 is an old comment from Patrick, but in today's Nim, it segfaults in both release and development modes for me. Rust's guaranteed memory safety means that Rust code (without explicit unsafe, the vast vast majority of code) cannot segfault.
> I don't understand why people think that Nim is "terribly unsafe" when in reality it's like any other language
For example, unless I write a bad cext, I cannot get Ruby to segfault.
None of this makes Nim a bad language. All languages have tradeoffs.
EDIT: Also, Nim is planning on turning those segfaults into runtime NilErrors and a nilChecks flag that will check for them at compile time, you can also avoid this by annotating Pointers with `not nil`
Maybe I've got the wrong impression and their docs are terribly misleading and there are safety checks all over. But I found the dics easy to understand last time I read them and the safety issues seemed clearly marked and more or less where'd you expect.
That is factually false.
"Perhaps it's not as safe as Rust"
And there even you have contradicted yourself.
"Perhaps it's not as safe as Rust but that brings specific trade offs most people dont wan't to deal with."
That much is true ... and can be said without telling falsehoods, like your first statement.
"I don't understand why people think that Nim is "terribly unsafe" when in reality it's like any other language"
You are confused by your own strawman.
All languages have faults. Engage with your critics, own your faults, and either correct them or justify them based on your principles.
EDIT: To give an example, pcwalton also initially criticized Go for not being memory-safe for values of GOMAXPROCS greater than 1. However, the Go team later implemented the dynamic race detector, which, if you've followed pcwalton's comments at all, you know that he is actually quite impressed with.
(I suppose, in the future, pcwalton should just generate a throwaway before commenting on Nim.)
Most games use worker-queues(in which case you can use non-GC objects) to deal with architectures like the CELL and for better cache coherency. In that case Nim is a pretty good fir.
Of course, you shouldn't use shared memory unless you need it. But often you need it. Look at how game developers have demanded shared memory in JavaScript, for example. Modern multicore CPUs do a lot of work to make shared memory work, and work well.
> Most games use worker-queues(in which case you can use non-GC objects) to deal with architectures like the CELL and for better cache coherency.
I agree with you that GC is often not the best solution for shared memory concurrency. But I think you really need to design the language around "no GC" in order to make that really ergonomic relative to C++. The entire C++ language and library ecosystem is based around making manual memory management (relatively) easy to use; going back to malloc and free is a hard sell.
Have a source for that? I find it pretty dubious.
If you're looking to multicore for performance with javascript then you're using the wrong language. Correct memory layout and access patterns will give you a real-world 50-100x win.
Look for asm.js threads on HN: virtually every time it shows up someone brings up shared memory multithreading.
SharedArrayBuffer is the direct result of this popular demand: https://blog.mozilla.org/javascript/2015/02/26/the-path-to-p...
Everything that has to be thread-safe uses channels. I use channels to sanitize everything for a purely synchronous game loop. As an optimization, in cases where there are atomic operations available, there are places where concurrent code can mutate values visible to the game loop, but this is strictly an optimization technique, to be used judiciously. (So only values like "speed" can use this technique. Anything that's a reference is verboten.)
How would that work with multithreading? (Assuming you had a thread-safe GC, which Nim's isn't.)
I didn't realize Nim's GC wasn't thread safe. In my current architecture, you'd only have to worry about the part using channels to sanitize things for the synchronous game loop. If everything outside of the game loop was written such that most everything was allocated on the stack, the GC would never have to collect anything outside of the game loop. So maybe it could work as a port. I couldn't say for sure, though.
It should be easy to bootstrap by first using Go's parser to dump some AST that Nim code can then translate, and then once that version is done, just reference the Go parser using this new impl. The only real struggle with the project from what I can see is deciding which standard library items to use the Go transpiled implementation (probably most of them) vs which need to have Nim backends (e.g. runtime package). Meh, just rambling thoughts in my head I was considering playing with given the time...
Nim's macros have one limitation that prevents an accurate implementation of another syntax: they can't fully modify the existing syntax.
See how I had to use "scase" inside "select" blocks because the existing "case" keyword insists on having "of" after it. So Nim's semantics put some limits on the amount of hijacking one can inflict on it through macros.
macro goImport(path: string): stmt
// TODO
discard
goImport("golang.org/x/crypto/nacl/box")what ?
I seriously doubt that Nim is out of the box binary compatible with another runtime's fundamental types.
Basically, I'm not sure of the benefit of building go on top of nim. They appear to fill the same space in orthogonal ways.
I'm hoping to see Rust get green threads/tasks/goroutines too. I'm working on a GC myself, and hopefully someone is trying out a green thread scheduler.
Nim also has lightweight coroutines using `async` and `await` (http://nim-lang.org/docs/asyncdispatch.html) - you can run a bunch of these within one thread.
Also, have a look at gevent for Python.
gevent is too much magic. It aims to give you async without changing any of your code. To accomplish this, it monkey-patches the entire Python standard library, in a way that is 99% compatible with Python, but the 1% will constantly surprise and infuriate you. Its compatibility shows no signs of increasing given how much its development has slowed down.
You can use gevent as a quick hack, but you will hate yourself if you have to maintain gevent code.
I use gevent both for small and large projects, and haven't had any complaints. The monkeypatching pains my soul just a little bit, but I've found no better async framework for Python yet.
It's a mature language. You can look into that?
It does. Don't forget that this is gccgo so it is possible to use plain C functions as goroutines. Nim is translated to C and with the help of a macro I convert Nim functions with an arbitrary number of arguments into ones with a single void* arg that gccgo wants for its 'go' keyword implementation:
extern void* __go_go(void (*f)(void *), void *);No.
> How does the Nim code yield the thread for other goroutines, does it have to register a callback?
There are no callbacks. Yielding happens automatically when launching another goroutine, when sending, receiving or selecting on a channel. You can also yield explicitly with go_yield() - the better named equivalent of Go's runtime.Gosched().
It's easier to understand if you realize that all those operations with goroutines and channels end up being done in the Go runtime.
>No.
Are you sure? Once a thread enters cgo it is considered blocked and is removed from the thread pool according to these sources [1][2]. I previously found a thread where Ian Lance Taylor explained it more explicitly but I couldn't find that now. Is that not what is happening though when __go_go invokes your function pointer?
I do not understand how the Nim code can live in the segmented stack of a goroutine, nor how the Go runtime could know it is time to grow that stack.
[1] https://groups.google.com/forum/#!searchin/golang-nuts/gorou...
[2]http://stackoverflow.com/questions/27600587/why-my-program-o...
Yes. See the chinese whispers benchmark with 500000 goroutines and a maximum resident set size of 5.4 GB on amd64. It has the same memory usage (and run time) as the Go version compiled with gccgo.
> Once a thread enters cgo
This has nothing to do with cgo. It's a different mechanism specific to gccgo.
> I do not understand how the Nim code can live in the segmented stack of a goroutine
Good thing you asked. I just ported to Nim the peano.go benchmark described as a "torture test for segmented stacks" and... it failed. The fix was to pass -fsplit-stack to gcc when it compiles the C code generated by nim.
> nor how the Go runtime could know it is time to grow that stack
From what I can tell it's done in __splitstack_makecontext() and friends from GCC's libgo.