Porting a NES Emulator from Go to Nim
hookrace.net
hookrace.net
Win!
Once you've built a C-to-JS compiler, and a Nim-to-C compiler, you've got a perfectly-functioning Nim-to-JS compiler. There's no game of telephone gradually confusing things; the fidelity is perfect no matter how deep the pipeline goes.
(This is also what's amazing about layered packet-switched networks, come to think of it. The core switch transferring data from your phone to AWS doesn't need to speak 4G or Xen SDN; it's just coupled to them, eventually, by gateways somewhere else, and it all just keeps working, one quadrillion times per second, all over the world.)
For example, here are a list of ways in which the C-to-JS compiler is far less than "perfect": http://kripken.github.io/emscripten-site/docs/porting/guidel...
In software, though, "imperfect" means "will work perfectly, 100% of the time, even after a quadrillion tests, within a well-specified subset of all possible inputs; will fail in arbitrary unknown ways outside of that subset."
It's a difference to being "within tolerance"—even within tolerance of a physical system, stresses are still going on. Physical systems have a lifetime, and repair procedures, for a reason.
But in software, you don't have to worry about "stress within tolerance." In fact, even if someone else builds an "imperfect" software system that accepts inputs with undefined results, you can just wrap it with an input validator, and suddenly it's a well-defined-100%-of-the-time system!
(Of course, in implementation, software has to run on hardware, which is the other kind of imperfect system. But, surprisingly, you can write software to compensate even for that, with failover and quorum-agreement &c.)
Unix Philosphy right there (right?)
My favorites:
> I decided to re-implement Javascript in Javascript. It failed. Here is my story
> ReactOS running IE6 in a JavaScript x86 emulator: we put a browser in your browser so you can browse while you browse
> Introducing js.js: a JIT compiler from JavaScript to JavaScript
I thought that Go would be the last language I'd write by hand. Previously I wrote C++, which was a dead end in that I could never use tools to parse it and translate to a new language. But with Go it should be much easier to do that if/when I ever decide to switch to something else.
The performance of the emulator in browser (compiled via emscrimpten) is very impressive! It felt like solid 60 FPS to me. I wonder how the Go version compiled via GopherJS would compare? Have you tried?
I haven't tried GopherJS, but I don't have much Go experience apart from reading it. On a related note, I read that porting the Go version to Android would be difficult and is still far in the future: https://github.com/fogleman/nes/issues/7
- http://hookrace.net/blog/what-is-special-about-nim/
- http://hookrace.net/blog/what-makes-nim-practical/
Summary by benhoyt: https://news.ycombinator.com/item?id=8822918
* Run regular code at compile time
* Extend the language (AST templates and macros);
this can be used to add a form of list comprehensions to the language!
* Add your own optimizations to the compiler!
* Bind (easily) to your favorite C functions and libraries
* Control when and for how long the garbage collector runs
* Type safe sets and arrays of enums; this was cool:
"Internally the set works as an efficient bitvector."
* Unified Call Syntax, so mystr.len() is equivalent to len(mystr)
* Good performance -- not placing too much emphasis on this,
but it's faster than C++ in his benchmark
* Compile to JavaScript* Run regular code at run time - does this just error if you try to use const on something that depends on a runtime value?
* Add your own optimizations to the compiler - isn't this very dangerous, especially if optimizations from various sources of code get combined? Even if your optimizations are valid (which there seems to be no guarantee of), computer math is notorious for being different from real math. People unaware of the nuances of integers rolling over or floating point seem like they could easily shoot themselves in the foot here.
Yep. For example `const foo = stdin.readLine()` would result in "Error: cannot evaluate at compile time: stdin".
As for your second point. I haven't used this feature personally but I have heard that the compiler will tell you exactly where these optimisations are applied. You can also disable these optimisations very easily during compilation.
* Yes, with compiler plugins
* Yes, with macros
* Maybe. I think that you could use a compiler plugin
to do arbitrary transformations on all the code in a
module, but I haven't seen an implementation.
* Yes
* What garbage collector?
* Yep
* Kind of. Any method can be invoked as a function with
the <type>::func(val, args...) syntax, but the
opposite's not true.
* Check
* Yep, via Emscriptenhttps://www.reddit.com/r/rust/comments/2yfsi9/rust_to_js_wit...
* Haskell can run regular code at compile-time (But it is slow, using ghci/TH)
* Haskell can add optimizations (rewrite rules and plugins). I think Haskell innovated rewrite rules and Nim probably was inspired by that?
* Binding to C easily is also a Haskell feature
* Type safe sets and arrays of enums: Is this not just a library? Haskell sets and arrays of enums are type-safe by default?
* Unified call syntax is true of Haskell too
* Compile to Javascript -- Haskell can do that too
- Predictable performance (lazy evaluation is problematic)
- Easy to achieve high performance
- Easier to read, even for non-specialists
- I'm much more productive in it
- Multi-paradigm: Nim is mostly imperative, but OO and functional can be mixed as well.
That's the point! I know many languages and Nim is by far the most productive one. It's like coding in Python with the assurance that the compiler will catch many errors that would cause runtime errors in Python. That assurance makes coding in Nim faster than in Python because I don't have to think so much about avoiding runtime errors.
Does Nim compile to C99 ?
If someone were to ask what's wrong with Nim, what would you say are the things that need to be fixed?
There are quite some compiler bugs remaining in Nim and the standard library could be improved.
Transpiling to C gives you the opportunity to emit code that is straightforward clean and fast C for most operations, which avoid an abstraction overhead.
Before you reply to disagree, make sure to very, very carefully think through the implications of the last sentence there. It is not as if compiling to C is particularly hard; it is literally "homework assignment for a normal college senior in a compilers course"-difficult. Many languages have gone through a phase where they compile to C. Usually they deliberately leave C at some point, another thing to carefully ponder the implications of before replying in disagreement.
Yeah, and you could do this too at the start of your emitted code from the transpiler:
sleep(UINT_MAX)
which would also make it slower than an interpreted language. But I specified I wasn't talking about that, or about "just inlining the core interpreter loop", but about transpiling letting you also emit straightforward and clean code without extra abstractions for most operations.
A new language like Nim doesn't come with all the baggage of Python's runtime pre-existing, and doesn't need to recreate it.
That's reality from a company with a shitton of money and a vested interest in doing it successfully.
They can't do it in the general case because reality disagrees with you.
What I said: "unless done badly" (my words on the original comment), it's fast, as it gives you a chance to emit straightfoward C code for most operations. Now, regarding your example:
First, HipHop is not transpiled to C. It's a runtime with a JIT.
Second, what does "HipHop not supporting the entirety of the PHP runtime" has to do with anything?
I take it you mean that the fact they had to skip some parts of the runtime for increased speed, proves something related to what I wrote. But (besides the JIT thing) PHP is not greenfield like Nim. They have a runtime, and it has to work in a certain way. PHP wasn't designed with transpiling in mind, and HipHop had to follow it closely as a design goal. I never said runtime decisions you have to mimic to be 100% compatible can't slow you down.
> HipHop for PHP (HPHPc) is a PHP transpiler created by Facebook. By using HPHPc as a source-to-source compiler, PHP code is translated into C++, compiled into a binary and run as an executable, as opposed to the PHP's usual execution path of PHP code being transformed into opcodes and interpreted.
Still, how does the original version of HipHop as a transpiling compiler invalidate what I said?
I said (check my original comment): unless you do it badly, transpiling to C gets you to run faster, because it lets you translate most operations to straightforward and fast C code.
And that's exactly the logic they followed and what they achieved: they transpiled PHP to C, as opposed from running it with PHPs runtime interpreter, in order to make it faster, and it worked.
That they skipped some parts of PHP runtime behavior/semantics, which if they didn't it would slow things down, doesn't clash with what I said. In fact it's already covered in my comment: "it let's you translate MOST" (not all) operations to fast C code.
You're changing the context.
> while being fast since it's compiled to C. - them
> being compiled to C doesn't imply fast :) - me
> Unless done badly, it does. - you
You've switched to "faster" because it's a more defensible position, but that isn't the idea that was originally being responded to. That's why jerf said the following, emphasis mine:
> No, it doesn't. You could easily compile Python into C (basically, "just" inline the core interpreter loop), __but it will still be slow__
Which is fast compared to what? Being compiled to JS?
It's not like being compiled to a "fast language" automatically makes that compiled program fast. You also have to consider how complex the runtime is, and other variables which are probably a direct result of the semantics of the language.
So it is Go++ if you wish.
But I am not sure about its concurrency. Go does have channels which are nice abstractions for doing concurrency right.
foo = cast[ptr Foo](alloc(sizeof(Foo)))
# do stuff with foo
dealloc(foo)
@if musl:
passL = "-static"
gcc.exe = "/usr/local/musl/bin/musl-gcc"
gcc.linkerexe = "/usr/local/musl/bin/musl-gcc"
@end
$ nim -d:release -d:musl c src/nimes
$ ls -lha src/nimes
-rwxr-xr-x 1 def users 219K Mai 1 22:00 src/nimes*The Nim -> C -> emscripten path is very impressive. Kudos.
I'm not sure what Go does on OS X, but I know that it's not possible to statically link the c library on OS X.
Question about nim: from looking at https://github.com/def-/nimes/blob/master/src/nes/cpu.nim , I wonder: is there way to give the no. of cycles and instruction encoding with the "op" template, so those 256 byte arrays get built automatically?
cycles[opcode] := ncycles
# using the pragma here, but we could use a 'static' block instead
var cycles {.compileTime.}: int
# ---
proc doSomething =
static:
# invoked once at compile-time (at this definition)
cycles += 1
proc doSomethingGeneric[T] =
static:
# invoked once at compile-time (per unique generic call)
cycles += 1
macro doSomethingAtCompileTime(n): stmt =
# invoked at compile-time (per call)
let ncycles = int n.intVal
cycles += ncycles
# ---
doSomething() # this call doesn't effect 'cycles', it's declaration does (+0)
doSomething() # ditto (just here to prove a point) (+0)
doSomethingGeneric[int]() # this call effects 'cycles' (+1)..
doSomethingGeneric[int]() # ..but only once (+0)
doSomethingGeneric[float]() # this call also effects 'cycles' (+1)
doSomethingAtCompileTime(5) # this call effects 'cycles' (+5)
doSomethingAtCompileTime(12) # ditto (+12)
static:
echo cycles # prints '20'
# note: 'impl', 'spawn', and 'invoke' are macros
type
Foo = object
val: int
Bar = object
val: string
impl Foo:
proc update = echo "Foo: ", me.val
proc render = echo "Rendering Foo!"
impl Bar:
proc update = echo "Bar: ", me.val
spawn Foo(val:123)
spawn Bar(val:"abc")
invoke update
invoke render
invoke blah
# ouptput:
# Foo: 123
# Bar: abc
# Rendering Foo!
# Invoke Error: No event 'blah' defined.
If you look at line 531 of the CPU source code (as of today, anyway), there are multiple 256 byte tables that give instruction encodings, lengths, and cycle counts.
What I was asking is: "Is it possible to put these as a parameter of the 'op' macro so that it builds these tables automatically"
The answer might be "No" if e.g. a single op has multiple instruction encodings. But assume that there is only one encoding per op.
An equivalent question is - can static: sections assign to an array that will be available at runtime? From your example, the answer appears to be yes.
The answer is yes, but it's a tad trickier than just accessing the compile-time list from run-time code (which doesn't make sense, and is illegal). Instead, use a macro to generate a bunch of run-time checks against a specific value. Eg:
var eventNames {.compileTime.} = newSeq[string]()
proc defineEvent(name:static[string]) =
static:
eventNames.add(name)
macro checkDefined(name): stmt =
# begin new statement
result = newStmtList().add quote do:
echo "Checking for '", `name`, "'"
# loop over every known event name and
# build a run-time 'if' check for each one.
for n in eventNames:
result.add quote do:
if `n` == `name`:
echo "Found it!"
# add some events to compile-time list
defineEvent("foo")
defineEvent("bar")
# define some runtime values
let eventName1 = "foo"
let eventName2 = "blah"
# check runtime values againts compile-time list
checkDefined(eventName1)
checkDefined(eventName2)
# output:
# Checking for 'foo'
# Found it!
# Checking for 'blah'
static:
# define a compile-time list first
var names = newSeq[string]()
# add some values (at compile-time)
names.add("foo")
names.add("bar")
# define the compiler vars as run-time const
const runtimeNames = names
# define some run-time variables
var name1 = "foo"
var name2 = "blah"
# check runtime variables against const variable
if runtimeNames.contains(name1): echo "Has Foo!"
if runtimeNames.contains(name2): echo "Has Blah!"
A simple version already exists and works (for Python primitive types and Numpy arrays, via the Python C-API), but it's embedded in my company's proprietary Python+Nim (mainly Python) codebase. I'm working in my spare time to extract the relevant code as a Nim library and release it as an open-source package on Github.
If you'd like to learn more about it, or you'd like to be notified when the first release is ready, please come and discuss it on the Nim forums! http://forum.nim-lang.org/
If you depend on certain Python libs that's understandable. But it should not be too hard to translate them to Nim since the syntax of Python and Nim are not very different.
(See also this classic Joel Spolsky article, "Things You Should Never Do, Part I": http://www.joelonsoftware.com/articles/fog0000000069.html )
This would be the case whether you schedule the re-write in a single blocking development effort (in which case, all forward progress would stop during that time) or broken into batches over time (in which case, it will be much longer until the new system is ready, and the old system will be a moving target as it continues to be developed).
Instead, the chances of a Nim-integration being beneficial to the company (and thus, your chances of getting approval from your engineering manager) are MUCH higher if you can simply write NEW functionality in Nim (or occasionally re-write small, self-contained inner loops in Nim) and the new Nim code integrates smoothly into the existing Python as a Python module.
This is the approach I've taken at my company (with my engineering director's approval).
In theory, you could even use skunkworks-Nim in your large Python codebase, as long as your Nim code presents itself as a good-citizen Python module, much like the tales of skunkworks-Scala being used in large Java codebases.
http://www.quora.com/Why-do-large-corporations-use-Java-or-C...