Congratulations.
I wish BEAM could be as effective as or close to the performance of JVM for Numerical Computation.
I wish BEAM could be as effective as or close to the performance of JVM for Numerical Computation.
If you want to leverage highly performant numerical code, I suggest to link against a native C library from BEAM using NIFs via http://erlang.org/doc/tutorial/nif.html or connect a C program to the Erlang network via http://erlang.org/doc/tutorial/cnode.html
If you really like you can choose to compile things using HiPE which in my tests can make mathematical stuff about 10X faster... it sounds pretty unsupported these days though. Maybe just put messages on a queue and process them in golang or rust if you must have the performance, no need to let the BEAM have to handle such things directly and always good to measure where your bottlenecks are first.
You have to take immutability and multi-processing into account. Each process has a separate heap as a design assumption, so if you want to pass data between them, you must copy. Data is immutable as another design assumption, and this means that if you want to change something in a bigger data structure, you also must copy.
Theoretically, the compiler is free to optimize the latter - if it notices that some code can be safely optimized in a mutating way, it is free to do that, but then, someone needs to write that optimization - and the fact that this is not a priority for BEAM means that it is unlikely that someone will spend time to implement it if there are more pressing matters for them to do so. Then, not only it must be written, it must be proven that this optimization does not break anything.
Big binaries aren’t copied; they’re passed by reference to a shared ref-counted heap (where each process holds one reference to the binary on the shared heap, and the process’s heap can hold N references to the far reference pointer.)
In theory, more types could be made to do that.
In fact, I believe that handles passed back from NIFs actually reuse this infrastructure, presenting themselves as binaries such that sending the handle to another process is really just creating a new smart pointer in the new process-heap, holding the same raw pointer that the original NIF handle held.
And you can do a lot with that. Erlang’s new-ish `counters` module, where you get a handle to a mutable uint64 array, is essentially just a built in NIF that passes around a mutable NIF handle pointing to a uint64[] buffer, exactly as above.
There’s nothing stopping the Erlang runtime (or your own NIF code) from adding all sorts of other operations against native, mutable types, and exposing them as these sorts of abstract data structure handles. You could, for example, have a `matrix` module for doing matrix math; or a `data_frame` module that can participate in every operation a Pandas.DataFrame can (with all such operations implemented as native code); or a module exposing SHM buffers; or even a module exposing GPU driver primitives (e.g. vertex/shader/texture buffers).
Everything that the bytecode would do to such ADTs would be slow-ish, but the point would be to load stuff into them in your glue code, and then in your hot loop, just bump already-loaded ADTs together in fast, native ways.
I have no idea how they prioritize the new features to be built but IMHO safely optimizing code in mutating way is the reason why Haskell is so fast despite the heavy abstraction primitives it offers.
Also, what is the fun in writing perf code in another (native) language if host language is (or can be) capable by itself ?
I used the reference of Haskell just to make a point that "Immutable Abstraction can be backed by safe mutable implementations"