The Zen of Erlang (2016) | Hacker News Reader

The Zen of Erlang (2016)

ferd.ca

236 pointsjxub23 comments

Does anyone know of erlang being used for scientfic computing? The ideas and "let it crash" scheme outlined in the article are really cool, and it made me think how it would actually be very useful for making robust data pipelines in scientific contexts. For example, in astronomy you would have a telescope spitting out raw data, and a pipeline processing the data to create more inspectable intermediate products. Typically these are brittle C/C++ programs, with lots of state and branching logic, and further they become extremely hard to refactor because their function is so integral to running the facility. Erlang seems like it would be very useful for a control and pipeline system like this that needs to stay alive and not crash, but it's hard to tell (for me) if it's well suited to numeric/scientific computing tasks. Do people know of libraries or resources that would be useful for that? I did find this neat lsci[1] package that aims to call python modules like numpy/matplotlib (which would be perfect) but is sadly in a nascent state.

[1] https://github.com/lfex/lsci

Erlang's main weakness is computationally intensive mathematics. You could always write a NIF in C if you needed pure speed for a specific function. If you're not interested in writing safe c code it looks like there is a bridge from rust to erlang.

https://github.com/hansihe/rustler

Even if you write NIF C or in Rust. They can take down Erlang's BEAM VM.

> http://erlang.org/doc/tutorial/nif.html

Erlang itself shield you from actually doing stupid thing and follow the assumptions that BEAM have while running programs.

You're better off choosing a different language. Have Erlang deal with concurrency but don't have it do number crunching.

Rustler in particular boasts safety and an inability to crash the Erlang VM. Whether that holds true in practice, I don't know, but it's the first thing they mention in their README.

If you need fast small function use NIF, for large logic blocks better use C node its safe and isolated.

You should probably use erlang to start/monitor/kill process but not for doing the actual computing

I have thought about this too, even doing things like modelling climate change through distributed message passing (each process could act like a cube of the earth) but sadly I think the performance is going to be too far from C to be useful. HiPE has seen some very large performance increases in maths performance but still C is going to alway be a substantial amount faster.

> "Typically these are brittle C/C++ programs, with lots of state and branching logic, and further they become extremely hard to refactor because their function is so integral to running the facility"

Switching to Erlang won't inherently make those issues disappear. Erlang could definitely save you some C/C++ boilerplate code, but if your business logic is still stateful, it's not necessarily going to be easier to implement any of that with Erlang.

Erlang is probably more forgiving on the developers than C/C++ are, but that can be said of many other languages (ie. Python, Java).

If you're looking to pass messages around between processes/threads, there are plenty of libraries you can use today to start refactoring that C/C++ code:

http://nanomsg.org/

http://zeromq.org/

https://www.rabbitmq.com/ (most likely an overkill for the specific use-case you described, will be more relevant in a distributed computing scenario)

What you need should be a language/framework to better model the complex state of your system, and handle distributing processing, monitoring, fault tolerance, recovery and retries.

Consider a functional programming language with immutability and pattern matching, which should help a lot.

> Consider a functional programming language with immutability and pattern matching, which should help a lot.

So Erlang then

Erlang is not particularly performant, and it seems like what the top comment is describing requires something faster.

You only use Erlang for the orchestration part. Probably Fortran or something for the actual number crunching.

I'm suggesting Ocaml. ^__^

It doesn't have something like OTP (there are libraries like LWT for concurrency and IPC tho) but allegedly should have good math performances, from what I recall the last time I've played with the "big languages shootout" game...

Chapel and Julia have much better acceptance in HPC, than Erlang ever will.

The language is not designed for number crunching.

I think the typical experienced Erlang user would keep the C/C++ programs, but manage them through a fabric of Erlang code.

That is, each part is cut up into a small program, and these are controlled by the Erlang system so it acts like a supervisor of the programs. They are run under some kind of "visor" so their resources are limited to sane values etc.

When the C/C++ program does something bad, the Erlang system is notified of this. It can then take action in order to keep the processing pipeline running.

You could argue that in the current context of containers, this is easy to achieve with them. But I claim writing all of this in a programming language is far more manageable than trying to handle the rate at which the whole container-context is evolving.

There are some examples. See https://youtu.be/xj3smNjGLaE

If you like this, I recommend checking out Learn You Some Erlang (for great good) http://learnyousomeerlang.com/content

The blog author is the author of Learn You Some Erlang btw (just for others reading this, since it wasn't clear from your comment)

His other writings are good too. I recommend Erlang in Anger.

I've also written http://propertesting.com/, which I'm currently working to turn into a full book with Pragmatic Programmers (with Elixir translations included as well, so that everyone on the BEAM can benefit :) )

I really like the "let it crash" philosophy. I have been working with elixir for a year now and it makes software much more reliable and easier to write.

"We all know the saying it’s better to ask for forgiveness than permission. And everyone knows that, but I think there is a corollary: If everyone is trying to prevent error, it screws things up. It’s better to fix problems than to prevent them. " [1]

[1] - https://signalvnoise.com/posts/2440-we-all-know-the-saying-i...

That s a MUST READ

Reply on news.ycombinator.com