Hacker News Reader

Hacker News

Comment by rdtsc | Hacker News Reader

Full thread

rdtsc

Wow, lots of goodies. Erlang team has been unstoppable in the last few years.

Some changes:

* New state machine module: gen_statem (ssh module uses it as a first example).

* 3-5x times faster open_port(spawn, ...)

* Must faster and better tracing (also lttng support)

* Keeping messages off heap for processes

* mnesia-ext : pluggable storage for Mnesia. This removes the 2GB per shard limit.

* Allow limiting process's max heap usage

* Observer tool allows configuration update frequency and length of graph windows

* crypto uses EVP calls so should be accelerated by hardware were supported

* Parallel/faster module code loading

View on HN

> pluggable storage for Mnesia. This removes the 2GB per shard limit.

This is a pretty big deal isn't it? I've never used Mnesia but that 2GB limit was a common warning I came across in the docs and blog posts, despite noting it was a great DB to work with.

Yap pretty big.

Allegedly this has already been running at Klarna in production for a while. LevelDB backend from Basho (eleveldb, it has extra optimizations and improvements besides just Erlang wrapper) has been used.

Granted 15-20 some years ago perhaps 2GB was BigData(TM) ;-)

For a while I understand you could set up sharding, so each shard is 2GB and spread shards across nodes.

Lots of good stuff, also excited for:

* dirty schedulers improved

which means much better scheduling of long running NIFs from what I understand:

https://medium.com/@jlouis666/erlang-dirty-scheduler-overhea...

I couldn't understand if they are enabled by default or not still. I didn't see the announcement mentioning that...

Yeah, saw in the root link but R19 docs still show it as experimental and disabled by default, shucks.

> Allow limiting process's max heap usage

What happens when it hits that limit? Crash, or try to GC, or... ?

Haven't looked into details. But I think it is configurable. Killing is an option or just sending a message to the error logger. There was also something about it sending a special gc trace event when it hits max.

http://erlang.org/documentation/doc-8.0-rc1/erts-8.0/doc/htm...

Configurable, but can kill the process in question and/or log when it occurs. Their suggestion is to log it first and tune it from there since heap size isn't easily predictable.

EDIT: Scroll down a little bit to get this particular flag documentation.

Hah, Mnesia doesn't have a 2GB limit per shard now. But having access to a leveldb backing store means you can avoid keeping the dataset resident in memory. This gives you a nice memory/latency/disk tradeoff.

Oh right. That's for disk_only shards. Regular shards use the disk log and those don't have a limit if I remember correctly.

Keep doing it, Erlang team !!!

Reply on news.ycombinator.com