Erlang/OTP 27.0 Release Candidate 1 | Hacker News Reader

Erlang/OTP 27.0 Release Candidate 1

erlang.org

129 pointslobo_tuerto25 comments

Open articleView on HN

For an ecosystem as large as Erlang/OTP, it actually has surprising little investment relative to its size.

An area impacted the most, is R&D into improving raw performance.

It's effectively just Lukas Larsson & Björn Gustavsson trying to solve this problem for the past 10-years or more.

Their work is hugely appreciated, yet they get little to no support on this front.

It's definitely beyond my technical abilities to help, but I would financially sponsor focus in this area.

Does anyone know how I can donate/sponsor more focus on improving raw performance?

I don't have a direct answer for you, but the Erlang Ecosystem Foundation seems like a good place to start.

https://erlef.org/

> It's definitely beyond my technical abilities to help

I think this goes for most people that would like to help. This is very mature code and ripping out any portion and replacing it without a massive amount of groundwork isn't going to fly, Erlang is used for semi-realtime mission critical stuff all over the globe. That's 'tread very carefully' ground.

One can help on a technical level without providing fully baked solutions.

Finding (or developing) real or at least reasonable uses of Erlang where performance is lacking in an unreasonable way.

Narrowing those use cases into a minimal test case.

(Much harder!) Finding a way to make such a test case work significantly faster, even if it breaks other things, can be a helpful step towards a polished improvement.

Some areas of OTP are very well polished and difficult to improve. Some areas are kind of the first thing that worked and it was good enough and never got much polish. There's less of the second kind than there was before, but when you hit those, it's often pretty easy to improve, and a bug report or proposed patch will likely be processed by the upstream team quickly.

I find Erlang amazing, but it's important to note that a lot of OTP is written in Erlang and usually easy to read. Most of the C code is fairly easy to read too, once you get used to it. It's a very approachable ecosystem, and I think people should be encouraged to peek under the hood, and tinker.

Here's a free idea for performance improvement, although it may not be good. Erlang's JIT is very simple, because it needs to run for every beam file at load time. So this idea comes in two parts. a) Could you get a minor reduction in beam loading overhead by caching the native code output of the JIT processing. b) Are there ways to improve the JIT output that are too slow to run just-in-time, but could be run to improve the cached output and loaded in the future. Of course, there's the elephant in the room of making sure cached native code is compatible with the current BEAM, in case one JITed with a newer or older version. And the current JIT is successful by being simple, an earlier version of JIT tried to output optimized native code and was too complex and was abandoned. I think a motivated person with a good amount of experience with systems programming, mixed language experience, and willing to beat their head against the wall of dlopen or however the JITed code gets loaded could learn a lot working on this. And if it works or at least shows promise despite not completely working, share it and see where it goes.

I think the most exciting part is the new documentation format, as someone who primarily works in elixir, I do reference the erlang standard library every now and then so an easier and familiar documentation format is welcomed.

I just hope that "h :ets" works in iex later (for those who don't know, it shows the help/module/function documentation of the value specified)

That was implemented since OTP 24, in 2018, and works fine in iex already.

https://www.erlang.org/doc/apps/kernel/eep48_chapter

https://www.erlang.org/eeps/eep-0048.html

https://github.com/elixir-lang/elixir/issues/7198

Fascinated to see the mutable tuple optimization. Tuples are dense contiguous vectors in memory, which provide O(1) access - like binaries, but unlike lists.

It would be amazing if tuples could also provide O(1) mutable update, in special situations when the original value is not accessed again.

The Release Notes show record syntax example.

Most tuple changes in code are done through pattern-matched deconstruction, then independent re-construction, which is impossible for the compiler to recognize.

So the first step to wider optimization would be common use of 'setelement', very well hidden in the erlang module (i.e. not 'set_element', and no Elixir 'Tuple.set' or 'Tuple.update_at').

Then later, that function would get optimized away in many situations. Given that tuples can be up to 60m elements, it would give Erlang a big boost to have O(1) mutable vectors (note the `array` module is a functional tree, not a mutable 1D vector).

I am thinking about graphics and data analysis - a huge win. Elixir could also benefit tremendously. Various packages, like 'array' module in Erlang, and 'Nx' in Elixir, would be radically changed. Maybe Elixir could extend '[i]' syntax to access tuples.

Let's go for OTP 28!

Most interesting might be the new "tprof"-tool that allows for heap profiling

This will be huge. One of the biggest issues I had running an erlang system that moved lots of arbitrary sized, untrusted data through it in a parallel way was understanding what was going on in the process heaps and reference counted shared binary heap. Especially the shared heap since there's no way to set an upper bound on it, and when you exceed it the VM crashes.

erlang:process_info(Pid, binary) for all processes with some post processing was usually pretty helpful for me, although I don't think it tells you much about ets tables, which is where my big bloat usually was. If you store binaries in ets (or mnesia) and the binaries live for a long time, it's useful to run them through binary:copy/1 if the binary is a sub-binary that references a large binary that's otherwise garbage. Or at least this used to be useful; sometimes my experiencs get obsoleted when OTP updates have efficiency changes.

It's exciting to see the Elixir community contributing to this release: the key feature called out was the new documentation features, which were authored by Jose Valim, the creator of Elixir.

https://www.erlang.org/eeps/eep-0059 Author: José Valim <jose(dot)valim(at)gmail(dot)com> Status: Draft Type: Standards Track Created:02-Jun-2021 EEP 59: Module attributes for documentation #

congrats on another awesome release. such an incredible runtime system. I keep solving the same problems with half assessed solutions that I would otherwise get for free from Erlang.

If they add fstrings, some type of easier map get/put syntax eg #map.value = 1, and maybe a shorter hand fun syntax, then Erlang feels like it's gotten all the conveniences of python. Amazing how far things have come

Reading the change logs speaks volumes to the maturity of this ecosystem. A lot of really great quality of life improvements here and the always welcome performance improvements.

Congratulations and thank you to everyone involved!

Nice to see cross-pollination between Erlang and Elixir

indeed. elixir brought a lot of new life and momentum into the ecosystem.

I dabbled in erlang a few years back and while I saw some potential in it, I just didn't see it as my daily driver for new projects. Then elxir came around and felt like the semantics of clojure and the syntax of ruby running on top of the vest virtual machine for server apps. match made in heaven.

I'm curious what the default memory usage will be with the raising of the default process limit to 1_048_576?

I think it's a good move, if nothing else people who do benchmarks with 1 million processes but don't know those settings are changeable will get further.

I don't think there's a significant cost to raising the limit. If you actually run 1M processes, that'll use some memory, of course. The Erlang Efficiency Guide [1] says a process uses 338 words when spawned, which is 2704 bytes on 64-bit machines. It's certainly possible some per process overhead was missed in this accounting, I did not measure to know for sure; and also I think the published guide is from OTP 26.

[1] https://www.erlang.org/doc/efficiency_guide/advanced#memory

Nice to see JIT improvements.

Open Telecoms Platform. Not one-time password. I had to look that up.

OTP has been around probably longer than you have.

Not true. Erlang is from 1986. I programmed BASIC in 1980 and Pascal in 1983.

Of course I have heard Erlang. Wikipedia says Erlang/OTP is a synomym. That I managed not to notice for 38 years.

Essentially a synonym ya. Within the BEAM community when people say OTP they often are referring to Erlang's massive standard library (which includes stuff like process management, a key/value store, a relational database, a state machine, and a ton of other goodies) whereas Erlang refers to the language itself. Eg, when programming in Elixir we use OTP, not Erlang.

Very cool!

OTP doesn't mean Open Telecoms Platform anymore. Yes it's on Wikipedia because that was the historical name and use case, but the Erlang maintainers and the ecosystem foundation the EEF refer to it simply as OTP, Erlang's standard library.

From Erlang's website:

>OTP is set of Erlang libraries and design principles providing middle-ware to develop these systems. It includes its own distributed database, applications to interface towards other languages, debugging and release handling tools.

Reply on news.ycombinator.com