Erlang/OTP 17.0 has been released | Hacker News Reader

Erlang/OTP 17.0 has been released

erlang.org

186 pointsrachbelaid34 comments

Learn You Some Erlang has a new chapter on maps for the occasion: http://learnyousomeerlang.com/maps

Thank you! That book has been an amazing resource. One of the few ones I still got in a dead tree format.

Even if you're not super interested in Erlang, I can't recommend this book enough: it's clearly a labor of love, and very nicely written. It's worth supporting that kind of effort.

Oh yeah. Erlang or not it teaches a lot of very useful principles -- fault isolation, building distributed system, functional programming. Those can be applied to general system design.

Reference manual for Maps: http://www.erlang.org/doc/reference_manual/maps.html

Manpage for Maps: http://www.erlang.org/doc/man/maps.html

Older documentation for Maps (EEP): http://www.erlang.org/eeps/eep-0043.html

Important note found on the reference manual page:

Maps are considered experimental during OTP 17 and may be subject to change.

The documentation below describes it being possible to use arbitrary expressions or variables as keys, this is NOT implemented in the current version of Erlang/OTP.

Exceptions returns badarg instead of badmap, this will change in the future releases.

I especially like this property of maps that cannot be done in either records or JSON ( had it been supported as a primitive)

    M4 = #{{"w", 1} => f()}.  % compound key associated with an evaluated expression

And of course self-referencing is a welcome addition (especially on the terminal)

    Fac = fun(0) -> 1; (N) -> N*Fac(N-1) end

The syntax would actually be:

    fun Fac(0) -> 1; Fac(N) -> N*Fac(N-1) end

Note that the name `Fac` is not visible outside of the fun's scope, and to use the same variable name outside, you still need to bind it:

    F = fun Fac(0) -> 1; Fac(N) -> N*Fac(N-1) end

Looks like this is pretty much identical to JavaScript?

Pretty much. Javascript has the same scoping rules for functions used as expressions:

    var f = function fac(n) { if (n==0) { return 1; } else {return fac(n-1)*n; } };
    console.log(f(3));
    console.log(fac(3));

This will log '6' and error out on the last one (fac is undefined). This is the same scoping Erlang uses for anonymous function names as far as I can tell, with one difference: Erlang's function name is a variable that can be used as such internally. In Javascript, the function name doesn't refer to a variable per se:

    var f = function fac() { console.log(fac) };
    console.log(f);
    f();

Which will log 'fac()' and 'undefined'. In Erlang:

    1> F = fun Fac() -> Fac end.
    #Fun<erl_eval.44.71889879>
    2> F().
    #Fun<erl_eval.44.71889879>

Both instances (F and Fac within the right scope) return the same anonymous function.

Are you sure you didn't make a mistake, and confuse the undefined that f() returned with an undefined output from the console.log(fac)? It looks to me like the function is a perfectly normal variable.

    var f = function foo() { console.log('Me:', foo); }
    f();
    Me: function foo() { console.log('Me:', foo); }
    undefined

You're right, I've been fooled by Firebug merging in multiple identical output lines together and missed the (2) it had put on the right hand side of the screen. That makes a lot more sense.

The addition of maps is really awesome. In my applications I've had records, which are kind of like C structs, being passed around to represent state. But if I need to add features the records become very constraining, because adding another key to them invalidates any previously existing records when you do a hot reload. So you have to jump through a bunch of hoops to get the records migrated, or just restart the whole system, which is less than ideal.

With maps this whole class of problems goes away.

>With maps this whole class of problems goes away.

A friendly warning from long time dictionary (ab)user. They introduce their own class of problems. Basically the strong vs weak type ones.

Agreed. Erlang's maps have two different operators when setting values on them, one if you care that the key already existed and one if you don't, so in that way you can still somewhat enforce strictness of keys. But you can relax that strictness when it comes time to roll out changes.

I see that, when you create several maps that have the same keys but with different values, the key-set is shared between instances, and this is basically what brings this particular implementation of maps up-to-par with replacing Erlang records in their use-case.

I'm curious whether this efficiency-in-storage translates to effiency-in-messaging, though:

• When you pass a map N times to another process, is the key-set copied N times? (Not that bad if true, so I'm guessing so.)

• When you pass a map N times over the distribution protocol, is the key-set serialized and transmitted N times? (This'd be pretty bad if true.)

The first one: Yes, it has to. Otherwise the keyset would be shared among processes and that is generally not possible in Erlang (Immutable binary data being the exception)

The second one: yes and no. The format is described at http://erlang.org/doc/apps/erts/erl_ext_dist.html and maps are laid out as a serialized construction like you say. Two points balances this out: There is an atom-cache which allow you to cache atoms and make their representation small. And you can zlib-compress the data.

It is not set in stone yet and might even change in a later versions. The term format is versioned so you can upgrade it later if need be.

Right, what I was asking in the second question is basically whether there exists, or is planned, a keyset cache to go along with the atom cache. I'd think that, if the atom cache is a good idea, a keyset cache would be good for exactly the same reasons.

If you do that, it is better to create an arbitrary caching construction of subtrees and then reap the benefit by tighter packing of data in general. I agree a keyset cache could be really nice to have going forward. It would resemble what happens in-heap.

But the rule of the Ericsson OTP team is to get it correct before making it fast.

Atoms and keysets both have pretty much the same caching semantics, though: they get repeated over the wire with pretty good locality, and don't have conflicting terms busting the cache in-between. That's not really true for anything else, which makes me guess that an arbitrary term-branch cache would be pretty useless.

The idea of having arbitrary subtrees is to support an efficient compression scheme. But you are indeed right that a keyset cache would be extremely effective at limiting the size of maps.

Just updated the HOWTO for installation on OSX Mavericks with a working Observer: http://blog.equanimity.nl/blog/2014/04/09/erlang-r17-on-osx-...

Here's the README:

http://www.erlang.org/download/otp_src_17.0.readme

It's also worth calling attention to Lisp flavored Erlang (LFE), which has come a long way recently. They've specifically addressed language artifacts that would be more welcoming to someone coming from a strong Common Lisp background.

For seeing where it's heading, read Duncan's website: http://technicae.cogitat.io/search/label/lfe He gave an excellent presentation on this at Erlang Factory conference in San Francisco last month.

Otherwise, the main website is: http://lfe.github.io

Unfortunately, I won't be able to use a lot of the improvements in my projects, since it'll take some time for everyone to adopt it. But maps and dirty schedulers (!!!) are both very welcome.

Do heed the warning in the README wrt dirty schedulers...

  Note that the functionality is experimental, and not
  supported. This functionality will be subject to backward
  incompatible changes. You should not enable the dirty
  scheduler functionality on production systems. It is only
  provided for testing.

Yeah, I was speaking with one of the developers on IRC. The API can still be changed, so now is the time to try it out and get my 2 cents in ;)

Erlang is more than a language, it is a philosophy, and the language is merely an expression of that philosophy. I strongly recommend that everyone read "Making reliable distributed systems in the presence of software errors". This is where Joe Armstrong describes the ideas that fed into the creation of Erlang. I learned a great deal from it:

http://www.erlang.org/download/armstrong_thesis_2003.pdf

A few quotes:

-------------------

The central problem addressed by this thesis is the problem of constructing reliable systems from programs which may themselves contain errors. Constructing such systems imposes a number of requirements on any programming language that is to be used for the construction. I discuss these language requirements, and show how they are satisfied by Erlang…

History:

1998 — Ericsson delivered the first AXD301. The AXD301 is the subject of one of our case studies in Chapter 8. At the time of writing (2003) the AXD301 has over 1.7 million lines of Erlang code which probably makes it the largest system ever to be written in a functional style of programming.

…Telephone exchanges are expected to be extremely reliable: Typically having less than two hours of down-time in 40 years

…We organise the software into a hierarchy of tasks that the system has to perform. Each task corresponds to the achievement of a number of goals. The software for a given task has to try and achieve the goals associated with the task.

Tasks are ordered by complexity. The top level task is the most complex, when all the goals in the top level task can be achieved then the system should function perfectly. Lower level tasks should still allow the system to function in an acceptable manner, though it may offer a reduced level of service.

Programming a hierarchy of tasks needs a strong encapsulation method. We need strong encapsulation for error isolation. We want to stop programming errors in one part of the system adversely affecting software in other parts of the system.

The essential problem that must be solved in making a fault-tolerant software system is therefore that of fault-isolation. Different programmers will write different modules, some modules will be correct, others will have errors. We do not want the errors in one module to adversely affect the behaviour of a module which does not have any errors.

…Our applications are structured using large numbers of communicating parallel processes. We take this approach because [it] provides an architectural infrastructure — we can organize our system as a set of communicating processes. By enumerating all the processes in our system, and defining the message passing channels between the processes we can conveniently partition the system into a number of well-defined sub-components which can be independently implemented, and tested.

Concurrent processes with no data sharing provide a strong measure of fault isolation. A software error in a concurrent process should not influence processing in the other processes in the system.

…In our system concurrency plays a central role, so much so that I have coined the term Concurrency Oriented Programming to distinguish this style of programming from other programming styles.

In Concurrency Oriented Programming the concurrent structure of the program should follow the concurrent structure of the application. It is particularly suited to programming applications which model or interact with the real world.

Concurrency Oriented Programming also provides the two major advantages commonly associated with object-oriented programming. These are polymorphism and the use of defined protocols having the same message passing interface between instances of different process types.

…In the real world sequential activities are a rarity. As we walk down the street we would be very surprised to find only one thing happening, we expect to encounter many simultaneous events. If we did not have the ability to analyze and predict the outcome of many simultaneous events we would live in great danger, and tasks like driving a car would be impossible. The fact that we can do things which require processing massive amounts of parallel information suggests that we are equipped with perceptual mechanisms which allow us to intuitively understand concurrency without consciously thinking about it.

When it comes to computer programming things suddenly become inverted. Programming a sequential chain of activities is viewed the norm , and in some sense is thought of as being easy, whereas programming collections of concurrent activities is avoided as much as possible, and is generally perceived as being difficult.

I believe that this is due to the poor support which is provided for concurrency in virtually all conventional programming languages. The vast majority of programming languages are essentially sequential; any concurrency in the language is provided by the underlying operating system, and not by the programming language.

The inability to isolate software components from each other is the main reason why many popular programming languages cannot be used for making robust system software.

…”It is essential for security to be able to isolate mistrusting programs from one another, and to protect the host platform from such programs. Isolation is difficult in object-oriented systems because objects can easily become aliased. (An aliased object is one where at least two other objects hold a reference to it.)” — Bryce

Bryce goes on to say that object aliasing is difficult if not impossible to detect in practice, and recommends the use of protection domains (akin to OS processes) to solve this problem.

…In a paper on Java Czajkowski, and Dayn`es, from Sun Microsystems, write:

“The only safe way to execute multiple applications, written in the Java programming language, on the same computer is to use a separate JVM for each of them, and to execute each JVM in a separate OS process. This introduces various inefficiencies in resource utilization, which downgrades performance, scalability, and application startup time. The benefits the language can offer are thus reduced mainly to portability and improved programmer productivity. Granted these are important, but the full potential of language-provided safety is not realized. Instead there exists a curious distinction between “language safety,” and “real safety”. ”

In this paper they introduce the MVM (an extension to the JVM) where their goal is:

… to turn the JVM into an execution environment akin to an OS. In particular, the abstraction of a process, offered by modern OSes, is the role model in terms of features; isolation from other computations, resources accountability and control, and ease of termination and resource reclamation.

To achieve this they conclude that:

… tasks cannot directly share objects, and that the only way for tasks to communicate is to use standard, copying communication mechanisms, …

…Many of the features of the Tandem computer bear a striking similarity to the design principles in the OTP system, and to the fundamental principles of Concurrency Oriented Programming which where discussed earlier.

Here are two quotes from the paper, firstly the design principles on page 15

There is considerable controversy about how to modularize software. Starting with Burroughs’ Espol and continuing through languages like Mesa and Ada, compiler writers have assumed perfect hardware and contended that they can provide good isolation through static compile-time type checking. In contrast, operating systems designers have advocated run-time checking combined with the process as the unit of protection and failure.

Although compiler checking and exception handling provided by programming languages are real assets, history seems to have favored the run-time checks plus the process approach to fault-containment. It has the virtue of simplicity—if a process or its processor misbehaves, stop it. The process provides a clean unit of modularity, service, fault containment and failure.

More modern work with object-oriented systems has also recognised the importance of isolating software components from each other. In [21] Bryce and Razafimahefa argue that is is essential to isolate programs from one another, and from the programs which run in the host operating system. This, they consider, is the essential characteristic that any object system must have. As they point out in their paper, this is a difficult problem in an object-oriented context.

Yes, yes, yes. It really is a philosophy. Great summary on his incredible (and quite readable) thesis.

Great ... maps is a worthy addition to the language :)

Get it at Erlang Solutions -- there are packages for Ubuntu, CentOS 6 and others.

After R16, we actually ended up packaging our own Erlang debs at $WORKPLACE (libssl depencency mismatch in latest wheezy; mismatched sources for xmerl & others; packages generated from a commit different than the release, with most app version numbers wrong; missing crypto applications because of some unknown reason, etc...)

roll on elixir 0.13

So I guess it just missed the boat on ubuntu 14.04 LTS, right?

There are instructions for using 'apt-get' and friends for installing current releases:

https://www.erlang-solutions.com/downloads/download-erlang-o...

While R17 wasn't ready for this approach immediately upon the announcement, I'd expect it any day since other pre-compiled packages have been made available already.

Essentially:

wget http://packages.erlang-solutions.com/erlang-solutions_1.0_al...

sudo dpkg -i erlang-solutions_1.0_all.deb

sudo apt-get update

sudo apt-get install erlang

Reply on news.ycombinator.com