Big changes to Erlang
joearms.github.io
joearms.github.io
"Key := Val Updates an existing Key The key must be present in the old map. If Key is not in the old map Erlang will complain loudly. Using two operations instead of one has several advantages: A spelling mistake cannot accidentally introduce a new key"
Lord knows how many times I accidentally created a new key with a spelling typo, and then later when I wanted to retrieve the data, I was like "What the hell is wrong? I know I put data in that key, so why is nothing coming back? I guess I have discovered a rare bug in the compiler that no one has ever discovered before."
Using actual English names actually minimizes the risks of mistakes.
And of course, using a statically typed language makes all these concerns moot since the compiler won't let you make these mistakes in the first place.
No, you can make exactly the same bug with a static type system. Structurally typed records are just that – structurally typed. If you add an extra field you never use, the type system doesn't care. You need further static analysis to catch that.
(Beside, Erlang ships with a decent static analyzer.)
I.e., you declare:
-record(foo, {bar = ?default_bar, baz = ?default_baz}).
One big inconvenience with records is the fact that you have to type record's name every single time you have to do anything with it. And writing MyFoo#foo.bar is cumbersome, compared to MyFoo.bar in most other languages.
Another issue is that records exist only before pre-processing. Even compiler is unaware of member names, let alone the runtime. This makes introspection and live code operations (by connecting a shell to a running instance) quite inconvenient.
Which is not at all to say that Erlang's tuple/record trick is anything but a clever trick. It is kind of neat to see how far you can go with just that, though.
For eg. in Perl:
use Hash::Util 'lock_keys';
my %hash = (key1 => 'foo', key2 => 'bar', key3 => 'baz');
lock_keys %hash;
$hash{key1} = "FOO"; # updates fine
$hash{key4} = "XXX"; # throws errorErlang is still not really all that popular, but has always had a lot of Important Code in production thanks to Ericsson, which has made it more difficult to change even though there are some things that even the creators acknowledge as not particularly elegant.
It is both a blessing and a curse. If a service or program doesn't crash and cause a lot of drama and headaches it also doesn't get noticed in a certain way.
I forgot who mentioned it (one of Erlang Factory talks probably), but for example if you access the Internet via your smart phone, there is like a 30% change that it goes through Erlang at some point (due to going through Ericsson's servers).
yes there is probably > 30% chance of that. the space that you are looking for is mobile-packet-core, and is dominated by ericsson, csco (via starent acquisition), and to a lesser degree by alu, nsn etc.
Google basically solved this problem for themselves with gofix. I'm surprised Erlang doesn't have an equivalent, given that it would basically just be a reuse of the "parse transform" code.
Python added a set builtin in 2.4 and a literal syntax in 2.7, Objective-C added NSNumber, NSArray and NSDictionary literals only last year.
It is like adding built-in maps to C++ or Java or C. Python set was similar. And there was talk about whether {} should be an empty set or a dict. It stayed a dict. I remember it.
But also in mid 90's already large production systems were written in Erlang while Python was still experimental. Python 2.0 was released in early 2000s or so.
"Significant" and "large" are different things. A change can be small and significant, and large and insignificant.
> It is like adding built-in maps to C++ or Java or C.
It's like adding built-ins to anything. Your point?
> Python set was similar.
Similar what? It was a new builtin and literal to a language which did not have a builtin (let alone literal) set.
> And there was talk about whether {} should be an empty set or a dict.
Which is relevant how? Also not really, `{}` becoming an empty set literal was taken off the table more or less immediately when the introduction of set literals was being discussed.
> But also in mid 90's already large production systems were written in Erlang while Python was still experimental. Python 2.0 was released in early 2000s or so.
You're joking right? Python 2.0 was released in 2000 (October), which would quite obviously mean that Python 1 had preceded it by a few years (6), and most things are usually considered not experimental by the time they reach version 1. Python had stopped being experimental more than a decade before sets were added.
If you'd read my comment you'd understand. The point was that adding a new built-in to an almost a 30 year old language is unusual. Adding a new built-in to Dart is probably a lot easier. So not it is not "like adding built-ins to anything".
> > Python set was similar. > Similar what?
Wild guess but well I don't know the topic we are discussing -- maps to Erlang. And by extension adding a major built-in type to any established language.
> > And there was talk about whether {} should be an empty set or a dict. Which is relevant how?
It is not. Just making conversation remembering from years back. Sheesh...
Now if say Nimrod or Dart acquired built-in sets (I don't know maybe it already does) it wouldn't be too remarkable. They are pretty experimental.
Also there is a difference between having a library and a language built-in. All those languages including Erlang, have libraries that provide associative data structures.
Support the runtime, support the syntax and semantics, change the pattern matcher, write extensive tests, add it to the term_to_binary/binary_to_term protocol, add it to distribution, enable reading of maps in various function calls, enable NIFs to inspect maps, define how to copy maps between processes, change the core compiler.
It isn't so simple.
Maybe less likely, but if you've been writing a lot of code using -> and then in one case need := I can see it happening quite easily.
The problem with accidental key typos is that at their worst they are completely silent, and if the value you intend to replace was another valid value, finding the bug might not be easy.
Priceless (and completely true). I really think this attitude is why Erlang is such a success.
do stuff <- getLine
putStrLn stuff
getLine >>= putStrLn
putStrLn =<< getLineYes I would. I would much rather see "getLine >>= putStrLn" than a 2 line do block.
Update: Python is heavily used in Biology and related research (so they said) and transition at MIT seemed like reasonable, but I still believe that Scheme shall be the language for serious introductory courses, like classic Berkeley CS61a was. Now PLT/Racket folks are doing great job of introducing proper concepts and developing "good habits" to students.
The thing that I am less clear about with maps is their relationship with Mnesia tables going forward. Mnesia tables take their schema from records. I suspect we will go down the route of having records that contain 'privileged' fields (effectively ones you can have indexes on) and field that contain maps which allow to 'extend' the schema without performing difficult state transforms.
It will certainly help when you need to add a field to a record and thread that change through a running system without stopping it.
Does anyone know how often Learn You Some Erlang is updated? I'd definitely take another pass at the language with these features in place. Kudos, team!
Maps should be a replacement of data structures like dicts and gb_trees, but I personally do not see them as a replacement of records within a module, where I feel their restrictiveness is welcome, for two main reasons:
1. especially to crash early in live code upgrades, despite, I'm sure, a lot of people disagreeing with me.
2. The module isolation inherent to records makes people think at a protocol level and with their API much, much better than the common pattern of saying "screw it", sharing the state around, and breaking abstraction all over. I like how it constrains the programmer to think of what should be passed around in messages, and that maps may remove that "think hard" part of the problem.
Maps should be especially nice and enable more complex dictionary manipulations, nested key/val mapping, and so on, and in terseness of operations. More elegantly, they could be a decent fix to 'use ETS to optimize K/V operations', although they won't benefit from the same parallel access.
I plan to explain this and possibly revisit some code snippets from the book in an add-on chapter, and also show what I wouldn't change.
Regarding Funs, I probably will just add a section to the anonymous function part, and see if I ever used recursive anoynmous functions before. It's likely that I avoided them on purpose in the book and as such, won't need to add too much there.
Let me know if that sounds good to you.
Long live records!
It would be nice if records became a concrete thing in Erlang.
Why? Why do you need to pattern-match on a structure whose keys you don't know a priori? Why should the syntax prefer one map-like data structure over another?
Don't get me wrong, I love that maps are in the language now – as a replacement for records, where syntax and pattern matching make sense and are used all the time, and the data structure issue is much less sensitive. Please don't hamper their performance by forcing them to support the use case for which dicts and gb_trees are intended!
You can know a data structure's keys in advance while still having it dynamic and not a record: proplists, key/val lists, and so on are like that. The user has to know the keys to use them sometimes. The data structure definition doesn't.
The weakness of dicts and gb_trees is two-faced: they're both slower than maps promise to be, and they do not have pattern matching.
Records and maps are distinct in my opinion. I, for example, won't expect to be able to use Dialyzer and do type checking of specific key's values (much like with dicts, trees, k/v lists), but will do so with records.
Maps are pretty much dicts and trees with pattern matching added more than anything. They even have the same issues with considering the equality of integers and floats.
Records still have their use case as is right now. Maps have more in common semantically with dicts than they have with records, where the similarity is syntactical. I prefer to prioritize semantics over syntax.
So? Having a similar interface doesn't mean they have comparable implementations. Even in the maps runtime implementation, we see a benefit of knowing that a structure is homogeneous: run-time type information can be lifted out of the structure, saving memory and time.
You can know a data structure's keys in advance while still having it dynamic and not a record: proplists,
proplists are like, the poster child static heterogeneous structure, precisely because I do know all the keys in advance! It's just that in a given instantiation, some of them map to the value "undefined"!
The weakness of dicts and gb_trees is two-faced: they're both slower than maps promise to be
Then reimplement the dict module. How does syntax help here?
and they do not have pattern matching.
Again, why do you need to pattern match something whose keys are determined at runtime? I've never wanted such an ability in a language; I don't see the benefit.
Records and maps are distinct in my opinion. I, for example, won't expect to be able to use Dialyzer and do type checking of specific key's values (much like with dicts, trees, k/v lists), but will do so with records.
Why not? OCaml can perform typechecking of structurally typed values. That's a solved problem. If there's no plans to support it in Dialyzer I'll add it myself.
Maps have more in common semantically with dicts than they have with records, where the similarity is syntactical.
I think we disagree on what a "map" is. You say a map is one thing (a replacement for a key-value store), but when I see syntax that looks, walks, and quacks like a structurally typed record, I see a structurally typed record, and I'm utterly baffled why those things should have anything to do with each other.
Maybe I'm being pessimistic, but I think frames have pretty much zero chance of being implemented so long as maps have syntax support.
We will be able to do X = 3, #{X := Val} = Map, ... to extract the value of 'Val' under the unknown key 'X'. This just hasn't been implemented yet, but is part of the total specification.
What you see is the incomplete implementation of maps, not up to the total spec, because this is a RC1 implementation. They will support part of the record use case, but the entirety of the dict use case with more flexibility there (including comprehensions, from_list, etc.)
They are, by specification and once complete, going to be far more feature-equivalent with dicts than records.
Yes. I don't understand the use case for that.
That is, let's say I have an in memory map of IDs to counters (for whatever reason).
I can write an implementation to increment and get the counter value for a given ID (assuming this is a gen_server and this is the call implementation) like so (well, possibly like so. I haven't used maps yet; haven't looked at the early r17 builds yet) -
handle_call{{incrementAndGet, Id}, _, State) ->
#{Id := Val} = State,
NewVal = Val + 1,
{reply, NewVal, State#{Id := NewVal}}.
handle_call{{incrementAndGet, Id}, _, State) ->
Val = orddict:fetch(Id, State),
NewVal = Val + 1,
{reply, NewVal, orddict:store(Id, NewVal, State)}.
PS - Because it pattern matches, if I decided I didn't want to throw if the thing doesn't exist, I can do cool stuff like -
handle_call{{incrementAndGet, Id}, _, State) ->
NewVal = case State of
#{Id := Val} -> Val + 1;
_ -> 1
end,
{reply, NewVal, State#{Id => NewVal}. #{Id := Val} = State
Val = orddict:fetch(Id, State)
Those are different questions, the latter of which not one I feel capable of answering, given that I wasn't involved with the discussions.
I assume the reason for wanting to replace dict/orddict is because they feel very much a second class citizen in the language (see at bottom for a comparison of why pattern matching trumps separate modules), dicts are inefficient for small collections, and orddicts are inefficient for anything but small collections.
As to why they went this route instead of frames, I recall seeing it in the discussions (I believe the Erlang mailing list but I can't recall), but I never paid that much attention to it.
But then, to the first, why not just keep the dict/orddict syntax, a completely frivolous example, solely created to demonstrate why pattern matching is cleaner then dict/orddict -
%This just increments a map if the password passed in matches, sets the counter to one and the password to the passed in password if the map does not contain a password, and throws if the password does not match, but does exist.
incrementAndGetIfAllowed(Map, Password) ->
case Map of
#{password := Password, count := Count} ->
{Count + 1, Map#{count := Count +1};
#{password = _} -> throw(bad_password_can_not_increment);
_ -> {1, Map#{count => 1, password => Password}
end.
%We have to extract first, then do our logic separately. Behaves the same as the prior function.
incrementAndGetIfAllowed(OrdDict, Password) ->
CachedPassword = orddict:find(password, OrdDict),
CachedCount = orddict:find(count, OrdDict),
case {CachedCount, CachedPassword} of
{{ok, Count}, {ok, Password}} ->
{Count + 1, orddict:store(count, Count + 1, OrdDict)};
{_, {ok, _}} -> throw(bad_password_can_not_increment);
_ ->
orddict:store(password, Password,
orddict:store(count, Count + 1, OrdDict))
end.
To demonstrate why maps need syntax support, you need an example which pattern-matches keys.
Credit for this idea comes from Richard O'Keefe who pointed this out years ago. I don’t think any other programming language does this, but I might be wrong.
That is a nice language feature to enforce, I love it :) This is however basically the same thing that V8 / most Javascript runtimes do under the hood[1]. Essentially, as you construct objects and add/remove keys, you back them in the engine with more-efficient shared objects that have the same structure. Add a key D and it moves from <shared structure with A,B,C> to <shared structure with A,B,C,D>. Since most code ends up doing the same kind of operations on a bunch of similar bits of data, you can save a lot of compute-time by assuming that and having a slower fallback when it fails (like using an actual dictionary instead of a Struct-like thing).
That said, Javascript has zero enforcement for this, and the runtimes may have already shifted to something different. They are quite different beasts. Just pointing out that the idea has been around.
[1] it has been a while since I've looked closely, and I may be mistaken. Call it 95% certainty.
For the record, one of these is another thing JavaScript got right, sort of, with named function expressions:
setTimeout(function innerName() {
if (shouldRecurse())
innerName();
});
typeof innerName; // undefined
(set-timeout (fn []
(if (should-recurse)
(recur)))(map (fn whatever [a] (println a)) [1 2 3])
ps: this code is stupid but I'm too lazy to write useful clojure from my phone, you get the gist
set_timeout sub {
__SUB__->() if should_recurse();
};
use 5.016; # or newer
use feature 'current_sub';Hence it's literally the exact same thing as a dictionary, but is named a hash in Perl.
Figuring out how to store arbitrary key-value pairs should be one of the first steps in learning a new language, given that every language has a different name for it.
Associative Array is probably my least favorite name for it, since it implies arrays aren't associative. It was a clojure book that first made me realize an array is a map that only allows ints for keys.
> {}.__proto__ === Object.prototype
true
> Object.create(null).__proto__
nullActually, it only allows nonnegative ints for keys.
Also, it has the added undesirable property of requiring memory proportional to the largest key value, rather than number of keys.
Most languages conflate these, because they kind of look the same, but they are entirely different use cases, with different use patterns, permit different optimizations, and their generalization (a dynamic heterogenous structure) is pretty useless.
The relational model [1], which has been around since 1969, does distinguish these. The static heterogeneous structure is called a "tuple", and the dynamic homogeneous structure is called a "relation".
If only language designers would stop conflating the implementation of these structures with how they are used, we could drop the cringeworthy names "associative array" and "hash", stop calling tuples "maps" (which is a specific kind of relation), and stop calling relations "tables" (which are two-dimensional arrays, which are another kind of relation).
Erlang at least as of R17 has mostly kept this distinction (Erlang maps mostly support the static heterogeneous use case). However the majority of users are keen to jump on the "maps" bandwagon as a way to get syntax support for dynamic homogeneous use (which makes utterly no sense to me, as most of the syntax is for pattern matching and construction, which makes little sense in the dynamic case!). I am glad R17 does not include many of the proposals for such syntax that I've seen on the mailing list; and instead sticks closely to the original static heterogeneous design proposed by Richard O'Keefe.
The way around this of course is to tag the values with their type. This is usually done implicitly in dynamically typed languages (e.g. Javascript), and always must be done explicitly in statically typed languages (e.g. OCaml). But once you do that, you have a single type for all the things in the structure (a tagged union type) and it's now a homogeneous structure.
Either that, or you can split your mapping into multiple mappings, one for each type of thing. That has the benefit of permitting better optimization (you don't need to store per-value tags, among other benefits).
(Even in dynamically typed languages, it's better to explicitly tag things anyway. If you have a structure that can legitimately contain two kinds of things, one which happens to be an integer, and one which happens to be a string, then it's pretty likely it might later need to contain a third kind of string, which, if it happens to be an integer or a string, well, you're screwed.)
A relation is "homogeneous" because its rows are all of the same type (i.e. they have the same columns and types).
This is in direct contrast to the rows/tuples, a set of which form a relation: the "keys" of a row (i.e. the column names) are determined a priori and are fixed ("static"). (Yes, even in a schemaless database – your program almost definitely operates on a fixed set of column-keys.) Whereas, the values in a row almost always are of different types ("heterogeneous").
There are of course two other combinations – static homogeneous and dynamic heterogeneous. The static homogeneous case is the degenerate form of the static-heterogeneous and dynamic-homogeneous: the keys are fixed and the values all the same type. You can view such a structure as either dynamic-heterogeneous (ex.: a structure which tracks run-time statistics) or static-homogeneous (ex.: a structure containing configuration settings that just happens to all be of the same type) without losing anything.
The dynamic heterogeneous case is not really a useful one: generally, dynamic structures support some form of aggregation or iteration, since the keys by definition are not known beforehand. But if the types of each value may differ, such operations are ill-defined: how can you operate on a value whose type you do not know? I contend that any claimed use of such a structure is best broken down as a combination of the two: either a static-heterogeneous structure containing one dynamic-homogeneous map for each value type, or a single dynamic-heterogeneous map whose values are static-heterogeneous tuples which include a tag indicating the type of the value.
But there seem to be some relevant points left out.
First of all, a relation is a set of tuples. You can say that a set is just the degenerate case of a dynamic homogenous map in which the values are empty and you use only the keys. That seems like a useless generalization though, and more likely to cause confusion than clarity.
Let's say you have a simple relation of person to product: this person has bought this product. It would be awkward to try to group them like: "this person has bought these products" because you could just as well invert it to "this product has been purchased by these people" and there's not a good reason to choose one over the other. Making it arbitrarily asymmetrical just makes it harder to reason about, so that's not a good choice. And neither attribute is unique. So, you are left saying that the key is {person, product} and the value is empty (or "unit", if you prefer).
And yes, that is the key of the relation, so that's not wrong. But what have we gained through this exercise? We are calling a binary relation a "map", but confusingly it's not a map of the first attribute to the second -- it's a map of both attributes to nothing!
You could say that relations representing something more map-like are common enough. But even in that case, it's common to have multiple candidate keys (which can be true even for normalized relations), and it's hard to say what's mapping to what. Again, more confusion than clarity.
I think it's best to just call a relation the set of all tuples which satisfy the relation predicate. That's the model.
To give separate examples without using keys, you can compare a (integer-indexed) tuple to an array. The former is static-heterogeneous; the latter dynamic-homogeneous. And thankfully, most languages (even mainstream dynamically-typed languages like Python) seem to get the distinction, even though ignoring types or implementation, they are nearly identical.
Simplicity and flexibility. Erlang records have to be declared and they're nothing more than syntactic sugar for tuples.
> At a first glance it seems as though records are nominal types
They're not, not in the way you'd usually expect anyway. They're little more than macros for accessing tuple fields by name:
-module(test).
-record(foo, {bar, baz}).
main(_) ->
R = #foo{bar=1, baz=2},
io:format("~w~n", [R]).
{foo,1,2}
S = {foo, 2, 3},
T = S#foo{bar=3},
io:format("~w ~w~n", [S, T]).
{foo,2,3} {foo,3,3}
-type foo() :: #{ 'status' => 'update' | 'keep', 'c' => integer() }
1> rd(person, {name, email}).
person
2> P = #person{name = 'John', email = 'john@example.com'}.
#person{name = 'John',email = 'john@example.com'}
3> P#person.email.
'john@example.com'
4> rd(person, {name, email, age}).
person
5> P#person.email.
** exception error: {badrecord,person}You have to think about what changed and prepare for an upgrade from any version you may encounter, rather than move along with incorrect data, similar to if it were corrupted.
I fear Erlang programmers will use maps in a way that allows them to be sloppy, rather than just exploiting the flexibility they allow. Double-edged sword, in a way.
It is up to you, try one and another one. Some people have big issue with Erlang's syntax. I actually kind of like it, so I prefer Erlang.
Others find Elixir more approachable, it reminds them of Ruby for example. Elixir has macros so you can do some nifty things with them. Those kind of blow my mind when I read them so I am afraid I would get too "clever" with them.
You can call Erlang from Elixir fairly easily so you can take advantage of libraries you find for Erlang.
So kind of up to you. Whatever you like or whatever gets you more productive and interested.
That and it's written in a brilliantly approachable style.
I found that too; I wonder if the programming community would be better off if more people were pushed towards Erlang? Similar to how every programmer should know C, so that when they write python they have some idea what's happening under the hood, it would be good for every programmer to know Erlang/OTP so they learn how large-scale reliable systems should work?
(Is there any newer official documentation for maps? I couldn't find anything.)
Compared with maps adding a fun-call chaining operator like |> should be child's play, right? ;)
Seriously, what are the functional programmer's tricks to handle long function call chains in languages without the |> or similar operators?
Anyway, maps are a more important addition to the environment for sure. So thanks again.
http://joearms.github.io/2013/05/31/a-week-with-elixir.html -> "The Pipe operator"
I really get several arguments against such "syntax sugar". I would not call it useless, though.
So how to handle/structure/avoid code with function call chains in absence of such operators?
An F#-like pipe operator:
X |> F.
F(X).
fun map(F)/1.
begin _1 = F, fun (_2) -> map(_1, _2) end.
The problem with what Joe likes is that in X |> foo(bar), the call foo(bar) is transformed to foo(X, bar), making what really happens less obvious, a big no-no for Erlang.
Care to provide some "marketing" about what they allow for? (besides and in addition to what is already in the eep-0037(?))
Why are they important? How will they support developers in writing and understanding erlang code? Do they provide even some internal/resp performance benefits?
Reg. the piping syntax: how is your proposition making the "transformation" more obvious for the reader of some application code?
To me it seams partial appl. is the more powerful feature for sure. Together with a rather "simple" |> operator it could provide the fun piping functionality. It looks like a map function is needed to wrap other functions taking part in the piping, right? Is that the "make it obvious" part you mentioned?
The piping syntax from elixir (from the joe blog above) on the other hand binds the |> operator to a more specialized, less flexible "fun call" transformation providing fun piping only, right?
Anyways, thanks for insights and again really kudos for the named funs.
Also, coming from a functional background, where I read "X |> F()", I think "(F())(X)", not "F(X)".
code {background: #F7F7F9; color: #999999}
(CSS edited for clarity)