I have never programmed Erlang but it feels like it is the currently only language that is some kind of secret weapon. It has similar aura than Lisp had before that with Erlang you can do stuff beyond "normal" languages.
It has a good marketable brand and 'appearance' for sure. A good Aura and level 3 magicks, but once you start leveling up your spell tree, like other comments in the thread have alluded to, the cracks start to show and you bleed out mana trying to scale what you initially thought would be an effortless process.
A relational/object hybrid data model that is suitable for telecommunications applications. A DBMS query language, Query List Comprehension (QLC) as an add-on library. Persistence. Tables can be coherently kept on disc and in the main memory. Replication. Tables can be replicated at several nodes. Atomic transactions. A series of table manipulation operations can be grouped into a single atomic transaction. Location transparency. Programs can be written without knowledge of the actual data location. Extremely fast real-time data searches. Schema manipulation routines. The DBMS can be reconfigured at runtime without stopping the system.
I've found some practical aspects a bit painful though. It took me a while to figure out how to make backups, migrate schemas to different sets of nodes, that sort of thing. Also there are size limits on disk-based tables which are a bit limiting, and while you can use table fragmentation to get around that to some extent it doesn't seem straightforward to use (I haven't dared so far). I also don't like the way it deals with netsplits - when nodes reconnect it tends to require manual action to resolve.
There's now mnesia on leveldb
I actually prefer it. Too often systems out there don't tell you what they do in the event of a netsplit. They may have marketing copy somewhere that tells you what they try to do, but then you see the Jepsen tests and realize that's a lie, or is naive, or whatever.
Mnesia makes no secrets about it. In the event of a partition it stays partitioned, operating as separate nodes. It's up to you to figure out what to do about that. You can grab a Raft implementation to perform leader election, with the tradeoffs that entails. You want it to self-heal and deconflict based on arbitrary logic, for an eventually consistent system? You can do that too. You want to pretend it doesn't happen, stick your fingers in your ears, "LALALALA" until it does and requires manual intervention? That's fine too! What it DOESN'T do is give you a false sense of security while handwaving away the decisions and implementation concerns that were made to determine CAP behavior, which I find pretty much every other distributed data store does.
It definitely has pain points in learning it, and it also has some very definite limitations, but as a baked in, minimally biased distributed store, it's a battery I really loved having included.
For some tasks (scientific computing, UIs, offline analytics) Erlang might not be the best choice. However, within its domain of applications, it really shines.
The really interesting part is message passing and easy peasy baked in communication that allows to to send messages to other erlang nodes. So building applications that talk to each other over networks is just a normal function of the language and not a higher level dependency such as a library.
The concurrency is also pretty neat as erlang treats all threads of execution as an erlang process. So merging that with the distribution and simple message passing you can easily build stuff that scales without much external tooling.
It's almost an OS unto itself. There are two interesting IoT systems which use it to such length such as GRiSP which is erlang beam ported to bare metal using RTEMS and Nerves which runs beam directly on top of a Linux kernel making beam the serspace.
Here are some more outside of Erlang, that I know of and I'm sure there's more.
APL/J/K - mathematical algorithms
Prolog - logic, tons of business rules
Forth/Lisp - bottom up programming, when you have an idea of the primitives you need to tackle a problem, but not exactly sure how to put it together.
Assembly - When you must absolutely run impossibly fast especially on very small CPUs
AWK/Perl - slice and dice text files
For these languages, I don't substitute for any language. I'll never slice and dice with Java, Python, Go. I don't care. Awk or Perl. I'll never implement tons of rules in any other language, I don't care what logic library they implement without first doing so in Prolog.
For the solo programmer, the above languages hold very true. I have played around with Erlang since Prolog influenced it's syntax, but I'm yet to have to build a large scale fault tolerant system that needs it, but the knowledge is tucked away at the back of my memory should I ever.
I feel that Python, Javascript and Go can also be some sort of secret weapon if used in the right place.
Like what ?
Probably because that claim makes no sense.
Do not assume you'll be able to build your own Whatsapp just because of Erlang.
The story is greatly underreported and all focus is only on "they run Whatsapp on Erlang with just ~50 engineers".
Highscalability lists just some of the patches and optimisations they have here [1] and here [2]
Here's an incomplete list of patches only. There's also tuning and optimisation:
Erlang: Fixed head-of-line blocking in async file IO by patching BEAM, added round-robin scheduling for async file IO, added multiple instrumentation patches. Instrumented scheduler to get utilization information, statistics for message queues, number of sleeps, send rates, message counts, etc. Made lock counting work for larger async thread counts. Patched to dial down spin counts so the scheduler wouldn’t spin.
BSD: Backported a TSE time counter. Backported igp network driver.
More Mnesia (Erlang) patches discussed here: [3]
Are you ready to do this for your Whatsapp?
> elixir/phoenix also achieved the same "2 million connections on single server" without needing those optimizations.
There's more needed to run a chat server than just "2 million empty connections".
---
[1] http://highscalability.com/blog/2014/2/26/the-whatsapp-archi...
[2] http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-...
[3] https://www.infoq.com/presentations/whatsapp-scalability/
I don't need to thanks to the fact that a bunch of those patches are now part of Erlang.
There's also signinficant tuning and optimisation, both for the Erlang VM and FreeBSD.
There also things like (quotes from Highscalability):
"Mnesia: Using no transactions, but with remote replication ran into a backlog. Parallelized replication for each table to increase throughput."
"When Rick is going through all the changes that he made to get to 2 million connections a server it was mind numbing. Notice the immense amount of work that went into writing tools, running tests, backporting code, adding gobs of instrumentation to nearly every level of the stack, tuning the system, looking at traces, mucking with very low level details and just trying to understand everything. That’s what it takes to remove the bottlenecks in order to increase performance and scalability to extreme levels."
Or even the things like "What has hundreds of nodes, thousands of cores, hundreds of terabytes of RAM? The Erlang/FreeBSD-based server infrastructure at WhatsApp". Oh, wait. Erlang's default distribution mechanism grinds to a halt when there are more than ~60-80 nodes. And Mnesia has a 2GB limit on table sizes. So you have to work around those limitations yourself.
There are no magic bullets. Erlang will only take you so far. The rest (80-90% of the way) you have to take on your own, and you have to know what you're doing, and what needs to be done: patches, tuning, workarounds, limits of the systems you work with etc.
I've seen people say these, and I have no idea where they come from. If you have a decent network, dist works fine at well over 80 nodes, but everyone says it doesn't work. pg2/global has some sharp edges if you're trying to have many nodes acquire the same global lock when you have a lot of nodes (a few hundred) or a smaller number if you have a lot of latency between them. There's options though -- maybe you don't need to acquire the same lock on all nodes, or maybe you can look in pg2.erl and global.erl and wiggle the locking code until it no longer live locks.
The Mnesia supposed 2GB limit is a bunch of hooey. Yes, disc_only_tables has (or had) that limit, because dets has that limit. Yes, it's a sharp edge, because there's no warning about it. However, a 2GB dets table is awful to work with anyway. You want to use disc_copies or ram_copies for big tables. Also, mnesia_frag is well supported, so if you really wanted to, you could make your disc_only_copies table 1024 fragments, and have 2 TB of dets, if that's how you wanted to role.
And yes, if you're going to hyperscale, you're going to need a couple people who know how to figure out what your system is doing. Is there a language/environment where that's not true?
I claim, without real proof, that Erlang's BEAM VM and OTP standard library are easier to understand and tweak when you do hit problems. You'll note however, that Rick Reed's first presentation was when he had been at WhatsApp for about a year, and he had zero experience with Erlang before that.
You have a very naive view of large-scale distributed systems. For companies that truly need them, "time" and "effort" are never a consideration. Furthermore, the challenges facing these companies in battling project delays and cost overruns are the 100% organizational and political, not technical.
My entire intention was to say Erlang is easy to get off the ground with quickly.
Most companies outside of the Silicon valley bubble are resource limited.
a) Erlang (as opposed to something else) is only useful for large distributed systems.
b) If your organization is at the scale where you need a large distributed system, then the problems in your project aren't related to code or to coding speed.
b) Needing a large distributed system is a requirement that one can get to from lots of situations. You are making way too large of a generalization.
b) No. Large distributed systems is not something you stumble onto, it's a result of many years of organic growth.
I'm not sure that's true. Sure, a few of the most famous software companies in the world have built their own at considerable cost. But numerous other huge, important companies are dependent on large-scale distributed systems whose major drawbacks (reliability and maximum scale, usually) and major benefits (simplicity, time/resources saved on not having to hire tons of specialists, quick development time) are based precisely in the time and effort constraints under which those systems were developed.
In my mind, Erlang excels in a few ways:
Less often remarked, binary parsing in Erlang is rather nice. Parsing out an IPv4 packet looks like this:
<<4:4, IHL:4, _DSCP:6, _ECN:2, _TotalLength:16,
_Id:16, 0:1, _DF:1, MF:1, FragmentOffset:13,
_TTL:8, Protocol:8, _IPChksum:16,
SrcIP:4/binary, DestIP:4/binary, IPRest/binary>> = Payload,
OptsSize = (IHL - 5) * 4,
<<_IPOptions:OptsSize/binary, IPPayload/binary>> = IPRest,
More often remarked; because of the language constraints, most notably a lack of shared memory between processes and immutable variables, most programming ideas end up expressed in a way that's amenable to massive concurrency, while being comprehensible. Most Actors have easy to understand behavior --- they may accept messages, leading to them sending messages and changing their internal state. From there, you may need to puzzle out the overall system behavior, but often times, getting each individual Actor's behavior correct, leads to correct (if hard to verify) system behavior.
Hot loading code reduces deployment time, which increases developer productivity. When you've got a million users connected to a machine, it takes a lot of time to move them to other machines so you can do a traditional stop / start cycle; hot loading means you can fix things in seconds (and, of course, it means you can break things in seconds too). You can, of course, hotload in C, and probably other languages, but very few people do it. It's comparable to pushing PHP files though.
And, of course, the most important thing is ejabberd is in Erlang, and it looks like it has what we need for a chat server, and I heard some other people scaled it really far. ;)
It's because it's not super easy to properly architect in, it's hard to test, and it requires supporting running multiple versions of the code concurrently (and the ability to migrate data on the fly).
Keep in mind that we rolled out the patch incrementally to one or two machines and then to more. If it looked good we’d roll it out to the rest of the cluster. From there we’d make note of the result and get the change ported into a proper release.
Each time I hear stories that both clustering and hot upgrades, module loading, etc are hard or rarely used I wonder if the person is just repeating someone else’s rumor. They’re great features and work fine if you do a little homework (just like learning anything, don’t treat it like magic).
This was from my time at Cloudant/IBM, though its far from the only case I’ve seen. We ran over 1000 machines this way with some clusters growing to more than 200 nodes using distributed Erlang (something I keep hearing is hard or impossible, it’s not).
Badfun errors are a great example of something tricky if you're not used to thinking about how the compiler translates these to private functions via lambda lifting. It's the same reason funs are not something you should rush to pass around over a disterl cluster. Recursive functions which keep fun's around in a loop also become a problem if they're long lived.
I usually point out that first class modules or MFA tuples are more idiomatic in Erlang than opaque first class functions as values but we're off into the weeds here. It's a good example of where effort and gotchas become a barrier for many.
Lots of languages have actor systems. But how many have preemptive multi-tasking on these actor systems? I can't think of any at the moment. I am well versed in Akka for Scala and the lack of preemptive multi-tasking for actors is a big pain in the ass. Erlang's advanced BEAM and support for this is it's main selling point, imo.