Adopting Erlang
adoptingerlang.org
adoptingerlang.org
It's an ongoing effort to gather all the resources that will help you use Erlang in a business. The booksite is divided in three sections focusing particularly on Erlang/OTP’s higher level concepts in the current open source ecosystem, how to use it in production (while setting up a pipeline for continuous development and delivery), and how to build a team when you’re starting from scratch.
We're working mostly linearly in each section, but they're not all progressing at the same pace. We have nothing yet on "building a team" although our outlines are all planned; the "hard things to get right" chapter was supposed to come in later but we saw enough questions about these topics to rush it out.
We might release things from each section as they are ready and we see fit. In the meanwhile, we're always appreciating feedback and people checking in to see what's new in terms of content.
They know too much, they often don't recognize how many pieces of assumed information they're injecting into their so-called explanations. You often get circular explanations in addition to circular reasoning and it can be off-putting.
A better use of time is to get someone who is just started to exhibit competence to write the docs. They won't assume the jargon, they can't assume deep knowledge, and because they're living right at the pain point they can be especially sympathetic. And if what they explain is dead wrong, this is your chance to correct a horrible misunderstanding before they get too far along. Pointing out the 'false friends' in a system early on can be pretty powerful. Educationally or as an opportunity for improving the system.
In other words, I'd encourage you to invite new people to contribute. Just keep an eye on what they do and be ready to jump in with corrections.
There's such as thing as too verbose, but it usually comes from writing stuff in the wrong kind of document: there is always a place for in-depth explanation, and there is a place for terseness to the point of familiarity.
One last thing: if your topic has a jargon of its own, a particular meaning to words, introduce these as early on as possible, and be excellent in consistency throughout docs (in addition, consider a definition link/info widget for each such term on its first occurence in any document, letting people 'jump right in' to any section/article). If you are going to force people to re-wire their brain to suit your particular vocabulary, you should maintain 'immersion' at all times.
If find this especially true in not-so-well-defined paradigms such as OOP when approached by various programming languages. You need a "map of terminology" more than anything else to quickly grasp how to use language XYZ.
Exactly why I hate Brew with passion. In the doc you find: formula, bottle, cask instead of package and... no idea what the others mean. I have no time to learn and remember the mapping, which exists for no other reason than the author trying to be cool.
It's teaching you the wrong way, making you almost illiterate about pm when you enter the real thing (here Linux/UNIX world, or Python after BASIC, vanilla web concepts, etc.)
People who are bored or curious can pull on the thread at their own pace.
Anything that helps 'lateral' access to a knowledge base is helpful, because you don't need to go through all the bottom-up (learning) or top-down (prior knowledge) to explore and reach a given problem space. You can simply 'cheat' directly to 'stage N'.
Once there, you might be missing key information: principles above you, terminology and base config below; this is where a glossary (hyperlinked automatically ideally) is a life-saver. Follow 1-2-3 links and you've got the gist, back to 'stage N' to the next step, rinse and repeat.
When documentation is badly done, this approach may result in endless circular references, feeling 'trapped' at or around 'stage N' — you explore that stage all you want but gain no understanding of how it fits into the whole. But when it's well executed, such documentation is a treat — RHEL comes to mind as being very practical to use in that regard, for a rather complex/deep doc base.
(though maybe to clarify, that's not a subtle dig. I haven't read it yet, though I'm trying to get into Elixir so I'm bookmarking)
The effort mentioned here of getting newer people to read the content and/or produce it is something I've done a lot in the past, and one of the ways I've used in on-boarding people as well. We (the co-authors of Adopting Erlang) tend to do it with topics where we don't know everything the other authors know, and provide feedback to each other about things that lack clarity.
I've also taught Erlang stuff in multiple places and started knowing some of the things people find confusing as a kind of pattern, which may get easier to address early on with time.
The thing that can never be done well, though, is predicting the things someone with an entirely different background from yours would find confusing; the context carried by their experience influences the things they understand and the information they extract from a text. Teaching to a functional programmer is different from teaching to a developer who has done OO their entire career. Teaching to someone doing front-end work is going to be a different experience from teaching to someone doing mobile or back-end work. The same may often extend to sociological, economical, or cultural factors as well.
All of this means that each teaching experience has new surprises as well and you can't just cover them by thinking very hard. Frequent reviews, both technical and from editors can help, though, along with your own experience when teaching.
Haven't you ever seen someone ask a question online that had a "newbie" answer that is generally correct, only to have a ton of people flood your response with all the caveats ignoring who their audience actually is. The default for most people is to use your current knowledge.
To make matters worse, what often happens is that the instructors believe they have taught something, and they don't witness the actual learning take place, so they continue in their belief in their practices.
I think they deserve more thorough coverage because 1) you really need them in a lot of applications, and 2) they're not in OTP, so picking one (especially if you're not really an Erlang expert) and employing it in the right way are a bit trickier.
A simple example of an application that probably needs a circuit breaker is a web application that can keep running even if the database is not available, even if all it does is return an error message to the browser.
The cheatsheet about what the processes do when they trap exits/are linked/are port of OTP is awesome. I never remember that very well and was looking for a handy reference to have. Now I have it.
Docker and Kubernetes references are great as those are often involved in today's development and operations, and everyone is probably wondering what's the best way to combine those with Erlang.
There is a large overlap with Elixir stuff, mostly at a higher level; whatever covers OTP, handling supervision, releases, docker image practices, approaches to testing, production practices, etc. will very likely be useful to Elixir users.
But the lower-level details like handling dependencies, build tool commands, test framework specifics, the code snippets and samples, and so on are all going to be Erlang-specific.
I use both languages frequently for work and there's far more in common between them than there are differences. If that's not your cup of tea and Elixir is really all you're after, you can look for Ben Marx's Adopting Elixir (https://pragprog.com/book/tvmelixir/adopting-elixir) book. We are not affiliated with it and don't aim to cover the same content (we just have similar titles), but it's a good option that goes Elixir-first.
I've never met Fred, but I feel his book "Learn You Some Erlang" is clearly in another category: it's a "labor of love". I can't speak to his precise motivations, but the book is so rich and thorough, that it's clearly something he worked way harder on than would have been strictly necessary just to whip out a quick book establishing himself as an Erlang Expert. Maybe it was "the Erlang book he wished he'd had when he got started"?
As such, it's one of the few tech books I've bought, and done so quite happily, in recent years.
I've got:
Learn You Some Erlang, which has been published with No Starch Press. This was my first one, which I assume is the one you're talking about. Lots of drawings, something I no longer really have the time and energy for these days compared to early in my career. It's the big basics of Erlang. A bit dated now, particularly since the ecosystem evolved around it, but all the basics are still well-covered I think.
Erlang in Anger, only available as a self-published short e-book available for free, meant to teach people to operate Erlang in production.
Property-Based Testing with PropEr, Erlang, and Elixir; my latest published one (with Pragmatic programmers), covering property-based testing both with Erlang and Elixir at once. Pragprog have let me keep the initial website online, which only covered Erlang.
And Adopting Erlang is the latest one. We're still writing it (this time I'm a co-author rather than the sole author), and there's no physical or ISBN'd copy of it available at this point in time.
All of them have to be labours of love, just because it's a lot of work and effort, and at this point in my career, my name's well-known within the Erlang and Elixir communities, so you could say there are diminishing returns. I still enjoy writing though, it's just harder to make time.
edit: sorry I glossed over Freds comment and see what you mean now - ignore me xD
While I can argue in favor of tooling being roughly on-par (sometimes better, sometimes worse), I can't really discuss syntax since this is often just related to taste more than anything.
My stronger argument would usually be in favor of the community you end up with. While both communities are starting to merge and grow closer over time, your average Erlang dev is likely to work on infrastructure components, and your average Elixir dev is likely to be working on web apps. I personally like working on the former better, so there's more limited interest for me in terms of most Elixir jobs anyway, even if it's not like no infra work exists in Elixir.
Both can have a great time working on APIs of any kind, and both can reuse most libraries of each other's community, so as I said, this is starting to become a moot point though. I personally have no problem with either.
That being said, Mix does have its advantages in other ways (easier to extend for one-off scripts, for example, since it won't actually need a whole plugin; the ability for mixed-language projects with Elixir and Erlang), and they're clearly the reason (with Hex) why we have good package management today.
Again, I'm not ready to say one language has better tooling than the other; it's just that I feel it's a far cry from the situation from a few years ago where Elixir was just miles ahead -- the gap has closed since then.
A section on "stuff that's not in OTP, but many people end up using".
For instance, poolboy comes to mind, alongside the circuit breakers I mentioned in the other comment.
lager for logging.
At one point, there was a lot of stuff in the basho github repo that seemed pretty widely used.
It's been a bit since I've used Erlang so maybe there are some others that come to mind as "you're likely to see this included in a production Erlang system".
When I open the link I see the table of contents on the left and a bunch of empty white space. Scrolling down, the content eventually comes into view, but scrolls on top of the table of contents, making it impossible to click.
I'm blocking the google fonts site, so maybe that's part of the problem.
As an aside, it's been a little bittersweet seeing all of the attention and momentum going into Elixir. Yes it's great for BEAM and Erlang certainly benefits from those contributions as well. But there's just something so beautiful about Erlang, it deserves more attention than it gets. Thanks for keeping the dream alive!
Less “magic” (fewer macros that may change semantics, straightforward config handling), focus on declarativeness, no variable re-binding, more powerful test framework (common_test), seamless Dialyzer support, better logging framework for structured logging, simpler/minimal syntax, personal preferences for tooling (until recently, direct support for releases was lacking in Elixir), and force of habit.
I also tend to believe Docker makes little sense, where OTP releases are the alternative.
What's the opinion here? This question only pertains to OTP development.
You can use Docker to provide a common/consistent base on which you'd build the nodes running your OTP applications. If you have zero non-OTP dependencies, then this is overkill (you'd be better off deploying directly to some server, be it a VM or bare-metal, and managing configuration/deployment on that level), but most real-world applications do have other Dockerizable dependencies (databases come to mind, though there's ongoing debate on whether or not putting Postgres in a Docker container is actually smart). Also, using Docker to abstract the runtime environment for BEAM does help considerably with deployment to various PaaS platforms (e.g. Heroku, Bluemix, Elastic Beanstalk, etc.).
Personally, I lean toward a couple big servers dedicated to BEAM instances (running all the OTP apps), and a couple big servers dedicated to some SQL database (usually Postgres). Whether you maintain those servers as Docker containers or EC2 instances or DO droplets or physical servers is an implementation detail that doesn't really matter that much as long as it works and it's easy for you (or your DevOps / sysadmin teams) to maintain.
On that note: I don't think it makes sense to have a bunch of little containerized microservices each running a separate BEAM and a separate set of OTP applications. OTP already takes care of that sort of orchestration, and OTP applications already are (or can be) microservices. Just throw 'em all into a big server and let BEAM's scheduler and OTP's supervision model do their jobs.
That said, they still do not do similar things. OTP does not take care of orchestration if you must run more than 1 node.
You are still benefiting from OTP's supervision and scheduling when using docker or k8s, the difference is in what other features and the size you need.
No one is saying always deploy to k8s, but they are not overlapping.
Nor does using BEAM make it uniquely fitted to running on physical machines compared to other languages like java, go, C++, Rust, ... If your application is suited to running on a few droplets the language you are using is not likely the deciding factor for that.
> If you don't and you work in even a medium size company you likely work with what they have, which is increasingly kubernetes.
And that's totally fine, per my second and third paragraphs.
> OTP does not take care of orchestration if you must run more than 1 node.
No, but it (with BEAM) gets you most of the way there. The only thing not explicitly handled is spinning up the machine running that node (though you can certainly have the app itself drive that, e.g. by using the AWS API to spin up another EC2 instance with an AMI preconfigured to boot up and start ERTS). Once the machine is running, the application can use that node as a spawn() target and start throwing processes at it.
> Nor does using BEAM make it uniquely fitted to running on physical machines compared to other languages like java, go, C++, Rust
BEAM (or at least some sort of Erlang VM) is (in conjunction with an application framework that can capitalize on it, like OTP) is exactly what pushes the needle toward running on a couple big machines instead of a bunch of little machines (and again, whether those big machines are containers or VMs or physical servers is an implementation detail). Java and Go and C++ and Rust don't have Erlang's/BEAM's process model; they're dependent on the OS to provide concurrency and isolation (though there's no particular reason why a JVM implementation couldn't support an Erlang-like process model, on that note; similar deal with a .NET CLR implementation, or any other VM).
The compelling reason to go with a bunch of small machines instead of a couple big machines is resource isolation (e.g. enforcing memory/disk/CPU quotas for specific applications). It's definitely possible to do this on an OS process level (i.e. by applying those quotas to the BEAM processes), but if you're already using something like Kubernetes, no reason to not use that particular hammer to smack that particular nail.
Jose Valim, creator of Elixir, also wrote about this recently http://blog.plataformatec.com.br/2019/10/kubernetes-and-the-...?
https://www.scylladb.com/2018/08/09/cost-containerization-sc...
Erlang does have the option to bind schedulers to processors but I don't know of any real world usage of it:
Binding Schedulers: http://erlang.org/doc/man/erl.html#+sbt
Custom CPU Topologies: http://erlang.org/doc/man/erl.html#+sct
You can run BEAM apps in a container the same way you’d run any other app. It’s possible. But BEAM is less like a PHP/Python/Ruby etc interpreter and more like an abstraction for the OS (some might even consider it to be one). When you’re in Erlang you think in Erlang and the runtime abstracts things like processes, threads, and network partitions away... so you tend to want a big beast BEAM instance (or a few big beasts) rather than a lot of tiny ones (as one might scale up a horizontally scaled app like one on Heroku dynos).
It can be done. It’s just kinda going against the grain.
They are great to have when you need them but should not be chosen lightly.
Part of the point of this booksite is to show integrating Erlang into a modern environment and not being the oddball deploy at the company. It is the lack of libraries for services used for deployment/monitoring and lack of documentation that is the reason for this.
Also, there is no abstracting away network partitions.
Docker makes a lot of sense, actually, because it allows you to run integration tests easily and ship Erlang releases in the same way as everything else. There are exceptions, of course. As for OTP releases, they are a good way intermediate step before packaging code to RPMs, but upgrades should be done the usual way: via blue-green deployment. Don't even argue with me :-) I happened to support a huge system that uses OTP upgrades, and the amount of complexity and tricky bugs, that live code loading introduces, is immense. No one is smart enough to do it :-)
> If you expect failure to happen on an external service, do not make its presence a guarantee of your system. We’re dealing with the real world here, and failure of external dependencies is always an option.
In a nutshell, docker and k8s are to OTP what region failover is to k8s tech. They operate on different layers of abstraction and impact distinct components.
OTP allows handling partial failures WITHIN an instance, something K8s can’t help with.