How Elixir helped Bleacher Report handle 8x more traffic | Hacker News Reader

How Elixir helped Bleacher Report handle 8x more traffic

techworld.com

76 pointsstanislavb71 comments

Yet another "Originally wrote in X, rewrote the application (in Y), and it was so much better!" story.

I think the problem is that all of the focus arrows in on X vs Y, and not "rewrote the application". My opinion, and my experience, has demonstrated that once you understand the domain, a re-write in any language (including the original) is going to result in an improvement of similar magnitude.

They could have re-written it in Cobol and probably seen an improvement of similar magnitude.

But of course, "Y is n-times better than X" makes for a better headline and looks great on a Key Result slide...

Correct. Unless we're talking about extremely specific software, new performance typically just comes from newly learned skills. More efficient database calls, removal of long running loops, yada yada. I've revisited software I wrote for a company in my school days, only to decrease SQL calls from one minute to less than a second.

I agree. I'm not sure if the system would show a 10x improvement, but every time I've done a rewrite-- usually in the same language as the original-- almost everything improved dramatically (UX, performance, maintainability, etc). I now think that V1 should be considered throwaway, or at best research intended for exploring and understanding the problem space.

Putting all the success in terms of "rewriting" is as misguided as accounting it all on the technology change. I have seen rewrites using the same technology fail due to scope issues. I have seen rewrites fail because they could not master a new technology.

It is easy to paint everything with ones or zeros but it is most likely a combination of factors.

A team with more knowledge of the domain, a management willing to try new technologies and a technology that excels at concurrency, which seems to be the problem they had at hand, sounds like a good combination for success.

Except Phoenix and Elixir are just faster out of the gate than Rails and Ruby!

"tl;dr Phoenix showed 10.63x more throughput over Rails when performing the same task, with a fraction of CPU load"

https://littlelines.com/blog/2014/07/08/elixir-vs-ruby-showd...

You can tack on a Phoenix app on top of a Rails app's database and immediately get big improvements as Rails can be a bottleneck.

I briefly looked into this, but it looks like there are a few differences in ecto migrations which mean that it's not perfectly smooth to use the same database.

Seems like you can choose either rails or ecto to manage DB migrations, but not use both simultaneously. This makes migrating to elixir more challenging.

I'd recommend setting the `:migration_source` in your Ecto repository configuration and picking a different table name than Rails. But be careful, having two different sources changing the same database can be a disaster waiting to happen.

> Except Phoenix and Elixir are just faster out of the gate than Rails and Ruby!

Sure. So is Java, C#, C, C++, Haskell, Node.js, F#... I could go on. But let's be honest with each other here. What is the real chances that their speedups were really due to a computation bottleneck? And frankly, if it was due to a computation bottleneck, a language on the Beam VM was probably a bad choice.

> get big improvements as Rails can be a bottleneck.

Rails can be a bottleneck. It probably wasn't, though. I can (and have) take an average web app and short circuited a re-write by adding a few choice indexes to the DB. Or I could rearchitect it and move some data processing outside of the critical response sections and do the same. Or I could move a few choice resources onto CDNs and improve end user performance. Or do better compression on PNGs... Or change an O(n) function into a O(log(n)) function...

The list of potential improvements that have nothing to do with the speed of the language in question is almost endless. Ruby and RoR are pretty easy to scale, and if I were a project manager, I would personally have a hard time justifying limiting my hiring pool for the sake of a mere 8x speedup (unless I was honestly operating at a really silly scale).

techempower benchmark even has PHP crushing elixir

> My opinion, and my experience, has demonstrated that once you understand the domain, a re-write in any language (including the original) is going to result in an improvement of similar magnitude.

I have this software that I've been rewriting since I was in university---so about 16 years of rewrites give or take.

Every time I pick up a new language, I rewrite it in the new language. It's an image gallery with bells and whistles, talking to databases, and video conversion. It's not totally trivial.

What can I tell you [1]?

My Java version that I wrote as a 2nd year student was the fastest overall for the next 8 years. Rewrites in Smalltalk, PHP, and Ruby were all worse by many orders of magnitudes. The Haskel rewrite actually made it perform worse [2]. Architecturally they were 'better', but that's only API expression.

The only time I've gotten a 'win' is when I replaced the core database technology (moved from Postgres to Elasticsearch), replaced caching tech (moved from Redis in App-layer to Redis in Nginx) and with Elixir (like Java its fast enough that there's no need for caching).

Elixir has a lot of built in tools and libraries that make handling concurrency a breeze. To give an analogy, it's like working with Lua in Nginx---it feels like you are writing your application right inside of the webservers event loop.

[1] Disclaimer: I am a language hipster, and Elixir is the last language I tried.

[2] Haskel and I do not get along. Odd really. I get along with other functional languages.

Have you tried re-writing the Java version again with a modern version of Java, and with other lessons learned? Just curious.

That said, cool set of experiments, and IMO a great way to learn new languages.

I'm currently writing a java server after working on an erlang server and I miss erlang daily. "Concurrent" libraries like vert.x feel just odd with their callbacks although lambdas made this a little less awkward.

No, I haven't done that but it is a great idea since the language has really gone through improvements.

It'd be interesting too see how well the PHP version does now adays. It was written in 2007, PHP 5.1, and the main issue back then was memory consumption when handling multiple processes. I've heard PHP 8 is much better in that respect, with lower amounts of memory used in name brand applications like Worpdress.

Do you use it for something?

Even if not, I love this idea for learning new toolsets.

> Do you use it for something?

Nope, I just fell in love with a class project and expanded it greatly.

If you decide to go this route, I'd suggest picking an old class project too. Class projects are pretty much cut down to whatever a student can build within 2 months, taking into account that their time is split 5 different ways and they are inexperienced in the language.

It's the perfect amount of depth so you're not writing a toy project, but you are also not overwhelmed and consider this a real "side project" or 2nd job.

If you want something smaller---do a chat app. They're dead simple, and will teach you the network stack of whatever language you are using. Add some bells and whistles like image sharing to make it like Yahoo 2.0, and you'll have something fairly meaty.

Do you respond like this when people talk about rewriting the hot loop part of their Python in a C extension, too?

Ruby is, sadly, just slow. From my testing, Elixir/Phoenix is out of the gate 10x faster than Rails. Now, they saw a 30x speedup here, so I wouldn't be surprised if some of that came from a rewrite, but it's definitely not the case that all of it was.

Ruby is slow. But a database is orders of magnitude slower. So is a O(n) function when compared to a O(log(n)) function. So is transferring unoptimized files; or not serving static files over a CDN...

There's a lot of optimizations which had a larger effect on the speedup than the choice of language.

The thing is that all your database communication is going through your language as well.

Accessing a slow DB in elixir will be faster than accessing a slow DB in ruby.

I don't think "slow" is the real issue here, but rather concurrency models for IO-bound applications.

The way Rails apps often work is that something like unicorn is used to maintain a pool of worker rails processes, which handle requests synchronously. That worker process will block while it waits for IO requests such as DB and HTTP requests.

In this model, the worker memory footprint can end up being the problem, rather than speed of execution. Particularly in large Rails with many gems, loading the application can take up to hundreds of MB. This sets a fixed limit for the number of concurrent workers/requests.

Rails doesn't _have_ to be used this way, but trying to write a concurrent-safe application in Ruby/Rails can be an uphill battle, because many gems do not work this way.

Elixir does not have this constraint, because only a small part of the system (a single actor) is blocked waiting on I/O. This makes it much easier to support many more concurrent requests.

(N.B. I work at Bleacher Report, but this is just my opinion)

You're not wrong - that's certainly a factor - but Elixir is also just massively faster.

Yes, understanding the problem domain helps. But we already knew the domain and had done refractors in Ruby before then: this isn't our first rodeo ;)

Elixir is just much better suited to the highly concurrent tasks we use it for. Seeing 5-10ms response times on some of our services would just not be reasonable in Ruby/Rails. We know - we tried!

But you're not wrong: understanding the problem IS key to building an efficient solution. It's just that the tools you use will also have an impact. Elixir is a great tool.

Funny thing is that I have yet to see the other side of argument - when people had a bad experience with Elixir. It seems so nicely fit into its problem domain that basically everybody that starts doing it is happy with results. I guess that says something.

The bad side

1) no support from mega corporations 2) not that many developers to choose from 3) while based on BEAM it's still too "new" 4) too slow for raw money crunching

4th one depends on the implementation, and it isn't a disadvantage since most scripting languages have the same characteristic.

How many implementations are there?

I can give you 3 already: 1- doing it with SQL, 2- Distributed way with processes, 3- Doing it with custom NIFs etc.

Speaking as someone who came to Elixir from Scala/Akka on the JVM, yes it is absolutely a disadvantage, and no there is no implementation of the Erlang VM which rivals the JVM for speed. In Erlang/Elixir you get speed through horizontal scalability (which is easy) and/or writing parts of your code in another, faster language (which is harder).

Why did you leave Akka for elixir?

There were a few reasons. I never loved Scala, and I especially didn't like the let's-write-Haskell-in-Scala obsession with static typing that affects half the community. The JVM and Akka both have a lot of knobs to tweak, whereas the BEAM seems to have saner defaults.

I didn't need rocket fast performance and I didn't need any JVM libraries, but I did like the actor model of concurrency and process supervision, so when it came time to look for greener pastures, Elixir was a solid choice. I've been using it in production just over a year, and it's been excellent.

Fair enough. I've dug into the haskell-as-scala stuff and found it academically interesting but somewhat divorced from reality with respect to how immediately useful it would be to most projects.

The reality is that raw money (number?) crunching is rarely the problem for modern sites. Even when it is, the number crunching code is often easily isolated and can be modularized using another more performant language and then Elixir can call that module as needed.

Also not many jobs. I've been looking for an Elixir job in Canada for months.

IMHO the best way to start working with Elixir is to persuade people in your current place to give it a try. In the age of microservices it should be doable and later on it creates the market. That's what I did anyway.

This isn't going to change for years. If you want an Elixir job, go freelance.

1) You can argue that it makes tech much less dependent on needs of one particular entity. It can be a good or bad thing, depending on your views. 2) This one I do not agree on principle. I wish we finally stopped judging devs by "language experience" and start judging them by "domain experience". For early stage startups it might be a disadvantage, all the rest should not be concerned about it. 3) Actually it's pretty stable. It's constantly being expanded (i.e. GenStage, but I've been working with it for year and a half now and never had issues with breaking changes. 4) True, but this is a niche. Number crunching is important in some domains, but what you need from web app is usually making things work in predicable way. Everything else is more of a specialized problem which needs custom solutions.

Judging from past experiences here on HN and other programming echo chambers, I'd wager that Elixir is currently at the beginning of its hype cycle (somewhere between 'technology trigger' and 'peak of inflated expectations' if you go by Gartner). So I'm almost certain that you will eventually see "Why Elixir was bad for us..." posts upvoted on here or in other places.

Elixir hasn't gone through the usual hype phases because it's just syntactical sugar, less boilerplate and more modern than Erlang making it more palatable for Ruby/Python devs.

The BEAM VM is what makes all this possible and the BEAM VM has been around for decades.

If you search for Elixir on HN and sort by popularity, 10 of the top 20 results were submitted during the past 12 months (Which in itself might be just related to an increase in users here on HN, but compare the results to languages like Java or Haskell).

Many of my dev friends with hipsterish inclinations jumped on the Elixir hype train in the past year or so. I really don't see how this is different than past "hype phases" of other technologies.

Oh, sure, there'll be some people who regret it, but the key difference is that it isn't going to run into some of the same problems that plague other new languages because it fundamentally isn't a new language.

No matter how good Elixir is, at some point the hype will reach a point where many people will try it having wild expectations, and will get disappointed. This is just the way hype works :)

It was interesting reading some opinions from older Erlang folks. They are mostly supportive but disliked a few things about Elixir such as reassigning variables which they thought broke immutability.

I think if you replace Elixir with Erlang, the beginning of the hype cycle argument goes away since now we are talking about rock solid software backend that is decades old (from the 1980s) and powering a large chunk of the telephone switches in the world.

If I am not mistaken Elixir is primarily a front end that generates code for the Erlang VM (BEAM) supported by the Erlang OTP backend.

Having said that the questions remains, why doesn't Erlang get the same hype? I am hazarding a guess that a majority of developers still work in the imperative style and switching to the functional side takes a bit of relearning in the way you think.

Elixir has a much more "standard" syntax so its naturally gonna have a lower barrier to entry.

It's not really the syntax. Elixir's semantics are simply better than Erlang.

Erlang accumulated a lot of worts, but it never really got rid of them.

I don't think Syntax matters all that much. Smalltalk's syntax is pretty odd, but its regular and simple to explain/understand, so people quickly come to like it. Smalltalk's VM's though are a huge barrier to entry.

Erlang is the reverse: great VM... great architectural decisions... great libraries.... questionable language semantics.

I don't understand this argument. Elixir is still its own distinct thing, the fact that it compiles to BEAM byte code is secondary in assessing whether it is currently in a hype cycle or not. Just like Clojure, Eta, Scala, jErlang or the myriad of other languages that run on top of the JVM can be hyped independently of the JVM or Java.

Erlang did get the same amount of hype, probably multiple times in the past 20 years. So did many, if not most reasonably well-known languages, programming paradigms, architectural solutions, databases, frameworks and so on.

I do not make any judgement about the merits or maturity of Elixir itself. Elixir is perfectly fine.

As developers we should have a healthy dose of skepticism when it comes to blog posts that claim 10x speed improvements with technology X. Was it just the rewrite? Does your use case match mine? And so on... In short: Everything is a trade off, and there is no silver-bullet.

It's early in the adoption cycle.

Not that Elixir is bad, but if you are writing something in it today you are ahead of the curve. As soon as there are a lot of mediocre developers (if we get there) that start using Elixir because they heard it can do no wrong, you'll hear complaints.

This is undoubtedly true. One thing to note though is that I believe that Elixir/Erlang model of programming just fits well with the demands of web applications. There is a lot of cognitive load you're relieved from just because the used abstractions actually seem well suited for the task.

Bad side: you have to do everything yourself.

Ruby on Rails, ASP.Net, Django, they all have a million and one StackOverflow answers, blog posts, and books written about them. I found out the hard way that Elixir and Phoenix docs aren't as prolific. That and the names are less Googleable.

Some people may like that challenge, but it does add a level of difficulty.

In my experience reading the Phoenix source code is much simpler than Django or Rails so less need for stack overflow or help. There is much less to go wrong as well.

You do realize your experience isn't typical, right? Most developers aren't going to look through the source code for the framework they use to answer a question about usage. They're going to google it.

Unlike with past languages, I've also found myself looking at the source code with Elixir, but it's because docs are first-class citizens. I think this is atypical of languages in general, and it's actually why the source code comes up when googling for the info. The answers are in the code in the form of:

``` @doc This is some useful, markdown documentation, which actually contains usage info.

  ## Usage
    
  ## Examples

```

> I found out the hard way that Elixir and Phoenix docs aren't as prolific

Odd, because I have found the opposite to be true about the docs. I've found the docs proper to be quite prolific, well maintained and well-written with examples. This stems from docs being first-class citizens of the language, a dead-simple docs generation tool, and their dogged commitment on excellent documentation (which is why docs are first-class citizens to begin with).

> That and the names are less Googleable.

That said, the _SO/Google_ aspect (not docs) is _significantly_ less prolific - I totally agree with that. But I've found where with other languages I often find myself on SO (C# and JS mainly), I haven't even needed it because the docs are just so well maintained (and versioned). This, plus their community is _extremely_ active, on the elixir forum and github (also slack, but I use that less).

So now, when I google for elixir info, I'm basically googling to get to the official documentation quickly. If there is any question I have beyond that, I check the elixir forum, rarely (but sometimes) needing to post a new question. Perhaps I've just had a different experience wrt this.

You're right, I should take a look through the source more often.

I chalk it up to my C# dev background, where source code was either completely impenetrable, a Sacred Thing that nobody should feel privileged to look at, or both.

I'll have to give it another look sometime.

Yes, I also grew up with C# (after Delphi, which was basically C#'s unmanaged memory version, largely architected by mastermind Anders Hejlsberg). So looking at source just wasn't a thing. Now with Elixir, not only is it a thing, but they have a super active community and great source code/repository maintenance.

I've tried to get other people to get involved with Elixir but most people don't understand Functional Programming and haven't encountered the problems Rails/Ruby or say Python has.

I think people coming from OOP find FP hard and seem to struggle with things like Ecto and definitely Macros.

Once you have mutable state everywhere and problems with the GIL and throughput on routes being slowed down too much so you start moving everything into queues (celery or rabbit). Then you realise that all of this could have been avoided and made much much easier to test if it were all written in Elixir.

> I've tried to get other people to get involved with Elixir but most people don't understand Functional Programming

I really don't get this. Elixir is barely FP—just immutability and first class functions. Is it really that hard for a lot of people? It's not Haskell or Clojure (""weird"" syntax).

I'd argue the hard part of Elixir is OTP and taking advantage of distributed computing.

> haven't encountered the problems Rails/Ruby or say Python has.

This is a good reason, honestly. If Ruby is fine, why change? It'll just make learning a new paradigm and OTP less fun because there's no need to learn it.

My God that's a nightmare of an office. Open-plan and a wall of distracting televisions. How do they get anything done?

Or do employees just WFH when they need to actually produce stuff?

Had the exact same thought, it looks downright dystopian. Not sure how anyone can get anything useful done in an environment like that.

That's our content team on the 6th floor. They actually use all those TVs. It's the prettier floor for photos...

Engineering is on a different floor, still open plan, but has lots of small meeting rooms for quiet work too. Also a pool table

I would like to build an Elixir training video course called "Elixir for Rails developers".

This is probably sufficient for an intro video series…

1) Installing, including all of the dependencies: 2) Making a new project. 3) Setting up a data model. 4) Building CRUD routes. 5) Building views. Can you use haml, sass, etc.? 6) What’s the equivalent of the console? 7) Deploying to production. How does that work?

If any Elixir enthusiasts are interested in teaming up, let me know.

This already exists in book, blog post, and video formats.

With all due respect, people said that about dropbox when it launched. Can you recommend a good, cohesive video series that clearly explains Elixir and is targeted specifically to Rails devs?

Out of curiosity: what does it mean to you "targeted for Rails users"?

BTW, I was a Rails developer before I moved to Phoenix. Given that Phoenix is modelled after Rails that is the easiest transition there is (though it gets more and more different with every release and 1.3 will be a big step - and those are good changes).

I mean things like: "Is there a REPL? I bet you're wondering how we deal with the console: what's the equivalent of bundle exec rails c". "What's the equivalent of an ActiveRecord migration". And so forth.

We want to build the Rails blog tutorial, but in Elixir+Phoenix, and specifically for Rails users.

Definitely, please send me an email: zack@zackburt.com if you'd like to be involved in the project.

Anyone else worried about the long-term outlook on Elixir for adopting it now in bigger companies? (leaving OTP out, of course)

* Bus factor of 1?

* No visibility of improvement process (i.e. PEP)?

* Long term support?

How would you resolve this? do consultancies that build projects for clients in Elixir are aware of that? or notify the client of that?

> * Bus factor of 1?

The core team is made of six members: @josevalim, @fishcakez, @ericmj, @lexmag, @whatyouhide and @true_droid. We have more than 500 contributors on GitHub, it is an accessible codebase. The areas where the bus factor is actually 1 are very rare (and when I can I delegate work to minimize that).

> * No visibility of improvement process (i.e. PEP)?

All planned improvements are currently in the issues tracker: https://github.com/elixir-lang/elixir/issues. It is a short list because it is a stable language and the focus is on the ecosystem rather than the language.

Proposals are sent before hand to elixir-lang-core mailing list for discussion.

> * Long term support?

List of consultancies currently involved with Elixir: https://github.com/doomspork/elixir-companies#consulting

Plus Plataformatec has been doing and supporting Elixir for 5 years and Erlang Solutions has been supporting Erlang for way longer than that and they also provide Elixir support.

Thanks Jose!

I was hoping to catch you here or on IRC :)

Hi! :)

I do quite a bit of event passing and processing from physical sensors and devices. It is proving quite useful there. I probably won't do all of our system with elixir though.

I am curious, could you expand a bit on this? What about elixir in particular is useful for the physical sensor application? What kind of devices are you running it on?

I would think you could switch your average legacy ROR implementation to a newly coded system on just about any other platform and also come out way ahead on resource requirements.

For instance, they could have gone with .net and gotten nearly the same results AND had a stable of millions of devs to pull from.

Hence why reports like above are helpful, so there are examples to draw from. Otherwise everyone would just be guessing. I bet many would love to hear about migration cases from RoR to .NET as well.

Reply on news.ycombinator.com