Comment by DoesntMatter22 | Hacker News Reader

Comment by DoesntMatter22 | Hacker News Reader

Parent Full thread

This fits my experience too. I've been able to crush elixir apps pretty easily, cowboy and phoenix.

I actually think Elixir really doesn't have great performance. TechEmpower, which is IMO the most real world standardized tests out there shows that Phoenix doesn't even complete. And Elixir+Plug+Ecto performs worse than Rails, which is an entire framework.

Everyone in Elixir land tells me "Oh those benchmarks don't matter". Yet they are heavily talked about, and referred to here and other places. They only don't matter if you perform terribly on them I suppose.

And they say "Oh we didn't care to put much effort into it", Yet Jose Valim himself tried to work on it and didn't fix it. He's written extensively about how this type of test doesn't really fit elixir, etc, but ultimately it's just doing DB queries, why does this not work?

I really think Elixir is mostly propaganda at this point. It's a huge mental paradigm shift and I have seen myself that it wasn't performant, and as you said you keep thinking "Oh I must be doing it wrong".

I just cannot fathom why anyone in a decent sized company would use with all the negatives it has going for it. YMMV

Proof below

https://www.techempower.com/benchmarks/#section=data-r22&hw=...

> And they say "Oh we didn't care to put much effort into it", Yet Jose Valim himself tried to work on it and didn't fix it.

This is inaccurate. I have started looking again into solutions only last week [1]. My suspicion was always the database pool size was too small but, when I tried to contribute 4+ years ago, fine tuning was hard because it took too long, so I didn't pursue it further [2].

My discontent with the benchmarks is that they are not measuring what people effectively run in production. Since you mentioned Rails, here is how a Rails application looks like:

https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

But almost nobody runs a Rails application like that in production.

And then you look at the configuration of most stacks and they have been explicitly fine tuned to match exactly the concurrency traffic and patterns that the benchmark sends through. But, for most web applications, your web traffic is not homogeneous and you don't have a predetermined number of concurrent requests.

I still believe those benchmarks are not indicative of what you will actually see in production. Most companies who have gone from Rails to Phoenix, for example, report a 10-20x reduction in operation size and costs. But it is clear at this point people put way too much stock on these benchmarks. The irony of it all is that, if someone copies these setups into their actual applications, they will most likely perform worse. Oh well.

[1]: https://github.com/TechEmpower/FrameworkBenchmarks/pull/9302

[2]: https://github.com/TechEmpower/FrameworkBenchmarks/pull/5432 - here you can see me increasing the database pool to 40... but most benchmarks today run with 512-1024 connections (which, once again, is most likely not what you would do in prod). In any case, we need to bump our numbers accordingly.

Sounds like you are agreeing with me. You tried to work on it years ago and didn't fix it. Even when it was completing last year it's performance was half of what Rails was, which isn't known to be fast [1].

The database pool needs to be at 512 to 1024? Yet Rails seems to get away with far less than that and still have better performance.[2]

Even if, somehow, Elixir is actually faster, which seemingly lacks evidence at this point other than some anecdotes by people who have a vested interest in people adopting the language (primarily Elixir dev shops and framework/language authors), the fact that the community of Elixir devs cannot get decent results on these tests says a lot.

If these people can't get good results in tuning Elixir, who can?

Not a surprise to see people ITT also noting that they ran into the same types of problems that the tests reveal.

[1]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

[2]: https://www.techempower.com/benchmarks/#section=data-r21&hw=...

> The database pool needs to be at 512 to 1024? Yet Rails seems to get away with far less than that and still have better performance.[2]

Rails concurrency model is per process. So you will have `n_workers` pools and the amount of connections will be `n_workers * 3` based on the code you shared. The production machine has 28 total threads [1], which means Rails is starting 35 workers, according to their auto tuning code [2]. Overall, Rails uses 35x more pools than the Phoenix application and has twice the number of connections (105). It is absurd to imply that Rails is actually running with far less.

Second of all, the database pool needs to be 512 or 1024 if your app server can push traffic quickly enough to the database. If you are increasing the Rails pool size and it is not getting faster, then Rails is your bottleneck. Based on the runs I did last week, it seems the pool is the bottleneck in Elixir's case. We will know for sure in a couple weeks once the PRs land.

> If these people can't get good results in tuning Elixir, who can?

You can look at the results from the latest community efforts in main: https://www.techempower.com/benchmarks/#section=test&runid=1...

If you want to compare with Rails in particular, we beat Rails in every benchmark except two:

1. Single query: which will be tackled by better sizing our pools

2. Plain text: where we are currently sending the wrong payload (but when we sent the proper payload, as in the last published round, we beat Rails too)

And it does not include any of my proposed changes yet.

My point is exactly that the feedback cycle is long, so it takes time, and I was not able to invest time before. But given how intent some folks are to misinterpret benchmarks (and assign malice to the whole community), it is probably better to continue looking into it.

[1]: https://ark.intel.com/content/www/us/en/ark/products/120474/...

[2]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

Nice to see elixir put up decent benchmarks for once. It looks like it's about 80 percent faster. Still a far cry from "10 to 20x reduction in size and cost".

YMMV. In benchmarks like cached queries, it already outputs a solid 5.5x faster than Rails, and it is worth remembering that the Rails app is way too stripped compared to what folks are actually running in prod (which was one of my original criticisms).

10 to 20 improvement here.

I wrote, while learning, a monitoring system in Elixir. I put everything in one "Phoenix" app. What does it do?

* Query about 60 oracle databases in parallel every five seconds.

* Query a number of KDB databases every five seconds.

* SSH to dozens of machines every ten seconds

* Ingest statistics from icinga.

* Pull ticket updates from Jira every couple of seconds.

* Provide facilities for connecting in to screen on about 100+ remote machines, and interacting with their interfaces.

* API access to internal and external commands/functions.

* more.

It does much more than this, while serving live-view sessions to users, on a dual core machine (CPU: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz x 2), while averaging about 8% CPU for everything it does. The SSH component replaces a python version that does the same thing. The Python instance, by itself, pushes a 6 core box (same xeon as above but with 6 cores, rather than 2) to 100% CPU, and spaws close to 3,000 OS tasks.

Many of the other checks that are running in that 8% CPU Elixir process replace python, node, and java checks, each of which required more ram, demanded more CPU, and were spread across eight to ten dedicated machines per check. We have been retiring machines, and reducing CPU load, so I'd say we do see the 0 to 20 reduction.

Now, my solution might have been written better, but that was what I came up with while learning Elixir.

That seems like a good fit for Elixir tbh

The techempower repo is literally a gift to the world. I've seen and learned many ways of optimizing code ...almost none of which should ever be used in production or would ever make it past a PR at most companies. Still, it's amazing in the amount of non-realistic approaches used. It really is a work of art.

I hate when people downplay TechEmpower. They are the best benchmarks going, and awesome because anyone can contribute and as you said there is a lot of learning that can be done there.

You are right. There are many positive things about TechEmpower and keeping it open to all and running takes an incredible amount of effort, which is definitely appreciated and I failed to recognize. That was unfair of me.

Although I still strongly disagree with the initial observation that they are "real world". Most applications in the real world receive heterogeneous traffic, from a varying and unknown number of clients, with logging, potential contention on resources, and other aspects that are not taken into consideration.

Reply on news.ycombinator.com