Elixir saves Pinterest $2M a year in server costs
paraxial.io
paraxial.io
In TFA, it gets better though: "Steve: That’s pretty easy. When I started on the spam team, we had close to 1,400 servers running. When we converted several parts to Elixir, we reduced that by around 95%. One of the systems that ran on 200 Python servers now runs on four Elixir servers (it can actually run on two servers, but we felt that four provided more fault tolerance). The combined effect of better architecture and Elixir saved Pinterest over $2 million per year in server costs. In addition, the performance and reliability of the systems went up despite running on drastically less hardware. When our notifications system was running on Java, it was on 30 c32.xl instances. When we switched over to Elixir, we could run on 15. Despite running on less hardware, the response times dropped significantly, as did errors."
Would be curious to know how they tried to optimise the Java stack.
Because on every benchmark I've seen the JVM is faster in every which way than Elixir. Except for memory where often people will over-provision the JVM rather than look at where their code might be over-allocating or leaking.
The big one however is going from 200 Python servers to 4 Erlang ones: a 50x reduction is quite something and a Python-to-Python rewrite would not have allowed to achieved a 50x gain:
> All this is possible because Elixir, and the Erlang platform underneath, are fundamentally designed for always-online software with many users. When you use the right tool for the job, the benefits are clear.
it depends.
The elixir programming paradigm might be one where you are able to more easily write efficient, but still highly concurrent code, where as it would take more work to do the same in java.
Still I'd prefer Elixir. The BEAM VM just runs lighter.
https://medium.com/pinterest-engineering/fighting-spam-with-...
In short, the Elixir is doing something completely different. And they are also not counting some new pieces like the database cluster and kafka based aggregators, etc.
A java-to-java rewrite could have been as much if not more painful than a java-to-elixir rewrite, especially if the service is highly concurrent.
Rewriting synchronous code as asynchronous in Java is a lot of work and not fun at all IMO.
Given the recent arrival of virtual threads in Java 21 this may not be necessary any longer but at the time I think it was a perfectly reasonable choice.
If your rewrite or refactor gives you a 10X in performance, that's not an optimization, but a bugfix. Unless you are a researcher who have just found a revolutionary algorithm.
Just because they were slow doesn't make them bugs, nor does using something faster in a rewrite means it's a bugfix.
I'd recommend to adopt this mindset even if one doesn't have those constrains - without erring on the side of "faster than necessary" though - because it's usually difficult to assess how much time and money are leaking because of the inefficiencies one accepts in the name of questionable reasons. They were sort of lucky to have a bill to show it to them.
A proactive stance in this regard goes a long way.
If they rebrand Python to Metal or some other name, people would recommend it left and right. It's just suffering from bandwagon criticism. Yet it remains one of the top 3 languages for years, covering several domains.
However, it does not have a great concurrency story relative to languages that were built concurrency first (Go, Erlang), and it's fair to acknowledge that.
https://medium.com/pinterest-engineering/fighting-spam-with-...
That's an enormous "except". Absolutely gigantic.
Are you looking at a benchmark that compares real-world usage, or a microbenchmark like hello world or a small JSON payload?
You're "Not even sure how that's possible" as if implying PHP is slow?
Even old pre-7 PHP was much faster than Ruby, Python, and others. And most PHP libs are just wrappers over C code.
No, PHP 7 was an impressive step forward in speed because it switched to better bytecode and object representations internally, but Python wiped the floor with most versions of PHP 5 for exactly the same reason. (PHP 5.5 with opcaching was roughly comparable.)
But this comparison overall is like trying to find the strongest two-year-old...
How is that slow? I'd like to meet the developers that consider this slow.
If one is looking for a strong pickup truck, they don't care that a 18-wheeler is stronger, as that's in another category they have no use for.
One might want to pick among "scripting languages with suitability for web coding and great adoption" for example...
Playing with Elixir, I feel similar speed of development, but it also comes with speed of running, which is incredible.
The fact that Elixir, a compiled language, known for speed, is slower than PHP is surprising. As far as "Most PHP libs are wrappers over C code". That's just not true. Most PHP libs are in PHP
The metric is speed. Compiled vs interpreted is an implementation concern.
You can be compiled and slower than interpreted (and vice versa).
>The fact that Elixir, a compiled language, known for speed
Elixir is not compiled into binary. It's compiled to bytecode for the BEAM vm.
And Elixir was never known for speed, nor was Erlang. They are known for parallelism, availability, fault tolerance, smart schedulling, etc.
Long story short, they are running in debug mode, badly written, not optimised, with bad OS level settings. And every time the community have tried to contribute fixes, the experience has been... Really bad.
So we stopped trying. If things have changed we could try again but ... We just wrote them off
There is a lot of focus on raw performance on web-related services, when in reality most of their running time is spent waiting for IO. If there are two things the BEAM excels at, is IO and turning almost any problem into half a dozen processes that are scheduled and run in parallel, if not geographically-distributed, with 1/50th the effort of any other language.
We live in a world with 32+ core CPUs. If your load is not spread uniformly all over those cores, you're losing a ton of performance. Handling requests over separate threads, like 99% of languages do, still isn't enough if all the business logic runs on the same thread.
I'm currently writing a web crawler in Elixir, and it is easier to design it so every request is done and processed in parallel, than to write a naive sequential one you'd do in any other language in half a day.
It would be a dark day to discover my AWS t1.337metal was blocking 63 cores on I/O when nearly every modern lang has a wealth of async functionalities just awaiting to be exploited.
To junior developers reading this, such statements are never true.
According to others in this discussion they also made architecture changes (DB, Kafka etc.). Do we know if that improved the performance?
There is no objective way we can tell if Elixir had any performance impact. It could have been due to the rewrite, the architecture change or a combination of both.
Yeah the BEAM with some Rust NIFs is a great combo. I'd definitely consider it in the future for many types of problems and anything involving a HTTP or GraphQL interface.
Take the hot code reloading and actor model-based concurrency as a prime example. It's like getting AWS-level functionality without the steep bill for a lot of companies.
Though, I gotta admit, it used to be a hard sell for CPU-heavy workloads, especially number crunching. But Elixir is stepping it up with their Nx library, so that's changing.
Examples of companies cashing in on BEAM's efficiency:
Bleacher Report: Went from 150 servers down to 5. No joke.
Discord: Handles millions of real-time users without breaking a sweat or the bank.
Financial Times: Their content recommendation engine got both efficient and cost-effective.
Change.org: More petitions, fewer servers.
Podium: A million SMS messages a day and didn't have to massively scale hardware.what are "extremely powered machines"?
> It's like getting AWS-level functionality without the steep bill
which part of AWS functionality? load-balancing Beanstalk-style is free. AWS compute is not free, but neither is compute free with Elixir or whatever stack you run.
Similarly, distributed Erlang allows Elixir to run across multiple nodes. This could cut down the need for extra AWS instances or orchestration layers like Elastic Beanstalk. And when it comes to deployments, Elixir's hot code swapping can simplify what might otherwise require rolling updates or blue-green deployments with Elastic Load Balancers in the AWS ecosystem.
On the concurrency front, Elixir is designed for handling a high number of users and tasks simultaneously, which might reduce your reliance on EC2 or Lambda. Phoenix, Elixir's web framework, even has real-time capabilities baked in, so you don't need extra services like AWS WebSockets for that.
Finally, Elixir's actor model can serve as an in-memory message queue, which could potentially negate the need for something like AWS's SQS. So, while you're still incurring compute costs, the need for additional AWS services could be lessened, thereby simplifying your architecture and perhaps lowering overall costs.
Erlang and BEAM was designed to handle Ericson's telephony services. Piping-lots-of-stuff-in-parallel in fault tolerant fashion is the main use case around which it was designed.
Of course this is anecdotal.
When a person makes a claim, especially such a ridiculous one, it is perfectly valid to outright reject that claim without any argument. Why? Because no argument was provided in the first place.
Also I dislike FUD spreading so there's that, I am trying to engage them in a more-informed manner.
Though I am not sure whom you are addressing, my parent poster or theirs.
Hype and FUD are different sides of the same coin
Do you feel that Elixir is being needlessly hyped?
You'll note I'm not selling anything here, and no one is paying me a commission.
A junior that got "swindled" by my claims and spends a weekend learning Elixir becomes a better programmer and earns another feather on their cap. How tragic.
Junior devs, if you want to become a senior greybeard like me, learn anything that tickles your fancy, and ignore anyone that says it isn't worth your time. Even learning COBOL will make you a better programmer. I can only promise that Elixir is more fun than COBOL.
I write concurrent code without much extra effort compared to non-concurrent code. Elixir makes it easy.
Elixir/BEAM do have some benefits that are worth considering for many projects. But they absolutely are not special in this regard, and that's the junior-developer trap about which the person to whom you replied was referring.
The BEAM VM itself is a different beast compared to the JVM. It's more like its own mini-OS designed for real-time multitasking. Each process has its own garbage collection, and it's all non-blocking. So if one process goes belly-up, it doesn't take the whole system with it. Imagine a network of telephone switches; if one gets zapped by lightning, the rest keep chugging along. That's the level of fault tolerance Erlang and BEAM were designed for.
Now, speaking of fault tolerance, let's talk about how easy it is to mess up an Akka system if you're not careful. Say a new Java dev joins your team and doesn't get Akka's actor model. They might introduce shared mutable state between actors, which is a big no-no and can lead to all sorts of race conditions. Or they might do something like put a blocking operation inside an actor, which can hog resources and mess up the whole system's performance. Akka's great, but if you don't follow its principles, you can still shoot yourself in the foot.
So, the beauty of Elixir and BEAM is that a lot of these good practices are enforced by the VM itself. You get that fault tolerance and concurrency baked right in, without having to rely on every engineer knowing all the best practices.
Hope that clears things up!
In Elixir, on the other hand, you could create a module which is like a server process. You can start this server in a procedural flow, or you can "connect" it to a supervisor by giving it the startup information needed and a strategy to be used on how to restart the process in case it crashes for some reason.
A client process (or processes) with it's own module can then send messages to the server which will handle the incoming messages in its inbox sequentially. If you squint at it from an angle, modules look like classes in that they provide a way to separate code logically.
This way of doing concurrency takes getting used to and has a higher initial learning investment. But it feels cleaner and is less prone to user errors. In go for example you have to be careful of closures and shadowing which will result in shared memory and hard to debug errors, even though the initial investment in learning Go is much lesser.
Is it possible to get to the same result? Yes. It is not, however, anywhere in Elixir's ballpark of "easily". Do not discount the power of language-level, not just support, but encouragement. Especially when you're working with junior devs, if "the right thing" and "the thing the language wants you to write" are not in alignment, everything is much, much harder. Erlang and Elixir actively encourage easily parallelized code. Java activity encourages tangles of objects.
Erlang and BEAM, the things underlying Elixir, were specifically engineered to be used in a reliable, distributed fashion including a gigantic amount of idioms and support that no other language comes close to.
Erlang's bit syntax is hands down the best byte stream serialization that still exists--both from a perspective of performance as well as expressiveness. OTP is a documented set of idioms and behaviors for building systems that are meant to be highly distributed and deal with failure gracefully. These include things behaviors like in-place upgrades, supervisors which shut down and restart failing processes, error delegation, etc.
Erlang was meant for genuine "five nines" reliability. No other language comes close (maybe Ada does--I'll let their proponents chime in about that).
What you need to be fast is a systems engineer.
Not for long. Something is looming on the horizon in Java-land.
Only if you think that the BEAM is similar to being able to easily spawn a function on a separate thread and having channels.
Last I checked, goroutines had no mailboxes, supervisors, process monitoring, registry, and its scheduler has a much smaller scope and featureset than the BEAM's.
I swear it's obvious when people comment about the Erlang ecosystem without having really used it for anything.
BEAM's got this whole ecosystem built around it, right? Mailboxes, supervisors, process monitoring, and a registry—these are all first-class citizens in the Erlang world. And let's not forget the scheduler; it's like comparing a Swiss Army knife to a simple pocket knife when you look at BEAM's scheduler next to Go's.
It's kinda funny when people talk about BEAM and Erlang as if they're just another runtime or language. (it's more OS like than traditional VM like) They're really more like a whole philosophy of how to build robust, fault-tolerant systems. And if you haven't actually built something substantial with it, you're likely to miss out on what makes it so special.
Couldn't agree more with your take.
In my experience I think my statement still stands.
The key VM feature you're missing is process isolation. Without it, supervisors are not really possible - you can implement something that vaguely looks like them, but it won't provide the fault tolerance guarantees.
Imagine for example an Erlang process that leaks files. At some point it hits an EMFILE, gets killed, and the supervisor restarts it. The system will then go back to operating normally.
Now imagine a Goroutine doing the same. It hits EMFILE, exits, and the supervisor restarts it. This doesn't help anything: it just hits the same error and the system is unusable. There's no way for the VM to guarantee cleanup when a Goroutine exits, because it doesn't isolate Goroutines and track which one owns which resources.
Links and monitors are tools to extend the same behavior to user-managed resources like DB connections and so on. The responsibility for cleanup if a process crashes while holding a DB connection falls on the DB connection library, not on the error handling inside the crashing process.
Are you aware Erlang was designed to be a VM(the BEAM), a language for that VM(Erlang) and a comprehensive set of patterns and infrastructure for fault tolerance(OTP)?
All these as a whole are Erlang and the whole reason it has such reputation. I recommend reading https://erlang.org/download/armstrong_thesis_2003.pdf
> But they aren't core to how the BEAM handles concurrent processing.
They are, unless you want to reduce it to "it's just a pre-emptive scheduler with greentrheads".
> In my experience I think my statement still stands.
They're similar in that both have (mostly)pre-emptive schedulers, that's about it.
IO lists, which are the foundation of anything that build a string incrementally, are managed with vectored IO syscalls out of the box (readv/writev), which you'd have to handle yourself in most other languages, or resort to allocating and endless memory copying which Erlang is able to avoid.
I mean, if your business logic is inherently serial it makes little difference if you run it in a single thread or if each serial segment in between IO requests is run in a different thread. One way or another it's not going to get parallelized.
When would this EVER be the case? I'm struggling to imagine a scenario where logic doesn't/can't run in the worker thread.
That said, I'm sure a 2x performance improvement could've been done in Java as well if they did a re-architecture. They could also have made a lateral movement and go to a different JVM language, like Scala that also has an actor concurrency model + accompanying syntax.
But a knee jerk response of: This is mostly just good because they rewrote/rearchitected it, ignores the benefits of using a language or technology that fits the new architecture better.
This is the key point that people miss when pretending that languages are interchangeable. The entire point of making a programming language is to make certain types of ways to solve problems easier to express. This constitutes a language's "pretty path". By providing such pretty paths, languages necessarily make less desirable paths, which will be painful to slog through.
If you try writing a functional pipeline in Java, you're going to have a much worse time than doing the same in Elixir. If you try to do Object-Oriented class towers in Scheme, it's going to be painful. Etc, Etc. You can write a Rust program and a C program that compile to the exact same binary, but I can put a whole stack of cash on which one's going to be easier.
They say "The combined effect of better architecture and Elixir".
So the post is mostly useless.
In a rewrite with a different design/architecture, that new design typically accounts for most gains, rather than language.
A language may make some parts of that rewrite simpler.
2 million a year is several developers compensation saved every year. It also opens the door to more savings down the road, potentially, as existing workloads may discover they can use the same approaches to reduce cost / overhead.
Fairly safe to day not at all.
The JVM is extremely fast, very efficient and very scalable (you can write java code that scales linearly with available cores). If performance or scalability is a metric to care about, it is nearly impossible to outscore java. You can, with very skilled C/C++ developers, but it's going to be difficult to find those people and it'll be a lot of work. If you need extreme performance on a reasonable budget, you can't do better than java. I know java isn't hip anymore, so this is not a popular truth, but it is.
So then Python comes along and you find loads of devs for that. Once Python is entrenched, businesses have a hard time telling their devs to actually learn something new. And few devs will already explore things like Elixir on their own in their free time. And so they continue to hire Python devs.
(One could also replace "Python" with "Java" or "NodeJS" or similar, the principles remain the same.)
They did savings by re-implementing their services and attribute those savings to the new tool / programming language.
I wonder what the saving would look like if they chose another tool for the second / optimized system. I doubt it would differ much if they went with Go, Java or stayed with Python.
EDIT: I’m being rate limited because I guess my comments are too spicy for the HN mods, but anyway I agree that there’s no reason other non-Python languages would fare much worse than Elixir.
For example, a Go has a very good asynchronous story as well and while it may not be exactly as good as BEAM languages, it makes up for it in straight line execution performance.
I’ve personally rewritten a few carefully optimized Python applications in pretty naive Go (without significant rearchitecting) and witnessed 50X-1000X performance improvements. And moreover Go allows for more optimization beyond what Python allows (e.g., consolidating allocations and lifting them out of the hot path).
Many other systems don't have this property. They fall down under pressure.
Gosh, a huge chunk of HN is always so dismissive. At least read up a bit beforehand, man. The criticisms should be informed and benefit the readers, not only express a generic skepticism.
The system they created now is totally different from the one they had. It’s more efficient by an insane margin. Choice of language seems like it would have trouble breaking the top 5 major reasons.
I migrated Rails apps to Elixir before, we reduced from 15 servers to 3, and 1 was basically "if crap hits the fan", we could have gotten away with 2 easily.
It's worrying that a supposedly high-quality forum like HN receives comments with no substance. If you have an actual counter-argument, let's discuss. If not, well, not an interesting exchange.
It's very hard to provide evidence unless we make a screen-share call where I show you real time dashboards of services being bombarded with thousands of requests per second and for you to see for yourself how the median latency numbers climb from 25ms to 45ms and then fall back to 20-30ms after the burst load subsides.
I find it difficult to just describe this because as much as I've seen it many times in practice, it's also practically impossible (NDAs and compliance nightmares) to demonstrate it to a programmer outside the companies I've worked with without violating all sorts of laws. :(
But yes, basically: a super latency optimized runtime, a GC that's not very sophisticated but it elegantly dodges most GC problems by simply releasing all memory linked to an actor as soon as it quits (and Erlang/Elixir encourage you to spawn many of those in the right conditions; not for every single thing though), and one of the very fastest dynamic languages in the world, probably second only to JS's V8.
All of that is combined with me working with several other programming languages and their hosting solutions which were tripping over themselves when 1000 req/s started coming in (looking at you, Ruby 3.X and Puma and a few other servers; or PHP, or Python).
TL;DR: reliability is much better, latency is predictable.
Weirdest thing is: people don't believe it. If you only knew the CTOs I worked with: they were extremely data-driven and they would not allow me or anyone to just pull all of that out of their bottom. All had to be proven with numbers, and me and my teams did that, many times.
I understand the skepticism somewhat, but you and a few others seem to look at Elixir through the lenses of "too good to be true", and IMO you should try relaxing that skepticism to some extent. And try to be little more sympathetic because again, I literally cannot give you the hard cold data without violating at least three laws.
All I’m saying is that if you can drop 1330 servers just like that, there might be something more going on than Python’s slowness.
This is from experience. I have seen people create slow and fast systems with just about any tech. I can make Elixir crawl, I can assure you of that.
I have seen Python apps use 10 servers and reduced it to one as well. Same tech, just a more efficient mindset. It’s IMO a bit too simplistic to say systems with GCs fall over when under load.
Also nobody used the word "magically" before you did. Note that.
What's your argument exactly? That Elixir is overrated? Or something else?
Furthermore, I am not insisting on my standards of the quality of comments. I am under the impression that's the expected quality of comments on HN at large.
To be clear, I think Elixir is marvelous. This post spiked my interest in it actually. Sorry if I come across ignorant. That’s because I am.
I am not evangelizing tech -- I am a polyglot and I use what I find is best suited for a job, and Elixir happens to cover quite a lot of ground. That's all really. I also use Rust and Golang quite a bit.
I simply get ticked off when people start demeaning something without seriously working with it or even reading a bit beforehand. Sorry if I mistakenly put you in that group.
So yeah, Golang's GC is world-class, no argument from me.
The beam also has a spectacular failure mode: OOM whenever messages come in at a higher rate than they are processed. The lack of backpressure mechanisms mean a huge amount of beam language developers spend way too much time recreating their own way for dealing with this or pretend it is not a problem at all. This means too many libraries in the ecosystem behave totally differently under load.
Frankly the Elixir ecosystem, which is not without merit, is more interested in perpetuating the myth of magic scalability by virtue of the beam.
I am on ElixirForum every day and worked with Elixir for 7 years and have never seen anyone "perpetuate myths". I've seen some people willing to "increase adoption" which was always met with resistance by the wider community -- we believe growth should be organic.
Pretty sad stance from you though, I have no idea why people get so ticked off when another programmer wants to tell them about a secret weapon.
If you are not willing to try it, that's fair. Say that. Claiming you know stuff about the ecosystem while a guy who is there every day is not seeing that at all comes across as... strange. Biased. And not arguing in good faith. :(
I am not willing to drag others, such as those that wrote the repos, into a technical discussion with people out to act as you are.
If you try telling people that did encounter that phenomenon that in practice they wouldn't/didn't then you shouldn't be surprised if they question why they started talking to you in the first place.
You cannot just hound people with demands because you don’t like what they are saying.
I genuinely see only one thing: I asked you to elaborate but you are convinced that I am pretending to discuss while I, again genuinely, actually did want to discuss.
You asserting something about me, a person whose mind you cannot read is confusing and quite aggressive, in a very uncalled-for manner too. But as I said already -- have it your way, I disengaged because it became apparent you are not interested in discussing. OK. It's your right.
What's not OK is you claiming that I am not interested in discussing however, and I maintain that I was interested in discussing.
> You cannot just hound people with demands because you don’t like what they are saying.
1. I am not "hounding" you for anything, I asked a question.
2. You are again assuming my motivation and I assert that you have gotten it wrong. You that I "disliked what you said" is a borderline personal attack and an off-topic. I was confused why you claimed what you did and wanted you to elaborate, to find out what made you think like that and if I can change your mind with a few anecdotes and some facts (that are hard to look up because they require scanning a forum; yet they are there and are visible to everyone who engages with the platform).
BTW, if you really have known anything at all about the Elixir ecosystem you would know that its creator, to this day, engages with users on ElixirForum and asks for their feedback on what they find lacking. That sort of engagement and genuine discussion spirit that you claim I (as a part of the Elixir community) don't have.
That alone invalidates your point entirely.
I am disengaging second and final time, let future readers decide for themselves.
He at no time called him biased or said he was directly acting in bad faith
As I plainly explained, right at the top, it is not theoretical. There is no point engaging people with evidence if they are so dismissive of basic facts.
But then that is also true here. Your claims about him not saying what he plainly did are just bizarre.
I said, very plainly and visibly, that my parent commenter's unwillingness to back up negative claims COMES ACROSS as biased and ARGUING (NOT "acting") in bad faith.
Come on now, this stuff is not hard, the message is literally up there. Not sure why you had to editorialize it and thus misconstrue it?
Also concatenating binaries. Don't do that, use an iolist
The OOMs were largely being caused by calls to and from other services (i.e. kafka) so the answer proved to be in controlling the rate at which things come in and out at the very edge.
From what I saw I got the impression the Beam devs assumed memory and CPU usage go together so a system that is under load memory wise would also be CPU wise, but this isn’t the case if your fan out and gather involves holding large* values on which the response is based, even if for tiny amounts of time.
EDIT: *large meaning "surprisingly small" if you're coming from other universes.
In golang the approximate equivalent is a buffered channel that would start blocking because it has run out, but the beam will just keep those messages piling on to the queue as fast as possible until OOM. This is obviously a philosophical tradeoff.
* I should qualify that each request here had a life in low double digit ms, but there were millions of them per second, and these were very big machines.
Edit: huh. I could swear the VM had memory limit options. Guess not. Time to rewrite it in zig!
That team would have thoroughly endorsed a zig rewrite! It was a very odd situation where most of us liked erlang the language but found the beam to be an annoying beast, whereas most of the world seems to be the opposite.
My favorite thing about Python is writing prototypes. The big risk with writing a prototype is that it survives into production. Using Python ensures that the code will be replaced by real code
Python is great for other non-production code like Jupyter notebooks, numpy experiments, etc
I'm not sure by which metric we are judging whether a language is real or not, but I'm fairly sure that almost anything anyone comes up with will include Python, considering it's one of the most used languages in the world at this point, and used for a fairly large variety of use cases.
- No sound typing
- No proper package management
- Runtime type checking only
- Interpreted
I guess thats what he means with "Not a real language".
I disagree.... Its a tool that has it place. But its often used as a hammer.
* GIL - Your use case more than likely reimplements one wheel or another. Whatever compute you're doing should be deferred to the right tool (DB, queue, etc). Otherwise, if it's I/O, you're on par with Go and NodeJS (See FastAPI 3rd party quarterly benchmarks as an example)
* No sound typing - I don't know what this means. If you're concerned with typing, you can use pydantic for highly performant type checking and input/output validation.
* No proper package management - Poetry is excellent, been around a long time and is the unwritten defacto tool. Pipenv imo is close second. This argument feels forced as no one argued Go isn't a language before gomod was solidified. The community was fragmented and people wrote/chose their own tool.
* Runtime type checking only - In a world of interprocess communication, I don't see how this is relevant. You're not using Pytyon to write firmware. If you're not writing tests and just depend on successful compilation, you're writing bad code. Tests not only cover your point but are an excellent self doc. An added value, imo, that isn't spoken about enough. Regardless, tests cover this.
* Interpreted - A good last point to nail in the "your comment says nothing". What does this imply? That it's easier to debug with an interpreter? That cold starts are slow and your design is flawed given the tools?
Anyway, with C and Go bindings, most arguments against Python fall short. It has its place, yes, but a much wider one than the bandwagon regurgitates.
I don’t even know what “proper package management” means in this context. Certainly there’s an official package management system and modules. C++ doesn’t have the latter (very fragmented user efforts and not commonly used in my experience) and barely the former (no adoption). So that means C++ isn’t a real language?
Python does have type checking by the way. Same as TypeScript - you annotate your types and you can run a program to verify your annotations. This is basically how TypeScript works although the typing in the latter is more mature by way of @types packages to let the community supplement adding typing information to third party packages. There’s no relation between the two so it’s not sound (in both cases), but in practice it’s quite useful.
Anyway, it’s a no true Scotsman argument.
My comment still stands on saying that Python isn't a real language being a pretty non-sensical statement.
It shows that Python with Django is literally 40 times slower than the fastest framework. Python with uvicorn is 10 times slower.
The use of languages like Python and Ruby literally results in >10x the servers being used; which not only results in higher cost, but also greater electricity use, and pollution and carbon emissions if the grid where the data center is located uses fossil fuels.
Not to mention, dynamically-typed languages are truly horrible from a code readability point of view. Large code bases are IMO difficult to read and make sense of, hard to debug, and more prone to bugs, without static types. I'm aware that Elixir is dynamically-typed, but it (along with JS) is an exception in terms of speed. Most dynamically-typed languages are quite slow. Not only do dynamically-typed languages damage the environment as they're typically an order of magnitude slower, they also lower developer productivity, and damage the robustness and reliability of the software written in it. To be clear, I'm in favor of anything that increases productivity. If Kotlin were 10 times slower, I'd be happy to pay that price, since it is genuinely a great language to work with, is statically typed, and developers are more productive in it. I'm not sure how Elixir mitigates the downsides of dynamic typing (maybe lots of 'type checks' with pattern matching?), but it would definitely be super-nice if a well-designed (Kotlin or Haskell like?) statically-typed language targeting the BEAM existed...
I'll definitely take a look at JustJS, that's an impressive ranking!!!
https://just.billywhizz.io/blog/on-javascript-performance-01...
(Related GitHub thread: https://github.com/just-js/just/issues/5)
I was mostly responding directly to these 2 statements from my parent comment:
> They did savings by re-implementing their services and attribute those savings to the new tool / programming language.
> I wonder what the saving would look like if they chose another tool for the second / optimized system. I doubt it would differ much if they went with Go, Java or stayed with Python.
Which I didn't entirely agree with.
> greater electricity use, and pollution and carbon emissions if the grid where the data center is located uses fossil fuels.
to
> dynamically-typed languages damage the environment as they're typically an order of magnitude slower
is quite a stretch.
Do dynamically-typed languages inherently damage the environment? Or is it the fossil fuels?
Not that the appeal to the environment matters, because later on we have this:
> If Kotlin were 10 times slower, I'd be happy to pay that price, since it is genuinely a great language to work with, is statically typed, and developers are more productive in it.
My opinion is that slow languages that use 10x the electricity, with no ROI for the 10x energy use is bad.
High energy use, even if it's clean energy, implies a higher environmental toll. If a country were solely using nuclear and solar, higher energy use results in (1) more nuclear reactors constructed, and (2) more solar panels built. The manufacture and construction of both has an environment cost. Of course, with fossil fuels, the damage to the environment is potentially a lot worse.
> Not that the appeal to the environment matters
I don't think higher energy use is inherently bad. If we can improve the quality of life for human beings, then IMO a higher energy use is justified. I don't really believe in degrading our quality of life to lower our energy use.
My problem with many popular slow languages is that they have a negative ROI for the higher electricity cost. In exchange for 10x the energy use, you have a language that results in less-readable code (a serious issue), that causes more bugs / less-reliable software, etc. We literally a get negative ROI in exchange for 10x the energy use. Which is absurd and illogical.
If Hindley–Milner type inference had been more prevalent in the 1990s, I have a feeling dynamically-typed languages would have never take off. We're moving back to static typing with mypy, TypeScript, etc., but I'm hoping we move away entirely soon from using slow languages for writing servers serving large numbers of users.
If we actually got something in exchange for it, it wouldn't bother me so much.
Dynamic typing doesn't cost much more money on average and thankfully the cost of energy itself is a motivation for companies to do rewrites. If they are paying a lot in server costs and electricity then they do typically rewrite to reduce the amount of servers.
For companies they primarily need to worry about running at a profit and getting to market quickly which dynamic languages do extremely well, and the costs in electricity and carbon aren't very high when your scale is small.
This (or similar variants of this) is an assertion that's commonly made about dynamically-typed languages, but I don't think they hold any water.
Less readable code (due to the lack of types) makes it a lot harder to add new features, and harder to debug code as well.
Several years ago, I briefly worked on a fairly large codebase at a startup that was written in a Ruby on the backend, and CoffeeScript on the front-end. There were only around 60,000 active users. Yet, the dynamic typing made adding new features, or fixing a bug a truly painful and fragile experience. Needlessly painful, and slow. It literally reduced developer velocity.
I think once you cross a few hundred lines, dynamic typing becomes a handicap rather than an advantage.
All of this doesn't even touch on the energy use. Which I'll admit is irrelevant to most companies. Server costs even for popular web/SaaS/etc tech companies are often a tiny tiny fraction of overall cost, with most of the company's annual operating cost being employee salaries. (As I had stated earlier, I don't mind a language being slower - if it actually provided any advantages–like improved developer productivity, in exchange for that slowness.)
Facebook, twitter and plenty of of billion dollar businesses were built with dynamic languages and many would argue they may not have even existed if they were written in staticly typed languages due to the slower up front time expenditure.
I like staticly types languages for large projects but enjoy thr development speed of dynamic ones. If you don't like dynamic that's completely fine
This is pure speculation. There's no way you can back this up with any sort of real data.
The inverse isn't true either. StackOverflow was written in C#, and it's been doing perfectly well.
I've addressed another commentor making a similar presumption as you, here: https://news.ycombinator.com/item?id=37312297
The only thing I'll concede is that using a statically-typed language without a good type inference system, might slow people down a tiny bit.
The serious downsides of dynamic typing means you might still win out (in terms of developer productivity, code readability, code reliability, and ease of debugging) even if you use a language like Java instead of PHP.
Ultimately billion dollar companies have been built on dynamic languages. There is nothing stopping you from succeeding with dynamic languages. There just isn't. There are tradeoffs and these companies made them.
Yahoo used C++ for instance (it would not be my choice, even though it's weakly statically-typed).
But, yea, like you said--there are tradeoffs.
I think dynamically-typed languages has the allure of letting you build quickly initially, but the downsides of dynamic typing start hitting pretty soon afterwrd.
I write a lot of small scripts in Python. It certainly is easier to throw something quickly together, especially when the data model is amorphous, with dynamic typing. But that doesn't mean I'd use Python for a large project.
Did you even ever see implementation behind techempower benchmarks? There's NOTHING realistic in them. Those applications literally hardcode static content length header values to be faster. They are pretty good show of how low you can get to squeeze out performance but not one sane person will write code like that.
There's also a good amount of active discussion going on here: https://github.com/TechEmpower/FrameworkBenchmarks/discussio...
Here's the code for the JS (just-js) benchmark: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
And here's the code for the Python uvicorn benchmark: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
From some light skimming, the JS and Python code doesn't look particularly abnormal or overly-optimized.
Take some Rust frameworks: they write out pre-computed header strings to the output buffer, completely bypassing what the framework's documentation recommends. Examples are actix[1] and ntex[2]. No one would ever do this in real life.
Now I like Rust, and it'd likely be some of the fastest even without these shenanigans (Axum and may-http don't do that, I believe). But I don't know if other languages/frameworks have benchmarks implemented in the same non-idiomatic way just to look better.
[1] - https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [2] -https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
If first several bytes of the header are going to identical for many requests, a super-optimized framework would ideally memoize it, and write it directly out, as this benchmark is doing.
(And if the framework itself is doing it, then any user of the benchmark would just inherit that optimization, and not have to resort to non-idiomatic optimizations...)
I can't take seriously a benchmark that doesn't even show memory usage.
Hahaha, yep: https://github.com/TechEmpower/FrameworkBenchmarks/issues/39.... Nothing to see here.
There's unfortunately no other comparable as-comprehensive web framework benchmarks that I'm aware of.
This is the opposite case, when lessons learned in the first write are actually useful for a second rewrite.
I think the commonly associated soundbite is "build one to throw away."
They could have also blown everything out of the water with C++ - or probably even golang - but if elixir can do it on 2-4 boxes it's fast enough.
To someone who starts their job on an Elixir codebase, it is just not a smooth onboarding at all. While the performance aspect is unparalleled compared to most of the popular scripting languages in the last decade, the price to pay to settle into Elixir seems huge to me.
When you get junior (or even non junior) developers onboarded in a new language, you have a unique opportunity to break them of bad habits and expand horizons.
Yes, there is a cost to it as it extends in the short term the time it takes to get developers ramped up, however the long tail payoff is huge
In what terms, exactly?
Anything the business can control for: architectural designs, server costs, approaches to building out features / services for the business etc.
When you can mold someone's experience via a new language to model a domain, they become very efficient to it, when they have no prior notions to fall back on.
How many times have developers gone down the wrong path because of X did it this way? type thinking. When you can sufficiently remove that so all that is left to think about is the problem space, you do make more gains around that problem space.
My thesis from (albeit anecdotal) experience, is that when you have developers working in a new paradigm (often, this corresponds with a new language) you have better chances at establishing these things than having to consistently try and override a developers prior notions about how something should work / look.
The trade off is higher ramp times and slower on-boarding, of course. In the short term, it can be more costly.
We just did a Prometheus migration that I suspect will take us 5 years to break even on the development effort investment. And I'm not even counting opportunity costs, which were immeasurable.
I like Elixir and I want it to do well, but bad articles make that harder, not easier.
I probably might even sign-up some day if I weren't repelled by this being required every time I come. I even stopped reading Quora, and, most recently, Twitter because of this - they started requiring signing-in while I don't want to stay signed in and be tracked even though I actually have respective accounts.
People have written browser extensions to remove Pinterest from search results, as it is almost always a dead-end
Pinterest is useful just not for us nerdy guys. I am not sure why Google keeps it though or how they benefit, unless Pinterest uses AdSense exclusively, then one can determine that its some sort of partnership. You would think Google would be smarter about who to send over to Pinterest if thats the case.
So much for the idea that men are more "vision-oriented" than women.
I've just searched about two years of browser history and i haven't landed their once.
Nobody is saying they are forced to visit the site.
Google, I google anything and they show up (ESPECIALLY IMAGES). Blacklist one domain in search? Don't worry they've got 200 other ones.
I ended up doing a wildcard blacklist with uBlacklist
Not sure exactly why/when the switch over happened but Bing has dominated Google for image and video search for years.
I have experienced this many times when looking for a specific image on Google Images, I will click on the link that goes to Pinterest only to find the image is not there.
Oh, and if it could stop recommending pro-anorexic content to my teenager, I'd like that to. It's banned on the LAN but I can't enforce that elsewhere.
It's a dumpster fire of SEO dark patterns and unmoderated shitty content and shouldn't be ranked so high in searches. Google should have routed around them years ago.
I'm not looking for an _app_ when searching in google, I'm looking for a link to a _web site_ with original image.
Yes it's a joke... just like Google's results in 2023.
"One of the systems that ran on 200 Python servers now runs on four Elixir servers"
This alone is a major telltale.
However, could they have done a 50x performance improvement in Python? And what about the other numbers, like speed and concurrency?
That said, I'm confident they crunched the numbers and did the tradeoffs; after all, adding another language and/or architecture will make your company more complex, makes hiring more complex.
Here is the math they did.
Pros: Looks good in my promotion packet.
Cons: Need a bigger garage to fit all of these new sports cars.
And sure, you can get some performance improvements by rewriting things in those languages, at the expense of losing the entire python open source environment.
So I would need way more information about the previous system to take this even remotely seriously.
Python scales quite reasonably for most small to medium companies.
Even on I/O bound operations, in Python you have to choose between the error-prone async framework if you want to improve resource utilization or you stick to the synchronous world and accept extremely low resource saturation.
Either way, I can entirely believe that another language would beat Python on both counts. I’ve seen similar results rewriting a Python system in Go with extremely minimal rearchitecting.
The silliest thing is that the title credits the improvement to moving toward Elixir rather than moving away from Python (or maybe their case really is ideal for the BEAM VM and wouldn’t translate easily to, say, Go’s runtime model although I doubt it).
That doesn't mean that the move into Azure wasn't the right one at the time, or that it was more expensive than not going into Azure. It's simply that the market evolves.
The cost and the incredible performance gains we got by moving to a bunch of local computers was enough to make the whole thing pay for itself in about two months. Yep, physical computers costed less than two months of cloud. Plus the gains in productivity from having to wait minutes instead of hours.
Maintenance was never a problem, and we didn't need to hire new people to take extra care of the servers.
My current company is thinking of doing the same for AI servers. It's just too expensive in AWS.
Some of us never left ;)
50 millions children under the age of 5 are already obese (and I think 93% of obese children shall stay obese their entire life).
So... Anorexia may be a hill to die on and it's not fun for parents of anorexic kids but priorities, priorities and priorities.
The real eating disorder worldwide leads to obesity, not anorexia and I wish all the hate and energy spent thin-shaming was instead redirected towards fighting the more important eating disorder.
The numbers from the WHO here are scary and only ever growing:
https://www.who.int/news-room/fact-sheets/detail/obesity-and...
But the broader point beyond my specific snipe -- which is personal and driven by deep anger over a serious problem affecting me personally -- the broader issue is not only that Pinterest is on the whole an anti-social actor because of its garbage moderation and promotion of harmful content in its algorithms, it is more broadly a bad actor in the web ecosystem generally.
They hijack search results by scraping and stealing content, hiding the original link, and try to trap you in their circle of links.
In this day and age of people getting sued and slapped down for legitimate webscraping, it boggles my mind that Pinterest gets away with what they do.
On the whole a swamp of an unethical company. F*ck pinterest.
(But love that the commenter couldn't help but turn the thread into an underhanded comment against "fat people". Oh so typical.)
Wrong. Not in most cases, but binge eating disorder is a thing.
And I think this has become far more common than people will admit. We're burning out our pancreases through processed & high sugar, low-GI foods because the industrial food supply is poorly managed and under-regulated.
That plus cars.
But I hope parent commenter never has to live personally through finding out just how life destroying the predominant cultural attitudes around body form & food around us are. I wouldn't wish it on my worst enemy. Obesity is by far the least of concerns.
If you always do things right the first time, you don't get to brag about putting out the fire your started.
"rewriting in another language reduced the number of servers by 95%" is hard to beat, but at the same time, this saves "only" 2m a year, or about 0.3% of FY22 cost of revenue (per another post)
Pinterest per employee revenue seems to be around 1m, which basically suggest that this could even be a worse than average allocation of resources.
My takeaway would be "don’t bother with this kind of optimisation before you reach a scale where you can afford to do marginal improvements"
if you do this with 4 or 200 machines, does not make a big difference.
Maybe some of that also applies to 4 servers, but to a way lesser extent.
Having to maintain 95% less servers is worth it even if they didn't save any money IMO.
This also could lead to them reducing their engineering team that maintains these services which would reduce costs even more.
It's not esoteric. It's a modern programing language, with 35 years behind its Virtual machine.
1 in 3 phone calls you've ever made was routed through a telephon switch written in a lang implemented in BEAM virtual machine.
WhatsApp was written in a lang implemented on the BEAM.
> I'm willing to bet that way fewer people in the company/industry understand the new codebase and can make changes to it.
It's really not hard to learn. It's not Haskell, It's not Coq, It's not Brainfuck.
You could pick up the syntax and the major semantics in a weekend.
> Steve: We chose Elixir because we were looking for a system that was easy for programmers to understand and could take better advantage of our servers. I was intrigued at Elixir’s combination of friendly syntax, powerful metaprogramming features, and incorporation of the Actor model.
Am I wrong or is this guy in some sort of bubble where only functional languages are taught?
But if you are in a bubble where everyone uses Haskell and talks about Monads, then OPs statement may be valid
Also, I find it hard to believe that anyone who knows N+1 programming languages would have a hard time understanding Elixir quickly, it looks like most mainstream programming languages used today, with slightly different syntax for some things.
Take a look at https://elixir-lang.org/crash-course.html and you'll see what I mean, it's basically Ruby with some slight modifications.
I wrote this blog post about what I think are the three most important differences between Elixir and Ruby: https://phoenixonrails.com/blog/elixir-for-ruby-developers-t... . There's definitely a learning curve, although it's not insurmountable.
It /looks/ like Ruby in a lot of ways, but treating it like Ruby w/ modifications is a mistake that I've seen lead to a lot of really bad usage that (at best) fails to take advantage of the underlying Beam VM, and at worst [and more commonly] actively works against it.
I’ve also worked on several projects where I’ve pair programmed with a dev new to the language. They’re usually pretty productive within a few days. Not writing their own DSLs or going deep into OTP, but productive in terms of writing application code.
Phoenix is pretty straightforward for those who have experience with Rails, Laravel or a similar full-stack web framework.
The green threading runtime was removed in 2014, shortly before the 1.0.0 release in May 2015.
The reason for the waning influence was exactly that: couldn’t be reconciled with systems use cases effectively.
I hate async/await with a passion.
I bought a book about Elixir and genuinely tried to learn it, but noped out quickly as in I find programs written in it to be unworkable mess quickly.
It's like Perl on crack.
If you wanted something that was easy you'd go got Golang or Java, not a language without types.
Lots of reasons to love Elixir but this doesn't sound like one
I feel like statically typed langs, (I'm looking at you Java), are a bit more stubborn to work with, especially around API design, prototyping, greenfield stuff.
I do like langs like Haskell and Standard ML where they are statically typed, the the type system is mathematically sounds, and the types are inferred.
I want my type system to be inferred and bomb proof or to just get out of the way. Its the best of both world.
* Reading and understanding code is much easier because the types are written down. You spend much less time figuring out what variables can contain.
* Navigating code is much faster because tools like go-to-definition, code completion and find-all-references work reliably
* Refactoring code is a lot easier - or in large projects actually tractable. In large dynamically typed projects something as simple as renaming a variable can be an impossible task.
* Obviously the one people talk about most is catching bugs. The degree it does that depends on how strong the typing is (e.g. Rust will catch many more bugs than Java). But they will all catch the embarrassing things that dynamic languages can't like typos and missing arguments.
If you've only ever written short greenfield projects you might not appreciate some of these benefits as much as you should because you wrote all the code yourself so all the details are still in your head. It's a bit like saying seatbelts are an unnecessary pain because you haven't ever been in a crash.
> types are inferred
Some local type inference is good, but Haskell / ML style global type inference is kind of the worse of both worlds. You have to satisfy the type checker, which is harder because global solver errors are always harder, and you don't get the documentation benefits of static types because the inferred type is frequently a generic type.
Rust went with local type inference only for very good reasons.
This is all preference. Ive also worked on ruby on rails projects that are 15 years old with 500K LoC
i admit changing stuff can be pretty sketchy if your project doesn't have great test coverage. but that all ultimately comes down to the culture of the programmers working on that code base.
I think things like dynamic/ vs weak typing or functional vs imperative have a much greater impact on code quality/ease of coding than that of static vs dynamic.
I personally think programming in Java, c# is painful, but a lang like Crystal or Standard ML to be very pleasant. and vice versa, i think vanilla javascript is painful for all the reasons you mentioned but langs like Ruby, Erlang, or Elixir to be very pleasant.
https://edward-huang.com/programming/software-development/20...
“Some languages have a greater association with defects than others, although the effect is small.” Languages associated with fewer bugs were TypeScript , Clojure , Haskell , Ruby , and Scala ; while C , C++ , Objective-C , JavaScript , PHP , and Python were associated with more bugs.
Not reliably. It can work in a small subset of situations. For statically typed languages it always works.
> “Some languages have a greater association with defects than others, although the effect is small.” Languages associated with fewer bugs were TypeScript , Clojure , Haskell , Ruby , and Scala ; while C , C++ , Objective-C , JavaScript , PHP , and Python were associated with more bugs.
This is mixing up too many things. For example C++ has a relatively decent static type system, but obviously it's going to have way more defects than memory safe languages.
This is a much much better study:
JavaScript's weak typing does it no favors, agreed. But that's not a universal dynamic language issue. Ruby, for example, doesn't have those type coercion headaches.
Now, about Haskell and Standard ML—these guys offer a different flavor of static typing. It's not the Java-esque rigidity; it's more flexible and, dare I say, enjoyable.
On the tooling front, I've seen dynamic languages with solid IDE support and go-to-definition features. It's not a static-only perk; it's about the ecosystem's maturity.
That study is a neat data point, but it's not the whole picture. We should consider multiple variables like paradigms and type strengths, not just the static vs. dynamic lens.
I do also want to say in the defense of Elixir (and Erlang), with the advent of the dialyzer lib, it's a gradually typed language, and soon to be (fingers crossed,) a lang with a pretty unique type system. It will be both dynamic but have the same guarantees as a statically typed lang.
https://elixir-lang.org/blog/2023/06/22/type-system-updates-...
(Small tutorial on Set theoretical types, most of this is above my understanding though) https://pnwamk.github.io/sst-tutorial/
I bet it was an architecture problem more than a language problem.
I’d be expecting to leave most of that in place and fix the hotspots that are dragging performance down. Maybe some compiled code in certain places.
Almost certainly a detailed analysis would have yielded 20 things that could be tuned.
Posts like this are almost something to be ashamed of “we rewrote an entire subsystem because we wanted to use a language we like”. That’s really failing in your responsibility to the company.
I would tend to agree. I don't understand these sorts of decisions to rewrite huge sites in niche languages and frameworks. You tend to get one engineer who really likes some language, has some clout, and is able to pitch it as a good idea. And then the blog posts about how it saves the company untold millions, while hiding all of the associated costs.
Not only is it niche, so your costs going forward will be higher, it even required investing in new "tools and strategies" just to bechmark it.
"Elixir has proven so efficient that testing the limits of our services became a challenge unto itself, requiring investment in new benchmarking tools and strategies".
I wouldn't personally gamble on a robust job market for Elixir developers in a decade. And those there are will be able to command higher wages.
Sure, they will be out there. But so were COBOL developers for Y2K.
Finding people is a nightmare.
Really I can’t think of anything that would justify using a niche language over javascript/Python/C++/Golang/C#/Java.
Surely one of these will get the job done.
I agree with the skeptics about this article, it’s a flashy headline that can’t be accurate, per the points being raised.
- no idea what's wrong
- No idea what's right
- We know what was right, but the world has changed now.
Once this is done (maybe proof of concept, maybe research maybe years of fighting the politics) we can build out something that's not wrong.
If we time it well and get it right it's a unicorn. It's Facebook. if we time it right but get it merely not wrong, it's friendster or myspace.
And of course by the time we have learnt all that, mobile hits and we have to relearn
All user facing APIs and cloud provisioning state is written in elixir and runs on pretty much just fumes. Cloud systems interaction is written in golang because cloud stuff.
Email is in profile if you’d like to chat. Always happy to talk elixir and golang
I wonder how much time and team hours were spent learning the new framework and it's caveats vs if they'd just spent a little time optimizing their existing stack.
All too often common with non-perf folks making sweeping changes without rigorously measuring along the way. I guess it makes for good headlines.
"We built this thing really shabbily and look how much we saved using a different stack. Oh and we also architected it properly second time around."
No, that's not what happened! What happened is you put more thought into it the second time.
I hate this management BS; if the project fails, blame the engineers, if the project succeeds, praise the tech... They just do everything to turn engineers into commodities and most of us just play along.
So it may not even be Elixir that is saving most of that money. Who knows.
i had been developing in ruby for over a decade and about 5 years of Node exp at the time.
im not a language enthusiast and mostly write in go today.
I remember when twitter had fail whales constantly and they rewrote it from ruby to java I think. At the time everyone assumed it was all ruby's fault, and that might be partly true. But it's also true that the engineers who rebuilt it understood the problem much better now, and knew the major pain points. They also completely changed the architecture to be better suited to the problem. I submit that a lot of rewrites could happen in the same language and still have major gains.
All that being said Elixir is great, and particularly well suited to these kinds of problems.
And they said they could do it with 2. That is anywhere from 30x to 75x. Although I think the current work on Fibre and Async could have reduced it by 10x to 20x as well. Rails just need to adopt it. It is unfortunate they didn't try to optimise it but instead went with Elixir.
Yes, those numbers are for code, network speeds are not directly impacted the same way. But again, does not sound unreasonable.
Startups rarely die because of server bill specifically. More often it's lack of PMF, slow iteration, expensive labor. $2M is pocket change for Pinterest, they probably spend more on office snacks, so even if you grow to that scale, the savings in this example are not exactly life-or-death.
What is life-or-death, however, is how easy it is to hire for, how many quality libraries (implementation speed) and how many library maintainers work on any given stack. If it's easier to hire for Python or even Java, I'll use one of those. For stability, we used Elixir's Oban library to schedule jobs for financial transactions, and a bug in Oban regularly crippled our card authorizer, leading to our customers unable to use their cards and million-dollar losses. Oban, with all due respect, has probably 10x fewer maintainers than comparable solutions/libraries in other stacks. Maybe if we had a "true Elixir guru" on our team, we could have fixed it or even rewritten it from scratch, but we live in the real world, so we ripped it out, replaced it with a more boring solution and are much happier for it, much fewer after-hours pager duty panic attacks. Is it hitting the servers harder? Maybe, I don't care, it just works and the cost difference is likely negligible.
Python and Java have a multitude of web framework authors and yet none have made anything with the capabilities and ergonomics of Phoenix. I don't think they will either. It would take features those languages lack.
i guess it totally depends on "the thing" in question, but do you have any references for that assertion? that common web framework-ish, orm-ish stuff takes a lot more devs in python and especially in java?
The first reference for this assertion is anecdotal. I've been a professional dev for 13 years and grew up with family members and friends in the same line of work. I've seen and heard of many, many projects built in different languages and anecdote after anecdote has been in the direction of people getting more done more quickly in higher-level languages. Writing similar things in Elixir/Clojure/etc goes faster than it does in Ruby/Python/JavaScript, which goes faster than using Java/.Net, which goes faster than using C/Fortran/Pascal. You haven't been around for those anecdotes I've heard, but a public one you may be familiar with is Discord. They scaled to over 5 million concurrent users on just 7 server engineers IIRC. Two years later, at a much larger scale, they still only had 4 engineers focused on infrastructure and 40 total: https://news.ycombinator.com/item?id=19238221. How many cases can you think of where a team of 7 Java or Python devs built something equivalent?
Secondly, there's research. Google "Function Point Metrics". Most research on programming productivity is paywalled but I shared one paper that isn't on one of my first Elixir-focused YT videos 5 years ago: https://www.youtube.com/watch?v=1e2_NXLxi-E&t=412s. Obviously this isn't perfect since it doesn't include what is appropriate for what domains or how well they scale with project size. Still it's a useful data point, as are pure measures of expressiveness: https://redmonk.com/dberkholz/2013/03/25/programming-languag...
Finally, thinking from first principles, why would anyone be adopting newer languages if they didn't offer some advantages over older ones? Why would certain features, such as pattern-matching and macros regularly be adopted by languages of the past 15 years, despite being very rare in languages created in the 90s? Furthermore, why would startups—for which productivity is a matter of survival—be early adopters? The simplest explanation, in my opinion, is that there really are some productivity advantages and sometimes those advantages are enough to overwhelm the difficulty in learning something new and working in a smaller ecosystem.
Nowadays, kind of ironically after the consolidation, we have backend services in at least six languages I can think of. Which is just the reality of how things play out for a tech company doing acquisitions. But yeah we’re not making new ones in Elixir.
[1] https://www.sec.gov/Archives/edgar/data/1506293/000150629323...
Your condition is predicated on "everyone" at Pinterest revenue scale justifying the business case of throwing away their existing tech stack and adopting Elixir with an equivalent architecture---never mind that the article is coming from a company that sells Elixir consulting services---while major CSPs voluntarily bend over to notionally substantial revenue decline...and I'm bordering on bad faith??
At least I bothered to put a supporting numerical estimate on the hypothetical; you're just handwaving greenwashed bullshit with improbable, unsupported outcomes.
That being said, I don’t think we could have processed the traffic more efficiently in another language.
We were processing 12M requests per hour. You could run the entire thing on 2 vCPU we “over provisioned the crap out of it” by running it on 4 pods w 2 vCPU as the upper request limit.
> Our service didn’t quite grow exponentially in use, but it did hockey stick. It went from free, to a few hundred bucks, to around $12,000 just for API Gateway. No Kinesis. No Lambda. Just API Gateway.
> A good part of this entire system still runs in Lambda, although it will be moving into Elixir over time to make it easier to reason about and develop on locally.
> What everyone should do is think about where your service is going, and can you afford those costs when you get there. If you don’t have a team of ops people and you aren’t familiar with serverful stuff, spending $30k/mo on HTTP requests might be cheaper than an ops team.
[0] https://medium.com/coryodaniel/from-erverless-to-elixir-4875...
3-5k requests per second is not that much and highly depends on what you actually do.
A lot of things can be done just with nginx. nginx is very fast.
I’m nowhere close to performance requirements of Pinterest. But my auth service runs on a single 1vcpu, 512mb ram, and 10gb ssd. I use leveldb and swap on the 10gb. I’ve benchmarked it to handle 8-9k rps while delivering 150ms max response time. Not bad for a few bucks a month.
I'm running seven Elixir/Phoenix apps (one of these is umbrella with 12 apps) on shared CPU cheap VPS with 1gb memory. I actually ran out of disk space recently than anything else so far.
Jose: Can we have some of that money?
Steve: No.
And how much does the change cost them in terms of:
- Not being able to hire talent when they need because they don't know or aren't willing to work with Elixir.
- Spending extra time training engineers.
- Having to hire more senior engineers/those versed in functional programming vs generalists.
- The rewrite itself (I'm sure it didn't happen magically overnight).
Unless the number had a few more zeros at the end of it, there's no scenario where this project made real business sense vs being some principal engineer's vanity project. $2M is what a company like Pinterest spends on an off-site or a random exec's spot bonus. There are probably analytics dashboards that no one looks at that cost >$2M/yr to produce.
around 3.3 k requests per second ....is that a lot ???
This wasn't done to have mone to donate to Elixir but for revenue.
And besides that, you don't even know if they do
Then developers learn they can't pay bills with exposure tokens and employment doesn't offer the lifestyle they had hoped for.
But there is no worry, they get replaced by next cohort of gullibles believing in open source.
What I am trying to say, it should be illegal for big corporations to use open source without paying royalties to all contributors.
''' The combined effect of better architecture and Elixir saved Pinterest over $2 million per year in server costs. '''
Why do people write hollow drivel that doesn't actually say anything?
If you can wrap your mind around immutability of data structures and understand how to use enumerators instead of loops for iterators, you're golden.
Have you ever seen basic db optimization? Alone in my companies people were just using stuff wrong.
Performance and Architecture are a after thought in our industry. A normal developer doesn't think about it.
There was one query in my company which was running in one region slower than in another one and there was also an explain statment available. No one looked at the explain statement and thought "huh why does this simple select use so much memory". People weretrying to see why the regions themselfs were different not what the problem with the query was.
Yep, their blog post is carefully worded to say "The combined effect of better architecture and Elixir" but they didn't mention how much of it is related to architecture or what specifically they did with Elixir to make things faster. It feels like a marketing piece for their consulting services.
I mean they put:
> Rewrote an #AWS APIGateway & #lambda service that was costing us about $16000 / month in #elixir. Its running in 3 nodes that cost us about $150 / month
They saved 100x here by moving from an expensive architecture (Serverless lambdas) to potentially reserved instances which are reasonably affordable, at least for cloud standards.
I remember once needing to parse XML in Python. I started with the easy approach of using the first XML parsing library I found which was xmltodict. Eventually I stumbled upon lxml which improved overall performance by 20x and I didn't have to rewrite much code at all. Sometimes it's easy to get big wins in your existing language if you know what the problem is.
defmodule Math do
def sum(a, b) do
a+b
end
end
def sum(a, b):
return a+b
That said, code volume is never the issue, code clarity is. Right now you're comparing trivial examples, but what does in this case Pinterest's high volume spam detection code look like? I wouldn't judge a language on microscopic code snippets, or disregard them as "too noisy" when it usually isn't a deciding factor in programming languages.
Huh? TekMol explicitly points out that the python code also includes a module definition. It's just that the python module definition takes up zero characters in the source file. (Note that this is not always true, but in this case it is.)
It takes more than that, in the name of the file, but the Elixir file will also have to have a name, and the comparison ignores both of those equally.
-module(math).
sum(A, B) ->
A + B.Besides, the equivalent of Elixir's module (a struct with functions) is Python's class, not a plain file.
Sure, if you don't like having the name of a module/class at the top of a file that excludes a lot of language.
def sum(a, b), do: a+b