Hacker News Reader

Comment by msoad | Hacker News Reader

I love how Facebook and Instagram never went the route of "full rewrite" for their apps as they scaled.

I my experience "Language X is slow and we could save Y switching to Z" is always a false promise. You can pick parts of the system that are costing a lot and port them to other language/frameworks to capture the bulk of savings while keeping your developers happy working on familiar code. Or if you'e big enough like Facebook, you can go see why X is slow and if it is possible to improve at the compiler level. Never disrupt the developers flow (Twitter did this with their Scala craze back in the day)

View on HN

If Twitter's port to the JVM didn't result in 10x fewer servers with 10x lower latency vs an identical Ruby implementation we wouldn't have done it. Sadly not much progress has been made on the RubyVM though Twitter did try with improvements to GC and other changes. We even tried JRuby as part of the evaluation.

Shopify is working on this! They have YJIT, led by maximecb

Is TruffleRuby promising at all?

Enormously. The latest Railsbench results on TruffleRuby are ~3x CRuby.

Nice. 10 years later but glad that Ruby programmers might have an option that lets them program the way they want without having to choose a different language / platform for performance.

It is still no where near the level that Twitter would have stayed with Ruby though. Although I believe TruffleRuby being 3x faster than current CRuby is only the beginning. JVM got tons of improvement in the past 10 years as well it is uncanny.

Although Dr Chris Seaton mentioned it is not his goal to have Shopify Rails running on top of TurffleRuby. Not sure if he meant it was short term or long term. I think Dr Stefan Marr will be joining him soon, and Dr Maxime Chevalier-Boisvert is leading a team working on YJIT in Shopify. ( Did Shopify just hire all known Dr. working on Ruby performance )

Might as well grab Charles Nutter while they are at it.

> Sadly not much progress has been made on the RubyVM

Lots of progress has been made on CRuby. The official optcarrot benchmark (realistic-ish interpreter bound) is ~6x faster on Ruby 2.6 than Ruby 1.8.

CRuby performance running Rails improved almost as much. Zendesk got a 2-3x improvement from 1.8 to 1.9 and 1.9 to 2.6 doubles it again.

How does that compare to the same algorithms run on other languages these days?

Wasn't there a time when jruby was faster than the standard ruby implementation? On one hand, that's absurd, but on the other, a lot of effort goes into Hotspot, and you'll get those benefits for free.

Not really, it can turn out good from the JVM stuff for specific cases here and there, but it's mostly interesting for not having a GVL, thus enabling full parallelism for threads.

I'm not sure many companies have the luxury of doing what FB did. I can't imagine a world where choosing to rewrite your app, or choosing to rewrite the entire language, where rewriting the entire language would be the cheaper solution.

Twitter did this with their Scala craze, but Twitter struggled to scale it's Ruby app. Twitter didn't have nearly as deep pockets to write a performant alternative Ruby VM.

Joel Spolsky did it, city something?

I believe the product was called "CityDesk" and the in-house language that compiled to several different target languages was called Wasabi.

Edit: I was wrong, Fog Creek did have a "CityDesk" product at one point, but Wasabi was for on-prem FogBugz[1].

[1]: https://www.joelonsoftware.com/2006/09/01/wasabi/

> keeping your developers happy working on familiar code

That turns pretty quickly into "developers being unhappy that they have to maintain legacy code written for the old language/platform," though, especially with any sort of employee turnover.

It doesn't have to be this way. If you're going to go all-in on the monolith approach like FB and Instagram did for their product code (at least when I was there), you need to have a central team whose entire job is scaling the codebase. This means actively refactoring other teams' legacy code, making sure that tests are fast, etc.

Yeah I have seen this successfully done at a smallish (200 person) company. There was a "platform" team which was essentially an "un-suck the monolith team".

Refactor slowly over time, turn a ball of mud into cleanly defined domain boundaries, parallelize the tests, etc. etc.

Idk, I worked at IG for a year and wrote a lot of backend code in the Python webserver stack. I believe they would benefit massively by aggressively migrating services to the FB Hack stack.

A ton of work goes into making sure the Python code isn't slow, and there are complicated C++ services built as workarounds for the slowness of python.

Heresy! This is a Python thread, please summon the required positivity!

I disagree, this logic doesn't work always unless you have investor money lying around. For a bootstrapped business, it could mean living or dying. As an example, a client of mine was paying close to $5000 per month in server costs simply because of the scale of traffic they had on their site. By re-writing Wordpress in Elixir, I was able to bring their costs down to $1000 odd per month. It's actually cheaper than what any of their competitors are paying for, in servers as well.

This $4000 in savings actually allowed them to hire an additional full time developer to maintain their site. So, your logic only works for certain use cases, not all.

How much did you charge to rewrite WordPress in elixir though?

And how big was the risk to have the invested time wasted? How long did you block new feature developement?

Safe to say they would've gone out of business if they kept burning that kind of money for the remainder of the year. This was around covid era, they didn't have a solid revenue stream from the traffic they got.

I'm obviously not going to reveal actual numbers, but it's safe to they spent less than what they would have spent being on Wordpress for the remainder of the year. Time to completion is 3 months. No new "feature" development was blocked. Remember - this is a content website (Wordpress is mostly used for content AFAIK and less for building features).

The moment it was done, we just switched nameservers. It had no impact on client's workflow or day to day life.

If it was a content website on Wordpress, couldn't Cloudflare's $5/mo APO have done the job? It static caches the site at the edge. You'd get almost a 100% hit rate unless there are strange requirements.

edit: Adding link with info about Cloudflare's APO product https://support.cloudflare.com/hc/en-us/articles/36004982231...

yes it could, but it's not webscale!

gotta use aws

Well to be fair he is an Elixir consultant so he rewrote it in Elixir.

APO is not a drop in to make what better, there are alot of kinks, and you could spend as much in dev time to get that working right. YMMV.

It can be a drop in (it was for my sites), but it depends on the requirements

Good suggestion, but if your content site is a news site, then homepage and certain category pages require by the second updates. Cloudflare unfortunately can't help and all the requests are directly hitting your database at that point. That forces you to get an expensive MySQL instance to sustain that kind of traffic.

Cloudflare will automatically purge the cache when you update or publish an article. Also from my understanding of APO, the cache will get distributed to all their edge locations without each one needing to hit the origin server. So if you update an article it would only get fetched a few times max until it gets updated again. That seems pretty good. Also as far as MySQL, you could install a Redis cache in front. Personally we use Varnish and Memcached as additional layers in front of Cloudflare (it works out of the box with Cloudways)

Saying that you disagree and saying at the same time that you rewrote it in Elixir makes me think that wasn't a good use of the money. They will have problems hiring for it and it isn't even performant compared to other languages you could have written into if you really wanted to save money (i.e: golang).

Of course I don't have all the context, but, I cannot imagine any circumstance when rewriting some server side logic in elixir would be useful from a PoV of cost savings..

How do you know it wasn't a good use of the money? The client's site has been up for a year since then without any issues for 1/4th of the cost. Their investment in this stack wasn't much. Hiring wasn't a BIG problem, but it did take time. We used that savings to find a dev to maintain the site.

By RAW numbers, GoLang or something else maybe faster, for sure, but it will definitely extend the time to develop something like Wordpress because its frameworks I've tried (Eg. Gorilla) need more effort compared to using something like Phoenix that I use. Plus, a lot of devs were already onto Ruby, it's easier to onboard them into Elixir than GoLang.

> Of course I don't have all the context, but, I cannot imagine any circumstance when rewriting some server side logic in elixir would be useful from a PoV of cost savings..

I think you may not have worked with a high traffic site, just by swapping languages alone you get insane cost savings for that scale because it quickly adds up. Even if your new language is just 0.1 second faster per user, for a million users, it adds up really fast. As I mentioned, just by this switch alone we brought down the cost to 1/4th.

I believe GoLang or something else may have drastically reduced further, but it has the same downsides of Elixir (Harder to find talent for, learning curve). I'm a pure Elixir consultant, so I chose it. My client is more than happy with this outcome.

Yes, I kind of agree with everything you said actually! Apart from Elixir from a cost savings on infrastructure PoV, if that's how you wanted to frame it (since the context of the HN submission is about changing languages for better performance or not, and Elixir not being particularly fast).

How much of that traffic could have been avoided by a CDN, a caching plugin, Varnish, and php-fpm?

You can't cache everything on a news website, especially the homepage. The homepage requires constant updates. For rest of the pages, I agree and I do use caching.

Even a one second cache would cap your homepage to 1 req/s which can be cheaply and trivially handled by any stack.

Every single large news website is cached.

This is some of the original thinking around StackOverflow, if I remember correctly from the podcasts. Jeff was a C# dev and using what was familiar, but he also commented on how you could get good performance out of some crappy code and easily refactor later.

Their whole culture elevated performance and they handled a lot of traffic with relatively small amounts of hardware that they documented. I've got some familiarity with a Rails app where the devs seemed astonished at some of the details in their blogs about running SO.

One factor for Twitter was that Ruby was simply orders of magnitude worse for Twitter usage than I think Php or Python was for Facebook or instagram. Or at least one never heard of these providing horror stories where Twitter definitely had horror stories. In one of the few times I've been "close to the source", I heard Twitter's head of operations complaining the electricity had become their primary expense from starting X many Ruby instances per second. It was one instance per connection or something close, because Ruby was unstable and because Ruby never gave memory back to the system.

Yeah, it kind of reminded me the "python strict mode" they talked about before [1]. No designed to be cool but to be useful.

[1] https://instagram-engineering.com/python-at-scale-strict-mod...

Yes, Static Python especially relies heavily on strict modules, since they enable us to perform module-local analysis, which enables some cool optimizations.

This is something that people have known for a long time: a lot of money has been spent on making compilers make fast code. For many intents and purposes we know what kind of reasoning we want to be able to do to produce fast code. Strict modules is probably one of the sinples yet efficient optimizations you can do: being able to reason about where and what something is without having to do an expensive lookup every time it is used is brain dead simple to understand.

Yet many of the languages that evolved as "scripting languages" threw much of that knowledge out. In the middle of the 90s we knew how to make fast dynamic languages. Maybe not as crazy dynamic as something like ruby, but not far off. Or at least, they had the sense to codify it as a separate thing (CLOS in common lisp comes to mind).

Something like SBCL is more dynamic than you need(TM), yet produces code that is often at least an order of magnitude faster than python.

"Language X is slow and we could save Y switching to Z" is always a false promise.

The thing that keeps software interesting for me is that there are almost no absolute rules. So I'll agree that complete rewrites are usually a mistake and performance problems are often not in the language.

But I can't agree that rewrites are always a mistake. esbuild is a recent good example of how much difference switching to a faster language can make.

Did Twitter actually contribute to Scala core though? They definitely created and contributed to various Scala libraries but they didn't fork/augment the Scala compiler like Facebook did with HipHop.

They did contribute to the JVM but directly, not by forking it.

And are nowadays one of the reference users of GraalVM in production.

Oh right! I forgot they moved to that.

Not sure I can agree with that. Sinking resources in to re-engineering a language infrastructure, mostly because the language / infra is not actually a good choice for the target problem is an approach only very large engineering teams can tackle. As has been already been started (further down the thread). Breaking down your code / monolith into manageable components and re targeting for performance in the key spots is almost always the right approach. Unless of course you have the engineering resources and hubris of someone like FB

Rewriting in full for the sake of performance gain is indeed a questionable proposal. Rewriting hot spots being the more sensible approach.

A better reason for rewriting however is to migrate to a more maintainable language with focus on readability from one which focus on expressiveness (and hence favored for rapid development) as the project matures.

I think while performance is not a proper justification, if you get other stuff in the bargain it can be really worth it. Rust, for example - the justification to switch is usually not just speed but safety and the safety enabling more optimizations, parallelism etc. If all you care about is speed you stay in C/C++.

Sometimes the choice is also made when you already have to pay a transitional cost - at a previous employer we had to rewrite a bunch of code to move from an old version of PHP to a newer one, and a strong argument was made that at that point we should just adopt a better language since we already had to pay that cost. I think eventually a lot of stuff transitioned to Haskell as part of that process, even if other stuff stayed in PHP.

Reply on news.ycombinator.com