Hacker News Reader

Comment by robertAngst | Hacker News Reader

> they'd written a monolith instead of building a system where the message delivery was designed to be sharded; n

Do some languages handle this differently?

Isnt this based on how the code was written rather than a language natively sharding?

Yes erlang does. Unless you try hard and work against the language erlang systems are written differently than ruby/rails program.

I used "system" and "program" there specifically. One of the key things erlang "forces".

Languages do matter.

Yes, that was largely my point. While Rails at the time probably did sort-of encourage a monolith, there was nothing requiring it to be that way; it was their design choices, not Ruby or Rails that caused their scaling problems at the time they moved away from Rails.

When they were rewriting anyways, it's very much possibly that moving off Rails was the right choice anyway, but that was incidental to the far bigger problem of their broken architecture.

Sharding is basically a data storage concern, so it's not something that the language can really control.

There are however, frameworks that enforce some level of "shardability". The Datastore (now Firestore) in AppEngine/Firebase comes to mind, in that your data is shared all over the place (the details are hidden from you), and you can't write un-indexed queries that don't scale well. The Datastore has some limitations that definitely seem awkward if you're coming from a relational DB world (e.g. no joins, limits on the kinds of inequality queries you can do, etc.), but these limitations are there specifically so Datastore can guarantee performant distributed queries.

Sharding is not just about data storage. You can shard parallel units of work across multiple CPU cores in order to prevent single CPU bottlenecks.

Erlang encourages shardable solutions from OTP concepts being so strongly integrated into Erlang.

Absolutely not. While working for Amazon we had to think about a lot how to "shard" our fleets to have the least amount of items in every cache and maintain high hit ratios. If you let all traffic hit a single cluster your cache hit ratios will be terrible and most customers are going to experience worst case latency which is absolutely do not work for popular websites.

Reply on news.ycombinator.com