Comment by dcchambers | Hacker News Reader

Comment by dcchambers | Hacker News Reader

Parent Full thread

Every time there's a GitHub outage of any severity one of the top comments on HN is inevitably suggesting that it's probably due to Rails. It's getting pretty tiresome.

Calling it a "legacy rails" stack is incredibly disingenuous as well. It's not like they're running a 5 year old unsupported version of Rails/MySQL. GitHub runs from the Rails main branch - the latest stable version they possibly can - and they update several times per month.[^1] They're one of the largest known Rails code bases and contributors to the framework. Outside of maybe 37 Signals and Shopify they employ more experts in the framework and Ruby itself than any other company.

It's far more likely the issue is elsewhere in their stack. Despite running a rails monolith, GitHub is still a complex distributed system with many moving parts.

I feel like it's usually configuration changes and infra/platform issues, not code changes, that cause most outages these days. We're all A/B testing, canary deployments, and using feature flags to test actual code changes...

[^1]: https://github.blog/2023-04-06-building-github-with-ruby-and...

It's easier to blame <piece of technology> than to admit running a service at Github's scale is highly complex and takes deep expertise.

It's also that many Rails shops have performance problems: which isn't the same as saying "Rails is slow"!!. "Getting performance problems at some point" is almost a rite of passage in Rails; I'm certain every rails developer has pored over N+1 queries, caching, async jobs, race-conditions, gems and whatnot to keep the system running.

The only Rails projects that I worked on that never had performance problems are the ones that never reached any scale. All Rails projects that gained traction that I worked on, needed serious refactorings, partial rewrites, tuning and tweaking to keep 'em running. If only to tame the server-bills, but most of the times to just keep the servers up. Good news is that it's very doable to tune, tweak and optimize a Rails stack. But the bad news is that every "premature optimization is the root of all evil" project made a lot of choices back in the days that make this nessecary optimization today hard or impossible even.

What I'm trying to say is: Performance issues with Rails will sound very familiar to anyone who worked seriously with Rails. So it's not so strange that people reach for this conclusion if almost everyone in the community has some first-hand experience with this conclusion.

> The only Rails projects that I worked on that never had performance problems are the ones that never reached any scale. All Rails projects that gained traction that I worked on, needed serious refactorings, partial rewrites, tuning and tweaking to keep 'em running.

You'll be hard pressed to find any stack that doesn't require this.

Obviously.

A big problem with rails, though, is how easy it makes it to "do the bad thing" (and in rare cases, how hard it makes it to do the "good" thing). A has_many/belongs_to that crosses bounded domains (adds tight coupling) is a mere oneliner: only discipline and experience prevents that. A quick call to the database from within a view, something that not even linters catch, it takes vigilance from reviewers to catch that. Reliance on some "external" (i.e. set by a module, concern, lib, hook or other method) instace-var in a controller can be caught by a tighly set linter, but too, is tough.

Code that introduces poor joins, filters or sorts on unindexed columns, N+1 queries and more, are often simple, clean-looking setups.

`Organization.top(10).map(&:spending).sum` looks lean and neat, but hides all sorts of gnarly details in ~three~ four different layers of abstraction: Ruby-language because "spending" might be an attribute or a method, you won't know, Rails, because it overloads stuff like "sort", "sum" and whatnot to sometimes operate on data (and then first actually load ALL that data) and sometimes on the query/in-database. It might even be a database-column, but you won't know without looking at the database-model. And finally the app for how a scope like top(10) is really implemented. For all we know, it might even make 10 HTTP calls.

Rails (and ruby) lack quite some common tools and safety nets that other frameworks do have. And yes, that's a trade-off, because many of these safety nets (like strong and static typing) come at a cost to certain use-cases, people or situations.

Edit: I realize there are four layers of abstraction if the all-important and dictating database is taken into account, which in Rails it always is.

I wasn't really blaming rails per say, if anything their main database mysql1 seems to pop up in their post mortems more then anything else

Reply on news.ycombinator.com