30,000 users seems like a ludicrously small number of users to hit scaling problems. It sounds like Mastadon has not been designed for scale from the ground up, which is surprising for a project that hopes to be a popular social network.
The frontend serves 30,000 users. The backend processes their posts alongside the posts of everyone that replies to them and all posts from people that the “home” users follow. So, while the home user base is 30,000, the effective load on the back end is much more, depending on how many followers/following a person has.
You can find posts from people who were hosting their own single user instance that went down because of a popular post that got federated across multiple instances.
I either understand something wring about it or that still points out to Mastodon being slow and badly engineered.
Like, the updates are per server not subscriber right ? That's at worst few thousand requests, all of them can be essentially served from RAM. Even if you get 10k responses to your comment, 10k responses within say 5 minutes is still only like 30-40 req/sec, that shouldn't be much even for Ruby
That doesn't give confidence to the typical self-hosting Mastodon user who goes 'viral' somehow. So they should expect that if they were to have a post to go viral across instances they have to become a sys-admin for the day to bring it back and scale it up to handle the traffic.
No wonder normal users are not self-hosting their own instances to fully own and self-verify themselves on Mastodon and have to search for an instance to re-centralize to.
That is very disappointing and not a great sell for Mastodon so-called 'verification', but not at all surprising.
Mastodon is still a solution pretending to search for a problem, already reminding us of the failure of federated social networks in the long term and eventually withering away and re-centralizing with larger instances.
https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
Hardware having been obtained (via Hetzner, instead of in Kris Nova's home lab), the instance has scaled.
The article quotes blocking requests for 10 to 20 seconds, this causes everything to go slower.
The deep nesting and overall layout of the cache drives me nuts. The storage layout is designed for the state of filesystems ca. late 90's, pre reiser or ext3, with the major deficiency of pretty much demanding an unnecessarily expensive object storage setup.
The nature of most of the space on an instance like this going to cache remote content is also that it ought to not try to make caching decisions in the app, but leave the actual caching to a proper cache, which would've been trivial if the paths of the cached objects was mapped to the media-proxy URLs it falls back to if the file has not already been downloaded. Instead a bunch of effort goes into building a cache setup that people often end up putting on expensive object storage when it's data that shouldn't even need redundancy.
It was migration from one clustered filesystem to another and the temporary step was just Apache equivalent of try_files and NFS mount aside the new FS while the migration was running in the background.
If anything filesystems are better with some nesting vs "one big dir"...