Comment by danpalmer | Hacker News Reader

Comment by danpalmer | Hacker News Reader

Parent Full thread

> already upgraded the server as high as possible (16€ to 67€ per month)

No judgement on you or the dev using "as high as possible" here, I realise it could be a simplification in this instance.

However, I have seen a fairly strong pattern of fediverse servers being run by people with relatively few skills in system administration/web apps/operations/etc. It seems like there are a lot more people who are engaging with it on the same level as people renting multiplayer game servers, or installing Wordpress on a consumer-focused VPS provider.

And I think this is kinda great. I want the fediverse to be accessible at that level or even below! It does lead to issues like this though.

67 euros per month is a small server, and a single server is not the best way to run a service that other people are using. It's different when it's your personal site, but a social network?

I think the fediverse needs to figure out whether it wants to focus on making smaller servers easier or bigger servers easier. It's fine if most people are on million-user servers with teams running them, and it's fine if most people are on family servers that can go down without much negative impact, but this middle ground where people are on ~10k sized servers that individuals without the necessary skills are struggling to keep running on a volunteer basis seems to be a bad middle-ground.

Honest question ... How far can you push "one server"? Given that HN runs on one server (plus one mirror) and this apparently serves a not that small community well ... Given "1 million connections" are possible on one server ... Given that green threads are a thing these days ... Given that SQLite is so much faster than networked databases ... Given that a monolith is faster than Microservices ... Given that it's possible to use a compiled language instead of trying to speed up interpreted languages ...

What's the limit here? HN is a great example of what's possible, but can a single beefy server scale to handle the traffic of a Reddit?

Serving a social network is not the case where you'd want just one server. Mainly due to reliability concerns, spikes in traffic and different patterns at different times.

But if you like the idea of ignoring best practices, there are EC2 instances with 192 vCPUs that go for several thousands an hour. And taking as example Golang benchmarks https://github.com/smallnest/go-web-framework-benchmark where it's not uncommon to be able to serve 6 digits of 30ms requests per second with just 4 cores. Then the conclusion is, the limit is the sky, if you are stubborn enough.

> But if you like the idea of ignoring best practice

... do you know me? ;-)

Another dumb question ... for "several thousand an hour" I could buy a pretty decent server myself ... last time I checked (on https://yourdatafitsinram.net/) the largest physical server there was like 10 grand.

Combine that with the ability to actually throttle a physical server (ie you only pay what you actually use, not per hour) ... I guess I would propose the following scaling strategy:

1. get cheap physical server 2. get your software up and running 3. get a second physical server that is roughly twice as capable 4. use that second server as a "beta" and development server ... do a lot of profiling there 5. once your real world usage spike on the first server routinely reach 80-90%, switch production over to the bigger server 6. get a new development server that is twice as beefy as the former development server

I mixed up pricing. Several thousands a month. But if you could buy a 192 CPUs server for just 10 grand, seems like not a bad deal. Especially if this hypothetical project is able to attract the load we are discussing for longer than a few weeks.

Several thousands an hour is a bit of a stretch, maybe you meant monthly? but your point still stands.

For those interested: right now the largest instance I see is a u-12tb1.112xlarge (448 vCPUs, 12 TB RAM, 100 gb networking) which is ~$109/hr on demand or $40-80/hr with a 1-3 year commitment. This works out to ~$29k, ~$58k, or ~$79k per month.

If you don't need 12 TB of memory, there's a 6 TB instance for roughly half the cost.

Honestly, I am completely clueless, but what would a beefy server with 240 physical cores cost? (8 CPUs with 30 cores each)

edit: physical server that is

Whoops. Mixed up pricing. 192 vCPUs instances on-demand depending on the type indeed fluctuate around $10/hr, leading to several thousands a month.

Not to forget the data transfer costs though https://aws.amazon.com/ec2/pricing/on-demand/ And also the potential minefield of API GW and NAT costs.

this really depends on the software. a website using a "slow" interpreted programming language (php, python, ruby etc) in a "heavy" application that runs many tasks in the background on each pageload does not compare to a website that merely returns static information, either directly from the webserver or mostly from cache (no or much less need to waiting for cpu to process, or disks to fetch data vs ram). it is easier to just archive and provide access to existing content than to also run all the "invisible" services around that content.

You have to remember, that until 1-2 weeks that 16€ server was more than enough. This was a tiny community and most people (including me) had not heard about kbin, many not even about lemmy.

My eventual plan is to host my own 1-user instance and rely purely on federation.

In theory shouldn't there be able to be both? Some apps should be able to optimize for large-scale usage while others optimize for ease of installation and maintenance in your spare time, but all should theoretically (barring the inevitable drama) be able to federate with all the others.

I do wonder what the funding model is supposed to be for the large servers. Tiny servers can be funded out of pocket, but large platforms can't, and the resource requirements of fediverse apps seem to be much higher per-user than those of a centralized platform.

(I'm not sure if the resource requirements are inevitable or just a sign of an immature app.)

With +1gbps fiber available many people could even host them at home, I guess.

Hell, I currently don't even have gigibit fiber and self-host a Mastodon instance at home. Currently hosting Plex (meaning content-sharing with friends and family), Mastodon, Matrix, Airsonic, Wireguard, PiHole, and more. Everything runs off an HP EliteDesk with QNAP hosting all the media. The SFF PC is quiet and uses relatively little power.

I am glad I have no data limit though - I apparently went through >3 terabytes (split 30/60 between up and down) last month alone, mostly due to Plex and Wireguard.

Yes! Although I think that targeting both is unlikely to work in one app, which means we need interoperability and true federation between different codebases.

Currently this leaves a lot to be desired. It sorta works for status-based updates with Mastodon and a few others, but it's not a great situation as Mastodon isn't the most standards compliant player, and the standards themselves are pretty loosely defined.

Also from what I've heard about Mastodon, running it is surprisingly resource intensive, for whatever reason the server doesn't seem to be very efficient.

I wonder if this has the same issue.

Not sure about Lemmy. But for mastodon, there’s an incredibly high amount of servers to federate and keep a connection with. Then add all the storage for all the content being created. I can imagine it being heavy.

According to the docs [0]:

> Lemmy uses roughly 150 MB of RAM in the default Docker installation. CPU usage is negligible.

[0]: https://join-lemmy.org/docs/en/administration/administration...

I made my own server for mastodon, had to shut it down because it was pulling like 10GB per day.

It was fun but that was insane.

I agree here. I looked into setting up a public instance. Realized quickly if it gained tracking a single box is not going to work and you are immediately in a scale out setup of postgres at least.

Once you get to 3 postgres nodes, 2 LB's for the DB traffic, then Front ends with Load balancing, you are looking at 9 servers-ish minimum....

That monthly cost even in Linode is a bit more than I would be willing to front with no compensation.

That said, i may just setup an instance of kbin I can control and rely heavily on the federated portion to interface with the various communities. If my understanding is correct, I can access other lemmy instances even with kbin, such as beehaw etc.

Yeah, that seems to be the case. I'm on a kbin instance, can join magazines (subreddits) on other instances. Votes seem to propagate across instances.

The entire problem of fediverse versions of twitter & reddit reminds me of bbses, which I just remembered I loved and miss.

The negatives all just became positives, or at least drivers towards the positives.

I was digging through the lemmy perf issues a little and was surprised how they are doing things. Cool they got as far as they did though. Maybe I really underestimate how unique knowing how to scale things is

Reply on news.ycombinator.com