Hacker News Reader

Comment by JPLeRouzic | Hacker News Reader

> "After just a month our barely visible Fediverse presence was taking up 160 gigabytes"

The "160 gigabytes" text was linked to the Johnny Mnemonic wikipedia page, so it it possible they're just exaggerating and meant "it was growing at a rate we didn't expect or couldn't really sustain"?

This got me wondering, I created a Mastodon user @ defcon.social and my first couple of posts were images, just to test things out and get started with my chosen client - Ivory by Tapbots (the creators of Tweetdeck, RIP) after tptacek spoke glowingly of it here. Then I realised that by posting those I'm basically taking up space on a community service without paying for it, and I wasn't sure exactly what the etiquette was. On a commercial, monetised service like Twitter it's different as there are advertisers (or at least there were...) but afaik "defcon.social" is just a non-profit and I'd really like not to be a big cost sink.

Also I wondered if there's any overhead involved in following users across many different instances - if my @defcon.social user follows a bunch of users on various other instances does that place additional burden on their mastodon instance or just on my local client?

Might be time for me to read the defcon.social fine print and T&Cs (I only glanced over it) or even about Mastodon/ActivityPub generally...

From what I can gather every follower gets an individual copy of every post sent to their server who either deduplicates it and stores it in their database or just stores the message per subscriber.

So if 10,000 people on server A subscribe to someone on server B then server B will send 10,000 duplicate messages to server A.

I hope I’m wrong but that’s what I’ve heard.

Oh my! I’d heard ActivityPub was chatty but haven’t looked into the details. This, if true, sounds … not good.

Anyone with expertise here who can speak to the scaling of ActivityPub?

The size of an ActivityPub payload overhead is minimal.

On each individual server they are ideally saved based on their unique identifier (which are RFC3987 IRIs) and I doubt that there is any software that will do it individually for each inbox instead of storing one copy to which each inbox links to.

To minimize even this, each server can store just the IRI(which is mandated to be de-referenceable) and load the payload when prompted by UI. I doubt that anyone does this also.

Yes that makes sense! Thanks!

Reminds me of:

> This program posts news to thousands of machines throughout the entire civilized world. You message will cost the net hundreds if not thousands of dollars to send everywhere. Please be sure you know what you are doing.

From the rn usenet client, back in the day.

Maybe if more people thought this, a lot less posts would be made. I'm guessing that would be a net positive for society in general.

> Maybe if more people thought this, a lot less posts would be made

This is why we can't have nice things: https://en.wikipedia.org/wiki/Laurence_Canter_and_Martha_Sie...

Why are tech people so behind mastodon? If this is true this sounds incredibly worrying to scale; to the point where mastodon is limiting it's own growth potential

If. Did you research if this was true before posting a knee jerk reaction to something that might not be true? If not, you are no better than the person that posted the misleading comment.

Remember there's a fair bit of speculation here, I just thoughtlessly blasted my question here without researching (not sure where to start, tbh) and UncleEntity didn't seem too sure about the duplicated/redundant requests. It's entirely possible that Mastodon doesn't have this problem

I wish people wouldn't spread rumors like this. ActivityPub does have shared inboxes[0] and Mastodon does use them[1], so no duplicate messages will be sent in your example.

[0] https://www.w3.org/TR/activitypub/#x7-1-3-shared-inbox-deliv...

[1] https://github.com/mastodon/mastodon/blob/a5a00d7f7adff5e0af...

I mainly use twitter to keep tabs on some artists and none of them want to move. Reading this, maybe it is for the best. My follows would create a lot of overhead for wherever it is I park my account compared to someone who just participates in text.

I have my own instance, and it eats disk space, in a fun and sometimes difficult to recover way.

It caches all media that you view, and I assume it might prefetch stuff to. It has a tool that prunes out old cached images and such. However that script isn't setup to run automatically (or if it is, its far too loosey-goosey)

However the biggest killer is that it doesn't clear cached user page header images. Here is the output of my cache. Headers are not cleaned, and need to be deleted manually.

  Attachments: 1.43 GB (8.19 MB local)
  Custom emoji: 61.9 MB (0 Bytes local)
  Preview cards: 344 MB
  Avatars: 3.01 GB (71.7 KB local)
  Headers: 6.69 GB (172 KB local)
  Backups: 0 Bytes
  Imports: 0 Bytes
  Settings: 36.2 KB

Federated content is copied to every instance, so yes, follow a few people and you can see your instance's disk usage explode. I run an instance for some friends.

The usage can be contained with a few cron jobs to purge old cached content aggressively. Running

`tootctl statuses remove && tootctl media remove` once a week is enough to make it sane in my case.

To add to this, you can also set it up so entire federated servers are followed, and it may be that way by default. So follow ONE person on mastodon.social, and suddenly your toy instance is trying to copy a significant chunk of the Fediverse to its hard drive in real time.

That's done via relay servers, you can't just firehose an entire server without its cooperation.

You're correct. I forgot about that. I did try relaying for a while, that's what made the usage REALLY explode.

That's $3/month on S3.

My personal instance has been up two months with 12 GB space used, but only 88 MB of that is my stuff; the rest is just the cache of others' posts. That can be fairly aggressively cleared out, and I think Mastodon lets you configure an automatic purging of this data.

I was very tempted to create and operate an instance for me and my friends, since Digital Ocean have a nice pre-built image that's ready to go, but it never left the "that would be fun, I should read into that" phase. I hope you don't mind but I'm about to fire off a bunch of questions - if you're too busy to answer or just don't feel like answering, I won't feel offended :) How are you finding it, is it just yourself or do you have a few more users? Do you have to deal with abuse/spam much? Also what kind of hardware do you have it on - is it pretty memory/cpu hungry ever or is it quite well-behaved?

Mine's on masto.host, so it's managed; costs me a few bucks a month and I'm happy with it. It's just for me, so all my follows are federated; as such, spam/abuse has been non-existent.

Ah I hadn't heard of masto.host, ok that's another thing to look into. Thanks!

2 cores and 4 GB memory is more than adequate for a <10 users. My instance runs on an old Celeron NUC sitting on the shelf above the litter box and connects to the internet through a cloudflare tunnel. Works great, and is basically free to run.

Ah funny, I didn't think to look if there was tunneling that let me serve local stuff publicly (I have a NUC tucked away somewhere in my flat, I had it serving some stuff thru SSH tunneling to a Digital Ocean droplet until I realised "why am I not just running this on my droplet?")

Akkoma is a very good option if you're concerned about resource usage. I proxy media from other servers and then let it hit a managed CDN (bunny.net in this case). My CDN bill is less than 10 cents per month.

I do this, with the digital ocean image. It’s been pretty low overhead so far, I wind up paying like $15 a month at most including storage (not hooking up email saves a lot).

I don’t allow people who I don’t know to make accounts so I haven’t had any spam or abuse trouble.

I haven’t had to deal with any downtime or technical issues since I got it running, which is a bit of a disappointment tbh

Reply on news.ycombinator.com