RIP botsin.space
muffinlabs.com
muffinlabs.com
I'm not hot on the fediverse in general, and this just sours me on it a bit more. A bunch of dedicated admins keeping instances going, basically running hobby servers/websites like it was the 90s/early 00s, is never gonna work for the kind fo scale services grow to these days. I know not everything requires scale and lots of ppl are happy existing in their little silos, but that's just it, they're silos. Might as well be back on seperate forums for our seperate interests again. When you want the power of a mix of accounts/networks/interests everything balloons and can't be run with funds and larger centralization. Sigh. It's a tough one and has yet to be solved in full, with any existing approaches all sort of half-solutions. Maybe that's the way forwards in general (an internet of islands) but it sucks to have things going up and down and having to migrate around the net (with or without our own data) like nomads.
I don't really get the appeal of this. Different topics/interests often demand different moderation and forum features. Trying to shove everything into the lowest common denominator of social media results in things like people posting essays as screenshots of Apple Notes.
The only obvious benefit I see of this kind of large scale centralization is for marketing. And that's not a benefit to me as a user.
Traditional forums can solve the former by using social sign-ins, and the latter by having RSS. However, support for either of this is inconsistent, so it's usually easier to just use subreddits.
Finally, money. Forums cost money to host, while subreddits don't.
But then the Fediverse unfortunately runs up with the classic issue of network effects. For me personally, my Unified Home Feed of Reddit has more relevant content to me than my Unified Home Feed of Lemmy. And it seems it'll stay that way since Reddit's communities have thousands or millions more subscribers in general.
So, sticking to Reddit is easier. This isn't helped by the admin wars on the Fediverse which introduce messy and silent breaks in the network, often requiring multiple accounts to view everything you're interested in.
To the Fediverse's credit, their network is a lot wider so it still has some uses, that is, hosting communities banned on Reddit. But I'm not sure if some of the more "normal" subreddits have much incentive to move over.
On average I'd say account creation has become harder because everybody and her child tries to shunt you into some social sign-in but it still is not difficult to make accounts. "Modern" password managers make it a breeze to juggle many accounts, and frankly, what Netscape Navigator could do almost 30 years ago was already enough for that.
Same goes for forums. phpbb [1] can run on any LAMP potato and you get a whole boatload of potatoes for 2 EUR/month at Hetzner. I don't know what HN requires, but I guess it's not much more. The hard part of forums are the people needed to keep order and Reddit is not helping there, quite the opposite.
Sorry, I notice I'm grumpy.
I do acknowledge that there's a big benefit of having different accounts: different personalities! You don't have all your social eggs in one basket, so to speak.
€42.48 max. per month.
CPU: Intel Core i7-7700
RAM: 64 GB
Drives: 2 x 4.0 TB Enterprise HDD
I’ve run pgbench on some 12 euro/month VM at hetzner and it outperformed our 18k/year AWS RDS instance. Sure it isn’t managed, etc, but there is a lot of room between 12 and 1500 euros
Aside from that, if you’re running something on a shoestring and your own time, moving it somewhere else is a lot of work for little joy.
If anyone needs to migrate their own projects I've had good luck with feed2toot, to post RSS to a Mastodon account on a ordinary server. It's been around a long time now and seems reliable.
unsurprising that the bots would outpace organic users, but wow, what a ratio. i'd be curious to see this data charted over time
Also, there's a measuring change from active accounts to plain old users. I don't know the proportion that are active, but if I recall right, the fediverse as a whole had under 1 million active users. Assuming 500,000 active accounts that pull all the weight, it's 220 toots per user on average.
What is the best way around this for a hobby project similar to botsin.space. I don't mind the service going down in case of a DOS attack. I want to handle TLS myself though (so no free Cloudflare).
Most important thing is my good sleep at night, so no fine print that allows the provider to pass on the cost to me in case something goes wrong. (If that means higher fixed cost, that's how it is, I'm not asking for a dream house, just reliable cost control).
With AI scraping becoming more of a thing, a cloud platform could roll out a feature where an AI is allowed to scrape the daylights out of your site, but they must pay for the bandwidth or bandwidth + premium.
In this scenario you wouldn't wake up bankrupt but instead with a windfall of cash because TikTok decided to scrape all your stuff.
I definitely got joy out of setting up a bot on it. Huge thanks to colin for making it so easy.
One of the reasons I maintain a node with only one user is I fear the day I'll be responsible for other people's social media presence; I could easily see myself going "It's just a few thousand users" and the next thing I know I'm asking whether I can keep this thing going (and agonizing over what it'll do to my users to cut the service). And unlike Colin, I despise Rails and wouldn't have the patience to hammer on it when it starts to misbehave.
Props to Colin having the guts to take the risk.
It doesn't make sense to me that such a thing take so much dev and ops cost compared to IRCv2 servers, other than for the fact that modern webdev just so happens to be extremely bloated, especially when extremely competent and high spirited developers are giving up like this.
Are we doomed to keep adding more RAM and more disk and more bandwidth to catch up with ever-growing bloat?
In the case of Conduit, a Matrix server with a few private rooms and users consumed only 32 MB of RAM, using RocksDB for storage. The equivalent on Synapse required about 5x as much memory, despite using SQLite. In practice, Synapse instances will use Postgres since many appservice plugins specifically require Postgres and don't support SQLite. Not to mention, SQLite isn't optimized for frequent, concurrent writes.
I do sincerely think the choice of Rails, and the fact Ruby only got a compiler people use recently, means that most Ruby programs require fairly beefy processors and plenty of memory in order to keep up with a few hundred clients.
Of course, I am extrapolating based off my experience running Synapse (a Matrix server) with Postgres. There is a chance Mastodon scales much better.
I believe IRC doesn't operate like that. The messages delivered to each user don't need to be retained, and I assume the size of the largest channels is in the tens or hundreds of thousands.
I'm pretty sure the median IRC server runs for much less than 7.5 years. I don't think anyone expects volunteers to dedicate decades of their life to admin duty. and it seems fine and healthy for the ecosystem that he is telling people they have a few months to move their bots to a different server.
i haven't personally operated a misskey derivative, but based on my experience writing network servers on node.js it probably performs better than rails XD
the same applies for clients. there are nice native apps and some pretty efficient web clients, but they aren't the default on the most popular server software so nobody uses them.
If I’m not mistaken, Twitter uses Scala. That would have been a good start. For all the indie-ness, one of these clones could have been written in hand-tuned Rust or C# or Kotlin to respect the resources of people who would run them out of their own pocket. But sadly this has not happened yet.
Unfortunately it's written in ruby and it has no quotas on the amount of images and videos uploaded or downloaded per-account. Also it is not designed to scale horizontally or leverage any form of p2p.
The real culprit is the cloud premium. If you go with hetzner you have lots of runway too.
It has such an embarrassing failure mode that is unthinkable in statically typed compiled languages. Rails is not even a better choice at its main selling point which is developer UX, it has been many years since the rest of the industry caught up and surpassed this. Nowadays, as a primarily C# developer I’m always baffled by the crutches RoR developers have to deal with - one would just not tolerate these in .NET.
At the end of the day, if every line of code costs 100x more, it’s very difficult to come up with a good reason where such Ruby tax is worth it in the projects that cannot afford to throw more compute and memory at a problem.
The fact that he built it and maintained it is the proof for the claim. Without Ruby it simply doesn't exist, so it doesn't matter how much hypothetically better your favorite language is.
Thanks for hosting this all those years. I'll try to find a new home for my bot.
So, at a rough guess you need:
- Some kind of company entity for all this to belong to, possibly a LLC, and all the associated paperwork that goes with it.
- Bank accounts and some way to handle card payments
- Some level of requirement to provide customer service/support, at least for billing related issues
- Have to now also deal with card fraud, refunds, disputes, charge-backs - even if you use some service that will handle most of it, you'd still need some level of involvement
- Have to handle billing related tech stuff - issuing bills, ensuring accounts are activated/deactivated based on billing events
- And now you need to charge enough to cover all of the above, and your time, and the time of any professionals involved in the above
Starts to sound like at least a part time job. The OP may not want to go there.
For my personal projects that I provide as a service to others, I do them for the fun of it. In the past I've bailed or cut access to them when they've started to feel like a job.
It's almost novel now days getting sucked into something that shuts down. killedbygoogle.com is a meme partly I think because websites shutting down is just so uncommon in areas that we get personally invested in.
I run my own Lemmy instance just for my self and even that can be trying sometimes. I enjoy using it instead of reddit, but one day I will probably shut it down and be sad.
Managing all of that is easily learnable in a couple months if they have time, disposable income and few distractions but surprisingly few people who have site management thrust upon them know about these things in advance. To most people who think about running an internet anything the above are unknown unknowns. You can't go looking for things you don't know exist, so burnout is high.
I've seen no evidence that running a fediverse server is nearly so legally fraught.
The actual figure is a median of 1 subpoena per annum per 430k users if the majority of the users are under 30.
Of course, the real gold standard would be P2P, if only it could work. But... mobile phones can't burn battery running P2P clients in the background, everyone's under a NAT these days, and some types of software (like microblogging networks) would be horrifically intractable as a P2P system.
Oh well. At the very least, I really love the concept of Decentralized Identifiers (DIDs). I'd like to see more stuff like that.
The common comparable people raise is email/gmail, but we used the DID system and account portability to try to get a better outcome on provider migration. It's hopefully more like web/google -- which still has the centralizing pressures of scale benefits, but hopefully injects enough fluidity in the system to move the situation forward. Sometimes, you pick the design that moves the ball down the field, not the one that guarantees a touchdown. If somebody wanted to improve on what we've done, I'd tell them to focus on the federated queries problem.
On the other hand, in practice it seems like the AT proto infrastructure is still very centralized for now. DIDs are excellent in theory, but everyone is using PLC DIDs, which depends on the centralized plc.directory. You can run your own PDSes, but there's only one relay for now (that I am aware of.) I also don't think there is more than one instance of the Bluesky AppView, and the official instance is locked into the Bluesky Moderation Service, which seems to limit the usefulness of some of the censorship resistance of the protocol.
I'm not sure how much of that is social problems rather than technical, but I worry that since Bluesky and AT proto are gaining massive popularity with this status quo (millions of users!) it'll have a problem akin to Matrix.org, where in practice almost everyone is using the same infrastructure anyways.
It's still relatively early days, but millions of people is definitely enough to where you start to hit problems with having everyone under one roof. I really hope we get to see how things play out when more of the network is operated independently.
I'm also a bit miffed that Dendrite was positioned as a "next generation" Matrix server but now it feels nearly orphaned with missing support for newer features, issues with various appservice bridges, few updates at a very slow pace, and no migration path out in sight. I know it came with a fair number of disclaimers, but that still bums me out as it seemed like it would be okay for a small non-critical homeserver, and now it seems likely I'll have to engineer my own path out when clients finally stop working with Dendrite. (It already happened once...)
You have no idea how bad I want to love Matrix, but frankly if it was due to a focus on usability that decentralization "purity" has suffered, it simply does not show in the resulting usability improvements over years of time. Sorry to be harsh.
if you’re interested in progress on Matrix, https://matrix.org/blog/2024/10/29/matrix-2.0-is-here/ is where it’s at.
edit: I guess though that faster room joins weren't a part of Matrix 2.0. Actually, I don't really have a huge problem with the sync taking too long personally. So maybe Matrix 2.0 wouldn't bring that big of an improvement for me anyway.
I started to worry that it would not improve and I reexamined XMPP.
Finally I switched to XMPP that I had abandoned more than 5 years ago because I think there is a better chance that the community will finally offer clients with the important features, on all platforms, in the coming year and that it will last over time.
I will also say - there are ~100 self-hosting PDSes in the network, about 25 relay consumers, 3 alternative appviews that I know of (smoke signals, frontpage.fyi, and whitewind), the firehose & backfill are fully available, the specs are getting fairly complete, and the software is open source. This is a priority for us.
I don’t mean to be glib, just wondering if things can be “done better”
There's some additional overhead (doing HTTPS calls for verifying signatures, for instance) but that information can be cached pretty effectively.
Pushing contents is no more than posting HTTPS calls to every server in your follow lists, and possibly exposing said content in a GET API for pullers, though that's entirely optional.
Mastodon is heavy because of the way the backend is written (I blame Ruby on Rails for tha one), but there are fully featured ActivityPub servers out there that are orders of magnitude more efficient. Mastodon devs prefer the ease of development over performance but that's a choice, not an inherent problem of ActivityPub.
Versus something like Pleroma; which I've used since it's inception, being incredibly janky and lightweight, prone to breaking, but later versions have mostly ironed out most of those catastrophic bugs. It has it's own challenges as well, but it does historically scale better, is more flexible, and less intensive per instance
One demands a lot of money and time, where the other demands a lot of time and not so much on the money side. I'm not going to spend the time to give you a history of pleroma/mastodon instances, as it's a controversial history at best and there's a lot of people who know little, yet who will believe themselves an oracle. (ofc that could also be me so take it with a grain of salt and all) Though if you are willing to read through a bunch of highly opinionated accounts, and you pay close attention to what actually happened, the answer is pretty clear.
ActivityPub is intensive, but not the main culprit.
Correction: "The Mastodon software requires a lot of hardware resources in order to run properly"
Alternatives like GotoSocial, Akkoma, even Honk are much less resource intensive.
Mastodon is filled with such utterly basic UX issues. You move instances because the old one announces a shutdown? No old posts visible, no import possible. You want to see the history of an old account on another instance? The oldest toot you'll see is the first one that your instance picked up from that account. You have to switch to their instance to see old toots - there's a helpful link at the end of the feed, but it's still annoying. "Trending" topics only carry stuff happening on your server, and most of it is days old garbage. Search is horribly broken and inconsistent.
But this isn't "utterly basic" to solve on the backend due to how ActivityPub (currently) works. First you have to allow backdated posts[0] (not supported in the spec) which requires a mechanism to stop them being sent out (else they'll appear in current timelines[0]) but also you need a mechanism to send them out (to update the old URLs except how does the new instance know where the old instance sent the status? And how do you prove that you have the right to even request the change?)
These are probably not insurmountable but they do require a lot of thinking about!
[0] I ran into these importing an old Twitter bot into my Akkoma instance. I had to modify the server code and it was not a fun time.
They're basic for the user. I know a few people who left Mastodon for good after the second or third time they had to shift servers. That kind of stuff should have been thought of from the beginning...
> to update the old URLs except how does the new instance know where the old instance sent the status? And how do you prove that you have the right to even request the change?
The same way an account move is currently reported to the instances where followers reside and handled there - the account-move operation would only need to do a full re-scan of the old profile. That's a ton of traffic for people with followers from many instances, I agree, but the source instance could trigger the creation of something like a data dump that destination instances can download without hitting the API.
That gets you the old statuses, great. How do you then insert them into your existing instance? You can't just repost them because they'll appear with new timestamps (bad). You can't just repost them with old timestamps because servers and clients assume "just arrived == now" (bad). If you're using sequential IDs on your status table, good luck with that because I'm pretty sure someone has taken the shortcut of using that instead of the timestamp. Assuming you can work all those out, now you need to update the old URLs in the follower timelines to point to the new URLs (unless we punt on this and just let the old timeline sit around as it.) Except you don't know who got those statuses when they were posted - the old instance would need to have kept all the queue records for every post and be willing to supply them to the new instance. Or you can "eh" that and send them out to the new followers (except we don't want to do that because it confuses current instances and clients to get old timestamps at a new time) but that doesn't mean everyone will be updated. Or you can try and persuade the old instance to redirect each old status to its new URL once you've updated the software and protocol and clients to allow for status redirection, obviously, and worked out how to verify that server X is actually allowed to redirect A@Y's old statuses and isn't some hijacker / spammer / whatever and ...
I've not even given this much thought - I'm sure people who actually dwell in ActivityPub and security worlds can give a much better explanation of why it's not at all easy to implement.
> That kind of stuff should have been thought of from the beginning...
Yep, can't disagree that a whole heck of a lot more thought should have been put into the AP protocol from the start.
It was sad to see say, the "user friendly" webcomic go away. But I enjoyd it in the time. Just live the moment. Don't expect even the big websites and apps of today to last - not at least in the form you enjoy.
They don’t appear to solve any of the power dynamics of users and operators - users are still at the mercy of the operator - and they run on either altruism or monetization.
Mastadon appears to have successfully created N copies of the Facebook problem, which is definitely better than where we were.
Mastodon allows you to be the operator, if you so choose.
... and if you forget a critical update or you see it too late, you'll get hacked.
Self-hosting anything comes with serious challenges that most people only realize in hindsight.
But I think this just reflects the facts. Centralization works and is highly preferable for many users. Just like in the only big federated service: email.
Yes you can run your own. But there are a lot of costs in terms of time/complexity/knowledge/trust to that.
Outsourcing it to someone else is really nice.
You don’t need one big instance like Twitter was. Having a small handful of big ones works well too.
But the dream some people seemed to have where everyone should run their own instance alone or with a few friends was never going to happen.
I don't think users care about that at all, and if they have it explained to them, hate it. I think the real problem is that we haven't decentralized ownership and decisionmaking, instead we shattered big dictatorships into little fiefdoms, often run by local gangs (as one would expect.) Arguing that federation should automatically solve our problems with social media is like the US argument for "state's rights." You had one problem, now you have 50.
This is also exacerbated by the fact that people can't migrate. That would seem like it should be a developer priority to enable competition between instances, but instead people get irritated when asked about it at all. Every post locks you in farther to a particular instance. If people can leave on a whim, bad instances would starve. Instead of people being able to vote with their feet, the politics of mastodon all revolve around punishing other instances for various examples wrongthink by defederating. So now it's little fiefdoms at war with each other, you have to be in the in-crowd of your likely randomly chosen instance to have a say about it, and if you leave you lose everything.
Instead, everyone seems to be joining Bluesky now, which is also federated but doesn't mention it anywhere so users can just join the main instance.
I expect this will cause massive problems in the future when federation will start taking place on a serious scale and the risks of misleading users by using similar usernames on other servers start applying. People don't know the network is federated and there's no easy way to read up about it without diving into dev documents.
>Outsourcing it to someone else is really nice.
Yeah but the key thing is that you can choose your provider. Email isn't a walled garden that can be enshittified because you can just migrate somewhere else - yes it's a huge pain and has a bunch of drawbacks, but you can do it, and people do do it.
Moving to a different Mastodon instance is a way smaller transition than moving from Twitter to another social media platform entirely.
The Fediverse has a bunch of issues but I don't think we should think about it as "running your own", we should think of it as "choosing the provider that best fits your needs", as many have with Gmail.
XMPP tried to do it for chat. All the big players adopted it and then either realized that the protocol wasn’t complex enough for the features they wanted to offer or that it was much better financially to invest in a closed system. Sometimes both. The big providers split off into their own systems (remember, Google Talk/Hangouts/Chat and Apple iChat/FaceTime both started out as XMPP front-ends) and the dream of interconnected IMing mostly died.
RSS tried to do it for blogs. Everyone adopted it at first, but eventually content creators came to the realization that you can’t really monetize sending out full-text posts directly in any useful way without a click back to the originating site (mostly defeating the purpose), content aggregators realized that offering people the option to use any front-end they wanted meant that they couldn’t force profitable algorithmic sorts and platform lock-in, and users overwhelmingly wanted social features integrated into their link aggregators (which Google Reader was famously on the cusp of implementing before corporate opted to kill it in favor of pushing people to Google+; that could have potentially led to a very different Internet today if it had been allowed to release). The only big non-enthusiast use of RSS that survives is podcasts, and even those are slowly moving toward proprietary front-ends like Spotify.
Even all the way back to pre-Web protocols: IRC was originally a big network of networks where every server could talk to every other server. As the system grew, spam and other problems began to proliferate, and eventually almost all the big servers made the decision to close off into their own internal networks. Now the multi-server architecture of IRC is pretty much only used for load balancing.
But there’s two decentralized systems that have survived unscathed: the World Wide Web over HTTP and email over SMTP. Why those two? I believe that it’s because those systems are based on federated identities rather than federated networks.
If you have a domain name, you can move the website attached to it to any publicly routable server and it still works. Nobody visiting the website even sees a difference, and nobody linking to your website has to update anything to stay “connected” to your new server. The DNS and URL systems just work and everyone just locates you automatically. The same thing with email: if you switch providers on a domain you control, all the mail still keeps being routed to you. You don’t have to notify anyone that anything has changed on your end, and you still have the same well-known name after the transition.
Bluesky’s killer feature is the idea of portable identities for social media. The whole thing just ties back to a domain name: either one that you own or a subdomain you get assigned from a provider. That means that picking a server isn’t something the average person needs to worry about, you can just use the default and easily change later if you want to and your entire identity just moves with you.
If the server you’re on evaporates, the worst thing that you lose is your activity, and that’s only if you don’t maintain any backups somewhere else. For most people, you can just point your identity at a different server, upload a backup of your old data, and your followers don’t even know anything has changed. A sufficiently advanced client could probably even automate all of the above steps and move your whole identity elsewhere in one click.
Since the base-level object is now a user identity rather than a server, almost all of the problems with ActivityPub’s federation model go away. You don’t deal with blocking bad servers, you just block bad people (optionally using the same sorts of “giant list” mechanisms already available for places like Twitter). You don’t have to deal with your server operator getting themself blacklisted from the rest of the network. You don’t have to deal with your server operator declaring war on some other server operator and suddenly cutting you off from a third of your followers.
People just publish their posts to a server of their choice, others can fetch those posts from their server, the server in question can be moved wherever without affecting anything for those other users, and all of the front-end elements like feed algorithms, post display, following lists and block lists, and user interface options could either be handled on the client-side or by your choice of (transferable) server operator. Storage and bandwidth costs for text and (reasonable) images are mostly negligible at scale, and advertising in clients, subscription fees, and/or offering ancillary services like domain registration could easily pay for everything.
ActivityPub sounds great to nerds who understand all of this stuff. But it’s too complicated for the average social media user to use, and too volatile for large-scale adoption to take off.
AT protocol is just as straightforward to understand as email (“link a website domain if you already have one or just register for a free one on the homepage, and you can easily change in the future”), doesn’t require any special knowledge to utilize, and actually separates someone’s identity and content from the person running the server. Mastodon is 100 tiny Twitters that are somewhat connected together, AT actually lets everyone have their own personal Twitter and connect them all together in a way that most people won’t even notice.
Go into settings, click change handle.
Type in the domain you wish to change to. Click next.
It’ll give you some stuff to put into a DNS TXT entry on that domain. Do that. Click “verify DNS record.”
And that’s it. You’re done. Everything is “transferred.”
The history is transferable for the same reason a domain is transferable to another web host: what does URL stand for again? Uniform resource locator? That is, it’s how you locate something, not what that something is. In this case, the domain isn’t actually your identity: your identity is your DID, “decentralized identifier.” To hand wave slightly, all your content is signed with your DID information, not the URL you use. There’s a service that resolves domains to DIDs. So changing your domain means changing what that service resolves to. That’s why I put “transferred” in quotes above; when changing domains, nothing actually moves.
Now, if you want to change the server where your data is hosted, your PDS, it’s effectively the same thing: you spin up a new server, backfill your data by a backup or by replaying it from the network, and then say “hey here’s a new PDS” to the network.
All of this is possible because of the fundamental design choices atproto makes over the ones ActivityPub does.
Happy to answer more questions. But if data ownership and preservation is a thing for you, you should like atproto.
Of course, it still relies on the benevolence of the guy who runs and maintains the instance. He actually takes a fee out of the donations each month to pay for his time, but it's a token amount.
> But the recent Mastodon upgrade has caused a significant amount of performance degradation, and I think the only way to really solve it is going to be to throw a lot of money into hardware.
I found the latest upgrade also making some odd UX decisions. Content warnings got a weird new styling and it's not clear anymore how to hide images separately from hiding the text.
Are the mastodons okay?
There are good things too, don't get me wrong, like grouping notifications instead of getting a notification flood on a popular toot. That's nice. But what's up with perf regressions and (in my opinion) UX regressions?