I Wrote an Activitypub Server in OCaml: Lessons Learnt, Weekends Lost | Hacker News Reader

I Wrote an Activitypub Server in OCaml: Lessons Learnt, Weekends Lost

gopiandcode.uk

154 pointsgopiandcode108 comments

The author makes the basic mistake of most of the people implementing ActivityPub services: they want to map the logic of an existing type of web application and contort existing domain objects into encoding/decoding to an "impractically large number" of options. That happens because they want two things in one: a server and a client.

The ActivityPub specification needs to be read with a goal similar to an email server in mind. It should do one thing: receive JSON-LD objects in inbox, process them according to the specification, and(maybe) store them on disk.

The idea of "users", "friends", "posts", "feeds" etc, are concepts that belong to the clients on top of this server, not in the server itself.

This separation between clients and server will also allow better interop/graceful degradation of object types that the client/server don't specifically understand.

OK, but for someone who wants to build a useful tool that does what the author wants, "interacting with the Fediverse", such as federating with Mastodon, how useful is doing that one thing?

It depends on your goal. If your server is just a tool you use, you can ignore lot of concepts. There is no local timeline, there are no users, all follows belong to a single user, etc.

I can't find the link but a while back there was a post on the front page about how to get a findable, read only ActivityPub profile by just uploading some static JSON files. Not exactly a Twitter competitor, but you don't need much to start exchanging messages.

I believe you're looking for this: https://blog.joinmastodon.org/2018/06/how-to-implement-a-bas...

I did that myself. It's quite a distance from passively accepting requests to interacting with the Fediverse.

If you want to create one just for yourself, sure. If you want to create something for the rest of the world, probably not very much.

I get the "scratch your own itch" mentality, but not if you kneecap all efforts that try to build on top of it. :D

I think this is the wrong way of looking at it. If you're doing the whole stack, sure you will end up implementing quite a few things.

But consider that you can write a generic ActivityStreams server without supporting any of the ActivityPub activities. Now you have a generic platform to build on.

Tack on a tiny bit of support for e.g. addressing etc. as found in ActivityPub and you have what you need for federation.

With that generic platform, doing what you're suggesting is a matter of implementing a handful of Activities that mutates Objects and Collections.

What the author did is the equivalent of implementing a mailing-list manager by first writing a mail server from scratch instead of just writing the bits managing the list and sends, because he didn't have that lower level layer to build on.

There is indeed a lot of missing tooling to work with ActivityStreams/ActivityPub, that makes it painful now, and unfortunately a lot of ActivityPub implementers takes the same tack as the author and builds one big monolith instead of first building that lower layer.

(My only knowledge of activitypub comes from reading this article.)

To receive JSON-LD messages don't you need to send follow requests? And to do that don't you need to deal with the fact the spec is too complicated and most servers implement inconsistent parts of it?

To receive JSON-LD messages, someone needs to send them to you. Sending follow requests is perhaps the easiest way to do that, but those follow requests do not need to be initiated by the same code that hosts the inbox.

The point is there are several potentially independent layers and modules there: The message pump itself at least can be implemented separately from the decoding of individual message types, and separate from managing followers and following, the same way e.g. a mail server knows nothing about how to follow mailing lists, or decoding email messages past the header.

That sounds like a mess.

Reading through the other comments here it seems that the spec is in fact a mess…

Many people find whining about messes is more fun than doing anything useful.

It might sound like a mess if you see ActivityPub in terms of HTTP request, but ActivityPub is a protocol built on message passing, and what is a mess is trying to co-mingle the transport layer for the messages with processing those messages.

I'm not saying it doesn't have some warts and things that could do with being specified more clearly, but if you ignore a central aspect of the protocol and try to treat it as something it is not it will seem a whole lot messier than it is.

> It might sound like a mess if you see ActivityPub in terms of HTTP request, but ActivityPub is a protocol built on message passing, and what is a mess is trying to co-mingle the transport layer for the messages with processing those messages.

You are defending it strongly but the spec is indeed written like that. It's not even obviously message-oriented when you read it and the notion of following and managing followers is definitely put at the same level that transmitting new information regarding activities.

Don't get me wrong it can definitely be implemented but it's definitely not a good protocol nor is it well specified.

To me it was blatantly obvious that it is message-oriented when I read it given the vocabulary used. It could perhaps spell it out for people not used to it, but the use of e.g. "pub", "inbox", "outbox" and the use of addressing, including even "cc" fields in the spec should be enough for anyone used to writing message pump/queue based systems to see it as a pub/sub message based system. That it also talks of "actors" further underlines this, as the actor model is generally built on message passing.

As I said, it has warts and it'd be nice if the two layers specified had been separated out more clearly so it was more accessible to less experienced readers, but if a reader that don't see the message based foundation of the spec, it does speaks as much to the readers unfamiliarity with message passing systems as the spec.

Deeply disagree.

The spec first definition paragraph is about "Objects". There is mention of object retrieval long before message passing. The concept of messages is introduced in relation to Actors inbox and outbox. It’s not written clearly as being a message passing system and I am not sure it was conceptualised this way. Only from the server to server part you can start infering that’s what it’s supposed to be.

Well, then I guess it's not clear enough for you. For me it is clear as day. The Actor model is inherently focused on concurrently processing messages acting on objects, and the very section you refer to when it talks of objects shows an example mentioning actors and it's followed by a section on Actors.

The inbox/outbox terminology is further inherently tied to async message passing with a message pump/queue, and so on.

You may not be sure it was conceptualised this way, but it really is irrelevant whether or not it was conceptualised this way because the spec-as-written describes an architecture that to someone with experience in the field is most reasonably interpreted that way as it is inherent to the vocabulary used, and where interpreting it that way creates a significantly more cohesive architecture. You're free to interpret it otherwise, but it's now been pointed out to you that treating it this way simplifies the understanding of it - if you choose to ignore that advice, then so be it.

I might be sympathetic to the argument that it is may not be spelled out sufficiently clearly to someone without experience in this area or unfamiliar with the actor model, but there's a question of to what extent a w3c working group should feel a need to assume readers are not familiar with these kinds of models..

Strangely I think we are fondamentaly in agreement. I think the spec is poorly written for the reasons you quote. I am not surprised to see people struggling with the implementation. It's not a good spec.

So we have just the next question: If it's message passing, why was it build on top of HTTP? Wouldn't a RPC-like protocol as base be much more adequate?

This would just strengthen the suspicion that this protocol is badly designed, to be honest.

Because HTTP is ubiquitous and passes through every firewall around, and has extensive tooling. You need a really good reason to pick a different choice if you're going to expose your endpoints to the public internet today.

Do you know of a small sample project that does this as an example?

There are no "small sample" projects as far as I know. But if you look in my profile (or other comments in this thread) I did develop a server which only does ActivityPub, client to server and server to server.

This comment raised a whole bunch of red flags for me.

Fist and foremost: Saying that something is like an email server translates for me into "this is an under- and over-specified swamp at the same time, full of quirks, and actually not implementable in any reasonable way". Because that's what email is. I almost can't think of a greater horror than writing an email server from scratch…

I don't know enough about ActivityPub to judge whether it's really like email. I would strongly hope it isn't, as otherwise it would be a tech you should probably better never touch as a developer.

The next thing is: If an ActivityPub server only receives and sends some opaque BLOBs what's the whole point of it?

But when it's not about opaque BLOBs you need to map the structures in the spec to proper types in a statically typed languages as you can't manipulate them otherwise in any meaningful way. If it's not possible to do that because the spec is vague and/or there is no coherent data model behind it that would be just another reason to not touch this tech. Nobody needs the next underspecified, stringly-typed "email".

I really hope I'm reading this wrong!

The email comparison helps people to understand the directional way ActivityPub works, I don't know enough about email (whichever of SMTP or IMAP/POP3/samd you consider that to be) to make a comparison at protocol level.

> If [...]receives and sends some opaque BLOBs what's the whole point of it?

There are some rules about how to have side effects for said blobs. Some of the blobs themselves have side effects. That's mostly what ActivityPub is: rules about how to distribute the blobs in the federated context, rules to what to do with the blobs when they reach your servers (when coming from other servers, or directly from clients).

The vocabulary that ActivityPub is based upon, is another whole specification, called ActivityStreams, and which didn't originate in the W3C group. This vocabulary has three (*main) types of objects: Activities - which provide the backbone of ActivityPub (Like, Follow, Create, Update), Actors - basically different types of users (these are the entities that operate the activities) and, Objects - whatever the Activities operate on.

> If an ActivityPub server only receives and sends some opaque BLOBs what's the whole point of it?

There's still a difference between "try to black-box the incoming data as much as possible" and "treat the incoming data as opaque BLOBs and assume". The data is mostly JSON-LD which is a far cry from "binary large objects". It is always going to be "semi-transparent" as it will always be JSON. Whether or not you like the "-LD" extensions to JSON (they are heavy, they do have a lot of RDF baggage you may not desire), they give you a bunch of guaranteed "baseline schema" for the JSON objects that you can use for static typing that might be "good enough" for a lot of "meaningful manipulations" (such as following links to pick up related objects; LD => linking data) and that is all easily transparent.

A lot of the schemas beyond "LD" in ActivityPub are client/application-specific beyond most of the JSON-LD basics and should be easy to treat as a black box unless doing client/application-specific tasks. That's not necessarily "stringly typed", it's kind of a classic "serialization onion": The server at best needs to know that it is JSON and it may have JSON-LD metadata for relevant related linked objects (and a few other metadata fields common to "introspection", similar to "headers"). The client can dig deeper and know it is not just "any" JSON object but a more specific schema for a given class of thing the client cares about.

Thank you for articulating this very well, I was getting a bit frustrated at OPs contrarianism. :)

To be honest, this sounds indeed quite like the mess that email is.

If the server isn't just a "dumb 'BLOB' storage" it will need to handle application logic (sooner or later, as this is actually what servers are for)…

But given that the application logic seems to be mostly unspecified, kind of wild west, where every client application can do whatever it thinks it's users like, this will unavoidably end in all the problems you have with email, where the server needs to know about all the specific details, quirks, and idiosyncrasies of every client ever built.

The whole concept reads like an implementation of "'Postel's Law' fallacy".

The balance of client/server shared knowledge will always be a hard balance to strike in any protocol ever invented by people.

At least in contrast to email there is a stronger/higher baseline: HTTPS (primarily), transferring primarily one MIME type (application/json) and one "document" format (JSON-LD). The "headers" metadata and "content" are in the same parser format (JSON). That combination alone is far narrower than email protocols (which needs to support nearly any MIME type at any point in any envelope, the headers and content of envelopes have always been different parsers and much of the MIME parsers were bolted in after-the-fact making things worse).

> application logic seems to be mostly unspecified, kind of wild west

Other comments in this thread point to the larger "stack" of specifications beyond ActivityPub. ActivityPub itself is just one "layer" in the "stack".

That stack does rely on JSON-LD itself very heavily and many of the clients are in languages where JSON-LD tooling alone is "well-typed" without much in the way of further schemata. I can appreciate where in some static-typed language backgrounds most everything defined as stacks of JSON schemas starts to feel "stringly typed" and idiosyncratic. I don't agree with that sentiment and think JSON in general is far from the worst of "stringly typed", but I appreciate where that sentiment comes from here even if I disagree with it.

I think the surface area overall is a lot narrower than email and many parts of the "stack" are better defined, especially for specific client types. I think there will be some in-theory avoidable problems that resemble the idiosyncrasies with email, but I think in practice they still have a smaller "blast radius" than any similar problems in email.

Thanks! This sounds now a bit nicer.

At least the "stringly-typing issue" seems to boil down to having proper JSON-LD machinery in your language of choice. (In strongly typed languages some code-gen likely.)

Whether a new protocol should really be HTTP-based remains still a question to me. But OK, it's fashionable right now, everybody is doing it, so I can understand why someone would chose to go down this route. HTTP/3 is even quite good (if only a little bit to complex; but that complexity is actually inherent so likely better when it's contained in one place).

I think we have to generally assume HTTPS is the "new Telnet" for protocol bootstrapping and may be so for some time to come. Everyone has HTTPS debugging tools of every level "already installed" (curl/httpie/web browsers/fiddler/etc) and are mostly already trained in them. Every language under the sun has strong HTTPS libraries. Meanwhile, HTTPS gives a nice baseline for things like security (TLS was designed for multiple protocols, but let's face it is best tested and deployed in HTTP and always has been), caching, compression, many quality-of-life and quality-of-service tools. It's so much easier to bootstrap protocols from HTTPS as a baseline than "from scratch like an Apple pie" just as HTTP and Email themselves bootstrapped from Telnet. In the post-TLS era, bootstrapping directly from Telnet doesn't seem like an easy bargain anymore and yeah there are definitely advantages to getting your TLS and a bunch of other useful features/services "for cheap as nearly free" all at once in something like HTTP/3 (even if that is inheriting possibly "a lot of complexity").

You don't seem to have much appreciation for how elegantly handled email is.

I for one have awe & respect for works like Dovecot, which can elegantly decompose a couple different ingress-email pieces of work into special-purpose daemons that each can be scaled & operated independently. The system is practically ancient at this point but a model of how powerful & amazing clearly separated concerns & loose coupled systems are.

You've been persistently negative & calling this all a mess. But it's unclear what you think is elegant & graceful, that does work well. And most of your accusations about things being bad and awful and messy and wild west and postel's law fallacy... none of them have refutable claims. None of them make a position that anyone can argue with. You just spread negativity around & insult thinks in vague terms. It's not very hackerly of you to make a bunch of mean-spirited assertions that arent even backed up enough for anyone else to go verify or test.

Cut the FUD man. Adjust your attitude some. Don't be Mr Rain. Not cool. Be cool.

It's both true than Dovecot is an elegantly designed piece of software and the protocoles surrounding email are a complete mess with decades of bolted on additions and a most used implementation - Gmail - which likes to do things in peculiar ways.

I think the complaints you see here are entirely legitimate. Anyone having been doing this job long enough knows that designing good protocoles is insanely hard and exemples of poorly designed ones in the commercial world abound. It would have been more surprising for the feediverse to get that entirely right than the reverse.

I say that while agreeing with the original comment which points out that a good implementation should solely focus on the specification and leave everything else to the clients.

If one were to implement it on top of a RDF database then it isn’t any different than a typical backend taking in SQL (or sparkQL) and spitting out json to the client.

People seem to want to fit a square peg in the round hole and then you have servers bucking over a handful of users.

I spent a day or two looking into this when the hype machine was going strong and it’s really not all that bad (with the right abstractions, I suppose).

The application logic is not mostly unspecified, but implementers seriously need to start actually reading the whole thing, and that includes reading the ActivityStreams specs and JSON-LD spec, since it seems like a whole lot of the objections comes from ignoring that there are several more documents to read other than just ActivityPub spec itself even though these documents contain explicit references.

E-mails are not opaque blobs, and neither are ActivityPub messages. The point is that at it's lowest layer an implementation should care about receiving messages addressed to one or more Collections. That's it. It makes implementing a functioning ActivityPub implementation a lot easier.

The next layer up then specifies some rules for how to process those messages: Like on an e-mail server, if a message is sent to your "local" server intended for onwards delivery, the server must forward it on. Otherwise it is added to an OrderedCollection - effectively a mailbox.

The spec then sets out a structure for giving the messages an Activity type that determines further fields, and for some of these activities there are rules specifying how the relevant Actor's should act when those activities / messages are processed by them.

You can decide to do that synchronously when receiving the message. Sometimes that may be fine. But you can also strictly layer the implementation and deliver to a collection first and then asyncronously have workers process those messages. What you in either case ought to do for your own sanity is to at least logically separate the low level message pump (inbox/outbox) from the processing of activities.

For starters, doing this separation cleanly makes writing a scaleable implementation far easier.

> you need to map the structures in the spec to proper types in a statically typed languages as you can't manipulate them otherwise in any meaningful way

This is just not true. You can handle dynamic structures in statically typed languages just fine. It is in any case irrelevant, as ActivityStreams (which ActivityPub is based on) defines a typed vocabulary [1]. An implementation can choose to dynamically process extensions or it can choose to statically type the activities it understands and treat the rest as mostly opaque blobs other than the envelope/addressing -- this is exactly why it's beneficial to apply the layering as suggested with the comparison to e-mail and decouple the message pump from the processing of activities.

[1] https://www.w3.org/TR/activitystreams-vocabulary/

> JSON-LD

https://json-ld.org/ for anybody else not super familiar

Saw some comments on the protocol being fluffy and typical implementations resource hungry. This is an interesting guy to follow:

https://universeodon.com/@supernovae

He's the admin of universeodon, a mastodon instance with 13K MAU. He recently shared that in a month's time, 3TB of text was transferred just in ActivityPub events. Images a multiple of it. I don't know what the bill is, but I was pretty shocked by the stats...for "just" 13K users.

And the cruel thing is that it still doesn't work properly. Likes/boosts and replies do not properly synchronize.

But people are saying that the Fediverse could replace Twitter and Facebook and Tiktok and Instagram and what not, don't they?

How much hardware would you need for 100 million MAUs? (And that's just a fraction of the current social media users.)

If it really "scales" like indicated in the parent post this tech will never provide any alternative to centralized social media sites just for technical reasons no matter what people want or do.

Maybe someone experienced in effective distributed systems should start to design an alternative.

Otherwise there won't be any viable alternative to the commercial silos no matter how bad people would like one.

Mastodon absolutely does not scale that far and this should be considered fact. Far smaller instance owners sometimes complain about bills of hundreds of USD per month.

So the only way to scale up "indefinitely" is by having many small/medium-sized instances, but not really. In a 100M+ network, instances will suffer due to the wasteful nature of federation plus social media being append-only.

Costs will forever go up and this doesn't even mention the burden and liabilities of moderation.

Amidst all our hate for big social, we've forgotten about all the things they do very well. They are (financially) free. They are reliable and scale up without you noticing. You do not have to generally worry about your entire account and content being gone because some mod gave up. If you don't do anything funny, all your content is preserved, forever. Moderation works reasonably well, even if never perfect.

We've taken all that for granted. But it costs billions, an army of engineers, mods, legal, marketing, UX, top notch infrastructure to make it run and work this smoothly.

The idea that a bunch of enthusiasts can replicate this, is misplaced.

> If you don't do anything funny, all your content is preserved, forever.

Are you really not able to think of any dead social networks that deleted all of their content? Hell: Google alone has managed to delete at least two!

Obviously with forever I mean within the lifespan of the service, not literally forever as in a trillion years.

So yes, your 2008 Facebook post is still going to be there.

Yet, my 2011 Google+ post is not still there, so this seems like a crap shoot: your argument is inconsistent at best, but frankly feels more like you've bought into a lie.

> Mastodon absolutely does not scale that far

Maybe?

> and this should be considered fact.

I see absolutely nothing about ActivityPub that is inherently hard to scale. Adding more points of presence & smart fan-out seems like it could keep scaling indefinitely, from what I can see.

> Costs will forever go up

Let's simplify follower-ship and assume instead a simpler model of bi-directional friending. If you ahve M users each of which have N total friends, the naive fear is this: that costs will keep growing terrifyingly. But I expect it's actually more a sigmoid curve. As you grow into millions and especially many-millions of users, more and more you'll have your people following the same person - which reduces traffic - and more and more of your people followed by multiple people on another server - which reduces traffic. The actual growth curve here is more sigmoidal, with a big variability depending on the number of active instances fediverse instances out there.

> You do not have to generally worry about your entire account and content being gone because some mod gave up.

To get listed on joinmastadon you have to promise to give at least 3 months heads-up, during which time users have to be able to activate an account transfer to other systems. There was a case where mastodon.au nearly pulled the plug with a short notice, but someone else stepped up to take over the system.

The potential for this to be a huge problem is absolutely super real, but in practice, I have almost no real concern over this and think it should be broadly disregarded. The server I was on shut down, but the mod gave us almost a full year of continued service, and another year of read-only service during which we can transfer. Trying to drum up fear & doubt over the instability of this system seems irresponsible & premature, given how well things have gone.

> The idea that a bunch of enthusiasts can replicate this, is misplaced.

I agree that there are huge advantages & I think we absolutely have taken much for granted. Right now you're looking at this through shit-colored glasses though, and I think it's naive to wish so hard for this all to fail. It's convenient & easy to say it'll never work.

But there's so many strategies where we can start to turn the scale into something useful. If servers publish WebBundles of user's feeds, users can p2p distribute the signed http content among themselves, perhaps via WebTorrent. Rather than live in fear of our users and our scaling, we can rely on our users to help us tackle scaling. Yes we can, man. It's totally doable. Get off the ledge.

"and I think it's naive to wish so hard for this all to fail. It's convenient & easy to say it'll never work."

I do not wish for it to fail, I'm observing that it cannot succeed within the scope of our discussion, which was Mastodon as a "big social" replacement. We're talking 0.5-2B MAU.

> Costs will forever go up and this doesn't even mention the burden and liabilities of moderation.

Storage costs have historically dropped so fast that this just isn't an issue I'd worry about. I ran a mail provider with ~2m accounts around 2000. We had less aggregate storage than my laptop has on a single M.2 SSD now. With redundancy, the cost per GB for us at that time was around 10,000 times higher than it'd have been today. Our total processing power was lower than my laptop. Our bandwidth use was a tiny fraction of my home internet connection.

In other words: If it's an issue today, it soon won't be. Costs are low enough per user today to be viable, and they'll only drop.

> You do not have to generally worry about your entire account and content being gone because some mod gave up.

I do have to worry about what happens when I have to start again because engagement is cratering and I can't just migrate elsewhere or run my own, though (e.g. I get more engagement on Mastodon than on Twitter despite 100x as many followers on one of my Twitter accounts). Since I can (and did) choose to run my own instance, I don't need to worry about that again.

> The idea that a bunch of enthusiasts can replicate this, is misplaced.

This was the kind of argument used against open source 20-30 years ago. It's was just as ridiculous an argument then. This is magnitudes easier than what was achieved with open source, and it's also flawed because just as detractors of open source you're presuming that it will only be enthusiasts, and that e.g. nobody will start and operate commercial services for those who prefer that (some commercial hosters already exists for Mastodon for example).

I don't buy the "storage is cheap, bandwidth is cheap" argument.

The Fediverse is much like binary internet news. Something that is to this day not really operable for reasonable cost, especially not for small or even private providers.

A few hundred users make it necessary to shovel around TBs of data. For every involved server. This just doesn't scale, no matter how cheap bandwidth and storage will get in the next years.

You can not buy it all you want, but it's true. Consider that even if you were right that it takes only a few hundred users to require shoveling TB's of data (it's not, assuming we're talking monthly, unless you happen to be hosting e.g. George Takei and similar), the ~$5/month VPS I run my instance on can do that just fine at no extra cost. It's not necessary to pay insane AWS egress rates.

> For every involved server.

This is misleading. Not every server sees anywhere close to every post, because most servers are not "involved" with any given user, and that becomes increasingly true the larger the network becomes. Larger servers sees a large proportion but also receives only one copy of each post even though they're likely to have multiple recipients.

Realistically, the number we're looking at for the source server is about half the total network cost for that set of users.

> This just doesn't scale, no matter how cheap bandwidth and storage will get in the next years.

The cost per user of running a Fediverse instance is dropping, not rising, even as the network size has exploded. And as it is, the main cost of running e.g. Mastodon is that Mastodon, fittingly, is a huge, inefficient, resource-hungry beast. Not that federation is inherently costly.

My prediction is that the average cost per user for bandwidth, compute, and storage will be <$0.10/month within a couple of years. It's probably below that already if done efficiently enough at reasonable scale.

The bulk of the cost will be admin, moderation and ops, which scales linearly or sub-linearly with number of users.

> Realistically, the number we're looking at for the source server is about half the total network cost for that set of users.

Now think again what that actually means.

> My prediction is that the average cost per user for bandwidth, compute, and storage will be <$0.10/month within a couple of years.

So you think it's much more expensive at the moment, right?

But $0.10/month means already $100/month just for someone who has 1000 "friends". And that's not a big amount. Regular people on FB have often collected more "friends"…

A YouTube account with 10 000 followers is actually quite small. But that would be already 1 000 bucks! Most people on this planet don't earn so much money in a whole month. And this would be just the cost for some smallish hobby account…

So even when we take your numbers, it's extremely expensive! How is this going to scale beyond some privileged geeks in the western world?

> How much hardware would you need for 100 million MAUs?

Given an informed yardstick of $.02-.05 per MAU for a 10k instance, possibly $2-5M/month, though much depends on the profile of users per instance.

The support, moderation, legal, administration, and corporate costs would dwarf this cloud/hardware cost.

Would love to see how this compares to git.

Because they should have designed more compact and less talkative protocol. It was a similar problem with XMPP.

3TB for the text of 13K user sound crazy, you're right. But for the bill, strictly speaking about bandwidth, a 10mbps unmetered connection gets you roughly that. And 10mbps is pretty uncommon now because it's so low. So I'd expect the bandwidth bill to essentially be free (i.e. included)

Universeodon uses Fastly as a CDN. My masto admin experience is 80-90% CDN caching though obviously that’s images and static content.

Bandwidth is a function of the number of connected servers for remote followers. A popular person like George Takei makes a post and 15000 servers ask universeodon for the text. IMHO some sort of smarter fan out or message relay capability would help, much like email infrastructure.

Having said that, you can run 10k Mastodon MAU on a $200pm cloud resource budget, though double that adds headroom, a dynamically scalable architecture, a staging instance, elastic search and translation, more admin tooling, etc. Yes, some instances spend multiples of that per 10k but if you budget $400pm you’ll sleep well.

(Thing is, a 1k MAU would cost $100+ if you want to start with a scalable footprint).

Universedon had 70k MAU after the November surge and AIUI is still easily scalable to 100k+.

I don’t know about you, but a cloud running cost of $0.02-$0.10 per MAU sounds cheap to me. All large instances can cover it with donations. The “real” cost is moderation and administration labour.

> And the cruel thing is that it still doesn't work properly. Likes/boosts and replies do not properly synchronize.

It works exactly as designed. Personally I am fine that on someone else’s post I don’t need to see every reply across the world and fully synced like counts, but I can respect the POV that many users expect just that. This is not an activity pub limitation but rather a software choice by Mastodon. Many users, including @supernovae, push for functional changes around this.

A bit off-topic but the post title will probably attract relevant people.

What are the thoughts on OCaml on HN?

It replaced Python for everything longer than a couple hundred of lines long for me. Fast language, fast compile times, clean(-ish) syntax, strong typing system, good ecosystem, and now multicore support? Yes please!

I must be more nuanced, though: existing libraries in opam are generally very, very good (I really like cmdliner), but many things may be missing. There is no alternative to Django, for instance. No serious IDE, except emacs. The standard library was so lacking that there is at least an alternative. The situation improved, but there's still missing stuff compared to Python.

> There is no alternative to Django, for instance.

https://aantron.github.io/dream/, which is new and used by ocaml.org as well as OP

> No serious IDE, except emacs

and vim, and visual studio, and whatever else supports the LSP protocol via https://github.com/ocaml/ocaml-lsp

> The standard library was so lacking that there is at least an alternative.

While janestreet does have an publish their own stdlib, I personally try to stick to the stdlib whenever possible. Not to knock janestreet. I'm glad they're around and have contributed a bunch.

But overall I agree with you. It's been my favorite language to write in for years now. You can't just reach for off-the-shelf libraries for every little thing. Although the ones that do exist tend to be written halfway decently.

Dream is not a Django alternative. Django's powers come from probably one of (the best?) best ORMs in the industry, along with generated database schema migrations, and generated admin panels, to name a few. There's also the Django Rest Framework which makes putting together REST apis generated from your models super easy.

Quite possibly the best for everything up to moderately complex.

You'd want to enter sqlalchemy (python), DBIx::Class (perl) and arguably Sequel (ruby) into the competition for 'best' though (I'm sticking to dynamic languages here) - exposed power and ability to drop to SQL for only parts of a query vary substantially between options.

On a related note I've been keeping an eye out for a really good ORM in a typed language that is good with relationships without tons of boilerplate, I guess it's hard to do. Hibernate has way too many annotations for my liking, and I'm not into the code generation of JOOQ. I bet you could build it in Nim pretty easily with its compile time stuff.

Diesel might get there in a few years... :)

I know people are gonna judge the whole ORM thing, but writing raw SQL is nice for performance but not for getting something out quickly, or prototyping, when you have lots of relationships.

I stand by "an ORM should work as just a query generator and let you skip any object layer when you want to", "an ORM should be able to let you feed the object layer data from your own query" and "an ORM should let you mix and match programmatic query generation and literals to as granular an extent as possible" being table stakes if you have developers who actually like SQL.

There are a bunch of things I hate about the DBIx::Class stack (most of which were my poor decisions to begin with) but it gets those right. Example:

    $schema->resultset('Foo')
           ->where({ bar => $bar })
           ->update({
             baz => \'CASE quux WHEN 0 THEN baz - 1 ELSE baz + 1 END'
           });

would generate (the \'..' means 'reference to a scalar' in perl)

    UPDATE foo
    SET baz = CASE quux WHEN 0 THEN baz - 1 ELSE baz + 1 END
    WHERE bar => ?

and duplicate the relevant SQL generation guts across to a statically typed language wouldn't be horrible ... but every time I start trying to figure out how to get useful per-select-list types for query returns I end up tying myself in knots (easy enough for a simple-CRUD oriented ORM but once you're looking at expressions in SELECT that are query-specific it starts getting gnarly).

I think you may be right about Nim though and thank you for reminding me that I really need to get back to playing with that :)

(obligatory perl disclaimer - if you don't like perl, that's entirely fair, but in that case please steal the good parts of my/the rest of the DBIx::Class/SQL::Abstract team's ideas into your language of choice because developers who don't like perl deserve nice things too)

Oh, for sure on the SQL part. I do like being able to auto generate join tables and such, though!

Do you make GUIs in OCaml, and which libraries do or would you use?

And how about scientific computing (SciPy), deep learning (PyTorch etc.), or computational geometry (Shapely etc.)?

GUIs are a PITA like in most languages.

I think most people use something which binds to gtk (such as lablgtk) or Qt.

For scientific computing there is Owl, but I haven't used it personally.

Hmm, I'll stick with Python for now.

The ecosystem of libraries is just too good.

Perhaps if OCaml made it very easy to interoperate with Python, I could give it a chance.

There is pyml for interopt but I've never used it.

Lablgtk[1][2] for GUI, Owl[3][4] for scientific computing.

[1] https://garrigue.github.io/lablgtk/

[2] https://github.com/garrigue/lablgtk

[3] https://ocaml.xyz/

[4] https://github.com/owlbarn/owl

I've heard good things about OCaml in general.

But "no serious IDE, except emacs" is a non-starter imho, if it's true.

They should really invest in this. Otherwise the language won't attract any professional developers in the large.

Vscode works since there is an LSP server.

It is not true, as other commenters have said.

> No serious IDE, except emacs.

https://plugins.jetbrains.com/plugin/9440-reasonml (72k downloads)

https://plugins.jetbrains.com/plugin/18531-ocaml (2k downloads)

I'm not in the ocaml ecosystem enough to evaluate their quality, but anything on top of IJ is for sure a serious IDE

one of my favourite languages! not so much for its (excellent) technical qualities, but just as a matter of personal taste - it joined ruby and racket in a short list of languages that just feel nice to program in. (i suspect D would join that list too but despite being interested in it for a while i haven't yet had a compelling project to use it for.)

I haven't used OCaml much directly, but F# is a common enough tool in my toolbelt at this point. My experience of F# is that overall it's a good language family. The access to .NET's standard library (the BCL) and easy interop with C# are the biggest reasons F# is the tool I more often reach to as it already fits the ecosystem most of my other development is in, but I'd love to work more directly with OCaml should the need arise.

It's my favorite language by far!

Pros: type safe, GC, fast, (arguably) a simple and practical language if you have a functional mindset (much simpler and pragmatic than Haskell IMHO).

Cons: it's a niche language, so tooling/libraries/online help aren't on par with more mainstream languages. No canonical standard library (different codebases will use different standard libraries and even disagree on pervasive functions such as List.map). Whenever the code uses monad (e.g. concurrency monad / error handling), I find the language loses its simplicity.

Maybe it's true of every languages but I'm disappointed by some OCaml codebases where often two extreme cohabit

1. people who don't know the language and don't write idiomatic code (like, refusing to write .mli, abusing imperative features)

2. OCaml experts who over-engineer things and want to use the latest features and make the code hard to read/maintain

In a professional settings, it can be hard to have these two populations coexisting, and people tend to be quite opinionated when it comes to such languages (love it or hate it -> it's often a source of struggle).

OCaml is a great language but I would probably choose F# if I had to pick a language for a new project because of libraries.

My question is this: if I was to try to hack up an ActivityPub server in my platform of choice, how would I know how compliant it is? Is there any compliance test suite to verify this?

"Try and load it up in a client app" seems suboptimal.

"load it up and see" attitude is part of what made parsing and renderings HTML so hairy, and compliance test suites helped.

There was a suite of tests, that sadly fell to bitrot. One of the developers in the community created a parallel application that could test implementations, but then this too ended up unmaintained[1].

[1] https://github.com/go-fed/testsuite

I found the post well written and informative. Though I am clueless about OCaml it feels as this would be useful for anybody working on a new server implementation in any language ecosystem as it highlights what needs to be done and potential bottlenecks.

As for the activitypub spec and the currently popular implementations it doesnt take long exposure to the fediverse to realise there are some rough edges and historical accidents (e.g mastodon being actually the defacto interpretation of the standard). Imho now that there is substantial more mindshare devoted to decentralized social it would be opportune to revisit these things and if needed revise before they get backed in.

Im looking forward to a solid ActivityPub server written in Go or Rust that can run on modest hardware/small resource Docker hosts.

Well, there is one already as the reference implementation for a suite of libraries I wrote. You can find it at https://github.com/go-ap/fedbox. (Contributions welcome)

Does it only support C2S as the API? Are there any clients which actually support C2S rather than the Mastodon API?

It does support server to server, but currently it does not play well with Mastodon due to its limited support of HTTP Signatures algorithms. I didn't get bothered enough by this yet to actually fix it on my side.

And there are a number of clients that work with this specific brand of client to server ActivityPub but I wrote all of them. The one that can be seen on the internet is a link aggregator similar to HN and (old) reddit, you can find a demo instance at https://brutalinks.tech.

> It does support server to server

Ah, sorry, I should have said "client API" rather than just API there.

ActivityPub has a section which deals with how clients and servers should communicate with each other (called Client to Server - C2S - in the spec). So it's the same vocabulary and operations with slightly different side effects, but most servers don't implement it because it's not "specified enough". That's why developers generally just use the Mastodon API.

> [client apis / c2s / mastodon]

Yep, I know. That's why I was asking if there was any support other than C2S because that does somewhat limit your server to "basically zero client support" which reduces its usefulness as an example somewhat?

I think there is (was?) an attempt to rewrite Mastodon into Rust but I haven't heard much about it.

A single user Mastodon instance takes an unreasonable amount of resources. I don't know if it's just because of Ruby (Gitlab has the same problem, so it might just be) or because everyone is wasting money on expensive servers, but an RSS feed on steroid shouldn't take this much RAM.

Mastodon itself is designed for "flagship scale" (given lead developers run mastodon.social and mastodon.online, two of the biggest instances and the most "dogfooding" two instances) so it bundles an entire cluster of services: background processors (sidekiq), caches (redis, I think?), database server (postgres), optional ElastiCache, and more. I don't know how much Ruby itself accounts for expensive overhead, but just running all of those other things on a single server vertically for a single user instance is a massive, expensive overhead. (It's clearly built for horizontal scale where your background services and caches and database servers may all be different clusters of VMs/servers over vertical stack efficiency when "scaled down" from the "natural" "mastodon.social scale" that Mastodon is most optimized for.)

It's an interesting optimization problem reminder that scaling factors are different for different needs and not everything scales cleanly to every use case. A single user instance should be able to use a much smaller vertical stack, but scaling down from a wide horizontal stack is not necessarily the best or cheapest place to start when building something like that.

(There are some interesting projects I've seen to build single user instances with much less overhead, shorter vertical stacks. I'm curious to see where those efforts go. In my own usage of Mastodon my "single user" instance gets the benefits of the horizontal scaling Mastodon was built for because my hosting provider does a bunch of work to make sure that they take advantage of that economy of scale to host many small instances for cheaper than trying to run small instances in one-off VMs.)

This is the same problem that plagued mail servers. It takes so many different components which all have their own configurations to optimize (and memory/cpu footprints) that it ends up being too complex for most people to actually run an instance.

I'm also completely positive that a Go or Rust version on a $5 VPS with an in-memory database model (RocksDB, LevelDB, BadgerDB, etc..) could easily handle hundreds of thousands of users and gigs of content each day. It seems like it would be easy to expand that to a multi-node key-value store like TiDB, CockroachDB, etc.. if you needed to grow beyond a single host.

https://github.com/rustodon/rustodon#readme which has an awesome name but you're correct it appears that specific repo stalled out. I didn't check on the 41 forks of it

There are several websites out there which hope to list many ActivityPub servers (and clients) in many (programming) languages, and other implemtnation aspects...Like, here's an oldie but goodie website: https://fediverse.party/en/miscellaneous/ ...There are other wbsites of course.

Just select your desired lang. and review! Now, of course, it might be early days for some languages (e.g. for Rust, etc.)...But, one reason why some languages are used over others...is due to ease of deploying on VPCs and VPC-like hosts (...historically the land that php ruled ;-)

Enjoy, and I hope you find what you're looking for!

I've had a go at doing it in Go and the ActivityPub spec is so loosely defined that it's just a real challenge if you intend to actually unmarshal the JSON you receive

It's not completely impossible but you have to be okay with discarding a lot of unknown options or essentially reverse engineering the objects used by the servers you are federated with

That's not to say it's impossible, I was able to crawl the network successfully, but it hints at the reason that Mastodon and Pleroma use dynamic languages

I'd be very interested to see a flexible/complete AP implementation in any statically typed language

Fwiw WriteFreely is implemented in Go with go-fed but -- correct me if I'm wrong -- that library seemed more limited to me than what Pleroma and Mastodon support

> I'd be very interested to see a flexible/complete AP implementation in any statically typed language

Try Honk[1] or GotoSocial[2]?

[1] https://humungus.tedunangst.com/r/honk [2] https://github.com/superseriousbusiness/gotosocial

Neither is flexible, nor strives for completion. They are both implementations that try to map the ActivityPub vocabulary on an existing web-application domain.

They are not ActivityPub servers, but web-apps that use the ActivityPub vocabulary to federate, which is what I meant in the grandparent post when I mentioned the classic mistake of ActivityPub implementers. :D

Here's the implementation described in OP: https://github.com/Gopiandcode/ocamlot

OCaml is a statically-typed language. It falls somewhere between Go and Haskell on the spectrum of type 'strength'.

I'm surprised you didn't find my library because I managed to create a statically typed vocabulary library for Go that maps the specification verbatim: https://pkg.go.dev/github.com/go-ap/activitypub#Object

It wasn't easy indeed, and it locked me out of some options to support execution time vocabulary extensions, but hey, it works and it's relatively easy to use.

I'm surprised I didn't find it, too! My google-fu must be getting rusty. Thanks for the link, I'll have to have a deep dive when I get the time :)

I hope you get to it. The library itself contains more than just the vocabulary part and I would be glad of more eyeballs on the problem. :D

> Im looking forward to a solid ActivityPub server written in Go or Rust that can run on modest hardware/small resource

The "Lightweight" GoLang ActivityPub server is GoToSocial https://github.com/superseriousbusiness/gotosocial

The better-known lightweight servers are Pleroma and fork Akkoma, written in Elixir https://akkoma.dev/AkkomaGang/akkoma/

Some of this info I got via: https://social.treehouse.systems/@ariadne/110226729543740723

There's also Honk[1] which is written in Go but has slightly wacky source and doesn't support the Mastodon API (but does provide an inbuilt web UI.)

[1] https://humungus.tedunangst.com/r/honk

Ocaml code compiles to native binaries, just like Go/Rust.

Why specifically those languages? Others can also target modest hardware/small resource Docker hosts.

Sure, Zig, Nim, D, Erlang, etc.. could also do this, but Go and Rust are both big enough and just about as fast and low memory as anything that is available.

Java can be faster than Go, but not by much and I've always seen it to use 5-10x the memory.

Scripting languages like Typescript, Python, PHP and Ruby can't hold a candle to the speed of Rust and Go while also using significantly more memory. They also don't natively support multiple cores / threads.

Rust and Go represent the most approachable middle ground on all accounts of familiarity, performance (allocs and calculation speed), and large communities with libraries covering whatever you could want.

I mean, nothing beats C (yet), but that isn't actually what most devs are writing web services in. Rust overs better guarantees, but even that is too complex for the masses so Go is probably the next largest community that still offers great performance.

Java, PHP, Python & Node.js have massive market share, yet none of them are as ideal for the job.

Pleroma (written in Elixir) is one of the lighter, Mastodon-compatible AP servers available. I recently read a post (a toot actually, but I hate that term) by a Mastodon administrator observing that that Pleroma is often a common thread to problematic Fediverse instances because it can run on cheap VPS boxes on throwaway domains. Spammers/griefers can cause a lot of moderation problems for the same amount as a Twitter Blue subscription.

It is dubious endorsement, but I think it shows how much more efficient Pleroma is than other popular, easy-to-use-OOTB AP servers: 9 out of 10 price-conscious griefers use and endorse Pleroma

There's also LitePub, though development seems stalled (?)

link for reference: https://litepub.social/

Was it developed at all? I'm not seeing any business logic in the repo: https://hacktivis.me/git/litepub.social/files.html

It was and has some pros compared to AP...

Reply on news.ycombinator.com