Open-source Twitter-scale Mastodon implementation in 10k LOC | Hacker News Reader

Open-source Twitter-scale Mastodon implementation in 10k LOC

github.com

33 pointsnathanmarz19 comments

They can just put the rest of the code in this proprietary "Rama" lib that apparently doesn't count, and make it one line of code: Rama.StartMastodon()

That way you can call it "public domain" too, I doubt a single line is copyrightable. With all the real code hidden. Awesome.

Rama is a general purpose system. You can use it to build financial applications, note-taking apps, analytics, search systems, or pretty much anything else. The acceleration that Rama provides to building Mastodon at Twitter-scale applies similarly to other complex applications at scale.

Fediverse is a general-purpose platform.

My point is, they're (you're?) drawing an arbitrary line and counting a precise number. So far Rama has only been used for this one purpose, this is the kickoff marketing example, calling the system "general purpose" is a giant stretch.

Mastodon isn't one use case. It's a collection of radically different use cases. The way personalized follow suggestions works is completely different than how home timelines work, which is completely different than how tracking followers works, which is completely different than how search works, and so on. I can list dozens of features that work completely differently from one another. Yet they're all expressed elegantly, scalably, and with high-performance on the exact same set of abstractions.

If you peruse our blog post and documentation, you can see for yourself how general purpose Rama is. You can even try it for yourself with the build we released.

https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the... https://redplanetlabs.com/docs/~/index.html https://redplanetlabs.com/docs/~/downloads-maven-local-dev.h...

What mastodon specific functionality do you think is missing here that they've hidden away?

By this definition of open source, an open source windows or mac app would be a contradiction in terms, since any such app would rely on libraries and os calls that aren't open source.

An "open-source 3D renderer in just 1,000 lines of code" that just calls DirectX, I would call out, yes. Especially if it was made and released by the people who make DirectX, at the same time (or days before!) they release DirectX under a proprietary license.

Other apps, probably not.

Discussion on the blog post for this implementation: https://news.ycombinator.com/item?id=37137110

As best I can tell the Rama-awesomesauce is down in the backend modules, e.g. https://github.com/redplanetlabs/twitter-scale-mastodon/blob...

and the rest of the commentary I'll keep to my damn self

I'm trying to understand this as well based on the code you linked to and other comments - so the secret sauce of this implementation is the Rama library / framework / database? And that's closed source?

Also looks like Rama itself is in beta right now? When will it be stable? Will it remain closed source at that point?

No, sorry for any confusion: since this codebase is designed to be a showcase of how awesome programming in Rama is but it has (as the author claimed) 10k lines, then I would imagine an interested reader would want to know how does one use Rama's API to do Mastodon stuff, and skip past the Spring plumbing bits that are not Rama-ized (AFAIK)

Thus, I found some of the .whatever("*awesome") looking code down in the guts of the backend module but since there is no other obvious tour of how the Rama works in this codebase, I thought I would draw the reader's attention to the interesting bits

As for your other product related questions, that'd be much better as a top-level comment so nathanmarz can reply to them

The Spring stuff and Rama stuff is split between the api/ and backend/ folders. The backend/ folder is the more interesting part to look at.

The best resource for learning about Rama is its documentation: https://redplanetlabs.com/docs/~/index.html

Our blog post about our Mastodon implementation is a deep dive into how it works, and how the implementation is utilizing Rama to do some really powerful things: https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the...

From what I can tell, this is a production-ready differential dataflow solution with basically, a DSL. Is Java supposed to be the selling point as in, "oh it's just Java so you can run it on your Java things?" Why not just make it some GRPC-thing with SDKs or even its own well-documented DSL?

It can be programmed with any JVM language, such as Clojure, Scala, or Kotlin. The JVM is a well-understood and widely used platform, which is why Rama is built on top of it.

Being a library in a general purpose language as opposed to a custom language means you can do things like: generate dataflow code dynamically, easily use any JVM library, easily integrate with other external systems (e.g. databases) using their Java client, and easily mix regular Java code with Rama dataflow code. All of these are extremely important and should never be sacrificed. Losing any one of these massively increases complexity. Our Mastodon implementation relies heavily on being able to do these things.

Yes, I get that, and it makes sense. But given that this is a potentially very powerful, (but closed) platform, I see the JVM part as an implementation detail. If I'm going to be getting my whole team writing pstates and things, now I also have to tell them they need to write their code for the JVM. Temporal.io made a platform and wrote it in Go or whatever they use, but they released with a few SDKs for different languages. Would this just not be possible with Rama? like, does a serializing interface like grpc make Rama somehow not worth it?

One way to think of Rama is as a "programmable datastore", where your application logic goes into and is colocated with your data storage. So your logic has to be written to run on the same platform on which Rama is built, which is the JVM.

Is this Rama library for which this is a demo going to be Open Source? What license?

This keeps launching but there is basically not much to see. I guess: why should we care?

Reply on news.ycombinator.com