Ask HN: How about a Reddit alternative where every community is a Git repo? | Hacker News Reader

Ask HN: How about a Reddit alternative where every community is a Git repo?

3 pointsMrThoughtful9 comments

When I look at all the Reddit alternatives that come up, the problem seems to be that they all intermingle data structure, data storage, moderation, federation and frontend.

How about we build a Reddit alternative which just defines the data structure?

If each community is simply a directory, we could run such a thing as a git repo. Say we run the "aww" community with cute animal photos. Then the repo cold look like this:

    2023-06-12/
        Look at this funny cat/
            content
            comments/
                very funny
                i like her eyes
        Our dog climbing stairs for the first time/
            content
            comments/
                go dog go!

    2023-06-13
        A parrot singing a poem/
            content
            comments/
                but can he sing a book?

Every line with a / is a directory, every line without a / is a text file.

This already would:

    - Let everybody browse the content
      For example via github, gitlab, codeberg etc.

    - Let developers write content
      Fork the repo, add a file, send a pull request

Then we could take it from there and build frontends on top of it.

What does everybody think?

I love this idea a lot. It has huge promise.

There's a constellation of ideas around returning to the file system & destructing data, using flatter data & the file system to file data that have so much promise and hope. Using git as a next layer to syndicate.

9p is a well known pointer into this universe. There's less discussion/clarity around how an app or how a network ought become file system oriented, about what flat data architectures would be, but these humble ideas you have here I think show exceeding promise, are clear & reasonable a start that any dev of any language can see & grok (which is huge) & follow a long with. The sense is self evident.

One other humble pointer I might drop might be Hull: the alternative to shell that I'll never have time to implement. https://250bpm.com/blog:153/

I love how such a community as these could form & if things take a turn badly, folks can fork. They can abandon one host & shift to another instance to bave discussions on.

You might look at gitit, which is a git-based wiki. For a discussion forum with a lot of dynamic views though, I don't think git really brings advantages. You have to support different sorting and styling options, block lists, and so on. You end up wanting something more like a document database, or just SQL with some in memory data structures to track the state of busy pages.

When people post illegal content how do you quickly scrub it from the git history? Would there be risk of corrupting the repo if many moderators are simultaneously scrubbing contents? Would this cause some type of lock contention or perhaps CPU load issues? I ask because I have ran into corruption issues with git enough times that I always use rsnapshot to create file/inode level snapshots of them. Would you use different branches or repos per sub-forum?

I think this would need to be load tested and I suspect it would not scale as well as using a some form of a database for the metadata that just links to file storage.

Good question.

The first step would be to just write 'censored' into the 'content' file and commit. That would remove it from the current view.

Does it have to be removed from history?

If the history is not public, probably not.

Otherwise.. hmm.. the normal way to rewrite history would probably be too expensive. And would not play well with users who already have a fork of the repo.

How about locating the file in the .git/objects/ directory and truncate it, or write 'censored' into the file? That would be a very cheap operation.

Would that mess up anything?

If the history is not public, probably not.

I suspect this could be risky. If someone reports that illegal content was uploaded and it is known the site operates from git then it would be known that the contents would still be in git. The reason I mention this is that people who would view this site as their competitor game the legal system by uploading illegal material via proxies and compromised machines and automatically reporting it to authorities, server providers and dns registrars to get the nodes, domains and admins cancelled. This is how many small sites have been shut down and why many small forums have gone private. The competitors are anyone with a vested financial interest in the existing platforms. I think I would want all traces of the content purged in a timely manor to mitigate this risk if I were running the site.

Another tricky part of this is how git deals with binary blobs. The repos are going to get massive rather fast. I suspect truncating the binaries without making git aware may lead to errors. Even if they can be ignored it may get harder with time to troubleshoot the repo. I suppose you could make a prototype and load test it to see what happens when you upload a few TB of files and start truncating some of them.

I think one can also run a git repo without history. That would solve the "dirty stuff in the history" problem.

I don't think binary blobs should be included at the beginning. The "content" file should be just a text file.

In my opinion by implementing git without history and linking to binaries is just a reinvention of phpBB [1] which already has powerful moderation capabilities, a very advanced permissions systems and file storage that has been battle hardened for a couple decades. The UI can be made to look modern using modular styles and by uploading modern emoji which phpBB already supports. The most work would be working on a style to make it look modern.

phpBB can use Postgres or MySQL for the metadata and it links to file storage for attachments. There is also a very advanced permissions system with limits per file type. If the site grew big enough the file storage could be scaled with Ceph.

The only thing they leave out is some tuning/hardening of php.ini disabling unused classes and functions [2] and implementing good outbound firewall rules to mitigate most 0-day's. Those are fairly straight forward to implement.

[Edit] FWIW I tried vouching your submission that was flagged but then it was re-flagged.

[1] - https://www.phpbb.com/

[2] - https://www.cyberciti.biz/faq/linux-unix-apache-lighttpd-php...

Are they good questions though? It feels like github/gitlab/gitea solve 90% of these questions at scale fine.

Personally these questions came off as a rude routine to me, one that prefigure whenever anyone suggests networked social systems. Healthy skepticism is good, just but people who spread disbelief dont rank high in my estimation.

As for rewriting history, maybe use a month by month git repo, so if old content needs to be rebased the blast zone is contained. Job done. Then git submodules to collect each month.

It feels like github/gitlab/gitea solve 90% of these questions at scale fine.

Used as a social platform vs code repository? Surely Github have enterprise class storage and networking so if the question is around hosting the platform on Github then maybe that works at a technical level assuming one is willing to take the risk of Microsoft cancelling their social platform when someone complains about a comment and Microsoft don't want whatever divisive content is on their platform.

If self hosting I would be surprised if a single server or single VM could scale git to millions of people making comments, tens of thousands of comments being edited and thousands of comments being purged. My experience with managing large git servers is that it gets exponentially slower with mass manipulation of the repositories and that was on fairly beefy servers and that was just Jira, Fisheye, Crucible and Confluence and less than 10k developers, not millions of individuals. That is unless by self hosting we are talking about using automation to spin up many instances and even then it is not clear to me that git can scale horizontally like that. Even then that is not really self hosting, that is getting VC funding and starting a new company which draws even more attention and requires even more staffing to manage the content.

I think there would need to be some simulated load testing that attempts to replicate real world social platform behavior to see where git starts to fall over otherwise we would just be speculating in circles. I suspect if this were attempted there would be multiple layers of caching nodes not doing git and trying to minimize or optimize edits and purges to the storage and that may not even be enough.

Reply on news.ycombinator.com