Open Source federated link aggregator in rust | Hacker News Reader

Open Source federated link aggregator in rust

github.com

106 pointsmodinfo37 comments

Fun fact, earliy versions of lemmy were using my open source image hosting and postprocesing cdn (with S3,NFS,ftp.. storage) PictShare[1] until the devs raised an issue[2] about a huge memory leak when resizing this[3] 10mb PNG file which caused the memory to raise over 1.5 gigabytes and basically crashed the backend.

So they ditched PictShare and wrote their own tool[4] only to realize it has the exact same problem because of the way some PNG files are stored basically any resize operation causes huge RAM spikes no matter which software or implementation is used.

[1] https://github.com/HaschekSolutions/pictshare

[2] https://github.com/HaschekSolutions/pictshare/issues/120

[3] https://commons.wikimedia.org/wiki/File:Pizigani_1367_Chart_...

[4] https://github.com/LemmyNet/lemmy/pull/807

That image is already 300MB of uncompressed pixel data. That's a hard floor on the memory usage unless you do something crazy like build a streaming image processing pipeline.

With ImageMagick convert, it does take up 1.5 gigabyte of RAM and take quite about 5 seconds. With ffmpeg, it only takes up 150MB and finishes about 10 times faster! I am really curious what makes swscaler from ffmpeg so memory efficient compared to other tools.

Also PictShare supports S3 compatible backends, wasn't cool to see lemmy drop that so frivolously.

I was just thinking about something of much smaller scope that this might be able to do as a feature.

Basically, you have a list of articles that you want to read, in Pocket or a similar service. And you have a collection of 'link aggregators' where you get topical news and/or insightful commentary.

By combining the two, you can boost the signal of something you'd bookmarked to read later that has been resubmitted for discussion even if it's only received minor upvotes. This then provides the impetus to read it.

This was partly inspired by the idea that breaking news/speculation is more addictive but less filling than content were people have had time to consider it.

You seem to be describing LinkLonk[0].

[0]: https://news.ycombinator.com/item?id=28405643

Congrats and thank you for your hard work.

Now for a contrarian comment: whenever I read "in Rust", or whatever language, it does not make it cool or interesting to me. Rather, it makes it sound like it is a POC or training material. Which is unfortunate because eventually, as a user, I only care for features, not for tech.

Not everything has to be cool or interesting to you. To be fair to the authors of Lemmy, the only mention of Rust on the project page is "Server is written in rust", while the submission for some reason contains Rust like it's some highlight of the software.

Since we usually can't change too much of our environment, but we can change how we react to our environment, my recommendation would be to simply just ignore the "in X" in titles you see here, as what you say is true, it basically doesn't matter.

If the software is shit, it's shit, no matter if it's shit in PHP or in Rust.

I tried to be explicit that my comment was anecdotal, but apparently failed.

> "Server is written in rust"

Unfortunately this makes it a "new reddit clone" (with websockets and react SPA) not an old reddit or hn clone.

I like knowing what technologies were made with to some extent. It can give me a feel for what type of bugs I might run into or what the installation process looks like.

Agreed to some extent though, once something is done, what it was written in - should be irrelevant to the end user.

To be fair though, a few weeks ago there were people on here bashing rust claiming it's like impossible to use, async didn't work, the code is unreadable, shouldn't do web dev with it, etc. Meanwhile, here's yet another example of rust being just fine. So I'm glad they shared the technology they used.

Right, and it means something to me in that the code is easy to modify. So I guess the choice is you as a user or me as a user. I imagine this site, at least, leans towards people who hack on things (hence name).

True. Let me clarify that I approach software as a solutions first, meaning, I am a user first.

When said software is open source, it allows me to hack on it, improve it, etc.

What is imporrant is rhat if there's no perceived value for me as a user, no matter the tech, I'll most probably ignore it. On the other hand, I could learn any tech should I need it to propose more value to a software.

That is why I couldn't care less about the "in Rust/Python/Haskell/Zig/..."

I'm not opposed to people doing that, I'm just the kind of user that takes zero interest into it until I want to contribute to add value to the software.

Let's say I'm product-first rather than tech-first.

While I share your sentiment, some technologies automatically come with certain features.

It's the (evtl. statically) compiled versus the scripted ones. And among the scripted again is one ubiquitous and can be lean on dependencies (sh).

So IMO there is certain interest in how it's built. And it may attract contributors, inspire or serve as a warning.

I always find the language comments strange. I'm on HN, i want to see the language/tools/etc things are written in.

also worth looking at nostr which has a different approach to decentralization and doesn't trap your identity with your home server https://github.com/nostr-protocol/nostr

there is a reddit clone and a twitter clone implemented with this protocol as well https://nvote.co/ and https://nostr.com/

This has a tech solution, but the same people problem. Look at the % that have “leftist” in the description. Again, politics come before interest just like the existing sites.

I decided to still look. There seems to be no interest-based community. Just random posts likely interesting to an existing group of friends.

It would be fun to stand one up for like gardening, fishing, sewing, or another of my random hobbies. But I’m admittedly lazy and don’t want to own that (both $ and effort managing a community).

I enjoyed niche subreddits until each devolved into feuding, politics, or just repetitive “how do I start” questions killing the signal:noise. The commercialization of Reddit was well after my disinterest.

I think the very project is devoted to promoting leftist ideas though, which is itself a people problem. For example, the authors of the project has been adamant on having a hard-coded "slur" filter (until recently, it appears) because they want to stop those with a different opinion/language from using the software.

https://github.com/LemmyNet/lemmy/issues/622

Honestly, looking at the censored words in the config.hjson as of oct28 2021, I don't see anything political. Mostly filtering literally racist and sexual slurs in the same way television/radio/the workplace does... Only in 2022 would we be discussing the n word and it's equivalents as politically charged... In my opinion, the words in the slur filter I see are effectively bottom of the barrel for reasonable discourse... If you wrote them in an email at work you'd be terminated quickly...

The problem is the filter from Lemmy seems to suffer from the Scunthorpe Problem (https://en.m.wikipedia.org/wiki/Scunthorpe_problem )

Honestly, this is such a trivial change you could just fork and do it if you’re a programmer. It doesn’t make sense to get upset if upstream won’t support a trivial feature if that’s the only thing stopping you. Or rather, if they say no and I want it, I would just add it.

I'm not sure it's meant to be political, and instead A-political in that they don't want to be involved in the "fight for free speech" that seems to attract the most racist and bigoted dregs of society to projects that wish to partake. In the author's desire to be apart from that, you've labeled them in a way that may or may not be accurate. Being anti-racist doesn't have to be a "leftist" ideology but complaining about their stance on it and labeling them leftists sounds like you're admitting right leaning people tend to be more racist. I'm not sure if that's the picture you intended to paint. I don't see any labels for this being "leftist" on the project webpage or on the GitHub readme.

Lemmy is great! Despite the network still being fairly small, everything work really well already. Setting up instance was a breeze.

The devs are also very accessible which is nice.

Where's the benefit of federation for link aggregators? If reddit.com/r/programming and HN would also implement the ActivityPub protocol, wouldn't that turn all participating aggregators into one because all submissions and comments are shared? How can a community maintain its identity in the Fediverse?

This will be a controversial opinion on HN, but link aggregators, even the ones with comment sections, aren't communities.

Yes, I use a throwaway handle now, because I used a real username here and on Reddit long enough to learn two things:

1) the danger of being doxxed if you say the wrong thing is real and "the community" will ruin your life with glee

2) although I have made numerous valuable relationships online, this has never occurred in a link aggregator comment section

Considering these realities, I think link aggregators lack the repeated exposure to common members (excluding celebrities like dang) to create any real bond between individuals. No community exists because nobody ever gets to know each other.

No, the issue in this case isn't the Fediverse. It's link aggregators.

I remember Lemmy’s devs being adamant on having a hard-coded slur filter (https://github.com/LemmyNet/lemmy/issues/622). I think it’s been resolved now — not a great way to go about it though.

Damned if you do, damned if you don't?

Or what do you mean with "not a great way to go about it though"? Seems like the following happened:

- Developers implemented something well-intended, poorly-executed

- People pushed back

- Developers changed their mind and disabled the non-customizable slur-filter

Seems that's exactly how you'd want things to go. What's the alternative here that would be better?

I'm not sure it is "well-intended, poorly-executed". They are well conscious of what they are doing -- sacrificing flexibility to make a political point.

For "sacrificing flexibility" read "deliberately making the software harder to use".

The slur filter isn't so much an issue in itself as a red flag -- it makes me wonder what other stunts are they likely to pull.

> I'm not sure it is "well-intended, poorly-executed"

This is a clear unambiguous example of "well-intended, poorly-executed, then fixed"

For those who want to see what the slur filter looked like, it's still in their example config [1]. But even though it's now not included by default, the issue runs a little deeper imo.

From what I've seen, the developers of Lemmy are very political. They run and mod their own instances at lemmy.ml and communism.lemmy.ml, and their moderation has been pretty controversial [2] [3]. But worse, judging by the hard-coded slur filter, they seem to have no issue with mixing up their politics with the platform. Also note that it took quite a bit of pushback to get the slur filter removed, and the devs were not apologetic [4][5]. Who knows what they may add next as an "easy way to keep some groups of users off".

While it's nice that it's federated and open source, the question then becomes, are people prepared to monitor the source code and fork it + maintain it themselves if the devs decide to turn the platform political again?

[1]: https://github.com/LemmyNet/lemmy/blob/0.16.5/config/config....

[2]: https://old.reddit.com/r/RedditAlternatives/comments/czae8c

[3]: https://old.reddit.com/r/Anarchism/comments/cza14y

[4]: https://github.com/LemmyNet/lemmy/issues/1224#issuecomment-7...

[5]: https://github.com/LemmyNet/lemmy/pull/816#issuecomment-6446...

Beggars can't be choosers.

Whatever political choices they make is their right. People are free to move to other software!

Or fork off.

> the issue runs a little deeper imo.

Right, because the developers are "political"?

After seeing how they respond to criticism and how they act as moderators, I just don't trust them to stay politically impartial when it comes to building products. Of course other people may not care, but I do

Another github discussion showing their views and interactions with the community: https://github.com/LemmyNet/lemmy/issues/622

lemmy is awesome. I'm on lemmy.ml and it basically replaced my reddit browsing.

Reply on news.ycombinator.com