Hacker News Reader

There is actually a third point to this whole idea. Given that most Lemmy admins were not interested in being a home to the "mirrored" communities, I'm building a set of different Lemmy instances that will be focused on specific niches or interests, such as:

    - https://soccer.forum, https://nba.space, https://nfl.community, https://matchpoint.zone  (sport news, fan groups, discussion, etc)
    - https://metacritics.zone, https://roll20.top (pop-culture, movies, TV shows, games)
    - blockchained.world (web3)
    - https://selfhosted.forum (devops, homelab)
    - https://style.land (brands, fashion, consumer items)
    - https://hi-fi.community (audio), https://viewfinder.pro (photography)
    - https://hardware.watch (tech reviews and news, pc enthusiasts, etc)
    - https://gearhead.town (cars)
    - https://netheads.online (voip)

These lemmy instances are meant to be only destination hubs. They are not open for registration, but I am planning to let people decide what communities should be opened on those instances. I've created those because it seems that a lot of people are signing up to an instance and then fail at discovering the content.

clickable version

(sport news fan groups discussion etc)

https://soccer.forum

https://nba.space

https://nfl.community

https://matchpoint.zone

---

(pop-culture movies TV shows games)

https://metacritics.zone

https://roll20.top

---

https://blockchained.world (web3)

https://selfhosted.forum (devops, homelab)

https://style.land (brands, fashion, consumer items)

https://hi-fi.community (audio)

https://viewfinder.pro (photography)

https://hardware.watch (tech reviews and news, pc enthusiasts, etc)

https://gearhead.town (cars)

https://netheads.online (voip)

why do these sites all look like reddit?

Because they are all different deployments of Lemmy [0]?

[0] https://join-lemmy.org

They’re mirroring reddit content

ah, it makes sense now. thank you!

Isn't it expensive to register so many domains? I guess you could host them all as subdomains of alien.top or something. Anyway you're doing a great service, I still see people manually "mirroring" stuff to Lemmy from Reddit. It's nice to have an extra filter, but this automating some of that makes much more sense to me.

The idea is to have these instances completely separate from alien.top. alien.top is the flagship instance from this fediverser project, but these content instances are related to my actual connunick business.

Regarding domain price: aside from the 3-letter ones, the domains have been all reasonably cheap and they are an investment anyway. I have a vested interest in making the fediverse grow.

[0] https://communick.com

Love this! What mechanisms are you considering for selecting the communities to mirror per instance? M

Right now, I'm just going by feeling. There are some cases where it's not clear cut (e.g, should /r/mechanicalKeyboards go to style.land or hardware.watch?) but in most cases the mapping can be done by finding the "largest" subreddit in a topic and then checking their own list of related subs.

I wonder if you could team up with https://sub.rehab/ and evolve it.

Wow, I remember seeing this site during the first reddit protests but I completely forgot about it. I was using redditmigration.com for my initial pass, but it seems that a lot of the communities or instances recommended there have been abandoned. I will definitely take a look sub.rehab again.

The main thing Lemmy is missing for adoption is a good desktop and/or mobile app.

Since instances can defederate with each other it's often not possible to follow all your favourite communities from one account. So a good app needs to decouple this and enable follow any community without having an account, and instead store following data in the client.

Do you think there's a technical fix to this or do you think its an immutable aspect of Lemmy's design? Like, is there something facile that the client applications (Apps used to "browse" and interact with the Lemmy ecosystem/instances) can do or is it impossible with fundamentally altering Lemmy itself?

Yeah this is a design change that can only be fixed on the client application side because there will always be a reason created by an instance for why they don't want X content mirrored to their site.

Couldn't you just sign up on an instancewho's sole purpose is to remain federated with every other (major) instance and then "add" that account which the client app maintains along with one or all of your other accounts and wouldn't that super easily fix this issue even though it would be better for rhe app to handle all that without having an additional account to maintain?

Edit: also, did I just roughly describe a "mirror" without even knowing it from the spirit of what I said?

> did I just roughly describe a "mirror" without even knowing it from the spirit of what I said?

Yeap, you just did. :)

I absolutely adore when anyone accidentally does this. It should be a RedditSubbreddit and a LemmingCommunity

if following is all you care about, then you can just use the RSS feeds.

I don't think most Lemmy or Reddit user are satisfied at that level of engagement. RSS doesn't have comments and I don't believe any Redditor/Lemming seriously only goes for the "news" or being informed as the root goal although it would be a far healthier standard of engagement in my view.

Still, I don't see a replacement for community-oriented and community-curated discovery+parsing+critquing of content.

I'm just getting 'no recommendation at the moment' even for subs that are main/frontpage subs such as 'technology' and 'awww'

That's because I'm still building the reddit-to-lemmy mapping, and for some cases in creating the lemmy communities themselves.

There is also the fact that the largest Lemmy instances (lemmy.world) decided to consider alien.top as a mere "reddit repost" instance and decided to defederate with it, so I can not recommend instances from there as they will not be reachable.

This seems like a flaw of the federation system.

Why should defederation be two sided. You should be able to recommend public posts from lemmy.world, and they should be able to stop any outgoing communication within lemmy.world to alien.top, if thats what they want.

I somewhat agree, but the thing is, I can recommend them and users on alien.top would even be able to subscribe to the community, but lemmy.world would never send updates to us.

I actually wrote to the LW mods to see if there will eventually reconsider the decision (https://communick.news/comment/674064), let's hope for the best.

For my curiosity, why "access_type=offline" as well as "scope=submit" for connecting accounts? I got the impression from your description that it just allowed "claiming" the user-id on portal.alien.top but your scopes seem to imply you're going to submit any content from portal.alien.top back into reddit as me

Yep, the amount of access being requested seems excessive:

  Access posts and comments through my account.
  Access the list of subreddits I moderate, contribute to, and subscribe to.
  Access my inbox and send private messages to other users.
  Access my reddit username and signup date.
  Submit links and comments from my account.
  Maintain this access indefinitely (or until manually revoked).

I would expect this app to only need 2nd and 4th items in this list.

Two-way communication has been requested by some people and is the next item on my to-do list [0]

[0]: https://github.com/orgs/mushroomlabs/projects/1

I see https://github.com/mushroomlabs/fediverser/issues/9 but I would be much more likely to use it if the oauth2 dance didn't ask for aggressive perms versus a "ask for all the perms then revoke them later"

The issue you are referring to is on the reddit -> lemmy side, but I see what you mean.

I think I could make a separate application with less permissions and then just this one when two-way communication is ready and opt-in?

I would not claim to be a oauth2 expert, but to the best of my knowledge the scopes that one declares during the app registration on the reddit side are the upper bound of what said app is allowed to request, and thus if there was a checkbox on alien.top's side asking whether the user wanted bidirectional behavior, if unchecked the oauth2 handshake would omit the write scopes from the handshake and the token that comes back would be read only. I have mixed feelings about the "offline" part but if it was read only, I would be perfectly fine with alien.top using my access token to help spread the rate limiting load during any sync activities for the subreddits I was in

Furthermore, again, to the best of my knowledge, if an alien.top user changed their mind and wanted to opt-in to the bidirectional behavior later, you could have that user just re authorize with the same client-id and this time ask for all the scopes. Much less complexity than managing two separate app registrations

You are correct on almost everything, except on the ability on using your access token to spread rate limiting : requests still depend on the application id, so every request will use the app id quota. This is why the Apollo developer basically said that the pricing changes would kill any chance of having the app financially viable.

Regarding why I'm asking everything upfront, I will (partially) blame the library I'm using to authenticate with reddit, django-allauth. It can be made to make user configurable scope requests, but the default adapter just takes a fixed list of scopes from the overall application setting.

I do understand that it would be better if I asked for the proper permissions on demand, but to be honest it was now just a matter of prioritizing new features over addressing valid-but-minor objections that people might have.

> Access my inbox and send private messages to other users.

Yeah that's a quick no from me.

well, as you can see <https://news.ycombinator.com/item?id=38015078>, we're just "valid-but-minor objections that people might have." so ... that's, just, like, your opinion, man :-P

I would guess the short version would be "run your own copy with sane scopes" but that would require that I visit reddit again, sign up for their new developer-hating experience, and I just don't care about their content that much

Question though: if you are already interested in getting out of reddit and do not care about the content from there and/or interacting with it still, then why can't you simply sign up to the other dozen of well-managed Lemmy instances?

Also, think a bit from my perspective: every user that signs up to my instance but is not willing to have two-way communication is reducing its viral aspect and is costing me resources. I'm not saying that what you want is unreasonable, I'm just saying that there are others that could be more helpful.

If any and all security is a "minor concern" then I'm happy to pass.

If the application actually wants to send messages on your behalf (like it will for some users), you can not claim it's a security issue if it ends up doing so.

If you don't trust me or the application, it's another story. But please don't deride my work as malicious or ignorant of actual security practices.

The app claims to "check the subreddits you are currently subscribed" (sic) but instead asks Reddit for permission to post. This is a security bug! What are you talking about?

What has trust got to do with anything once I have observed the wrong thing happening?

> instead asks Reddit for permission to post

Not "instead" but "in addition to". And the reason it is requesting permission to post is separate from "checking the subreddits", it is (like I said) because there are already people who asked for the ability to respond to a comment on reddit from a lemmy thread, and I (admittedly) didn't bother (yet) to write two oauth flows for these two separate actions when one is faster to implement and easier for people to understand.

> I have observed the wrong thing happening?

Have you seen or heard anyone complaining about the application sending posts when it shouldn't? If not, there is nothing objectively wrong with it.

You might be concerned about potential issues with this approach and I completely agree that it is not ideal, but between spending time to (a) write a whole new OAuth adapter class to handle the different requests or (b) creating 15 new different lemmy instances and mapping 175 subreddits to Lemmy communities - which one do you think is more helpful to the goals of the project?

Security is not about bad things not happening, it's about preventing bad things from happening.

Look, you've already agreed that "it would be better if [you] asked for the proper permissions on demand", and that it was "a matter of prioritizing". Now you are trying to object and call it "deriding" if I don't pretend that you are following security best practices. You are not, you have explained why, what is this thread about?

By your definition, the most secure thing then is to not do anything.

> what is this thread about?

It really rubs me the wrong way that you are passing opinion as fact, especially when it is an open source project that is meant to help people. Instead of shitting on other people's work by writing things like "I am happy to pass", perhaps go take a look to see how you could make the improvement that is so important to you?

> By your definition, the most secure thing then is to not do anything.

Yes...

> perhaps go take a look to see how you could make the improvement that is so important to you?

No amount of contribution on my part can change your attitude towards security. If it is not a priority of the maintainer, the only move is to stay away.

I'm not saying you are creating a collection of spam accounts, I don't even think it's likely that you are. But if someone wanted to do it, this is exactly the way they should go about it. The only move you could take to prove otherwise (not that you owe it to anyone) is to not request posting permission until you need it, with written explanation.

> If it is not a priority of the maintainer, the only move is to stay away.

There is another move: submit a PRs with the change. I would gladly accept it.

Now that this is online for 3 days, I can give you some numbers: 11 people rejected the request and 135 accepted it. If I went by your preferred route, I would have zero rejections and zero conversions, because I would still be struggling with a self-imposed technical requirement.

My time and resources that can be dedicated to this are finite. It doesn't mean that I deny things could be improved, it doesn't mean I won't get to keep working to improve the existing product and it doesn't mean I won't address those concerns later. It just means that this is not meant to be some research project where I need to be satisfying some panel of theoretical experts who never care about the real practice. Right now, the majority of people are telling me "make it easier to find more communities on Lemmy and make it easier to convert more people even if that requires me trusting you", not "I don't trust you enough to let you send messages on my behalf."

> There is another move: submit a PRs with the change. I would gladly accept it.

That does NOT fix your SECURITY POSTURE. If security is not a priority, you will keep taking shortcuts. You might even have similar issues in your deployment. Sending patches over and over to a software under active development, to try to add something the maintainer doesn't care about, is just not a winning proposition.

> 11 people rejected the request and 135 accepted it

I myself didn't click "decline", I just closed the tab. How many people closed the tab when they reached the Reddit sign-in page? How could you possibly measure whether it's because they didn't agree to the scopes, whether they lost interest, or whether they forgot their Reddit password?

> If security is not a priority, you will keep taking shortcuts.

As with any other engineering discipline, it is about trade-offs. If you want to maximize "is it secure?" over "can it be used by people safely?", fine. But at the end of the day I managed to have this delivered and used by hundreds of people, who are now able to go on to tell others about it, and no actual security incident.

> How many people closed the tab?

How people got dropped by the funnel matters little to me compared to the fact that there is a funnel in the first place.

This is not a research project and no one is paying me to optimize this funnel. This is me building a tool to have a migration path out of reddit. Prioritizing having it available to people and other instance owners beats delaying it for a week on the basis of "some people will walk away because they object to the access request."

Those like you can wait for another week, or they can go fork the code if they want, or they can even ignore the whole project and join lemmy directly. But the product is not for "those like you", so why should I be focusing on addressing your concerns over implementing the things that have been actually requested by other people?

Very interesting. Will this cover all subreddits, or all above a certain size?

I've switched fully off of reddit in protest, and have turned to HN and Lemmy as imperfect replacements. The main remaining thing I've noticed reddit excels at is researching things like product recommendations.

Searching a keyword on BIFL, along with poking through various threads on whatever niche community might exist for the topic, has been one of the most reliable and useful ways to get a sense for what brands or models are best when considering a purchase.

So far Lemmy is not close to replacing this so it would be compelling if your project could change that.

There is no 1:1 mapping between subreddits and lemmy instances, so even the very tiny ones can be mirrored.

The main bottleneck is actually on the lemmy side, as I am having to create most of the communities, setting out rules, etc.

perfection is subjective IMO. I quit reddit when Apollo died. I use Lemmy, but no longer find myself getting sucked into a doomscrolling vacuum. For me, Lemmy IS the perfect replacement because I use it less.

Nice work! I think this is a much more elegant solution

Am I crazy to think this might raise legal alarm bells? Mirroring content is commonplace but I haven't heard of sites with an explicit path to take users from another site. One is grazing on their land, the other is an existential threat.

How is it that you are able to scrape all of the content of Reddit, while 3rd party apps have almost stopped working.

- reddit still allows 600 requests per 10 minutes free of charge.

- third-party apps need to make requests on behalf of all their users in real-time, this is making requests for a single user (the backend) and can be batched. Right now it it fetches data from 10 subreddits per request and it can be made to get even more at once.

- I am not scraping all of reddit, only about 80 for now. It can be done every 3-4 minutes well within the rate limits.

- if necessary, I could take the same approach used by the lemmit.online dev, and just request the JSON feeds (which do not require an API key) and parse the results on my own.

- If more people run this service, we can make them cooperate so that we do a "MapReduce" on the scraping: the requests to different subreddits are spread between the fediverser instances, and they can gossip back the results among themselves.

Are you not concerned about being sued by reddit?

cool. i've been wanting more people to move to the fediverse services so i have more people to talk to

So you want my reddit credentials ?

No. OAuth can solve this exact problem, you just need to authorize reddit to "login" to the portal website, and once you authenticate I can know that you really own the reddit account that you claim to have.

Oh nice ! I'll try it then.

Show HN: Fediverser Portal – Bring your subreddits to Lemmy