Linen – Make your Discord community Google-searchable | Hacker News Reader

Linen – Make your Discord community Google-searchable

linen.dev

80 pointsreichertjalex29 comments

Damn, I had this idea in my list for the past few months! Kicking myself for not moving on it sooner.

Congrats to the team, looks well done! It's a bit absurd that it's come to this but I guess that's where we're at now.

One question I had in my "to figure out" list was: how do you handle making users messages public, when they feel like they're joining something private?

Another question: do you sync deletions if a user goes back after e.g 1/3/6 months and deletes a message?

For Slack we're making all threads private and we follow update and deletion immediately.

For Discord we follow deletion after a day since we are doing a nightly sync for the time being.

Privacy is one of our main priorities since I get that we're taking something that felt private into something that will be persisted on the internet. Here is a few ways we are/planning on addressing this:

* We only store profile image, user id and display name for the user so we don't keep email or personal identifiable information

* We just shipped a new feature where we can anonymize the display name so it is even less identifiable - i.e. with names like `many-ancient-parrot`, `adventurous-billowy-kangaroo`, and `benedict-cumberbatch`.

* Right now we sync everything but very soon we are shipping features where communities can only sync from the new conversations.

* Upon request we delete all the users's messages and conversations based on the user ID. We're shipping new where the community admin can delete conversations themselves in the coming weeks.

* We allow community admins to decide which channels to show or hide since some communities choose to create new channels so it is clear that the conversations going forward here are public.

Do you anonymize only the display name? If you do that, you should probably sub in a generic avatar or something.

I use my personal photo on Discord communities I'm involved in, so you could randomize the name but it'd still be obvious it's me.

Now that you mention it seems obvious that should be a feature. I'll add it to the roadmap. Probably will be done in the next two weeks.

It's done now btw if you are curious.

> Make Slack and Discord communities Google-searchable

Isn't Slack an acronym for "Searchable Log of All Communication and Knowledge"?! Then they either failed miserably at making it searchable to the point of requiring an entire new business to make it so, or I'm not seeing the point of Linen.

There's a distinction between searching for info within a Slack community (Slack handles this well) and searching _from the outside_ for info that only lives within a Slack community. As more and more communities that used to live on public forums migrate to Discord or Slack, that knowledge is basically impossible to find from the outside (Google) if you're not already a member of the Discord or Slack instance. From what I understand, it's this latter use case that Linen is targeting.

"Slack handles this well" That's questionable… ':) — Though I get it now. And that's indeed a need! Reminds me of when we needed to setup IRC bouncers to not lose history. Thanks.

I love this solution for a number of reasons:

1. I find the posting experience on Discord far more seamless than that on a typical forum. From navigating channels to posting messages, I prefer the quicker navigation and viewing that it provides.

2. Discord servers tend to be more active than forums.

3. It fixes the gap between loads of pre-existing communities that would have been difficult to migrate over to a forum and document any sort of solutions that they provide in a searchable manner, something that even within Discord was rather difficult to accomplish.

4. You can also access loads of 'forums' now with a single account (you only need one Discord account in order to do so). A password manager makes this trivial, but the advantage that Discord has is being able to navigate seamlessly between many different servers or 'forums'.

5. I assume that it would be easier to document whatever gets put onto Linen than it would be to document Discord servers, but I'm not entirely sure whether this is truly the case.

Show HN: Linen – Make your Slack community Google-searchable - https://news.ycombinator.com/item?id=31168882 - April 2022 (57 comments)

This looks very useful, but is it allowed according to Discord's ToS?

No, it’s not, unless every participant individually and permanently grants a license to their content for this use before they’re permitted to speak in any channel that’s being archived by this integration.

Discord's developer TOS says that API users must "comply with all applicable privacy laws and regulations" and adhere to a privacy policy. It doesn't otherwise put any specific restrictions on what that policy can look like. It certainly doesn't say you can only use user content if the user "individually and permanently grants a license".

If you think that Linen would be violating the user's rights, as opposed to Discord's, then that's a separate issue, and one that would probably come to a question of fair use and/or implied license.

Out of curiosity, do you also think public mailing list archives are (or should be) forbidden, by the same reasoning?

The user doesn't implicitly grant the server owner a license to their content when they post to Discord, so reuse of that content by a third-party service has no implied consent permitting it.

I think that Linen is facing considerable legal exposure risk and is taking the Uber approach of "break the law and pray that no one sets up a class action lawsuit against them for their rights violations". Discord has set up their chat service to be private, and extracting that data and making it public without each user's consent is a violating of privacy that — unless an overriding public good is served — would lead to their fair use claim being overturned.

I expect Discourse to ban their official integration for being inappropriate for the platform (their terms of service permit them freedom of judgment in that regard), and then to warn and eventually terminate _entire Discords_ when their operators use the predictable metastasized proxy Docker image to circumvent that centralized integration ban.

Email, as opposed to Discourse, would inherit the concept of "the recipient of a letter is permitted to do whatever they want with it", same as we have had with postal mail for a very long time — as email can be reasonably considered postal mail.

(Disclaimers: I am not your lawyer, this is not legal advice. I strongly advise consulting with a lawyer on the risks and uncertainty surrounding Linen. I do not consider there to be sufficient case law to be certain what the outcome of judicial proceedings for any of the above assertions would be. My theoretical opinions and predictions may or may not be upheld in a court of law.)

What is the difference vs posting, let's say, to HN?

HN is a public website that publishes all participation to all public/anonymous visitors, and allows indexing by Google.

Discord is a collection of private chat servers that does not publish participation to any public/anonymous visitors, and does not allowing indexing by Google.

I wonder if this works with actual messages as well as threads in Discord, would be interesting when searching for information.

I tried searching for some exact phrases from the live demo, but didn't get any results in Google.

Maybe Google isn't crawling all the pages on the site? Possibly related to being an SPA?

Otherwise, I think this is a pretty cool project. I've always thought that Slack and Discord communities lack a good way to find them.

Try these queries: https://www.google.com/search?q=replace+beast+http+with+prox...

https://www.google.com/search?q=seeing+a+weird+issue+with+fl...

We built Linen on Nextjs since I wanted to server render all the content so it performs better for crawlers. But getting Google to crawl a large number of pages is tricky. It takes a while and even with a submitted sitemap Google sure takes their time.

Depending on the community after a month or two you end up seeing a few hundred pages being index by Google. Their ranking will vary based on how relevant the content is or based on some Google magic. Since I render the content that is newest on Linen it probably won't be on Google for a while.

What's the catch? It's currently free, but given that it's VC-backed, somewhere there's gotta be money made. Ads? Selling my community members' data?

And I also have questions about syncing: message deletions, channels changed from public-to-private, etc.

We have two pricing tiers:

* A free one that is hosted on Linen.dev/s/communityName i.e https://www.linen.dev/s/cfml

* A paid tier where it is hosted on the customer's domain ie. https://archive.pulumi.com https://linen.prefect.io https://community.supertokens.com

Most of our communities are on our paid tier where they get the content hosted on their domain and get their own custom branding. The main benefit being they get the SEO benefits of having a bunch of organic long tail keywords for their domain.

I am still figuring pricing out but I'm testing out volume based/community size based pricing.

I elaborated a bit more on how we handle privacy in another comment. Happy to address any other questions

The CFML Slack had a member in SF meet with the Linen team. The Linen team moves very fast and within hours they had our Slack indexed. I've used their search and I cannot say enough good things about them.

Very cool but this feels like something that it will be cloned as an open-source alternative soon enough (a la Firebase -> Supabase).

Do you have any plans to do it yourself?

I actually built Linen because my previous project was a fairly popular open source project. I actually don't think the hard part of this product is the technical part(like most startups). I have considered building this as open source and might still do it a bit later. But for the time being I want to focus on revenue since funding environment has changed quite a bit in the past month.

It's sad that Gitter didn't gain the traction in tech communities that it should have. It's already searchable (and you land on the conversation directly, not a cloned website), clutter-free and supported markdown before the others. But many communities simply chose to walk into closed spaces, now these workarounds are being built.

I used Gitter for a small open-source community for several years, but we moved to Discord just because it substantially reduced friction for non-technical user support. Not necessarily the biggest issue for many software projects, but for user-facing projects it was impossible to get people to sign up for Github accounts and connect to Gitter just to ask us a question. Discord has super-low-friction onboarding that really makes it easier for people to give feedback and troubleshoot issues.

Is it possible to search Gitter archives? It's real easy in Discord but I don't see a way to do it in Gitter, at least not without third party interfaces (I haven't tested if they work).

If they start supporting Matrix this should be possible!

It would be awesome if they supported Matrix - you could then archive many things via a bridge (https://matrix.org/bridges/).

Literally had this idea the other day

Reply on news.ycombinator.com