jacobobryant

hm, no idea. I haven't thought about revenue models much since I've envisioned this more as a grass-roots thing than a startup opportunity. I think the main thing would be just making it dead easy to get started. You wouldn't necessarily need to jump start a new network since the whole point of this is to be interoperable with existing networks.

A good first step might be making a personal site generator. Put in links to your twitter/youtub/substack/medium/whatever accounts, and then you get a nice looking site with all your content in one place. Maybe throw in a subscribe button so people can get email digests of all your content.

and then, almost as a side effect, anyone who uses the site generator also is publishing it all as an atom feed.

That's a big reason why I'm excited about repl.it. You can self-host a github repo in about 2 clicks (not including account setup). I could see it increasing the reach of self-hosted apps a lot.

Actually I made a little proof of concept for this a week ago: https://github.com/jacobobryant/Feedstuff

Heh, I published a post in a similar vein just today[1]. Lately I've been thinking the way to do it is have everyone publish a (paginated) Atom feed with all their public content interactions. So each item in the feed would be something like "published this blog post", "liked this tweet", "thinks this comment was written in bad faith", etc. And "follows this person" with a link to that person's atom feed. Then if you want to build a search engine/social network/recommender system, you can get started by crawling and indexing a bunch of feeds.

I'd like to make a web app (probably on replit since it makes self-hosting dead easy) that has plugins for a bunch of other sites. so you connect to all your other accounts, and then this app imports your data and publishes it to a single feed for you.

Obviously starting audience would be programmers, but I bet it could expand to a general audience too.

[1] https://news.findka.com/p/we-need-user-centric-data

I just installed it. I don't have much to add, but wanted to say: wow, this is awesome. I appreciated the How It Works section. I made a tiny open-source web app in a similar vein, though it just uses the naive approach (querying algolia on each page load).

Q: How many programmers does it take to change a light bulb?

A: Programmers don't change light bulbs; that's a hardware problem.

> Steve Omohundro had a talk recently where he described the need for "personal AIs" to help individuals resist manipulation from corporate AIs maximizing engagement.

This reminds me of The Big Promise of Recommender Systems (2011) [1]:

> However, when we look at the current recommender systems generation from the point of view of the “recommendee” (users’ side) we can see that recommender systems are more inclined toward achieving short-term sales and business goals. Instead of helping their users to cope with the problem of information overload they can actually contribute to information overload by proposing recommendations that do not meet the users’ current needs or interests. ...

> The window of opportunity is now open to innovate in a third generation of recommender systems that act directly on behalf of their users and help them cope with information overload.

I'm working on something in this space myself[2] (an essay recommender system). I think part of the solution is having recommender systems that are decoupled from publishing; e.g. a video recommender that suggests videos across multiple, unaffiliated sites, instead of the recommender that's built into YouTube.

[1] https://www.researchgate.net/profile/Marc_Torrens/publicatio...

[2] https://essays.findka.com

It's hard to say for sure without having first-hand experience with the code base. But rewriting messy projects from scratch is often a pitfall, and I would be very cautious about taking that route (especially given "I feel that I'm in way over my head" and "I feel competent enough in ... but not in creating a feasible architecture"). The last thing you want is to spend months creating a new project that ends up being not much better than what you started with, a scenario that happens all too frequently.

I would consider using this as an opportunity to learn and refactor a large code base. That's an extremely valuable skill, and it's hard to get it without situations like these (whereas anyone can get practice creating a new project on the weekend). You might have to spend a lot of time staring at the code, feeling like you're not making much progress, but it'll (hopefully) be worth it.

On the other hand, it is possible that the code base is just horrible and should be nuked, so use your own judgment :).

also rawg.io's api. I've used them in the past.

First I get the text of each article using Newspaper[1] (no rss involved). Then I used tf-idf and k-means to cluster the articles (I followed this tutorial[2]). Then I combine the clusters with my collaborative filtering model using feature augmentation: for each cluster, I generate N "fake users" who like each item in their cluster, and I add that to the rest of the rating data.

So in effect, content-based filtering is used to handle cold start (when a new article is submitted which doesn't have many ratings yet, the fake user ratings will dominate it), and then as real user ratings are gathered, it will switch gradually to relying only on collaborative filtering.

[1] https://newspaper.readthedocs.io/en/latest/

[2] http://brandonrose.org/clustering_mobile

Awesome, glad to get some confirmation. Findka's current sampling method might help with this already, by the way--since it starts out by picking a feed uniformly, posts from the guardian would take up the same percentage of your daily recommendations as posts from random blog, no matter how often the guardian publishes (at least until random blog's posts run out).

I've added OPML import FYI.

In certain contexts (like blog articles that aren't dealing with implementation specifics) I use "RSS" to mean "Atom/RSS", same as how "classical music" in many contexts also includes music from the baroque and romantic periods. I used "Atom/RSS" for a while, but recently my feeling is "eh".

Yes, exactly! The nice thing about framing information discovery as a stream-processing problem is that you can plug other methods into it. For a long time I was thinking just in terms of user-item rating matrices, i.e. collaborative filtering. You've got a list of users, a list of items, and ratings for some of those pairs. Pick 10 items to show a given user. But that doesn't give the user a whole lot of control.

If you think in terms of stream processing instead, you can treat collaborative filtering/other types of recommender systems as some black box that generates a stream. For example, Findka's ML recommendation algorithm picks a handful of articles for each user every day. It's just another stream, and it can be treated exactly the same way as any of the RSS feeds that users subscribe to.

So you get a list of sources, some of which are "regular" RSS feeds, some of which are algorithmically generated--and they can come from within Findka or from outside--and then the user should be able to decide how those feeds are aggregated. Bandit algorithms I think are a great fit (including contextual bandits!), but you can also include manual controls. e.g. you could say "feed X should be 10% of my main sink feed", or "feed Y should be no more than 5% of my main sink feed." Or give priorities--sample from all the feeds in priority 1 until the contents are exhausted, then move to priority 2 etc. There are probably lots of interesting things that could be done in this space.

Recommendation algorithms are often opaque out of necessity (I guess more generally, ML algorithms), but at least this way they wouldn't be in control of everything.

I'm currently using an off-the-shelf python library[1], so the entirety of my RSS parsing code is `[x.link for x in feedparser.parse(url).entries]`. There's also Superfeedr[2] which provides an API, but it's expensive (likely because they give realtime updates, which I don't need).

[1] https://pythonhosted.org/feedparser/

[2] http://superfeedr.com/

Awesome, glad to hear it :). Hope it goes well. I'll add OPML import soon.

Ha, glad I'm not the only one! Thanks!

All good points, and thank you! I think Findka is a better fit for people who are not already RSS power users (or RSS users at all)--the idea is to use algorithms to handle this kind of source curation automatically.

Ultimately, I'd like to provide the best of both worlds: automatic, algorithm curation by default, with controls that allow you to override things manually.

Good to know. I'm planning to add settings so you can control the behavior of each subscription (e.g. choose which ones you want a sample of, choose which ones you want the whole thing), so that workflow should be possible with Findka at some point.

Because I was responding to someone's tweet.

Tweet storm with a bit more info: https://twitter.com/Endless_WebDev/status/133715064374455910...

Great. I'm not claiming that existing RSS aggregators are broken, but this method is certainly better for me personally. Findka's meant to make things a little more automatic for those who want to spend very little time on managing their sources. The bandit algorithm makes it so you don't need to explicitly mark which sources you've decided are interesting. And the main, non-RSS part of Findka (article recommendations via collaborative filtering) helps expose you to new articles and sources.

This new feature is probably most useful for people who aren't already using RSS regularly.

Hey, nice to see this made it to the front page :). I launched this app on HN in October by the way: https://news.ycombinator.com/item?id=24921127

My current, somewhat vague end vision for Findka is to be a stream processing tool for end users. You subscribe to a number of feeds/streams/sources, and then Findka lets you transform them into output feeds/sinks. You’ll be able to choose how much you want algorithms to help out. The goal is "all your content in one place," but more sophisticated than what you can do now with current aggregators like Feedly.

Most of the essays currently are indeed blog posts from the last few years. For now I'm planning to rely on user submissions for the catalog--if you have some favorite classic/non-tech essays, you could submit them (there's a URL input box after you sign in).

Before today there were about 60 submitted essays; we're up to 117 now. So the recommendations going forward probably won't be dominated by my own submissions, but they will likely still be tech-focused, given the (initial) HN/programmer user base. Eventually (after we have enough users/submissions/click data), Findka should adapt to your preferences quickly. i.e. even if most of the catalog is tech-focused, the algorithm will look at what essays you've submitted and select recommendations from the appropriate niches. But at this early stage, the recommendations will be sampled more-or-less uniformly from the entire catalog.

I'm skeptical that using content-based filtering will make much of a difference at this stage. If I were ingesting a lot of articles from some external source(s), content-based filtering would definitely be necessary. But I think we'll get better results from manual curation. As of now, about ~90% of the essays in the database were submitted by me--so cold start shouldn't be a huge issue, at least for people who share my tastes.

But at some point, I'll definitely do some A/B tests with content-based filtering.

Awesome, let me know how it goes.

Hi. I pivoted to this recently from a cross-domain recommender system (see previous Show HN[1]--it could, for example, recommend movies based on which books you like). Long story short, people thought the idea was interesting (landing page converted at ~7%, and I got hundreds of signups), but the recommendations weren't accurate enough to get people to stay (retention was nonexistent; I also did some user surveys)--and I wasn't confident that I'd be able to make the recommendations much better with the resources I have. I was originally hoping that Findka would be good enough at least to sustain growth so that I could improve the recommendation quality gradually over time, but that didn't pan out.

Findka Essays is a simplified version of the old app. It recommends only essays (instead of any type of content), and you get recommendations through email only, not through a web app. I record which links you click on, and links in future emails get picked based on your click history.

Since the domain (essays) is much more restricted, a more-or-less randomly selected item should still have a decent chance of being a good recommendation (especially since right now I manually curate most of the essays). So I'm hoping that the system will be worth using even before we have lots of data to feed to the algorithm, unlike the old Findka.

Also, I genuinely love essays (reading and writing). I think they are the most valuable form of content, and I think helping good essays spread further would have a big impact on the world. So I'm pretty excited about this pivot.

For those interested, I've written a detailed description of the architecture here.[2] The web app is written in Clojure, using Biff[3] (a web framework I made), and the recommendation algorithm is a ~100 line Python file that uses off-the-shelf k-NN, with some of my own additions for handling exploration and countering popularity bias. See [4] for details.

[1] https://news.ycombinator.com/item?id=23541840

[2] https://findka.com/blog/essays-implementation/

[3] https://findka.com/biff

[4] https://findka.com/blog/essays-implementation/#recommendatio...

I think we're in agreement, assuming your premises are:

1. Reflection is necessary

2. In some cases, note-taking can aid reflection (such that marginal benefit > marginal cost)

I'd agree even if you change "some" to "most"--just not "all." I don't think you're arguing that, but I've seen people assert it.

Do you know if they actually don't bother? The article author makes it sound like tracking companies do in fact normalize gmail addresses. I suppose my custom domain addresses wouldn't be too hard to normalize either if you have a way to take a domain and get a probability for how likely it is to be a personal domain. It provides a little extra friction at least, I guess.

I do this with fastmail. Besides privacy, another benefit is you can set up routing rules. e.g. for subscribing to newsletters, use <whatever>.sub@example.com. Anything that goes to an address that ends in .sub@example.com gets marked as read and is delivered to a folder for newsletters. I do a similar thing when signing up for services. It's kinda like what Gmail does with ML, but you have more control.

I'm all for concision, particularly in technical writing. But not every article has to be like that. This article feels like a mix between an essay and a public journal entry, and I thoroughly enjoyed reading it.