NewsBlur: A feed reader with intelligence | Hacker News Reader

NewsBlur: A feed reader with intelligence

newsblur.com

204 pointsconesus84 comments

Hey folks, I'm a HNer. I created NewsBlur with Hacker News in mind the entire time. In fact, I thought up the idea during a summer NYC Hackers and Founders last year. For the past 16 months, I kept trying to pick a date when NewsBlur would be ready to submit. Finally, embarrassed as I am by the bugs galore, today is that day and I am glad to say that NewsBlur is far enough along to show off.

NewsBlur is an RSS feed reader that tries to do two things very well:

1) Shows you the original site instead of a context-less feed. Read the original and NewsBlur marks the stories you've read.

2) Filter stories you either like or dislike. A three-stop slider goes between dislike, neutral, and like (red, yellow, and green). Training is super-easy and all click-based (as opposed to you having to writing out what you like in a site, NewsBlur asks you, semi-Hunch-style, your opinions on facets of the site).

I started working on NewsBlur to see if I could do it. Put the AI together with the back-end feed processing and fetching, along with the nifty front-end of the original site. This is one of those projects where I just kept pushing in all directions until I felt I had something good, not knowing if I could do it at all, but believing the entire time that I was able to complete the project.

If you have a project kicking around in the back of your mind, just dedicate enough weeks and months to get it done and done well. Provided you do that, you're almost guaranteed to succeed in some way. Those first few months of writing NewsBlur, I felt like a fraud, wishing I could just get threaded feed fetching to not break every few minutes. Even what I considered to be simple python gave me headaches. (Turns out threading is evil, use multiprocessing instead.)

NewsBlur is entirely open-source: http://github.com/samuelclay/NewsBlur/. The iPhone app, the front-end JavaScript, and the back-end Python. Pick it apart, but know that it is more a learning experience than a business. Turning it into something profitable may not be easy, but I only need 80 premium users to make that happen.

It's interesting to see somebody tackling this (these) problem(s) of managing feeds through AI. As I said before http://news.ycombinator.com/item?id=1362554 , kudos and keep up the good work!

Whether or not its ultimately a success, you really developed a kick-ass product! It looks good, seems to work pretty well. If you did this all on your own, thats a tremendous accomplishment.

While I may have written and designed NewsBlur, I am using a ton of other open-source technologies:

* Django: Web framework written in Python, used to serve all pages.

* Celery & RabbitMQ: Asynchronous queueing server, used to fetch and parse RSS feeds.

* MongoDB, Pymongo, & Mongoengine: Non-relational database, used to store stories, read stories, feed/page fetch histories, and proxied sites.

* PostgreSQL: Relational database, used to store feeds, subscriptions, and user accounts.

* jQuery: Cross-browser compliant JavaScript code. IE works without effort.

* Underscore.js: Functional programming for JavaScript. Indispensible.

* And many miscellaneous jQuery Plugins: Everything from resizable layouts, to progress bars, sortables, date handling, colors, corners, JSON, animations. See the complete list on NewsBlur's GitHub Javascript folder: http://github.com/samuelclay/NewsBlur/tree/master/media/js/

Very classy acknowledgement.

Nicely made... shall have to look into it further.

First bug I noticed, most easily visible with Daring Fireball: click on a story from the bottom list. If there's more than one story headline in view, it also marks the bottom one as read.

Hi Samuel,

I noticed a small typo on http://samuelclay.com. I think you mean "Written with Django...".

Hah, that's ridiculous. Fixed and deployed. Thanks!

A really cool idea, though two points:

1. Can you rate individual posts (I'm at work so I can't spend too much time on it right now) instead of just tags/authors? It wouldn't feel like that much work to me as a user. You could then just read the tags/authors from the rated posts and sort them accordingly.

2. I was searching for a way to close the training frame, but was confused by the gray "Cancel"-button. Gray generally means disabled. I understand that you want to highlight the "Save and Next"-button. I would just change the "Cancel"-button into a link, to make it more obvious that you can click it.

4 very rare things just happened:

1) An application announcement on HN that genuinely appealed to me.

2) A single guy beat Google at UX in one of their best web applications (exposing the original site design for the feed is a fantastic idea!)

3) I paid for an application before it's ready.

4) I felt this was all so important I logged in to post this comment. :)

Kudos, you've done a fantastic job here. I really hope your environs permit you to keep this project going - it's a great idea, and I'm in love with it already!

One of the major problems I have with Reader is that certain low output feeds (making up the majority of my subscriptions) just seem to blend together - I forget why I subscribed to them, and have to go back to the original web page to try and figure out who the person is. Exposing the original page by default solves this beautifully. :)

Great work!

Edit: something I forgot. Visual display of the site allows me to instantly demonstrate RSS to my mother. Google Reader confused the hell out of her. :)

Very impressive, the UI stuff has clearly been thought about. At the same time, I found it a bit confusing. Also, landing straight in the app is cool, but it doesn't tell me how it's different, so I don't know why I would spend the time to figure it out.

It seems to have a lot of mysterious features like "instafetch" and "intelligence trainer" and stuff.

I guess I'm saying that my first impression is that it looks nice but complicated, like there's the promise of great stuff there but it seems like it would be hard to figure out. Compare with Google in the early days: great technology, crazy fast, but all that tech is hidden in the background and you don't see that, you just see the search box and the results.

Perhaps you've been too enamored with the cool/powerful features and haven't focused enough on keeping it simple?

But definitely awesome!

As a front-end engineer who used to work on Bloglines and helped create the Bloglines Beta, I say good luck! Your UI is very much like the Bloglines Beta and I realize this probably took a lot of work. Your challenges will probably be maintaining read state, security issues with trusting content in feeds, and keeping crawlers running so you don't miss items, searching content, handling archives, and more!

And of course, there's no money in this and the use of feeds are declining (so integrate Twitter and Facebook API).

Anyways, very nice. :)

Thoughts:

* FiveThirtyEight breaks your frames.

* I want a listing of all my unread articles to browse... I have >600 feeds and the list of blogs is long at just 64

* All NYTimes blogs seem broken? Freakonomics is super slow. An important feature of Google Reader is its aggressive pre-loading. When I click a headline, the feedback is instant. This keeps me reading (probably more than I should. Which is good for you, the feed reader seller.)

* On APOD, I just get a blinking cursor in the bottom window, and "original" mode works but "feed" and "story" don't seem to?

At this point, I'm just going to chalk it up to your site having issues, and check back again? http://img.skitch.com/20101026-tte8pqeksjx94syhuh7cbkt418.jp...

I really like your idea and will probably purchase it if it starts working. I'd also be happy to be a beta tester/constructive criticism provider/example of a heavy feed reader user.

Wow, thanks for the great bugs. So, the site isn't having any issues that it wasn't already having (bugs, the usual). What you're seeing is oddities in how the iframe is working. It's not a perfect process and I'm trying to figure out better ways to encapsulate the original site.

As for speed, I watch my logs like a hawk and have graphs for how each feed performs. No feed should take more than 500ms to load (and only a very very tiny percentage have taken more, based on my graphs from today. And nothing took more than 1.5s). Speed is a super high priority for me. Moreso than many features. Preloading is hard to do. I have no idea what you want to click on.

The iframe buster buster code works about 95% of the time. It should catch most NYTimes sites and switch you automatically to the Feed view. It doesn't always work.

Fixes for these bugs will be coming in the next few weeks as I sit down and identify the culprits.

You should still go premium, even if there are a few bugs still lurking. Check the features board. I have updates regularly and have for months. It's a long road ahead and things only get better from here.

Totally unrelated:

On the front page, there's no big "go premium!" button. When I went to the site to subscribe, I said "oh, um, where do I do that again?" and the answer was that I click on "choose your 64", which then gives me the option to subscribe.

It should be a big honking button!

Good call on making preset feeds loaded as soon as you open the page, lets you test it out right away. Instant engagement.

Also, it uses Google Reader shortcuts + imports feeds. Making the transition easy.

Excellent way to present a product versus a strong competitor.

I agree, but I feel like there should be some fill in the middle of the page. When I first clicked on it, there was a big white space in the middle. Maybe a carousel of prefetched articles would work?

Google Reader import ftw. I would have just looked and kept moving if I hadn't spotted that on the right.

Well played sir, well played.

That being said, you should fill in the email address when I need to create an account. Maybe make it greyed out since it is optional, but if I'm giving you access to do the import, I'm likely going to tie it to my email address.

Great idea. Noted for later when the server isn't being hammered. Oh my god is it being hammered.

Very nice app, and as with the parent, Google Reader import got me to try it.

However, I was put off by having to choose 64 feeds to "turn on" immediately after the import procedure. It meant my first experience of NewsBlur was the difficult and uninteresting bookkeeping task of figuring out which of my few hundred feeds I really, really cared about, and I pretty much disengaged from the app at that point. If I didn't have to make that choice right then, my first experience would instead have been simply my familiar feeds in a (rather nice) new UI - and at your current price point that would probably have been enough for me to buy a premium account.

(To give more context - I subscribe to lots of feeds, and rarely unsubscribe except from really noisy feeds, so I probably have a lot of dormant feeds sitting in Google Reader. I certainly don't recognise by name all of my feeds, or which ones I'd choose to keep if I had to choose 64. The autoselect by popularity is a good start, but for me isn't a good match with the feeds I actually get most value out of.)

I fully understand why you'd limit free accounts - particularly with your servers on fire :) - but I would suggest imposing the 64 feed limit after an initial trial period, rather than immediately, so people get that initial great experience.

Well, NewsBlur has auto-selected your 64 most popular feeds for you. All you have to do is hit the big green button that says "Turn Them On". This makes it obvious that the premium version is the way to go for your use case. It's very low-barrier and gives the user a conceptual model of what eventually has to happen for them to get more feeds.

Maybe I gave the wrong impression - I thought the UI for choosing feeds was well implemented, but having to choose is itself laborious, however pleasant the UI for doing it. (And as I mentioned above, "most popular" was not in my case a helpful heuristic.)

Put another way - if I had built this, the main reason I would have included Google Reader import in the free version would be so people could experience the value of my app without having to think - about what blogs to subscribe to, or what existing subscriptions to keep. Putting the paywall at this point in the first user experience removes that benefit in cases like mine. I might think about the benefits of your app and come back and upgrade - or I might forget - but you could have had me as an impulse buy.

I see your point crystal clear. But what happens when I don't make you choose a max of 64 feeds? I still have to fetch 100% of your other feeds, which could be in the hundreds. And I have to do this for every single user, which is currently in the thousands (ohh, my poor server). It just doesn't scale with my economics. If I were Google, sure, I could just throw money at more machines. As it is, I'm stuffing envelopes full of cash and sending it to Linode.

Ideally, I'd make all feeds free and updated every 5 minutes. Realistically, if I force you to choose, I can perform that much better for everybody else. It's a trade-off which I try to stack in my favor by giving the user an easy-to-use UI to activate different feeds as they choose under the standard account.

You could pay $12 a year and get unlimited feeds. That would make both of us very happy. :-)

You can make this much easier on yourself.

First, just give them 64 feeds and tell them about it on some form of status bar. The "updating your feeds 63/64" progress bar, but maybe yellow with "We're only updating 64 of your feeds. [a href="why-premium/"]Why?[/a]"

Also, you can choose those 64 feeds to update as a union of what they have subscribed to and what you're currently already having to fetch, thereby making the necessary amount of work for each new user theoretically less in the best case, and no worse in the worst case.

I think you should "shoot first, ask forgiveness later". Select just the 64 most popular feeds for the user, and then put a big button on top of their feed list saying "Limited to 64 feeds... see everything for only $12", or some better copy along those lines.

That is, I should go "wow!" before "awww, free is only 64", instead of the other way around.

> That is, I should go "wow!" before "awww, free is only 64", instead of the other way around.

Exactly what I was trying to say; you put it better than I did. (And your suggested fix would have worked for me too.)

This is great. One comment: I'm not sure why the landing page has a blank central column taking over 1/3 of the space with nothing. Apparently it never fills, what am I missing?

This looks great and releasing the source is certainly very generous of you. Definitely inspiring.

edit: I'm totally cloning this repo in case you change your mind :P

I mean, why not make it open-source? It's half-resume, and half-premium accounts for a nice service.

Okay sir, you've raked in 166 points on HN (33.2/hour), we expect a full emblogification on what happens next with this/you.

I'm going to be curious mainly about what kind of spikes you get in your traffic, paid accounts, and leads to other publicity.

Well, I have a plethora of munin graphs, about 26 new premium accounts (I need 80 to reach profitability), and my app server barely broke a sweat. (My task server, on the other hand, is only juuuust finishing fetching all of those new feed subscriptions).

Next week, after the traffic and love die down a bit, I'll write up a post-mortem with any advice I can scrounge from the effects.

Some great ideas, but still missing smart deduping. This is a real pain when having master feeds and aggregators that might also include those feeds: you may see the same story on the same site multiple times as each aggregation displays it. It would be nice if a tool recognized that it's been read and marks it as read in all the aggregation feeds someone might also have in the same folder.

You're right. But this feature is also completely out-of-scope for the type of reader I'm making. Maybe one day, if I'm sitting on a pile of money and have more time than I know what to do with, I may enter into the aggregation aggregation business, but it doesn't sound fun. I think a very small percentage of users want aggregation aggregation, whereas most folks are looking to keep up to date with an ever increasing number of writers.

I can tell you made some very conscious choices about how to position the premium version of this tool, but from my perspective I think you've erred too far on the side of caution. I would recommend 3 changes that would keep the spirit of "not in your face" premium pushing, and yet increase the take-rate on paid customers.

* Be up-front (on the landing page) about what you get for free and simply allude to the fact that a power-user option is available

* The dashboard needs to know if I'm using this enough to warrant a premium account. On a dashboard note: once I've clicked a feed, I need an easy path back to the dashboard.

* I need to be able to broadcast somehow that I'm using this. If I "like" a story, why can't I have a goo.gl shortened url to the story wrapped by NewsBlur post to my twitter/facebook??

Great work overall!!

That dashboard idea is priority #1 (next to finishing up the iphone app). But I want it to be a really swell dashboard. Graphs, relevant data, not just the server puking aggregation on you.

I'm hesitant to go overboard on branding and pushing premium. Notice the logos nowhere to be found when you use the site. It's like a desktop app for the web.

And I'm also hesitant to wrap content in URLs like that. I don't know, shouldn't the URL go directly to the original writer?

Wow, awesome. This looks very very slick, and the Google Reader direct import is just...nice!

Bug report, though: A couple of my blogs that I chose to "turn on" only show up as their comments feed. If I go ahead and turn them on, and click on the feed itself in the left pane, I get the site (fair enough), but the list of items in the bottom pane then only shows up the comments feed items. And clicking on them takes me to the comments. weird.

Haven't been able to figure out why. E.g. the XKCD blag (http://blag.xkcd.com) and Cheap Talk (http://cheeptalk.wordpress.com) feeds.

p.s: when this bug is fixed, and archiving support added, I'm totally premium-ing - hold me to it! Great work once again!

You've done a really good job here. A news reader that lets me both rate news, theoretically learns my preferences over time, and lets me read the entire article without clicking more than once, is a serious package. (Why can't other readers get those things right?) My only problem is I'd prefer a native reader app to a web-based one, just because the major upside of using a reader is that it's constantly fetching and storing my feeds so there's no wait when I want to go read some news. I may check out your iPhone app though.

If you want constant fetching, you want a premium account. Premium accounts get updated 10x as often, and allow an unlimited number of subscriptions.

The iPhone app is not submitted yet. Feel free to play with it: http://github.com/samuelclay/NewsBlur/tree/master/media/ipho.... I still have a few screens to make, and a ton of bugs to sort through. Maybe January?

How has Django been working out for you?

Well, the site is very front-end JavaScript heavy, so most of my time is spent on JavaScript. For the most part, Django has been great. I grew out of many relational SQL-based models and moved most of the key-value type data to mongo. (Stories, classifiers, feed load histories are all in mongo. Feeds, subscriptions, user accounts are still in postgresql.) And the feed fetcher is all custom code to run distributed and consistent.

Django has stayed out of my way and that's been nice. I've used Django on other projects (specifically Storybird: http://storybird.com, an collaborative storytelling startup) and it's been the moon for us. Absolutely incredible. And I write RoR at DocumentCloud and it too has been phenomenal. They're both great. Choose the language, not the framework.

Congrats on going from 'half-baked' to launch in 3 days!

Now that you've launched, can you take a look at the Hacker News 100 feed (http://feeds.feedburner.com/newsyc100)?

Also, I don't know if this is intentional, but the feed view for a subreddit shows the subreddit page, and each item in the feed just scrolls through that page (instead of taking me to the comments of each link)

Maybe I've allowed Dave Winer undue influence over me in this regard, but I can't see giving up Google Reader's All Items "river of news" view. Are there plans to offer this presentation style in NewsBlur for those of us who'd rather not visit the original sites? It doesn't seem incompatible with the AI management feature, especially if you can do better than Reader's "sort by magic" setting.

How are you handling sites with frameuster scripts ?

Handling iFrame busters? By using iFrame buster busters, of course!

When I detect an iframe buster, I prevent it from doing its job, but instead of continuing to show the original site, I switch over to the feed view. It complies with the wishes of the parent site, and I don't look like a bozo with such a glaring and awful bug. The feed view is still a great view. (It's the Google Reader view.)

That's awesome. That's perfect tradeoff between the wishes of the parent website and convenience for your users.

I'm assuming you fetch feeds using a queue.

* How do you add jobs to the queue ? Do you run a cron every n minutes to check which feeds need fetching and add them to queue ?

* When you add jobs do you check to see if a previous job for fetching the queue is still in the queue or do you use flags to mark feeds for which the jobs are in queue and eliminate duplicate jobs that way ?

Yes, I use celery/rabbitmq as a queue. But I don't bother checking for duplicates. If the feed is still in the queue by the time it comes back around for processing again, I let it run. That means that the queue is behind enough that it will even out in the wash.

So far so good. But when I click on a story there, it loads up that story -- and blows away NewsBlur. I can navigate back, but now I've lost my place.

In such cases, maybe you could doctor the links so that they'll open in a separate tab?

I'll make that a preference in the next couple weeks. That feature has been requested more times than I can count.

Just an FYI: my blog (see profile) seems to have busted your iframe. :) The code I'm using (not included in <head>):

if(top.location!=self.location)top.location.replace(self.location);

For what it's worth, it's "diggbar"-style garbage that I added that to thwart, and as much as I like your interface, I'm going to be leaving the iframe-buster in place.

Edit: it looks like A List Apart's feed also breaks, likely for a similar reason.

And I'll have to admit, the more I'm using this, the more I'm liking it.

So, I just signed up, been planning to write exactly this application for a few months now, but realized that I have to learn a lot more about neural networks to pull this off. You saved me a lot of time and effort.

Faced with the limit of 64 feeds, I even signed up to the paid version, but now it still only allows me to import 64 feeds.

Also, is an HTTPS version coming?

You can actually import any number of feeds you like, evne with the free version. The "Choose Your 64" dialog allows you to mix and match which ones you want turned on. Once you're a premium user, all of your feeds turn on automatically. The home screen should say you're a Premium user (and the house should be in color.)

And SSL is a bit of a ways off. I'm paying for the server out of pocket. I still need 62 more premium subscribers to be able to pay for the hosting fees (app server, db server, and task/fetch server).

Very nice so far. Seems to be stalled on the GR import. I like the UI so far. I don't know how much of the learning I'll make use of -- I usually only follow feeds where I want to read most everything, so further discrimination isn't really helpful to me, but having a good UX on a feed reader that isn't Google's makes me happy.

Yeah, there's over 10,000 feeds in the queue in front of all new users right now. I fixed a bug which was causing all new feeds to queue, instead of just new feeds that the user is activating. (New feed = feed never before fetched. So most feeds have been seen more). That 10,000 would've been a 2,000, but it still would've been too much in a single hour to handle. (That's 10,000 OVER the 14,000 feeds beings fetched already, so it's catching up quickly.)

Give it a few hours and it'll catch up and all will be well.

Seems to have mostly caught up. Very good. The only features I would add (you might not -- it's your app after all) are the ability to default to Feed view for all feeds, and an 'all unread' bucket sorted by inverse posting time. Otherwise, it's well ready to take over from GR for me.

Yup, soon. Those are going to be preferences and enough folks have requested it that it will make it in within the next couple of weeks.

This looks very cool, but can't replace GR(for me) for two reasons:

- The whole point of GR is I can read the original site even when it's not available - works nicely with restrictive filters etc.

- It's slower than GR - for obvious reasons.

However, it's great to see somebody taking a stab at an alternative news UI. Good luck on your project.

Interesting the fact that is definitely going again the trend of extracting content from websites to present it in a more readable format (instapaper, safari reader). I like the general interface, but I'm not a big fan of having to ready my rss crowded with ads and banners.

Nice thing is that you have the option to only read the Feed instead of the Original Site. By clicking on 'Feed', you get the same treatment as Google Reader. And NewsBlur persists your preference.

I am on a bit of an anti-Apple binge over the last week on HN, but I will point out that Safari Reader, like most Apple "innovations", occurred somewhere else first, in the Readability extension for Firefox and Chrome released almost a year before SR that thousands of people were using, which was being recommended non-stop on Reddit and HN. Can we please not let another mythology grow up around Safari Reader as with all the other technology which Apple appropriates.[1]

[1]http://news.ycombinator.com/item?id=1812967

I don't use Google Reader, so I would love to have an option to import OPML files. Edit: Scratch that. Found the option I was looking for. But, don't like that uploading an OPML resets existing feed list. It should ideally check for duplicates and add the new entries.

You can import OPML! When you login, just click on the green + button in the bottom-left. (There's a callout that says, "First things first..." right above it). There you can easily import OPML.

Yeah. Spotted it just after posting the comment (see above)

Yes, I know it should look for duplicates, with an option to erase. Unfortunately, this was too low on the priority list. But it's sitting there, waiting to be implemented sometime in the next few months.

Have you thought about using Google Reader as a backend instead of keeping your own database of feeds? It seems like that would cut down on your processing of the feeds and enable you to just work on the stuff you care about (the filtering and UI aspects of it).

Yes, I have thought about that. (Ohh, how many times I wanted to switch to this while developing.) But that means I would lose a whole lot of control, as well as the ability to perform real aggregation and intelligence on those feeds.

That, and I get to build a full-on distributed feed fetching and processing service, which has been a brutal, expensive, and wholly worthwhile experience.

Looks amazing to me, don't know yet if I'll be using it since I kinda prefer desktops apps, but I'm definitely going to try it for a while, and that's something that I almost never do regarding rss apps. Overall great great job. Best thing I've seen all month.

This UI looks very similar to MS outlook and I really it.However unlike outlook you cannot have all 3 panes vertically.On large screen ability display data into 3 vertical panes makes it more usable. are you planning to add this feature sometime in future?

-Why is there a "optionnal" label besides the password field?

-I really liked the little arrow, that we can lock on a page and allow us to create a bookmark like. It's innovative.

-It's really great that you released the source to the community!

Good luck!

I suspect the learning is not needed, I like the Google Reader learning because it's so straight forward. (what I read, I like)

I love the reading experience you've designed, though.

Awesome site! Upvoted.

Signed up and imported my Google Reader account, but some feeds are taking forever to load. Some other feeds don't show up at all after a while.

As an inveterate Google Reader user, I say well done! One problem: the "Preferences" dialog has no text, just two blank radio buttons (using Chrome).

Future features. They say ship while you're still embarrassed, right? Preferences are a high priority and will find their way in within the next couple weeks.

I get this error when I try to Google Reader import: http://grab.by/73TI

:-( I've seen that before and haven't been able to figure out why. I'm definitely passing the scope parameter. (scope=reader, or something like that).

After the hailstorm of traffic dies down, I'll look into this error. Thanks for the report.

If this ever takes off, you might be able to feature other blogs' feed on the demo page for a fee.

Says that an iPhone app is in the pipeline. Once that is done, please make one for Android also.

very cool! love the interface. just signed up and trying to import from google reader. fetching feed is happening but very slowly. hopefully its just because the server is overloaded temporarily.

Is it possible to: - change password - delete account

Heh, soon. Those are lesser used features that just haven't seen the light of day due to higher priority features. They will happen soon, though.

Very nice concept and UI ... great job!

One gripe: When you click an article it should open in a "_blank" window/tab. I hate having to click back.

This will be a preference within the next couple weeks. Lots of other priorities, but this one is definitely high up there.

Great stuff

Reply on news.ycombinator.com