Go read: fast Google Reader clone in AngularJS with Go on App Engine | Hacker News Reader

Go read: fast Google Reader clone in AngularJS with Go on App Engine

mattjibson.com

279 pointsmjibson119 comments

The funny thing about feed readers is that we actually only care about the UI, not the underlying feed fetching and parsing and whatever else happens on the server.

So, when I made my feed reader — this has got to be the most popular pastime of the last few months around here — I ended up taking the lazy approach and made a UI that sat on top of the NewsBlur API. Thus, I now have a NewsBlur account, all the messy date parsing and everything else gets done on NewsBlur's servers, and I access it through my own much prettier interface that works just how I want it to work (http://www.altfeedreader.com). For anybody still thinking about making a feed reader, I definitely recommend this approach.

Looks nice! On your landing page I'd recommend auto-cycling through the features on a slow timer until mouse-over.

It's not really funny — because when you actually do care about the underlying feed parsing, you end up searching in vain for a replacement among readers that are, as you say, lazy prettification on top of bad fetch/parse engines.

Google Reader had become amazing at getting updates virtually as soon as they were published. I've been comparing how well the competitors do, and some of them are more than 30 minutes later than Reader at finding and showing stories.

This is likely because Google had such a large userbase that they were able to update more people's feeds at the same time than a lot of other places like newsblur and feedly can right now.

I also imagine it's because google was requesting the feeds more often than others, or leveraging the feedburner system to know when they were updated and other such things.

Google Reader implemented PuSH:Pubsubhubbub (worst protocol name ever!). Most blog platforms ping using PuSH, so the update needs no polling and is thus almost instantaneous.

If anyone is looking to play around with PuSH, http://superfeedr.com/ has been a joy to work with. There's also a reference implementation available at https://pubsubhubbub.appspot.com/.

It handles all the "dirty" parts of feed reading (checking for new items, normalizing fields) and adds near instant updates for sites that ping hubs.

When I finally get around to writing my own feed reader, I definitely plan on using it.

Does anyone know which of the new replacement readers, feebly etc. support pubsubhubbub?

This is a question I have been wanting an answer myself for a while. Especially I am interested if Feedly, Newsblur, Inoreader, is using PUSH method

Some protocol developer had a sick sense of humor

You're welcome. :)

Neat. I had no idea that such a protocol existed let alone that it'd have such a ... colorful ... name that makes it hard to find :)

I wonder how hard this would be to get it implemented in some of the other readers.

The Google Reader blog (back in 2005!) had a great post detailing many of the XML validation errors their feed crawler found and must workaround:

http://googlereader.blogspot.com/2005/12/xml-errors-in-feeds...

The funny thing about Hacker News (and most other places) is that if you explain everything you end up writing a tome that nobody ends up reading.

That is to say, I completely agree with you. Google Reader had an excellent backend (being used by 90%+ of people meant there was a virtuous cycle happening), and I especially noticed this when I developed a prototype backend that I never actually used. Deciding how frequently to check feeds is complicated.

In the end, given that this is a side project, I'm never going to be able to compete with the big players, especially at scale, and to be frank, I don't find this side of it as interesting or enjoyable to develop.

The good news is that this frontend could be modified to work with another backend if there was a good reason to do so.

A lot of the major feeds out there don't even return valid RSS or atom. So, the problem begins when you have to start adding heuristics to your parser just to get it right.

Looks nice a few bugs when trying out with IE 10 though. Filters at the top don't work. Also a way to scope down to only a single RSS source would be nice.

It seems to work fine for me in IE10... I'm not sure what's going on there.

Also, it is possible to view the feed for a single source: clicking on a feed label in the subscriptions list shows just the items from that feed. If you have folders (like there are on the "try" page), clicking on the folder label shows items from all feeds inside the folder, but clicking on the folder icon expands the folder.

hmmm odd didn't for me. works on IE 11 though. Nice work. Love the layout. I might switch to newsblur just because of this :)

That's actually brilliant. I've been waiting for someone to come out with a UI like this.

Not sure how true the UI concern is. I finally got around to setting up emacs to fetch rss stuff through gwene.org. I suppose I'll miss the social aspect of reader, but I'm not sure.

Now... what I do like is having something that is very very easy to control with a keyboard. Even better, it plays well with muscle memory that I've already been building up. So... if that is included in the UI department, I guess I'm ultimately agreeing.

Yes, we agree — the graphical interface is just part of the interface that is presented to the user. It's a shame UI often means GUI.

Makes sense. And, honestly, I should know better. I just have gotten kind of annoyed with the graphical obsession that most sites have taken to getting this sort of stuff running.

The single most annoying part of switching solutions has always been modifications to my interactions, more so than any glitz. I think my defence against worrying about what different icons did/meant was just learning how to type what I wanted. Sorta sad it took me as long as it did to get to that level of using the computer. (I initially rejected vim/emacs because of how difficult they felt...)

Reader and gmail had hit a sweet spot of playing off my muscle memory rather well. To the point that I honestly forget about all of the extra graphical stuff they add around what I'm focusing on.

Anyway... feels like I should give this area a run. I think I agree with you regarding the backend stuff. I would probably just sit on top of gwene.org. But regardless of the source, making your own backend seems... unnecessary.

Love the way this looks. Do you have any support for keyboard shortcuts?

There's j/k for next/previous, but that's it. (Oh, and escape hides things.) Other than that, not yet.

I should have mentioned that I've also released the source code: https://github.com/davidjohnstone/alt

Oops, j/k didn't work for me because I had Vimium enabled.

facepalm

Thank you for open sourcing this. I've been itching to dive into some production ready Go code, and I've got a lot of interest in Angular.

I second this. I'm especially interested in how other people model their data for App Engine's store.

Did you have any difficulties there?

Yes. The data modeling was by far the hardest part of the app. If you look in the repo you'll see my first attempt was using a separate index table, which was convenient, but expensive. The current data model is quite nice in that the number of users doesn't matter - we don't keep track of who subscribes to which feed. But because of that, when getting the unread story list, we must do 1+N queries (where N is the number of new feeds). However, since we're using go and the datastore, all that work can be done in parallel, so it's not too slow.

I'd be very interested in reading a blog post about the data modeling process, especially if you find yourself changing it further as you receive more traffic.

> However, since we're using go and the datastore, all that work can be done in parallel, so it's not too slow.

I thought we couldn't run routines in parellel yet. Per:

'The Go runtime environment for App Engine provides full support for goroutines, but not for parallel execution: goroutines are scheduled onto a single operating system thread. This single-thread restriction may be lifted in future versions. Multiple requests may be handled concurrently by a given instance.' [1]

[1] https://developers.google.com/appengine/docs/go/

That description is misleading. All it means is that one go routine is executing at the same time between all requests. Up to 10 requests can be processing on any go instance. However, most of the time the go routines are just waiting for data to come back. Hence, we can fire up a bunch of go routines to do some work. It's the parallel vs concurrent stuff that Rob Pike has spoke about before. go app engine isn't parallel, but it is concurrent.

Looks good and simple. But I am still sticking with InoReader. Even though Feed readers should be simple but they should still provide enough bells and whistles for the people to adjust settings (look and feel) according to their own taste and not the taste of the developer. Below I see a comment which suggest that you only prefer to use Google to sign in. Well, most people won't agree and they'd rather you provide your own sign on system. In any case, the same logic works for UI adjustments etc. For all these reasons, InoReader by far is the closest you can get to Google Reader replacement.

PS: I have only recently discovered InoReader and I have no stake in it but I have been playing with many RSS readers as I am a long time RSS user of GR.

I don't care about downvotes but I can't honestly provide feedback as a user without comparing your products with other products.

Login-wise I feel exact opposite... I became quickly annoyed when I need to create yet another login. I prefer seeing "Login with Google"

http://www.mozilla.org/en-US/persona/

most people won't agree and they'd rather you provide your own sign on system

I don't, I have too many logins already. What are you basing this on, or is it just your gut feeling?

Another vote for Inoreader, its the closest usable interface to Google Reader IMHO, and I am satisfied at this point for image feed viewing and text/blog feeds etc. Really user friendly interface.

Not a Google Reader user (i used iGoogle until they discontinued it on smartphones...), but it's really a great site, congrats. I'll probably go get a look at the source since i'm an angular and app engine user (and not really serious with go...yet).

One small question : is there a reason why you need to go to the top right menu to add a new stream, instead of just a "+" at the bottom of the left pane list of feed ? Is that what google reader required ?

You can just press 'a'. And it's because I didn't have enough time to make it better, and that's the first place it went.

I actually like it, even though it seems to have fewer bells and whistles than Feedly, but I think it's even a bit faster, which is surprising, since most of the others readers I've tried have been slower than feedly.

I tried it, looks nice and works well.

One feature I miss from Google Reader is the ability to hide feeds that have no unread items from the sidebar completely.

I like that idea. I've made an issue for it:

https://github.com/mjibson/goread/issues/9

Just wanted to suggest the same.

Anotherwish of mine would be to not set a font size. It's all a bit too small for me. Sure, I can set a user-style, but it's easier if the site behaves right by itself.

One other thing: In the list of entries, the missing footer below an article is a bit irritating right now. Maybe it is just new, but I think it might be helpful to add a margin-bottom to .story-content.

Good work though, it is indeed fast.

Christ on a cracker. This is exactly what I've been hoping someone would make. Loaded all my feeds, without crashing. Keyboard shortcuts and working folders. And everything happens when you click on it, not 3 seconds later.

Where do I give you all my money?

Wow, this is great! Login with Google account, import data from Google Reader worked seamlessly, clean UI. Thanks!

Agreed, my only nit to pick is I can't seem to re-view old/read articles.

Click on a feed, then change the "X unread items" dropdown at the top to "all items".

This works, but it resets back to the 'no unread items' view if I reload the page.

Yeah, that's not saved as a preference yet. Some day...

The code is on GitHub; that would be a great pull request!

Good work on the UI - it has that clean responsive feel I really enjoyed about Google Reader. It took about 90 seconds for my 200 or so feeds to transfer from Goog. This is during the period this article is trending on HN - that is about an hour faster than Feedly managed when it was under load.

I just logged in with my two year old Samsung Nexus and it ground to a halt and took down Chrome. Hopefully the mobile experience is given a high priority - more and more of my feed consumption is doen mobile.

Overall Great work!

Thanks. I'm glad it's scaling - that was one of the goals. Yes, a high number of stories will perform poorly. Doing infinite scroll like google reader does is on my todo list, which will fix those problems.

First impression: Excellent. I see you got some inspiration from a certain Userscript and I applaud you for your choice. You managed to save lots of whitespace.

I have only one criticism with the UI. You removed the toolbar normally found in the end of each feed item. This is aesthetically pleasing but it introduces a couple of usability issues: 1) I sometimes accidentally click on an item and want to keep it unread, but the 'Mark as Unread' button is unavailable. 2) In full view, the only separator between feed items is a thin grey line. Please make it thicker and darker.

I have noticed a problem with all Readers that I hope you could address. Some XML's only show the latest item. If the reader doesn't update quickly enough, it would miss some items. I would appreciate it if you somehow made your reader update these problematic feeds more often.

Edit: When I click on an item to mark it as read, the only indication my click registered is that the unread count becomes reduced by one. Please show more prominent indication that the item is read.

If Reader parity is the goal then there seems to be a feature bug. Audio feeds like the New Yorker Comment Podcast and Science Magazine Podcast do not have audio elements in the feed. Though clicking on the title does bring up a download dialogue for the media file. Is this a feature that will be implemented in the near future? G-Reader was a great podcast tool.

Yes, I'd like to add that. I've added an issue for this:

https://github.com/mjibson/goread/issues/11

Very nice.. I've wanted a nontrivial example of a Go webapp -- the ones I write are strictly JSON API's that lurk behind our presentation UI.

You can see the actual dependencies used by hitting godoc.org:

http://godoc.org/github.com/mjibson/goread/goapp?imports

Actually building and running the thing, though, seems to be quite tricky - app engine forbids importing syscall, but half of the dependencies of this app seem to import syscall.

I think you're doing something wrong.

Are you using the Go App Engine SDK? Then you just run dev_appserver.py $DIR and it runs.

They might have stub source files with a `//+build !appengine` in them

Clean, fast. I like.

But I miss some shortcut keys. Especially SHIFT+J and +K to navigate among feeds. (Bonus points if you do it the GReader way -- keep moving the highlight among feeds until J or K is released, and only then refresh to show that last one as the new feed).

This is perfect. It was incredibly easy to move over from Google Reader.

If you decide to run some simple ads, I would be happy to pay for an ad-free version. I'd say this has potential to do a lot more than just pay for itself.

I would like to have a marker on the left pane to see which feed is selected.

Is this how Google Reader used to look like? I'm asking this because UI is very similar to CommaFeed (https://www.commafeed.com/)

Wow. I've never seen that. It does look uncannily familiar. Bootstrap will do that.

Love it! Signed up immediately, imported my Google Reader no problem, and so far very quick in use. I've tried a couple others over the last few months but this is, by far, my favorite. Bookmarked.

Thank you.

I've tried bunch of possible candidates to replace Reader and this is the first one that actually feels responsive enough for daily use.

And the fact that it's open source and written in Go is just icing on the cake.

Fast and simple, which is good, but it took me less than a minute to look at it and move on. Some of the key reasons:

* Impossible to reorder feeds on the fly just like in Google Reader. That's the #1 must-have feature for me.

* Inability to tag/label stories AND feeds.

* No integration with Instapaper and sharing services.

* Inability to star/like stories.

I couldn't care less about shortcuts and responsive design. Google made it easy to organize feeds and share/archive/search stories. That's what others have been unable to replicate so far.

Good luck! :)

Looking through your commit history, looks like you've been working very hard on Go read since you started in March. It looks and feels fantastic.

Thank you for going OSS with this project!

I checked out the source; what's the trick for getting the dependencies into the tree?

"go get" is not supported in appengine [1], and if I manually check out individual projects, I end up with extra code that will not compile in the appengine environment.

[1] https://groups.google.com/d/msg/golang-nuts/8BJ0fxrWZd4/PXyI...

Use your normal go install with a correct GOPATH set. App engine go will pick up on the GOPATH and use imports from there.

That worked, thanks!

"go get" is supported in App Engine, but only the download part, since the package will be built with your app (either through the dev_appserver, or when you upload it).

go get -d <package>

Thank you for the clarification, David.

I also had a problem getting the app to upload: "appcfg.py update" would hang during the build phase (may be Windows-specific): https://code.google.com/p/googleappengine/issues/detail?id=9...

Very nice. You mentioned the 'moving to folders' is coming soon? That would be great! Also - would it be better to make the add subscription easier to find? I was unsure where it was at first. Although I would use the keyboard shortcut 'a' now the average user may be inclined to look for an icon or something more obvious? Anyway - great job - just my 2 cents.

Yeah. I added a big button.

You know, of all the google reader clones, I haven't seen any implement my favorite feature, which is the 'gu' keyboard shortcut to start typing a feed name and 'enter' to go to that feed. Anyone know of readers that have this feature? Bonus points if it's a fuzzy search instead of an exact search.

NewsBlur has this, just hit 'g'. It's also fuzzy and allows you to make typos, yet it keeps the last useful results on screen.

Feedly has this (gg) in a semi-fuzzy way.

Am I really the only one who still goes to all of my favorite sites to read articles rather than having them in a feed? I had tried several times to see what all the fuss was about with Google Reader, and I just didn't get the enjoyment out of the presentation there compared to the sites themselves.

Depends on use case. If you have websites that only publish every couple of weeks (or several times a year), you want some sort of subscription to not miss what they have to say (e.g. new version announcement).

Good point, that is not something I had thought about.

> "...enjoyment out of the presentation..."

I think you nailed it there. I don't really care about the site presentation, just the content. It's interesting when the content that I'm reading in my feed reader mentions the "new design... working out the quirks" which obviously doesn't translate to the version that is on the reader.

I'm also lazy though... and often can't be bothered to even click on the "read more" when only the brief summary is contained in the syndicated version.

Well, one of the selling points of NewsBlur is showing the content with the design of the original site.

That said, most of my feeds use bland or generic themes. I really just care about their content.

Would there be a problem if Google just open sourced it's previous platform? They could open source the project and lots of developers could put their own take on the platform. It's probably simple to install on GAE (which they can monetize) and they wouldn't have to maintain the reader.

They said in the past that Reader used a lot of internal APIs to fetch, process and store the feeds, some of which may not be exposed even on GAE.

Please consider changing trigger point value for collapsing sidebar. Default Bootstrap trigger point value is an obstacle while browsing in full HD split-screen mode (two windows at once).

Feedly is also collapsing sidebar to early, this is a small thing, but causes a great deal of disruption in such use case.

Isn't the main lesson from Google Reader's being discontinued, not about UI or features, but rather about continued support? The main "feature" for a new reader should be it's ability and commitment to remain viable for the long term.

It's Open Source, so you're free to host it locally or go with anyone else who wants to pick it up and host it themselves.

I would probably first of all think the lesson is that whatever revenue it added for google it was too little to justify the maintenance of Reader.

For a much smaller company on the other hand it might actually be a lot of money.

It's Innovators dilemma kind of mutated.

Great work! I think this might finally be the one (and not a moment too soon!)

Is there any way to categorize new feeds? It imported my previous categories/folders just fine, but I don't see any to move new feeds to categories/folders.

Not yet, but soon.

I'm having trouble with the import from Google Reader. It was taking very long (>1h) and now it seems to have only imported a few feeds. I did have a lot of feeds, so maybe that is a problem?

Yes, having a lot or large feeds was buggy. That's been fixed just now.

Awesome, finally I'm not so scared of July 1st. Would be great if posts were marked as read on scrolling, without the need to click on them or hit the "space".

Am I the only person who uses g-u? It's been notoriously absent from almost all these "revival readers" I've tried, save feedly.

Hell yes! This is just what I needed. The other ones had strange UIs or were too slow for me. This was basically just as good as Google Reader!

I'm wondering what your experience was like with AngularJS? Pleasant or difficult? This seems like a perfect app for it. Congrats!

Pleasant. This is the 4th or so angular app I've written, so I've become somewhat familiar with it.

Love the simplicity, this is the perfect reader for me.

However, it seems down right now. Just getting "loading..." Anyone else?

Just reload. The datastore sometimes errors out and there's no retry logic on the client. It's also possible your feeds have triggered some new bug.

Nup. Tried diff browser/os. Let me know if I can help provide any info via DM.

I'm having the same problem as ffog and have tried several different browsers, reloading, clearing cache, etc.

Fast, clean and Simple with keyboard shortcuts - nice alternative to the other Reader replacements out there.

Nice. Very fast. Can't wait for sort by oldest first and ability to star/import starred items.

Excellent! In just few clicks i was able to migrate from google reader to this. Nice integration.

I'm seeing errors from users with giant feeds. I'll try to fix those within a day.

Looks great. One remark: links should really open in a new window (target=_blank).

Maybe I humbly suggest, or be pointed to the search function in this product?

No such feature yet. Sounds useful.

Search is the main feature that I find it extremely helpful with GReader. I usually search for something in my GReader, before going to the web search. This should be at the top of priority list.

Will test it when I can signup without depending on google.

Real nice. BTW, any idea how the HN RSS feed is ordered?

> The number of date formats we encountered is comical (so much so that I started a blog to document them). At current count, there are around 60 date formats I’ve seen.

Ha! Is that really not in the spec for Atom/RSS/RDF?

I believe it is in the spec, but just not followed.

But of course. Who approaches a problem like "I need to serialize dates into this standardized document format" and decides to go with "Let's concat strings in our own format" instead of going with "Let's check the spec and use someDateLibrary.dateToString('that fmt string')? (Or better, something like Go's time API, but that's beside the point)

I would use it if I didn't have to use a Google Account to sign in. Why not Mozilla Persona?

It's open source, if you care that much, swap it out yourself.

I have the opposite reaction: I only want to use my Google account to sign in. If a site doesn't have that, I'm less likely to use it.

Would you be more likely to use Persona if it acted as proxy to your gmail account, as it does on http://beta.123done.org?

I have no idea what persona is or what it's for. I'll look into it at some point and see if it can be integrated.

Reply on news.ycombinator.com