Extended Outage at Instapaper | Hacker News Reader

Extended Outage at Instapaper

blog.instapaper.com

58 pointslenova22 comments

From the article :

After spending multiple hours on the phone with our cloud service provider, it appears we hit a system limit for our hosted database that’s preventing new articles from being saved. At this time, our only option is to export all data from our old database and import it into a new one.

Seems pretty bad that it takes hours to discover that. Anyone knows what cloud it is?

instapaper.com seems to be hosted @ AWS

Which DB at AWS:

https://aws.amazon.com/products/databases/

Also shouldn't AWS warn them that this was about to happen?

Edit 1:

DynamoDB says: "There is no practical limit on a table's size. Tables are unconstrained in terms of the number of items or the number of bytes."

The SQL databases say this: "The Amazon Aurora engine will automatically grow the size of your database volume as your database storage needs grow, up to a maximum of 64 TB or a maximum you define. The MySQL, MariaDB, Oracle, and PostgreSQL engines allow you to provision up to 6 TB of storage on-the-fly with zero downtime."

So they either hit 64TB or 6TB of data I assume?

EDIT 2:

According to this they are MySQL based: https://medium.com/making-instapaper/rebuilding-instapaper-s...

Think you need to set up CloudWatch to get alerts.

And CloudWatch is free for RDS at 5-minute collection intervals which is more than adequate for alerting on things such as "Am I about to hit the space limit on my database?"

They could be on aurora though since many people just consider it to be mysql. It's a pretty painless migration from mysql as long as you are 5.6 or below. You can just replicate over to it live.

Post-mortem: https://medium.com/making-instapaper/instapaper-outage-cause...

Very odd and surprising it's taking that long to recover. I got the following email from them this morning which seems to indicate they will only be fully normal a week from now.

As we shared earlier today, Instapaper is experiencing an extended outage. After 31 hours of downtime, we were able to rebuild a database instance to get Instapaper back online! In the interest of coming back up as soon as possible, this instance only has the last six weeks of articles. For now, anything you’ve saved since December 20, 2016 is accessible. We’re working on getting the rest of your articles restored, but wanted to give you access to your most recently saved articles in the meantime. Again, there hasn’t been any data loss. The full database exports are taking much longer than anticipated. We’re aiming to get the full archives available by next Friday, February 17 at the latest. We appreciate your patience and understanding in the interim. If you've got any questions or thoughts, just reply to this email or let us know at @InstapaperHelp on Twitter. - Instapaper Team

I'd be interested to know what services like Instapaper are used by HN members.

Personally I've always wanted to like Instapaper more. I really do. I've given it a few extended runs, on it's own, and parallel to Pocket; but I always end up moving back to Pocket.

I recently switched from Pocket to a self-hosted instance of Wallabag (https://wallabag.org) and I've been very happy with it.

One thing I've noticed with these types of services is that they quickly start becoming intrusive and annoying. Ads are the obvious annoyance. But as an additional example, the Pocket extension injects code into certain sites (HackerNews being one) so that I can "more easily" "pocket" posted articles directly from the listing page itself. And in fairness it can be disabled, but it's enabled by default. And with things like this, I just find that simpler is better. I don't need all these extra features being pushed and modifying my browsing experience "by default".

I had some issues getting the Wallabag app working on a smartphone. Have you used that successfully?

I've had no problems at all.

I use it primarily on my Android tablet (Nvidia Shield K1), and occasionally on my phone (also Android, Moto X Pure 2015). I haven't noticed any problems. I use it for both reading and adding using the built-in sharing functionality in Android. For setup I simply set my URL, username, and password in the settings and used the "Autofill" option to handle populating the rest of the settings, and it works great. I'm running the server itself in a Docker container, with connections proxied through Nginx (also in a Docker container).

Hey dangoldin,

Don't hesitate to ping us on gitter.im/wallabag/wallabag if you still have any problems.

Thank you (I'm in the wallabag core team).

About "the Pocket extension injects code into certain sites", can you give me more details please?

What do you prefer with Pocket?

I've been an Instapaper user more or less since it launched. I have tried other services (like Pocket), but I've found their parsers inferior to Instapaper. Instapaper also seems to have focused more on the experience of reading, more than "supporting all kind of content". With the Instapaper plugin in Chrome it's quick to add articles to the reading list from twitter/HN/any page.

Two things on my wish list:

- Let me download my full history from their API and not just the last 500 articles

- Let me sort the reading list in their app on "last opened"

I'm using Instapaper exclusively due to their "download to .mobi" functionality.

I'm using it to collect longer texts (through a bookmarklet in my browser) that I can then later read off-line on my Kindle.

> I'm using Instapaper exclusively due to their "download to .mobi" functionality

Like wallabag does. And you can also export into pdf, csv, txt, epub, json, xml.

I used it a lot, both for the list and just the Readability-like feature until browsers added their own reading mode/reading list functionality. (First proposed by Mozilla, first implemented in Safari — like the download widget ;-) — and now also in Edge.) (Only Chrome is lacking, but I'm not too surprised an ad company doesn't add it to their browser.) Now I use Firefox's reader mode and synced bookmarks.

I use Pocket for the sole reason that there is an interface on my ereader to sync pocket articles. (Kobo Aura HD)

If there was the same for Instapaper I would jump. I find it generally does a much better parsing job. Pocket occasionally completely borks on an article and will just redirect you to the original. Other times it often misses bullet point lists (probably thinks they are navigation). Whenever this happens Instapaper almost always handles the same article with aplomb.

Aside: Let me take the opportunity to plug a little app I wrote called Erudite[1] that can download articles from Instapaper and Pocket, convert them to an appropriate format such as EPUB or MOBI, and add them to your Calibre ebook library. It is a command-line driven tool.

[1] http://evanmclean.com/software/erudite/ (Note that some of the documentation on the web-site is out of date. Doesn't mention Pocket and still mentions Readability, but the doco in the download is up to date.)

Wow, that's actually a very neat little app. I'll be using it for sure, since I switch between a laptop, phone, a tablet, and an old 6" kindle.

I found that Instapaper occasionally stripped out images when they should have been left in the article. I guess a lot of it comes down to how consistently the pages themselves are coded.

I'm guessing they mean space limit on a single DB, you would think a service like Instapaper would have some monitoring in place by now.

Interesting to compare both the different approaches taken in sharing details by GitLab vs. Instapaper, and the long-term reaction to it here.

The backlash against GitLab was obvious; I'll be watching the next Instapaper disucssion with interest.

https://news.ycombinator.com/item?id=13608267 etc.

Reply on news.ycombinator.com