Heroku Postgres | Hacker News Reader

Heroku Postgres

postgres.heroku.com

132 pointsryandotsmith77 comments

This is really exciting - the team behind this really cares about developers and their experience. They see a way things could be better, and regardless of technical difficulty, say, "why not?" Forking a db? Brilliant. Offload expensive task to passive followers? Obvious (in hindsight). Instant, brain-dead db provisioning? Of course.

Personally, the only terrifying thing about this is that when Heroku releases something, it's been in the works for a good long while, and they usually have a stream of incredible releases waiting right behind it. The mind boggles at what they're going to be doing next.

But we'll be first in line for it :)

When I read "forking" and "following" on there, it was a revelations: of course, that's so obvious! Those terms are much friendlier than "replication" or "master-slave". It describes the desired result, instead of the underlying technology.

If you had any idea how long it took to get that language right...

I'd be curious to hear the alternatives, and the overall process used to arrive at the current language.

I seem to remember the term "manumission" being floated around during the beta process... :)

Seconded.

Hmm, this would make a good blog post. I'll draft something up.

Please do. There seems to be valuable lessons learnt there.

If there is anyone from heroku reading, please pay some money to http://www.sequelpro.com/ developers to develop postgresql compatibility. All GUIs available on OS X for postgres are horrible

I love using Sequel Pro for MySQL, but also wish it had support for SQLite and PostgreSQL. It's a great app and better than all the other OSX clients that I can find on the internet, including the non-free ones.

It's open source, so nothing's stopping anyone from adding support for other SQL servers.

Is Navicat OK? I've been using it for a while simply because it was one binary that did SQLite, MySQL, Oracle, MS-SQL, and PgSQL.

I've used Navicat Premium with Postgres and have had no issues. Pricey, but it works.

have you considered Navicat ? there is also a free version and it runs on windows, osx, linux

I have contributed a little bit to Sequel Pro, and I'm a bit familiar with the codebase. Adding support for PostgreSQL is not trivial, a lot of MySQL is hardcoded into the user interface. There have always been many requests for PostgreSQL compatibility, but it would require A LOT of refactoring.

(Disclaimer: I haven't been involved much with SP for almost a year now, so possibly there has been progress in that time)

As one of the current Sequel Pro developers I can backup jabkobob on this one, it's not an easy task. It's not that we're ignoring the requests we get to add support for PostgreSQL or any other database system for that matter, it's simply a matter of time, of which most of the core devs don't have just now. We have a lot of plans around this area (as well as others) for the app, but it's going to take time.

Thanks for everyone's support.

We love Sequel Pro and would love to see Postgres support. Unfortunately, nobody has stepped forward with a credible claim to do the work and the existing developers are all, for obvious reasons, much more familiar with MySQL. If anyone is interested in working on that patch, I would personally chip in to a promising effort, and I would encourage anyone else interested to do the same.

Offer a public bounty ? Just the basic support for viewing data would be really helpful.

This is my biggest pain point while working with heroku.

What's wrong with pgAdmin ?

Ugly on any non-Windows platform, too complicated for simple use (e.g. enter a query here and show me a result or show me a schema), etc. Well, just compare Sequel Pro screenshot with pgAdmin3 :)

Disclaimer: I actually like pgAdmin3, especially the visualized explain page.

PgAdmin3 on Ubuntu here: gorgeous!

No can afford macbooks - something with Ubuntu please..

I would pay good money for a postgresql client as superb as sequelpro, and I think a lot of other developers would. Someone is going to cash in on this gap if sequelpro don't beat them to it.

How about a http://kickstarter.com project?

But it's kinda kickstarter in reverse since you have people wanting to give money but no one picked to do the work. It's a bounty.

I worked on and almost launched an open source bounty site a couple years ago. But the closer I got to launch the more I realized that I hadn't adequately solved all of the hard UX problems (mostly around a: picking who's going to work on it, trying to avoid both having two people separately do it due to know central coordination and having one person say "yeah, I'm going to do this!" and then not get around to it and hold the project back, and b: how to handle it when someone does technically address the bounty, but in a way that is unsatisfactory either because bounty-placers did not perfectly state their needs or because of poor quality.

I'm a big fan of DBVisualizer (http://www.dbvis.com/), and have used it almost daily for ~6 years now on OSX. It's not a perfect OSX app (it's cross platform Java, but quite good looking for what that normally implies). Lots of power, doesn't crash, decent looking, and it works for pretty much every relational database out there (including MySQL, PostgreSQL, and Oracle).

I own a SequelPro license (as well as licenses for a couple of other non-free tools, RazorSQL and Querious) and I always come back to DBVisualizer.

It would seem that their acquisition by SalesForce hasn't slowed them down at all, in fact I'd go as far to say that they've been even more productive since that happened. I'm curious about how that whole process has gone; has SalesForce provided more support/funding/resources? Or just removed distractions and allowed them to focus on their product?

Nice job Heroku.

Suffice it to say, any rumours of our demise were greatly exaggerated. So much more to come and yes, we are hiring, including for our Postgres team. (Resumes/github accounts/etc to jobs(at)heroku.com.)

Just a little usability nit, is the pricing per month, per year, lifetime?

It's not clear from the pricing page, and I'm assuming it's per month, in which case it's reasonable, especially given the value added services being offered, especially the ability to fork the database and the automated creation of read slaves.

It looks like a solid offering. It's probably not right for companies that are subject to HIPAA or PCI compliance requirements, and there's no information on the use of compiled extensions in the db which may limit it's utility if you're wanting to use PostGIS or other specialist datatypes.

Basic plan is $200/month.

I'm sure it's monthly. The price tiers are the same as the full Heroku platform (http://heroku.com/pricing), which is billed monthly.

I would say that at this point you're right, we're probably not right yet for companies subject to HIPAA or PCI regulation.

The advanced Postgres datatypes are pretty much my favorite thing in the world. We use HStore internally all over the place, as well as some more exciting projects that we haven't said too much about yet. As I mentioned elsewhere in the thread, if you're interested in trying out some of those extensions, contact me directly.

I'm a real fan of PostGIS and like ltree and hstore a lot. Heroku isn't quite right for my needs at this time, but I would definitely consider it if I had a use case that needed an extremely fast public facing postgres installation.

I am curious as to whether you do pro-rated pricing?

As of today, we only charge for the time you use. I don't imagine we'd ever need to change that, but never is a long time.

That is interesting, there are times when I want to use PG as an ETL tool and take a midsized ( < 2tb data ) set and load it up and slice and dice it and produce some output for other programs and then have it go away. So, I'll keep you guys in mind.

@Olefoo you can get PostGIS on Heroku via SpacialDB. SpacialDB is cloud hosted PostGIS and a convenient RESTful API. We have a heroku addon too currently in private beta. Check out our devcenter here: http://devcenter.spacialdb.com/ and let us know if you would like a beta invite.

That confused me too - the https://postgres.heroku.com/pricing page should definitely mention that it's per month. I'm also not clear on the difference between the Ronin and Fugu plans, both of which are 1.7 GB of RAM but one which is twice the price of the other.

They are built on different EC2 instances.

Ronin = Small Instance: 1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit), 160 GB of local instance storage, 32-bit platform. ~$60/month spot.

Fugu = High-CPU Medium Instance: 1.7 GB of memory, 5 EC2 Compute Units (2 virtual cores with 2.5 EC2 Compute Units each), 350 GB of local instance storage, 32-bit platform. ~$120/month spot.

Ikea = Large Instance: 7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of local instance storage, 64-bit platform. ~$240 / month spot.

Zilla = High-Memory Extra Large Instance: 17.1 GB memory, 6.5 ECU (2 virtual cores with 3.25 EC2 Compute Units each), 420 GB of local instance storage, 64-bit platform. ~$360/month spot.

Baku = High-Memory Double Extra Large Instance: 34.2 GB of memory, 13 EC2 Compute Units (4 virtual cores with 3.25 EC2 Compute Units each), 850 GB of local instance storage, 64-bit platform. ~$720/month spot.

Mecha = High-Memory Quadruple Extra Large Instance: 68.4 GB of memory, 26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each), 1690 GB of local instance storage, 64-bit platform. ~$1440/month spot.

I think Heroku will pay less than the spot price for the raw server, but fork out bandwidth, storage, EBS IO, backups and so on. Amazon sells its basic servers cheap, but virtually nothing is included.

It looks like their margins go up at Zilla, but maybe the high-RAM instances need more bandwidth and backups.

Do they still limit the usage of user defined functions and contrib modules, or did that open up? I would love to use Heroku for some lightweight data warehousing that I've got going, but it's still pretty dependent on functions/sprocs for performance reasons.

I looked for any more documentation about that, but the only official word I have from them in the past is the support ticket I filed last year stating that they don't support "additions" like user defined functions or the various contrib modules.

Our production service has always supported UDFs, though the free sandbox service does not.

Unfortunately, Postgres makes it difficult to install extension modules without superuser. That said, we've wanted to support contrib modules for ages. Lately, encouraged by the work Dimitri Fontaine has been doing on extensions, we have a pilot project going which includes support for hstore, pgcrypto, pg_trgm, and of course, postgis. Feel free to contact me at pvh(at)heroku.com if you're interested in taking it for a spin.

We're running user-defined functions on Heroku right now (using a functional index on a custom hash function to get fast space-efficient queries on a string column of a large table).

This is really awesome, just a few downsides (for me):

I don't use postgres - I hope heroku expands this kind of service to more databases (although I don't think it's likely in the near future).

The smallest database is also pretty expensive. I wouldn't mind a cheaper plan for less resources.

Only being able to create one database of each size is just weird, especially if forking or following. Can anybody confirm that this is a limitation? (I only read about this from one of the other comments here)

In case you're using Mysql, give Postgres a try -- it's really awesome.

I do agree about them being expensive. Their services in general are too expensive for me -- personally I prefer to manage my own EC2 instances. But they do provide value worth paying for depending on your needs.

For a similar service using MySQL you can try http://xeround.com/. (Disclaimer: I haven't tried it myself and have no association with them)

We are unlikely to offer MySQL at any point in the near future. We feel that across the board, Postgres has the best data robustness guarantees, the most powerful query optimizer, the strongest replication solutions, the most useful datatypes, and the greatest community driving, developing and supporting it.

We hear you on the cheaper plan; watch this space.

Last, there is no restriction on creating only one database of each size.

Honestly when you list the merits of Postgres like that you sound like a used car salesman. "Of course you think you want that but really you want all of this." We all get that Postgres is great. Saying "no we won't implement a feature you, our customer, wants" like this just rubs me the wrong way.

Is there some technical reason that you can't offer both?

What would you like Heroku to do with a MySQL on EC2 offering that Amazon is not currently doing with their MySQL on EC2 offering (Amazon RDS)? They are both local, use the same underlying infrastructure, and have Heroku add-ons, no?

Presumably, their decision to only support Postgres is partly competition (many others provide MySQL on EC2), and partly technical. How would you justify the technical merits of the decision, without sounding "like a used car salesman"?

Sorry, I thought you wanted to know why we chose Postgres over MySQL.

Those are the specific features I think that make Postgres a better choice for both use as my day-to-day database and why we chose Postgres over MySQL for building a database service.

Certainly you could build a similar service on top of, say, RDS, but without the features listed above, I think it would take a lot more work and not come out as well.

Hope that helps.

Is this still using the EBS RAID that you guys mentioned in a blog post a while ago? If so, how do you avoid any slow I/O requests (which plague EBS) stalling all I/O to the volume?

Actually, Postgres WAL support helps mitigate this reasonably well, and a well tuned application has something like a cache hit rate of >99% during normal operation.

That said, yes, EBS can have unstable performance. We monitor for several kinds of related problems and in the most dire of straights can perform a hardware migration to a quieter node which generally clears issues up.

"Forget daily backups, Continuous Protection redundantly archives data to high-durability storage as it is written, ensuring that it is safe no matter what."

I'm sorry, but daily backups are not only for high availability, but also for point in time recovery.

What if $dev drops the user table by mistake ? Do they provide backup for that ?

You can make snapshots of the data.

You can set up Point In Time Recovery (PITR), and replay recovery logs. It's not clear from Heroku's pages whether their solution is set up that way.

It sounds like their solution is set up that way.

From https://postgres.heroku.com/#protect : "Every change to your data is written to write-ahead logs, which are shipped to multi-datacenter, high-durability storage. In the unlikely event of unrecoverable hardware failure, these logs can be automatically 'replayed' to recover the database to within seconds of its last known state."

It sounds like their solution is set up that way.

I think the question is whether the customer will be able to perform a point in time recovery, and what the size and resolution of the backlog is.

Not yet.

Thank you for that clarification.

We have found that large production databases do not appreciate having frequent full SQL dumps taken from them. We offer an automatic daily snapshot option because we believe that for a production database it's really not wise to take dumps more frequently than that, though there's nothing stopping you from rolling your own.

This is really awesome! A couple questions though:

* What do I do if I need to tune/configure Postgres for my workload.

* How is the performance of transferring all of the queries and responses across the internet?

Keep in mind I've never used postgres before so these may be moot points.

We've done a lot of tuning focused on web applications already, our experience is that mostly people with problems just need to add an index.

Performance depends on where you are querying from. We've found that if you are inside EC2 network overhead is typically <1ms.

Although they only let you create one database per plan, you can use schemas to emulate multiple databases.

http://www.postgresql.org/docs/9.0/static/ddl-schemas.html

Just to clarify: You're almost right. You can, in fact, create multiple databases of any plan you please, but yes, within each of those databases you can use schemas to implement something akin to multiple databases without resource isolation, but with the ability to share data.

Any chance of PL/python or similar?

I guess I'm probably one of like three users of PL/python, especially since it's untrusted. Worth a check, though!

No plans to support untrusted languages as yet. Too many foot-guns built into that. You couldn't even believe some of the evil, insane things that we've done with that.

Trusted languages are far more likely. This one looks really, really cool: http://code.google.com/p/plv8js/wiki/PLV8

This is ridiculous, but I would love a REST interface to a SQL server. I do a lot of AppEngine work and although I love the datastore and write super-optimal queries, I would really love the ability to pull a SQL datasource in every once in awhile.

Check out the extremely cool HTSQL from Clark Evans!

Not exactly RESTful but you might find some utility in this, developed at and open-sourced by the NYT: http://code.nytimes.com/projects/dbslayer

(edit to add: this is mysql-only, unfortunately :) I forgot about that part when I remembered it off-hand.)

I just realised that Heroku have a problem: when I looked at the page, and looked at the subdomain URL, I couldn't tell if this was a real offering or a fake app put up by someone else.

Yes, it's the real deal, just like blog.heroku.com and all our other services.

In order to reduce this ambiguity we are gradually moving towards placing user apps on a different domain; because we approach these kinds of issues very gradually this will probably take quite some time to fully complete, but as of today all new Cedar stack applications are hosted on herokuapp.com.

Ah, good to know. I haven't got around to creating a Cedar-based app yet

This is cool, seems to have a pretty fair price structure. I still don't understand their shared database pricing. 5mb for free, then $15 for 20gb. I need like 100 megs for 5 bucks.

This blew my mind. Amazing stuff. Heroku is absolutely my favorite platform.

I think the Fugu plan is a typo, if you're reading this Heroku guys...

'has' a typo, you mean.

need GIS

And confused by "one database only". Does that mean CREATE DATABASE databases? If so why?

Ya me too, no mention of PostGIS.

PostGIS is currently in a pilot phase. If you're serious, please contact me as described elsewhere on this thread.

Hi co-founder of SpacialDB here. Cloud hosted PostGIS is already available as a Heroku plugin, stand-alone and an easy to use API. Check out http://devcenter.spacialdb.com/ for documentation and you can sign up via the Heroku addons page or http://beta.spacialdb.com/

Reply on news.ycombinator.com