Hacker News Reader

Comment by liotier | Hacker News Reader

> Yes, all databases are operationally managed by us. You never have to worry about maintaining the database or operating system.

I am not sure whether not having control over upgrades is a feature or a bug.

> We set up and maintain an optimized Postgres instance

Optimized for what ? Any non-trivial database is going to require tuning, for which there is no one best way - only custom choices.

View on HN

Hi, Database Labs founder here.

> I am not sure whether not having control over upgrades is a feature or a bug.

I think different people will have different answers to that question depending on their circumstances and abilities. Some want full operational control over everything, and others would much rather just have a database to use that's always upgraded, monitored, backed up, and so on. Our target market is the latter group.

Bear in mind that doing your own sysadmin work for your database is not free. Besides the direct costs of paying someone to do the upgrades, test the backups, and so on, it diverts human time and attention away from other endeavors. Human time is a tech company's greatest asset, and we are betting that a significant number of people are prepared to pay someone to handle all of that for them, so they can focus on their core business.

> Optimized for what ? Any non-trivial database is going to require tuning, for which there is no one best way - only custom choices.

Yes, you are absolutely right. Out of the box, our databases are tuned for what we perceive to be a "generic average web app workload" on the underlying host type that they use. I realize this is rather vague and does not cover many use cases.

Use case specific tuning is definitely on the feature list for future versions. Do you have any insight on how that could be implemented in such a way that it'd be a good value for you? In other words, can the broad array of custom choices be reduced to any smaller set of useful patterns?

Most broadly, write heavy and read heavy, and then OLTP vs. OLAP.

Implement them as sliders, people will think it's great that they can have custom tuning based on what percent read/write and OLTP/OLAP they think their workload is.

Thanks, that seems like a good starting point.

Do clients get Postgres superuser access?

Yes, users get full Postgres superuser access. You can freely create/delete/update databases, roles/users, privilige assignments, and so on.

How does that tie in with your admin of the database? What happens when a user changes wal_level or something? Just wondering where you draw the line.

We handle the system-level administration of the server itself. Wal_level, for example, can only be set at server start. We set that in the config file, which users can't edit.

Users receive superuser on a blank slate Postgres, and they can do whatever they like within the already-running Postgres server -- create/delete/update databases, users/roles, privileges, and so on.

We are seeking users who don't want to worry about wal_levels and things like that -- users who just want a Postgres server to use, with the assurance that backups, OS patches, and so on are all taken care of for them.

OK, that sounds fair enough. Just interested in whether you'll be running a patched Postgres to be honest :-).

There is of course set_config in 9.4 - superusers can then edit the config file through that (a superuser can overwrite any file the user running the server can write to anyway, but through more convoluted means).

Interesting. My reading of the 9.4 docs[1][2] is that set_config can't be used to change one-off settings that require a server restart, such as wal_level. It also doesn't seem to be able to change the config file at all. Is there some way to do that?

I'd also like to know more about how superusers can overwrite files. Do you have a link?

[1] http://www.postgresql.org/docs/9.4/static/functions-admin.ht... [2] http://www.postgresql.org/docs/9.4/static/runtime-config-wal...

Apologies - I was thinking of ALTER SYSTEM [1]. That writes to postgresql.auto.conf. It sounds like that file has priority over postgresql.conf [2]. You might well be able to just prevent writes by the postgres user (or whatever user you're running postgres as).

Regarding superusers overwriting files - COPY can write to anywhere on the filesystem. From the docs "COPY naming a file is only allowed to database superusers, since it allows reading or writing any file that the server has privileges to access." [3].

[1] http://www.postgresql.org/docs/9.4/static/sql-altersystem.ht...

[2] http://michael.otacoo.com/postgresql-2/postgres-9-4-feature-...

[3] http://www.postgresql.org/docs/current/static/sql-copy.html

Also, if you have any untrusted PLs available, a superuser can create one of those and do what they like.

    >> I am not sure whether not having control over
    >> upgrades is a feature or a bug.
    >
    > I think different people will have different answers
    > to that question [..]

Yes, as long as the features remain reliable. But sometimes, though rarely, an upgrades retires an API or worse, changes a default behavior. I guess that you have thought about that, but that is what I meant with my original question... Sometimes you have to choose between feature stability and newer versions (for new features or, more importantly, bugfixes). How will you handle that balance ? What guarantees does the customer gets either way ? Doesn't making it explicit to the customer comes in conflict with hiding the gritty details of database server operations ?

Our guarantee to the customer with regard to the upgrade process is that we will use our best efforts and accumulated knowledge to make the right decisions about how to manage the thorny questions around the upgrade process, to which there are no universally correct solutions, so that they don't have to do so.

Of course, it's very easy to say some nice-sounding words like that. The real evidence can only be demonstrated over time, after many successive trials, through building up an ongoing relationship of trust with our customers and repeatedly demonstrating that we make the right calls most of the time.

With regards to breaking Postgres changes specifically, they are rare and almost always announced well in advance, giving us ample time to message the users, prepare them, and manage the transition.

I see this stage of the database market as analogous to e-mail in the early 2000s. At the time, most people read e-mail delivered by a custom Sendmail, Postfix, or Outlook installation. Many felt that the numerous gritty details of running an e-mail server meant that every site needed its own mail daemon, customized to its particular needs.

There were a few email-as-a-service offerings like Hotmail, but many admins felt they were too limited for serious business use cases.

It turns out that most businesses actually just needed GMail -- a really well done email-as-a-service platform. Sure, there are still some places that really do need a custom e-mail setup, and they continue to use custom software. For 80% of all businesses, though, a Google Apps GMail account is more than adequate. They trust that GMail's admins will make the right call. We aim to do the same for databases.

I still run my own mail server... But I understand your point - the reasons I still do are mostly not technical.

well actually default postgres config is pretty slugish, their defaults are surely better than any untuned pg instance

that being said, don't know how this is any different than Amazon RDS

> don't know how this is any different than Amazon RDS

It runs on DigitalOcean, for one.

Not a plus.

It is, if you're also running on DigitalOcean. Intra-DC latencies.

Hello, Database Labs founder here. DigitalOcean users are indeed a key market for us, for this reason.

Besides DigitalOcean users, if your app is in a reasonably well connected data center, in many scenarios the additional network latency incurred by traversing the network from your data center to DigitalOcean is going to be minimal compared to the speed gain you get by running your database on SSD drives rather than on spinning disks or Amazon EBS.

Hello, Database Labs founder here.

Yes, our defaults are way better than an out-of-the-box Postgres. That said, we are acutely aware that significant further tuning specific to a particular database's workload can deliver even better performance. We are investigating the best way to deliver this to a broad audience now. Do you have any suggestions?

Amazon RDS uses Elastic Block Store (EBS) as its disk. EBS is extremely slow relative to the SSD drives used at DigitalOcean. Worse, unless you buy very expensive provisioned IOPS, you are competing for network bandwidth with all other EC2 customers in the physical network segment where your RDS database is hosted every time you do a disk access. This means your database will randomly become very slow at peak load times, with no visibility and no way to control it other than spending a whole lot of money on provisioned IOPS.

We've also gotten good feedback from a number of customers who simply do not want to deal with the complexity of AWS. Our API is far simpler; we don't have security groups and so on. If you actually need all that complexity, that's fine, but many people don't.

Finally, RDS does not support read replicas for Postgres (at all); you are limited to one database server. We are currently setting up replicas manually as requested, and will be building them into the user interface in the near future.

Are the SSDs at Digital Ocean enterprise grade and RAIDed? They won't corrupt data under power loss etc? Bearing in mind you're probably using asynchronous replication, people you can't rely on that for durability; people will expect that once it's committed, it's on on (at least 2) disks.

I think they use RAID 5.

Using an SSD for your database is tempting, but please be aware you must use an SSD that can flush to disk in case of power loss. You should read here:

http://www.postgresql.org/docs/9.3/static/wal-reliability.ht...

and also search the pgsql-general mailing list for the numerous discussions there about which SSDs are safe for Postgres and which aren't, e.g.

http://www.postgresql.org/message-id/1396467425.2481.2.camel...

and

http://www.postgresql.org/message-id/50A34426.5020304@strate...

Assuming DO is giving you the SSDs you need, you should include a FAQ reassuring customers about that.

Hi everyone, I'm Eric, another one of the Database Labs founders.

We asked for and received written confirmation from Digital Ocean on this question. Currently, all of their newer hardware uses full Battery Back Up (BBU) and RAID on all SSD arrays. Data centers where you are guaranteed to be on a box with their newer hardware are: NYC3, LON1, and AMS3. These locations are supported in the Database Labs interface.

Digital Ocean's other data centers have a mix of hardware. All hardware at a minimum has dual redundant power supplies and hardware RAID with SSD drives, however, not all hardware has a BBU on the RAID card. While it is extremely unlikely that there will be a full power outage that overlaps with a write time on a database, it is possible. We recommend, if this is a concern for anyone, that deployments be restricted to NYC3, LON1, and AMS3.

We will be monitoring this issue with Digital Ocean, and will let everyone know as more of their centers are guaranteed to be SSD + RAID + BBU.

Thanks for the suggestion of documenting this on the web site, we will be adding some information about this in the near future.

Amazon RDS for PostgreSQL also supports SSDs [1]:

"Amazon RDS now supports General Purpose (SSD) Storage You can now launch DB Instances that use General Purpose (SSD) storage, a new SSD-backed storage option for Amazon RDS"

[1] http://aws.amazon.com/rds/postgresql/

Reply on news.ycombinator.com