Comment by srvaroa | Hacker News Reader

Comment by srvaroa | Hacker News Reader

Parent Full thread

AWS takes care for most of them. All teams generally use cfn / terraform / to define that kind of infra (e.g. I need a DB with these properties) and they get applied as part of the standard deployment pipelines. We constributed support for cfn in Spinnaker to enable this.

A team in a different area to mine does offer Kafka as a service, but this pattern is an exception, and the org is actively moving away from it in most other cases. For example, a while back a team took care of offering "Cassandra's as a service", managed and operated for N product teams. They don't anymore, for reasons I explained in the article: - AWS catches up (e.g. they recently announced Cass as a service) - $commercial company has a Good Enough alternative (e.g. several teams are dropping Cass in favour of Dynamo) - It's operationally very expensive to maintain a team that does $storage as a service. - The cost/benefit ratio for doing our own managed storage only adds up when the managed solution is too immature, or lacks too many features that you need. The team offering managed Cassandras actually did this and moved to offering a managed Kafka clusters ~1y earlier than AWS released the first version.

Does that make sense?

So Terraform spins up a new RDS instance; what about all the management after that? Are teams left to their own devices regarding data/schema migrations, or does your team also provide a golden path there?

We provision database access through Hashicorp Vault. It's excellent, short lived credentials provided by Kubernetes service accounts (lots of glue here!).

After the RDS instance is created we need to manually create credentials so that Vault gains access to control it though, this is our mission to automate soon.

With credentials in place teams need to maintain schema creation and migrations themselves. We provide wrapper scripts go gain access with Vault credentials mysql shell or Perconas pt-inline-schema-change. Some teams create pre-deploy jobs or init-containers so that their service can run migrations automatically.

They are on their own. The team doing managed Kafka does offer some tools for data / schema migrations, but there is nothing for other storages.

Generally they actually don't raise this as a pain point, teams tend to be quite self sufficient in this regard and rely on things like https://flywaydb.org/ to deal with it.

From our PoV, at this point this type of feature would be in the high cost / low impact quadrant. Not because it doesn't make sense, on the contrary. It's just that it falls at a later stage of maturity than we're at organisationally. As I mentioned, Adevinta is an extreme case of sprawl. To invest productively in "a fully managed relational db with data/schema migrations" we'd need a certain degree of consensus on which db should we support. We don't have that luxury. Even more: we'd also need some form of consensus on "this is how you deploy and release software" which there isn't either (do you do proper CD? is there a manual judgement? are there deployments to a PRE/Staging env? ..). This heterogeneity greatly limits our impact (sprawl reduces the potential surface of impact for any decision we make) so we focus on areas where we can avoid this problem (e.g. everyone buys cloud, containers...). But also, as I said above, data/schema migrations is actually not a pain point teams complain about frequently.

Reply on news.ycombinator.com