How We Release So Frequently

All the principles in here sound exactly like what is SOP at Google (and probably Facebook and others). It's a pain in the butt sometimes, especially when making schema changes, but ensuring everything works properly even when you have multiple versions running in production can really help with confidence in making changes.

You do have to think several steps ahead when making big changes; the (internal-only) project I work on is in the process of completely changing our DB schema and we're redoing our API completely as well. We're attempting to keep our old API running in parallel while migrating literally everything underneath it, which is a fun challenge. It results in a lot of what can feel like busywork, but when the alternative is bringing down your service temporarily, it's worth it. An hour of planned downtime to do an offline migration can easily turn into several days when Murphy strikes. That's OK when you're first building a system, but once anyone is relying on you to get their work done, it is just pure pain.

I just want to throw out that there's nothing about this that won't work for !("high scale, complex") sites. I worked at a ~20 person startup and we arrived at the same decision regarding database migrations and, you know what, it wasn't that hard and it didn't take that much time and it made deploys a lot smoother.

I see a lot of other people mentioning the annoyance factor. Like anything else, you get used to it, and appreciate its advantages.

It's always possible to make changes to your schema without needing to potentially roll back the database. It may just be more annoying than you are willing to attempt. You might need to roll back your code to a previous version, but if you structure your changes right, you'll do it in multiple steps. (These are basically what's laid out in the article, but broken down more.)

  1. Change your code to write to old and new schema; keep reading from old schema. 
  2. Migrate data to new schema in background. 
  3. Add a flag to control where you're reading from. Default it to read from old location. Keep writing to both. 
  4. Flip the flag on some subset of your jobs. Ensure everything is still running smoothly for as long as you like. 
  5. Change the default to read from new schema. Wait as long as you like to be comfortable that the change is working properly. 
  6. Delete the code that reads from the old schema. 
  7. Delete the code that writes to the old schema. 
  8. Drop the data in the old schema.

At any point prior to deleting the old data, if you encounter problems you can roll back to an old version of the code. If your schema changes are incompatible, you can make an entire new database with the new schema. This may temporarily waste some storage space, but it's very safe.

Never rolling back makes a lot of sense, and ia often asserted by the fact that many migration frameworks dont even provide rollback facility e.g. flyway. A database migration is a major high risk change IMO and should be well thought through.

I am never thinking about doing rollback on production, downgrade is nice on dev machine when you work on multiple branches. I'd rather do database restore and code restore to pev version.

I would not roll out new code with migrations without ad-hoc db backup. Any new code as well.

Database schema changes can almost always be moved forward in a always safe manner using expand-contract pattern http://martinfowler.com/bliki/ParallelChange.html

As for reducing the risk of damage to the data itself from badly behaved application code, I think the best approach is to design your architecture such that you can't lose important data.

There are various other techniques than can help. I wrote about some of them here http://benjiweber.co.uk/blog/2015/03/21/minimising-the-risk-...

Releasing less frequently actually only makes the problem worse. Infrequent changes are often too big to have a chance of understanding their potential affect on the production system. You're also less likely to immediately know how to respond to a problem.

Flyway author here. Rollback is an illusion once your transaction has been committed. At best you can attempt to issue a compensating transaction to undo some of the effects. More details: https://flywaydb.org/documentation/faq.html#downgrade

If you add a column and real customer data is written to that column, then "rolling back" means deleting it. That isn't a problem that can be "solved".

Looks like a large distributed company...I have some experience with casinos...similar...

I'd be interested to know more about your experiences with an NSF "share"...which he mentions...?

Thanks!

It sounds like they are using NFS for distributing their code, which should be almost entirely read-only - the only place that writes new files to the file system is the single Jenkins machine that manages the deploys. It seems likely to me that this would avoid most of the risks inherent in running NFS at scale.

Yep. I work for a big company with a bureaucratic system engineering department that puts all program code on NFS (SANs).

It's been the cause of virtually all of our service outages and many of our performance problems---and it's completely obsolete in the era of Jenkins and Ansible.

That is something we're working on at the moment. It is read-only from the web servers' perspective, but it's definitely not without its problems.

I think NFS often gets an undue bad reputation. I work at a company which uses NFS at scale and it does the job without too much trouble. NFS is used as the storage for vmware datastores, xen primary storage and also for shared storage mounted between servers.

For the latter case, the mounting of these partitions can be automated with config management on the linux servers. You have to be careful with UID's and GID's but config management helps with this.

The filers supplying the NFS storage can be exploited to provide replication to other datacenters,snapshots and also provide redundancy with multiple heads serving the volumes.

In the past I've used Fibre channel ( found it overly complex) and iSCSI. iSCSI was fairly straight forward to use, but I've never tried to automate it. I guess there isn't a reason you couldn't however. For complexity I guess its Fibre>iSCSI>NFS.

Performance wise we don't have any issues with NFS itself, the bottleneck is sometimes the filer trying to keep up :-)

Anyhow, in complex environments, sometimes its good to keep things simple where you can. NFS helps with that, its stable, scalable and the performance is comparable to iSCSI.

Removing the need for shared storage on the OS where possible is the ultimate aim though.

We don't use database views, but kinda-sorta mimic them by separating the persistence layer from the API layer. It can be annoying to maintain (at the big G there are lots of SWEs who joke that their job is writing code to copy fields between protobufs), but the alternative is to couple your API directly to what you're storing in the database. That is a road that leads to pain.

The benefit over using views is that at your code is written in the same language, instead of having a while bunch of logic running semi-hidden in the database. If you have a bug in your view, you have to update your DB schema (or at least roll out new PL/SQL DB code or whatever). And if you're working with a planet-scale distributed app, it just plain won't work.

I guess your point being that these aren't novel techniques?

The author sure didn't invent them, but they aren't widespread enough yet. In fact e.g. Rails puts you in the opposite mindset (which is OK at early stages).

For a company like Sky Betting this is actually fairly monumental. Not every company looks and behaves like every trending SV startup.

http://www.cvc.com/Our-Portfolio.htmx?ordertype=1&itemid=112...

PS - They actually make pretty good money too: http://www.cvc.com/Our-Portfolio.htmx?ordertype=1&itemid=112...

Betting is a highly regulated environment. The kind of mistake that startups make all the time could result in a heavy fine of the loss of a license. In such a climate, this workflow is exceptionally modern.

That's a flaw in how you use node, not node itself. We run dozens of instances of node without internal process state, the need for sticky sessions, etc. You're losing a lot of load-balancing ability by not keeping this discipline too.

Whatever you are developing has no value until someone actually uses it. Therefore the quicker you can confidently release something(feature, bugfix, etc), the more value you gave deliver.

This idea comes from the concept of inventory waste from lean manufacturing

End result to a user...potentially yes...though they may never be aware of your efforts...

Provided all else is equal (bugs, feature requests, customer feedback,) yes. If it gives any business advantage, the sooner you release, the sooner it starts giving returns, and those returns eventually compound over time. So, yes.

>> It sounds like alot of extra code to write.

What he's saying is more about convention than writing code. For instance, instead of adding a column "abc" and doing:

foo.abc = 123;

they would do something like:

if (foo.abc) { foo.abc = 123; }

make sure all tests pass, and then migrate the db.

If you're asking about tools to migrate code, all popular languages have one. (I.e. django comes with one that's already really good).

Yes! Although I am obviously biased.

There are many tech teams at Sky. Each one is very different in terms of culture, tech, product etc.

As I understand it, Sky bet is quite separate from the rest of the company. This is probably a good thing...