Hacker News Reader

Comment by ransom1538 | Hacker News Reader

"A place I worked at had a symlink pointing to the app directory"

This is the way to go. Have your root web directory be a symlink. EG. /var/www/app -> /code_[git_hash]/ You can whip through a thousand vms less than a second with this method. Connect, change the symlink. Other options: Pushing out a new code branch, reverting with git, launching new vms with reverted images, rsync'ing with overwriting -- is slower, and more dangerous with prod.

"For the database, during a migration"

There is no such thing as a database migration on prod. There is just adding columns. Code should work with new columns added at any point. Altering a column or dropping a column is extremely dangerous.

View on HN

> whip through a thousand vms less than a second with this method

I forgot to mention a rather useful quality of this scheme when there are a whole lot of visitors: you can upload the code, switch the link on just a portion of the servers, gawk at the error logs and put the link back if you don't like what you see.

Yeah, very useful strategy. I've heard it go by various names (e.g. toe-dipping, one-boxing, etc.), but it's always been one of additional methods of helping ensuring safe prod deployments at any company I've worked for.

One downside - depending on your setup - is you may not have an easy way to hit the hosts directly/deterministically via any UIs in case you wanted to do any manual verification/debugging yourself.

> Yeah, very useful strategy. I've heard it go by various names (e.g. toe-dipping, one-boxing, etc.)

Strange terms. Isn't this just a canary?

Every company I've been with seems to re-invent the terms or swich their definitions slightly.

For me, currently, "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch". Wether the dedicated canary accounts end up hitting a one-box prod host or real prod host in the end isn't a factor.

The important thing is we incrementally expose our latest code commit to prod hosts via one-boxing to reduce the customer exposure if an acute problem somehow gets past the previous code deploy stages/tests.

> For me, currently, "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch".

I've never heard of this practice described as a canary before (shrug)

Yeah, I concede the common lingo within the company (or team even) may not align with either proper definitions or mainstream usage.

> "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch". Wether the dedicated canary accounts end up hitting a one-box prod host or real prod host in the end isn't a factor.

this sounds roughly like synthetic monitoring:

https://en.wikipedia.org/wiki/Synthetic_monitoring

synthetic in the sense of synthetic traffic, since it isn't traffic from genuine users.

Can you go into more detail about what is meant by this:

> alarms that feed into a master aggregate "switch"

what is the master aggregate "switch" ? what does it do?

> synthetic in the sense of synthetic traffic, since it isn't traffic from genuine users.

Yup - I think that lines up.

> what is the master aggregate "switch" ? what does it do?

We have a hierarchy of aggregrate monitors (or "switches") that watch n amount of either specific metrics or other sub-aggregate monitors.

In the case of production deployments, we watch a specific rollback aggregrate monitor for either a fixed amount of time or customer traffic that will auto-trigger a rollback if it goes into alarm (aka switches on).

We also have a master aggregrate monitor that will switch on if any sub-monitors get swtiched on for any reason. We typically watch this master aggregate alarm to auto-disable any promotions in our code pipeline.

> There is no such thing as a database migration on prod. There is just adding columns. Code should work with new columns added at any point.

If you can't alter a column, how do you prevent your database slowly rotting in terms of its design integrity?

Sure. After the code has been live with the new column for two (?) weeks - drop the column you don't need. Alters = add a new column, weeks later delete the old column.

>There is no such thing as a database migration on prod. There is just adding columns.

We have a DB schema that we not so affectionately refer to as the Standard Oil Octopus because of this methodology applied over ~20yr.

I agree with you in the general case but eventually hard cuts have to be made or you will perpetuate the existence of all sorts of legacy spaghetti (not necessarily in the DB, but in all the other things that use the DB). Like everything else there's a balance to be struck.

Yeah! I wasn't trying to suggest keep columns that are not used for years. I was trying to suggest during that release cycle don't alter or drop columns.

Reply on news.ycombinator.com