nbm

Imagine you have a news web site with decades of news articles. You add a new article, which changes the "Top articles" section on a side bar. With a static generator, you would need to rebuild every single article, which may take minutes or even longer, using both CPU and disk I/O.

With a dynamic generator, you might maintain a cache of the latest articles somewhere, and stitch it into another cached copy of the newspaper section chrome, and then insert the content. It might be more CPU in total, but the changes are live faster, and the total disk I/O would be much lower.

Also, you'd need more disk space with a static generator.

For most simple sites and blogs, there's no real resource use regenerating the entire site, so you get all the benefits of the static generator, and almost none of the drawbacks.

There's a weird power imbalance in these situations. The person reporting the problem is able to tell whatever story they want since they generally don't have a reputation to maintain (and will, if anything, likely improve their reputation), and the company has to be very careful - it's often better to not release any detail to avoid potentially worse fallout from releasing potentially private information (yes, the privacy of the person making the complaint).

This is especially bad when your company is on the receiving end, and you know the truth. For example, a previous company had someone making claims that some purchase was made on their credit card without their knowledge - but there was account email confirmation at account creation time, account email conversation with community support, actual logical proof from the post itself of something that required said purchase, and so forth. In another case, an account is closed due to some clearly provable illegal behaviour, but it was blamed on not paying to renewing something optional.

So since then I'm perfectly capable of neither believing nor disbelieving claims made by people, even when it happens to agree with what I was thinking already. It doesn't make me feel better or worse than believing them, I just assume I don't have the information necessary to make any real judgement about it. Life lesson, I guess, and useful to not get sucked into things like political theatre.

Google+ launched in 2011. Friend lists were on Facebook in 2009:

http://mashable.com/2009/10/09/create-facebook-friend-lists/

Intimidation, threat of harm, and incitement are generally illegal whether they're on the Internet, in a letter, or delivered in person.

The underlying problem is the use of a naive cache buster which would cause the old server to reply one way and the new one another. For both the people who want the old asset and the new asset.

If instead of being something the web server throws away before replying, the version number actually caused different assets to be returned, then you'd not have this problem.

One pattern is to separate out assets into a different package that is deployed to a separate host group and have your clients request a different host name, or have your load balancers use a path match to use those servers for those requests.

Another is to push asset updates first to all hosts. All hosts, even without code update, will now be able to respond for the new assets.

Another is to use a local cache plus some backend service or database to serve the assets from the web servers - again, all hosts will now respond correctly for the old assets and the new.

Configuration is perhaps more complicated than binary deployment. With binary deployment, you can end up with, say, only the version in production, and then version that is about to be in production - the "old" and then "new". If you've made it from dev to staging with one binary, and you discover a bug, you go through dev and to staging again, just with a new "new" binary.

Configuration, especially configuration management, often needs a more staged/tagged approach (in fact, you may have moved from having n custom builds to having one build with n configurations). You turn on a feature for some people, for one cluster, for all clusters of one type (say, v6-only clusters), and so forth. The potential combinatorial explosion is huge.

For the feature-flag case, you can use a canary approach, at least.

It's a lot harder to canary a change on one of your two (or four, or whatever) core switches, though.

A pattern I've seen is to move from a single weekly deploy of disparate changes (say, server config management, switch port config, switch ACL config, ...) to multiple smaller deploys (potentially done by fewer people) based on the type.

One "nice" thing about infrastructure is that most problems are fairly immediately apparent. There are also generally a lot fewer integration-style tests you need to consider. You can detect failures and roll back quickly. Unfortunately, you've usually had a huge impact when you fail. And it's also relatively hard to verify your change before you land it.

As long as you're not in the spacefaring, automotive, banking/insurance, and medical industries, it's probably the case that it's acceptable to have some downtime and bugs - nobody will die or have their livelihood destroyed by it.

Given this, your confidence threshold for a release is not approaching 100%, it's hitting some "good enough" value, where the work you're doing to test for the next 1% is 2x of the testing you're doing now and is "not worth it". As you burn through some sort of error/downtime budget, you'll adjust that level of confidence - as you have more problems, and take more time with responding to problems.

Continuous deployment's upside is a confidence in the release process (since you do it so often), and some assurance that you'll be able to find the problem reasonably fast (since you only have to look through a smaller number of changes). You'll have fewer bigger problems, and more smaller problems. There definitely are cases where 10 smaller downtimes of 5 minutes is worse than 1 larger downtime of one hour, but usually it's better to have the former.

This class of errors is sometimes fairly easy to catch with canaries.

If a button is obscured or inactive/non-functional for some reason, then chances are some metric is going to be statistically varied enough to call out while in the canary phase.

For more manual canaries, this same approach can be used for metrics like memory usage, latency, number of upstream/database connections, and so forth. Of course, that could be the _purpose_ of the change, which is why it likely will be checked with manual canaries (ie, not the canaries used for the continuous deployment process).

It's an internal project. It basically takes a "diff" from Phabricator, and goes through the process of applying it to master and ensuring that all unit and integration tests pass before committing it.

This removes the need for every developer to do a rebase, run test, attempt land, discover conflict, rebase, run test, ... loop.

I'm not sure about Instagram, but Facebook is a fan of rebasing in general. Nothing should ever appear as a commit in master that isn't something that should be used in production - ie, should never intentionally be broken in isolation.

In general, feature branches are relatively very short-lived, and will be code reviewed, rebased and landed as a single commit onto master.

Features are often feature flagged off anyway, so it is acceptable to commit partially-functional features to master while that feature is flagged away.

There is a concept of stacked commits, but each commit in the stack needs to be a working step towards the end goal, and as such can (and will) be landed in isolation as they are code reviewed.

Can you give an example? Your question seems too abstract for me to see if I have any experience with what you're talking about.

This is actually a problem whether you're using continuous deployment or not. You need to put a release out, which you have done some testing for, but you know that only real production traffic can shake out certain types of problems.

You don't have to have all three servers getting the same amount of traffic, and you don't have to have a single copy of your service on each server. So, you could reduce the weight of a single server that does canary traffic to reduce the pain, or you could run two copies of your service on a server, and have the canary copy get a trickle of traffic.

Another approach is to use shadow traffic - instead of handling the requests on the canary host, you handle it on the production host _and_ the canary host. You'd need to ensure the canary can't adjust the production database, for example - or maybe you only shadow read requests. If you don't get any errors, or you're able to prove to yourself that they function the same, you can then move to a more traditional canary.

You definitely need to adjust your continuous deployment implementation to your environment, whatever it is.

Assuming they're spread out over ~10-12 hours (some crazy morning people, some crazy nocturnal people), that's only ~4-6 commits per hour.

Most problems will be discovered by someone and reported in an hour, and most of those will also be discoverable in a dataset on a system like Scuba - https://www.facebook.com/notes/facebook-engineering/under-th... - and you can identify the first time that particular issue happened.

If you're lucky, it lines up exactly to a commit landing, and you only need to look at that. Otherwise, due to sampling, maybe you need to look at two or three commits before your first report/dataset hit. You can also use some intuition to look at which of the last n commits are the likely cause. A URL generation issue? Probably the commit in the URL generation code. You'd do the same thing with a larger bundled rollout, but over a larger number of commits (50, in the case of a daily push).

As mentioned in the article itself, Instagram uses code review (using Phabricator), and also uses automated continuous integration for running tests on each "diff", before it allows the change to be landed to master.

Beyond a certain size (basically, once the time the migration will take because of the size of the data it applies to is too large), migrations are a heavy investment - in elapsed time, I/O, and so forth. As such, they are planned to a degree that CD probably isn't the solution for it (for example, you probably can only have one migration in flight at a time).

They aren't done live as a single big process that have the potential to lock all queries/updates over their execution, but rather as a set of smaller steps that don't lock.

Facebook has spoken about its online schema change process before - https://www.facebook.com/notes/mysql-at-facebook/online-sche... and its follow-up at https://www.facebook.com/notes/mysql-at-facebook/online-sche... for example, and I'm sure elsewhere.

Most people using MySQL would potentially first use something like https://www.percona.com/doc/percona-toolkit/2.1/pt-online-sc... instead of trying to create their own.

The same principles apply to other data stores that have more rigid schemas.

My experience is that interesting things happen to organisations at size 20-30 (basically when the CEO and/or COO can no longer keep tabs on nearly everything themselves, no longer can maintain some relationship with all staff), and also when there are more than 33% of people with <6 months at the company. Hope this wasn't as painful as some of my experiences with just single dosages of these at a time.

This may sound weird, but you were probably the one with motivation problems in this scenario, not them.

Don't believe me? They're comfortable and secure. They're happy with their output. They're apparently paid well.

You were unhappy with your situation - you want things to be different than what they are, and it was grating on you (and may or may not have led to burnout from your description), until eventually you left.

I've been in a very similar scenario:

I joined a "corporate" that had been running an Internet site for a decade. They had not built a new system in years, and had a massive monolithic ancient app with hundreds of hardcoded rules for individual accounts, where every change required tons of effort and they maintained a QA team at least the size of the dev team. Very few of the dev team seemed interested in things like learning new technologies, trying out different development methodologies, finding ways to make the system better as opposed to the task list of feature and bug requests. (I'm not even sure it had revision control.)

Short version - I get annoyed quickly at, for example, rules about arriving at 9:00am (my train schedule meant I tended to arrive at 9:10am, or at 8:30am, and I didn't want to hang out at work...). I'm told that even though I'm being productive, I'm setting a bad example, and they don't want others to start doing that. (I'm thinking: "Why don't you just deal with bad performance, whether the person arrives on time or not?", as well as "Yeah, and I leave the office at 6pm, so I'm at the office longer than them!")

Slightly longer version - I work with my manager to get my team (also not happy with the status quo) together in a room somewhat separated from the dev team, and we pump out code for several months a lot happier, until ultimately we all got better offers.

There's a much longer story, but the point is that _I_ was the one unhappy with the situation, and this affected my motivation to the point of the occasional debilitating day of non-productivity. The original employees were happy with their situation, and many probably still work there doing the same job they were doing nearly a decade ago now.

It does depend on the contributors, and obviously depends on the culture of the company (ie, if everyone assumes managers are supposed to do it, nobody will do it).

You can't rely on any random grouping of people to decide well on what's important to do and how to effectively break up the work. But adding a random manager to that group doesn't help specifically. Adding a more experienced IC will generally help more than a less experienced manager.

Absolutely. There are many other ways people end up unmotivated, but the most common I've seen has to do with things that are within the realm of their manager - possibly with escalation to someone higher up.

I've encountered only five managers who fully exemplify this, but they had an outsize role in keeping good people who others might not have been able to keep, keeping them productive (at least enough of the time to satisfy their work commitments while they dealt with other issues), and attracting more good people (because of word of mouth from the existing good people). It seems the key here was to have the conversations when they were needed. The advice here "Make your job easier and don’t." doesn't correlate with these great managers at all. At least two of them to my knowledge have given reports the advice (and assistance) to find opportunities elsewhere.

I've been on the receiving side of "no ability to promote", and ultimately the best thing a manager can do in that situation is to have an honest conversation about what is and isn't possible.

In addition, don't just give up. Keep talking and helping out. It could be advice on how to stay motivated or at least not demotivated, carving out space for interesting projects or developing new skills, or even helping that person find a new opportunity outside the company.

Dealing with unmotivated unproductive people you have limited ability to actually manage is beyond my experience.

I've found that groups of individual contributors are actually very good on deciding what is important, what they should work on and when, and how to work together to get things done.

Managers can provide a useful role in setting the scene, reminding what the goals of the organisation, the department, and so forth are, and how this connects with what the team is doing. They are perhaps most useful in working with individuals who aren't working well as part of the team at the moment - giving them feedback, mentorship, building bridges, connecting them with people, training, and so forth.

In my opinion, managers hurt more than not if they override the self-organisation of teams, and this most hurts when you need two teams to work together - and one of the managers has (explicitly, perhaps) made it clear that doing things "off the plan" is not appreciated.

If one considers the primary purpose of managers to be to improve the performance of those that report to them, then this list starts poorly.

It _is_ terribly difficult to manage unmotivated people, but most people only end up unmotivated because of poor management. And it's usually cheaper and ultimately better to motivate a proved achiever who has lower motivation than to find a replacement.

http://www-cs-students.stanford.edu/~amitp/game-programming/... was a springboard for many people dealing with this problem. That whole site is worth a read if you're interested in game development at all.

The referenced video (https://www.youtube.com/watch?v=mLEawo6OzFM) is a great introduction to Facebook's datacenter fabric as deployed in Altoona.

FWIW, Sheryl Sandberg also leaves work at 5:30pm to spend time with her kids - strange that something she's been talking about for some time (before Bret left Facebook, even) isn't mentioned.

The Facebook hackathons used to start at like 6pm with the implicit expectation that one worked through the night (and weren't expected to work the next day), but these days they start during the day (the next one is 9:30am). There's still time to work all-night if that's your thing, though.

One thing I noticed at a previous hackathon was one of our engineering VPs showing their kids around. I've seen a few parents doing that - often their spouse will bring their kids to visit for dinner and to spend some time, sometimes they go back together, or the kids and spouse heading home for bedtime. Not sure if that's a "this is how I get through this horrible expectation people have of me" or "this is how I blend these two parts of my life that I value", but I'm hopeful it is the latter.

Okay, if we're only considering active health checks, then I'm not sure any load balancer considers a 5xx a success by default, let alone a "production" load balancer.

For a non-healthcheck 5xx response, it almost never is clear that this host in itself is responsible for the 5xx response. 5xx is the correct response when there is an error on the server side (ie, not an error on the client side, not a correct response), but it doesn't mean the server is a problem - it just means that the server experienced a problem in serving the request. That failure itself may be from one of many RPCs that server made to other services. As such, all web servers behind the load balancer for that request type will exhibit the 5xx response (at some rate, and depending on any state in connection sharing/reuse between the server and their upstream service), and all would subsequently be removed. Which isn't the correct response at all.

As someone who has had the job title "Systems Administrator" and the job title "Software Engineer", and currently has neither but still does exactly what he's always done - solving problems by understanding systems and, among other things, by writing code - I wouldn't consider load balancing and failure domains/types/handling as the sole or even primary purview of a systems administrator - especially in the case of large installations.

5xx may be the correct response - sometimes the server is asked to do something valid (ie, not a 4xx/client screwed up), but had an error when it tried it.

No load balancer I know of will remove a web server that returns a single 5xx from its healthy pool. It will need to use some heuristic as Rachel points out, some percentage that is based on the statistical norm. Otherwise it'll fail out too many hosts and cause a problem.

I think you're severely underselling developers. I've met people who have only had the Software Engineer title and never installed a Linux distribution who get this stuff at least as well as anyone who calls themself a systems administrator. Sysadmins don't have a monopoly on understanding failure cases and failure handling - false positives, false negatives, outliers and outlier detection, metrics to look at. It's a skill that comes from experience, which can happen (or not) whatever you call yourself or others call you.

I'm lucky enough that I get to focus on this type of problem, and while there's definitely an aptitude portion, it is also a teachable skill - I see my job partly as getting the team I'm working with to be able to do this stuff when I'm not around. That usually means finding two or more people in the team and cultivating their interest in it.

Pretty much all load balancers have health checks - active where they reach out to each server, or passive where they observe the responses of existing requests if they can.

One of the issues is making your active health check more like a doctor's physical than "'tis but a scratch" self-reporting. But also ensuring you're not dealing with a whole bunch of hypochondriacs.

Passive health checks at least have the property that they fail servers when the servers are unable to serve, even if the active health check does not consider some subsystem in its response. But alone they can easily be fooled by really fast non-error responses.

Anyway, saying "name of brand of load balancers" solves this problem is only covering the most basic cases. General solutions are at best only the first step of the full solution. You need to think about the edges - which I suspect is what Rachel is advocating.

Nope, this is my first use of XCB, and I've never used Xlib before either.

As you say, the documentation is barely to entirely non-existent.

I've been watching this religiously since it started. I manage to catch almost all the episodes live, and there's a great set of people on the Twitch/IRC chat as well. (I'm "nxsy" on there.)

I also wrote a Linux platform layer for it for fun [1] and for those who were skeptical about Casey's approach to portability (and there are a few people who pitch up who are/were aggressive about his choice of starting platform and/or his approach).

[1]: https://github.com/nxsy/xcb_handmade