coops

Someone like Amit doesn't need money; by the time you get as highly placed at Google as he was you will be wealthy. I believe that Uber has interesting engineering challenges but likewise have doubts about their profitability.

It's a real piece of hardware.

- crawlers

- websearch

- google brain

- video transcoding for youtube

- appengine (think snapchat)

- gmail

Almost everything Google does runs on Borg.

The best you can do is two-phase commit.

SRE here. If you can handle your work nobody cares how much you are in the office. At all. However, unless you are insanely organized you won't be able to do this every single week--most successful googlers have several projects going on at once and there will be some times when you need to work hard on more than one.

You cannot hire top-quality engineers who are willing to do shift work.

Yes, this is certainly a state you would desire to avoid. However, you didn't answer any of my questions.

Can you go into more detail regarding the prohibitive consistency issues? How do you maintain consistency in steady-state (ie. not during migrations?) Also, how do you make the call as to whether to bring your site down vs. attempting a live migration?

It would be interesting if someone from Github could discuss why they chose to do this migration by taking the whole site offline and doing the migration all at once. Did anyone investigate if this could be done without taking the site offline?

What's it like to administer a cluster of Mac Minis? Do any of the popular open-source orchestration tools work?

Consider that if you collect device fingerprints, you can detect users who live together, because they are likely to share devices. If somebody bad then gets access to that data, they could do creepy things to your users.

This is generally known as "device fingerprinting"--there are many ways to do it but they all involve probing for unique properties of a client via JS / Flash (listing installed fonts, drawing invisible characters and measuring via JS, etc.), then hashing them together to generate a unique ID for that user.

Some people think this practice violates users' privacy, and I'm one of them. This technology can be used to uniquely identify a user across multiple logins on the same site, or even multiple sites. It's quite widespread.

This paper[0] is mostly a survey of prominent DF providers and sites using this technology, and it's also a good primer on device fingerprinting techniques.

[0] http://www.cosic.esat.kuleuven.be/publications/article-2334....

I think what will actually happen to requests in flight is:

- partial data received by old HAProxy is lost as old HAProxy exits

- new HAProxy comes online, binds to port, receives fd

- iptables rule removed. new HAProxy starts receiving new requests

- in-flight requests from the old HAProxy are timed out by the kernel (TCP RST) as nothing is there to read request data from the old fd or send response data.

So I think this is actually "worse" in some sense than the other retry behavior since it's not recovered inside the same TCP session but instead forces the client to open a new TCP session.

I feel I must point out that this relies on your clients to retry packets that iptables drops, so at the least they'll have a slower experience than they otherwise would. I believe that this will also break requests that are in flight when the reload happens, if they have sent partial data.

Er, yes, hi Evan! Cooper here. I believe either you or Andy originally pointed this interesting HAProxy restart behavior out to me, in the context of explaining why you wrote Einhorn.

This is generally correct. In particular, when the HAProxy process is stopped/restarted there is a brief period during which the port is not bound by either process. (If the new process isn't able to get the socket when it boots it will sleep ~XXms, then try to bind/listen in a loop until it gets it or a retry threshold is hit.) During this time the kernel will reject incoming connections to the HAProxy port, so you are in danger of dropping incoming requests on the ground.

This release contains a neat feature: you can now bind HAProxy to a specific FD opened by its parent process. This means that you can babysit your HAProxy processes underneath a parent process that opens ports and get hitless HAProxy restarts, which I've long desired.

"A simple Hubot command can reroute our traffic to their network which can handle terabits per second."

Really? You have to round-trip through Campfire to control your network?

This is a terrible question so I'm not going to answer it, but if you are interested in reading about the corrupt history of Myrtle Beach (and the greater Grand Strand area) I strongly recommend Will Murdock's Banana Republic: A Year in the Heart of Myrtle Beach. http://www.amazon.com/Banana-Republic-Heart-Myrtle-Beach/dp/...

Funny to see this post pop up in a thread about how dysfunctional SF city government is. Myrtle Beach is effectively governed by an oligarchy of property development families.

sorry, no, you're all wrong. those employees are all taking unpaid leave to come work on healthcare.gov.

You must not have tried very hard. http://techcrunch.com/2013/10/31/oracle-red-hat-and-google-e... People are still flying out there to help with healthcare.gov, all the time.

For all their hand-wringing over the healthcare.gov debacle you'd think at least one of these DoBT guys would have gone in to help fix it. That probably would have gotten in the way of fund-raising, though.

I have done this the last 2 times I changed jobs. It worked out well for me both times. If you are considered to be a valuable employee, and you have management that is the least bit competent, you won't "just be replaced" for looking around. Why would they trade you, an employee who has proven herself to be valuable, for someone who just might be adequate? Also consider that we currently have a talent crunch on. No company with high standards for engineers is able to hire as many as they would like to.

It is a great idea to tell your employer that you're looking around, because it frees you to tell friends and former colleagues that you are looking for a job without having to worry that your employer will find out. In my experience you can get great job leads this way. This also gives you the opportunity to control the job search process such that you have multiple offers available at once, which improves your leverage.

When I do this I let every company that I am interested in know that I have a deadline by which I need to receive a job offer or not. Typically this deadline is my search start date + 1 month. After that I have a 2-week negotiation window, at the end of which I will accept 0 or 1 new jobs. This gives me a lot of leverage in soliciting counteroffers and minimizing stupid recruiter games like exploding offers.

Do be aware that if you follow this strategy some recruiters will bitterly resent you. This is because they know exactly what you are doing and how it minimizes the informational asymmetry that is one of their most important weapons.

most google search outages at this point are the result of some kind of bad configuration push.

if i recall correctly, likes were pretty dark-testable, since we didn't make them visible until we released them. so we were able to use a methodology similar to the infinite-scroll dark test--X% of items viewed by client in the app triggered a like on that item. we then cleaned up all the likes shortly before releasing the feature.

similarly, engineers who won't criticize or endure criticism of their own work are seriously poisonous for a company.

i'm a site reliability engineer at foursquare sf, and i think it's very accurate. dennis travels to our office frequently and takes the time to meet with each of us individually, even those of us who work very far away from feature-land. when we're meeting i feel super comfortable saying "this sucks and we ought to fix it." mostly we discuss organizational issues; i generally air technical issues in similar meetings with harry (our eng lead, posted above,) because making engineering work is harry's whole job.

i've worked at high-profile startups with founders who are very media-visible before and dennis is pretty special in my opinion. he seems to genuinely believe that all of us employees are important to making foursquare the best product it can be.

did the standard replica set election process not work for you? it is very rare for us to see a failed failover.

from this answer and your blog post it appears you were not using mongodb replica sets. is this true?