Hacker News Reader

Ffs, reply to the comments replying to you, that's what the feature is there for. Don't preempt them by editing your comment so people see your response first.

There is a penalty box for accounts that are determined unsavory, like mine, where they have a finely-controlled quota of commentary beyond which they are "submitting too fast" and told to slow down. Everyone has this limit, it's just for some accounts it has been set significantly lower. I suspect with that system you'll see this more, rather than less.

Easily move? It seems to me that you have no serious experience in the real world. There's something called "data gravity", and the non-secondary issue of how to migrate a "live" system (in production) from one cloud to another over the course of typically several weeks.

Moving from one cloud to another, even with containers, is never easy at large scale.

(source: I have worked at AWS for 6 years, at VMware for 2, and I've seen hundreds of clients go through this exercise)

I consider 'toomuchtodo a voice of authority on operations based on much reading and discussion (particularly on moving to physical, which we've discussed before), have performed the very exercise being discussed four times in my own career ranging from a couple cabinets to a couple hundred million in capital, completely agree with the entire comment to which you are replying, and feel that your jab about "serious experience in the real world" was totally unnecessary.

If moving operations around is insurmountably difficult, you built operations incorrectly. Put another, even broader, way with a few more implications: if you are totally reliant on one vendor for continuity of any part of your operations, you built operations incorrectly and are introducing unnecessary risk. If us-east-1 goes down and you cease generating revenue as a result, you have built operations incorrectly. That's really all there is to it. And yes, I realize this means 80%, maybe more, of the operations in in the world is built incorrectly. We just learned Snap's is[0]. Maybe even yours! And that's fine as long as you're working on it. Good news: said exercise is a good chance to fix it!

Now that half the crowd is inhaling to bombastically retort that undoubtedly controversial, yet completely true, paragraph, allow me to quickly redirect:

What's "the real world," anyway? Most of HN forgets a Windows/.NET ecosystem exists, not to mention extra-valley gigs in, say, Nebraska. Would you say the lone sysadmin holding together a hospital in Des Moines is gaining "real world" experience and able to meet you in discussion? Seriously, I hate "the real world" and the people who fire it as a volley during an argument. Even your career is not indicative of "the real world." (Nor is mine.)

[0]: Flagrantly so. I've spent the better part of an hour trying to concoct a scenario where that deal is even remotely in the win column for Snap. Still trying. You enshrined a business disincentive (nay, prohibition!) toward optimizing your opex into a five year contract and $400mm a year operating Snap was some kind of win? ... How on Earth? Even with a quarter billion DAUs...

I really appreciate the kind words, and wish I had more than one upvote for your comment. I'm not here for profit or ego, just to share what I have learned and experienced for the benefit of others. "I am old, here are my mistakes, do not make them" sort of thing. Whether anyone believes it, there's a serious Bay Area echo chamber (which HN overlaps with); there is a whole tech world out there that isn't startup culture/tools/methodologies.

If you're ever in Tampa or Chicago, let's grab a beer or dinner my treat. Would love to share war stories.

While I agree with your position in principle, I also come from a "legacy IT" world, the kind where systems are moved to public cloud because developers are several steps ahead of the DC admins, or because the CIO wants all net new dev to be on infrastructure he or she doesn't own. These are not "digital natives" (who have no excuse, imho). These are the thick, long tail or large businesses with aging infrastructure and a lack of willingness -- or ability -- to pay for top talent.

An additional point is that even the second tier PaaS/IaaS providers (like GE/Predix, for example) are trying to get out of the DC ownership business. There's no compelling reason for them to keep their own DCs when a) it's expensive to run & keep fresh, b) it's CapEx, and c) it costs them expensive heads to organize and manage everything.

Snap shouldn't fit the model of legacy IT. To some degree, myspace did and that's part of why they failed.

I also hate when people say "you should just be able to flip a button and switch cloud providers or regions or you're doing it wrong."

That shows about as much of a lack of real world experience as you are bemoaning. It doesn't work that way in real life.

It's an ideal to strive toward.

"Real world" and "real life" dismissals of said ideal are stupid, lame excuses for people who want to make a stupid, lame argument to cover up stupid, lame technical debt in operations by either (a) assailing the credentials of the speaker, as this entire thread has spent much time doing, and/or (b) providing a "well, everybody else isn't investing in this, neither should we" cop-out. And yes, you are doing it wrong. Rather than getting defensive, pulling the knives out, downvoting on sight, and trying to wipe away or justify doing it wrong, why don't you instead realize that it's motivation to do it better? We should all do things better, and every time a single AWS region goes down and takes out half the Internet I sigh because it's this exact, stupid, lame justification fest that results in that situation.

Work toward correct operations. You will never reach it. The cognitive dissonance of these two statements is totally acceptable. If you build a new service today and don't account for DR and security up front it just goes to show me that you haven't learned from the very public failures of those who have come before you, and that to me shows a lack of real world experience.

And yes, I'm aware of several shops that can literally flip a switch, and we're talking cage and ASN, not hobby-scale iOS backends. It's not like finding the holy grail and requires developers who "get" operations. It does exist. And it exists in real life! \u1F632!

The "real world" argument is a classic No True Scotsman fallacy.

"never easy" means something entirely different when you are spending billions of dollars on something.

Actually it is. I just moved about 35k instances and 6PB in s3, multiple many TB pg instamces from AWS to google. It is not that bad anymore. Containers and fast internet make it easyish

Alex, I'm curious to hear more about this experience. Did you use any physical device (Snowball, etc), or simply a very fast internet connection between the two? What made you move to Google?

Also if I could down vote you for saying that person has no experience I would. You just have the data to say that other than not picking the details

Whether you want to run fully in the cloud, or homegrown DCs or a combination, you need talented people who bring the required savings about.

When it comes to in house DCs, very few companies seem to have that top talent.

For example, most companies are simply buying regular Dell/HP servers, Cisco switches/routers, slapping them in cabinets and calling it a day. They simply do not know how to take advantage of high density platforms like open compute, or of SDN. They also throw millions at software vendors like Vmware/CA, instead of building their own provisioning or CPU/RAM/Disk aggregating solutions with open source or custom tools.

If they don't know how to do it, then the theoretical savings for certain companies, of bringing things in-house simply won't materialize. And then its better to throw money at AWS if your in-house "engineers" are also breaking things 10x more often.

In a company with a badly run DC, developers very quickly latch on to cloud benefits like not having to wait for 3 days for DNS request, and 2 weeks for a VM.

It doesn't take "top talent" to run the physical side of the DC. It takes processes and detail-oriented people who care.

At this point, with datacenters having been in wide production for the last 20 years, there should be plenty of 7+ year veterans.

(though yes, I have known in-house service providers that couldn't set up a new VM to save their life)

Yeah, some of the stuff in this thread is crazy. Are all these people new or what? Not too long ago, running on bare metal was the only serious option (shared hosting is non-serious). Cloud might offer some conveniences (just because AMZ and friends have made it so easy to give them more money), but the alternative is not that hard!

I agree the alternative isn't hard. I'm saying even very large companies just can't seem to do a great job of it. Which is why they run to AWS/etc in droves.

I've seen innards of very large DCs for more than a decade. At one of my first jobs at a fortune 500, I was responsible for everything from rack and stack to the command prompt. The expected turnaround time for a single physical server was 4-6 weeks until the application could be installed on it! One of the reasons was that they did not have automated DHCP/PXE provisioning. I started the process to enable it.. going through all the political, security mess, it was 9 months until it was enabled. I was gone by then.

An extreme example for sure, but if AWS revenues are growing like they are, then surely such issues are everywhere to some extent.

Because of google container engine (hosted kubernetes).

https://cloud.google.com/container-engine/

Also them starting the project along with the knowledge they have internally scaling containers helps.

Can anyone here provide some info or make a comparison with the Azure Container Service here, or any AWS option?

I'm going to go through all of the service offerings this weekend - from Docker Inc's Docker for Azure and Docker for AWS to the native container services on each.

Azure Container Service is simply a PaaS-ish offering of Swam, DC/OS, or Kubernetes. It still spins up VMs that you can log into, but handles deployment/provisioning of the product, and makes some assumptions about your use case. It's a great way to push a button and have a "real" deployment of those services to evaluate, especially if your goal is a platform-agnostic target. I work for an Azure-focused cloud consultancy and for any serious production environment we still build out a more custom deployment using a combination of Terraform, Chef, Cloud-Init, CoreOS, etc.

Thanks for the comment - I'm going to try it out. Want to email me your consultancy (email in profile) - I work with a bunch of diff companies who use Azue

> Also them starting the project along with the knowledge they have internally scaling containers helps.

FWIW, Most of this are advertising gimmicks though and Google has a pretty different internal infrastructure for orchestrating containers that has hardly much to do with K8s.

Google Kubernetes Engine still runs Kubernetes. I never looked at the Borg or Omega source code and had never been worked on a Google team. It is my understanding that there are some key insights developed from Borg and Omega that became part of the core concepts of Kubernetes that gives it an edge over other open-source orchestration systems. These include grouping containers into pods and using label selectors.

Yes, many of Google's technical leads working on Kubernetes and Container Engine are former members of the Borg and Omega teams, so Kubernetes and our hosted version, Container Engine, both benefit from what we learned building those other systems. (I think our 5 most-senior engineers have ~40 years of container management systems experience between them now?)

And it's not just the rather-large core team directly on GKE and k8s, nor the related products like Container Registry [1], Container Builder [2], and Container-Optimized OS [3]. GKE and k8s benefit in other ways too: Google's internal kernel team helps debug customer issues when we trace them to the kernel, and people like Kees Cook are helping with the upstream Kernel Self-Protection Project [4] that make container technology more secure. In addition to that kernel work, Google also has rather-decent security teams and they work with us to improve security in other ways too.

Finally, re: toomuchtodo's question, "Why opt for Google if you're going to use containers in Kubernetes?" Because we hope you find that Container Engine is the best place to run Kubernetes -- and benefit from the other parts of Google Cloud Platform. If you ever find GKE is not that place, and you don't derive value from the rest of GCP, then exactly as toomuchtodo puts it: "You can even move to your own datacenter at some point (relatively) easily."

[1] https://cloud.google.com/container-registry/

[2] https://cloud.google.com/container-builder/docs/

[3] https://cloud.google.com/container-optimized-os/

[4] https://www.linux.com/news/google-developer-kees-cook-detail...

This is not correct. I've worked with both borg and k8s and k8s is effectively a rewrite of borg using the same container infrastructure. There are differences, but they aren't meaningful.

I can think of a couple that seem meaningful, like cluster state management architecture (borgmasters/checkpointing vs. everything lives under consensus in etcd -- ish), which seem to have introduced real difficulty in bringing Kubernetes to parity with Borg, particularly in the scale department. Then I see a comment like yours and realize again that thought was put into it by much smarter people than me, but that one remains a perceived change that confuses me as an outsider. I'm familiar with the flaws of the borgmaster architecture, but the etcd architecture seems like an oddly drastic rejiggering to address them; I say that with a surface-level understanding of both systems based on a very short exposure to Borg several years ago, so I'm probably completely wrong or out-of-date here.

Am I totally off-base, if you're able to speak to this? (Maybe it exists and I've missed it, but I'd love to see a blow-by-blow of the differences and their rationale, too, because that'd be valuable insight on how Google learns.)

I'm not sure what you're talking about with consensus and etcd. That doesn't have anything to do with the end-user experience using k8s on a product like Google Container Engine.

When I say k8s is like borg I mean: it has the same concepts of tasks, jobs, and allocs. The scheduling of those is handled by a k8s scheduler which resembles the borgmaster scheduler (a lot of hand waving here), and the containers themselves execute in an environment much like the borglet provides for containers.

Many of the valuable features provided by borgmaster and borglet are provided in k8s and you configure them through similar mechanisms.

Beyond that, how they are implemented specifically, there are a ton of differences but for an end user who is just using k8s, not setting up and managing k8s infrastructure, it's conceptually isomorphic.

"For less than what Snap is spending on Google cloud infrastructure, SpaceX built a rocket that can take a payload to orbit and return the first stage successfully (SpaceX has taken on ~$1.2 billion in funding over the last 14 years). Moving out of a cloud provider is comparatively hard?"

Brilliant.

>primitives that are difficult to reproduce on your own at another provider (witness how terrible Open Stack is; no one wants to do that if they don't have to).

I don't understand this statement. I have built systems that work with openstack and then scale up to public cloud. The primitive is the same as EC2, which is a virtual machine. What did you have difficulty with?

But that is one of the outs that Kube gives you. Start on Kube/GCE and then if needing, you can migrate to colo boxes, and dynamically even. The beauty of a Kube based cloud platform is you can literally start anywhere, go anywhere.

There are some companies using k8s not just for the ability to move, but as a hybrid starting ground. One bandwith intense workload is running with GKE for all state/databases, but uses a bare-metal k8s cluster for compute and bandwidth intensive stuff. They actually use federation and the bare-metal cluster basically looks for a local database doesn't find one and gets routed to the next available global cluster, which is GKE.

Due to the cache and read dependend nature of the database queries the latency impact is worth it.

Gitlab reversed their decision to go with bare-metal. They still plan to ditch Azure and the most likely candidate (already testing to move their CI infra) for the move is GCP/GCE and GKE from Google.

Regarding gitlab, they have decided against moving to bare metal for now. (Can't find the HN comment from the gitlab ceo to link here)

You can even move to your own datacenter at some point (relatively) easily.

Dropbox built out their own environment (and did it migrating 500PB out of S3)

Like most discussions in this thread, this statement is way too general. Dropbox moved their storage from S3. What about EC2 or other AWS services they were using? Did they abandon all of those too?

You're overestimating the amount of work it takes to run compute on anyone other than Google or AWS.