Walmart Will Manage Distribution Centers with OneOps, Jenkins, and Kubernetes
techbetter.com
techbetter.com
I've commented before that Kube absolutely takes over the bigger more complex cloud installations out there, you can see how many companies are betting their infrastructure on it.
The only thing that I don't see is standardization of the cloud, just like what Amazon did. You see too many companies doing too many of the same things and reinventing the wheel.
Personally, I would love to see smaller installation as a standard of how to take things into the cloud as a cluster. Imagine what Heroku did to deployments. You can't beat this ease of use. Deis and Convox are both trying but not really "hitting it".
As for Walmart, absolutely stoked seeing it from them. This move and what the white-house did with the digital shows a lot of promise and hope. I wonder how much of this is on top of "older" management and how much is just complete restructuring.
My colleagues at Red Hat might disagree, they're working on OpenShift.
My more immediate colleagues at Pivotal, IBM, SAP, Microsoft, Google, Cisco, Dell-EMC, VMWare et al might disagree too. We're working on Cloud Foundry and BOSH.
Disclosure: I work for Pivotal.
Here are things that should be standard (IMHO) and you shouldn't reinvent
1. Underlying infrastructure auto scaling (Google, AWS) 2. Service Discovery 3. DNS (internal and external for multiple sources) 4. Networking
The real issue that I feel no-one really answered is who's this for. If a SRE is the target audience than there's a lot more we're missing as a community.
The thing is, everyone has a different don't-reinvent list. And they want to be not reinvented in different ways. Then they discover all the things that they didn't realise they'll need to reinvent.
> The real issue that I feel no-one really answered is who's this for.
I see PaaSes as serving three constituencies.
Operators, who wish to christ that Developers would stop making their lives hell by breaking stuff.
Developers, who wish to christ that Operators would stop making their lives hell by blocking stuff.
Business, who wonder why everything takes so long, costs so much and breaks so often.
I sincerely hope that's not the case anymore but that lasting first impression has stopped me from looking into it since then.
Getting up and running with OpenShift is pretty easy. You can either use the all in one Vagrant VM[1] or you can download the CLI[2] and run `oc cluster up`[3] to install the docker container version.
Either one doesn't give you an "HA" install (probably want a cluster for that). I regularly deliver OpenShift installs, and in most enterprise environments an HA install takes 3-4 weeks, most of which is communicating to all the silos (networking, storage, security, virtualization, etc) what the requirements are.
Disclaimer: I work for Red Hat.
1: https://www.openshift.org/vm/ 2: https://github.com/openshift/origin/releases 3: https://github.com/openshift/origin/blob/master/docs/cluster...
I think #1 and #2 should be the same people. with PaaS I don't see any need for "operators".
If an engineer can't launch a new service with all the requirements EASILY, you're doing something wrong
You're == all of us, myself included.
Yes and no. What we're finding is that you need far fewer operators. But you still need someone to keep an eye on things and run 'bosh deploy' to upgrade the cluster or add a new service. The latter could be automated, but it's one of those things people like to do under supervision.
At Pivotal we run Pivotal Web Services (PWS) -- a version of Cloud Foundry which is usually less than a week behind the current release -- with three shifts of 3-10 people each. Two to five pairs, is how we think of it.
PWS has thousands of VMs and tens of thousands of applications running. Pre-CF, pre-BOSH, an installation of this magnitude would need hundreds of sysadmins to stop it from immediately bursting into pretty but expensive flames.
But in general, you're right. The contract with engineers is "tell me what you want and I'll give it to you". Cloud Foundry does that well.
Some virtualization use cases can be solved with kubernetes, but many things an enterprise runs do not work at all in the kube paradigm. The majority of applications are pets (i.e. stateful) and many applications run on Windows or have some other kernel requirements that mismatch the baremetal kubelet.
If you think Kubernetes 'absolutely takes over the more complex cloud installations', you're living in an echo chamber of 12 factor apps that doesn't line up with the majority of what I've seen in big enterprise cloud workloads.
Case in point: Walmart has one of the largest openstack deployments in the world.
For instance, would a complete Windows Server instance fit the Pet Set concept in the scheduler?
PetSets are simply ordinally ordered container collections instead of randomly ordered.
The usual concept in Kubernetes is that a container has a name like 'thing-randomsuffix4242'. It is pushing you towards making your pods stateless and not precious so the failure handing logic is simple. If it is blown away for whatever reason can easily be replaced by the scheduler and 'thing-randomsuffix242b' is never far away.
With a predictable ordering you can actually make assumptions you otherwise couldn't. For example, perhaps you want the convention that 'thing-predictable1' is the master and 'thing-rpredictable{2,3}' are the slaves.
so rather than 'kubectl -v', you will run 'hyperkube kubectl -v'
Edit: You can find Helm charts here: https://github.com/kubernetes/charts/tree/master/incubator
We had a talk at the OpenStack Summit about how we do OpenStack on Kubernetes, if you're interested: https://www.openstack.org/videos/video/cisco-containerizatio...
I'm heavily biased towards Deis as I've been using it for a while (and even wrote an UI for it[0]), but what do you feel it is missing? Feature wise, it's almost at parity with Heroku and stability wise, it's not perfect but it doesn't require a PhD in DevOps to maintain either.
I replied to the Deis engineer above with my thoughts.
Can you elaborate on this? Any feedback on how we "missed the mark" is always helpful.
(I am an engineer working @ Deis)
The problem with all projects, not just deis is that the end user needs to know too much.
I launch about 30 projects a year on top of Heroku and it never ceases to amaze me how simple it is. Yes, the applications are simple. Classic app->db. That being said, the setup is stupid simple.
When I started working on the-startup-stack[1], I was amazed on how much you need to do when you start from scratch. Things I forgot I even did since I have a system running in production with incremental changes for 5 years.
You need to configure networking, decide on DNS, decide on so many thing. It's a lot of work.
Here's what I want (and what I am trying to do with the-startup-stack[1]).
0. AWS KeyPair 1. create-stack 2. launch-app
Lot of what's missing is best practices and production-ready environments.
Going back to that same lunch discussion I had with the CTO of a big company doing a replatforming of their infrastructure: Generalization is the hardest thing here. Reasoning on who's the customer and what does their stack look like.
If you give Deis/OpenShift/Kubernetes to the typical YC founder (I am not trying to insult anyone here) that's trying to get their app in their cloud, it's just too much. It's too much of things they don't care about right now. By not caring about it right now they are likely vendor-locking themselves for a very long time.
We had this dream that users can use Deis as a self-serve dev shop but we've been seeing a lot of people interested in running their entire PaaS stack, so the docs definitely reflect the administrator more so than the platform user (like Heroku).
[1]: https://github.com/segmentio/stack
It came out of our infrastructure at Segment and works really well.
Deis and Cloud Foundry both support buildpacks, so to some extent, it would be possible to move off when the time is right.
Disclosure: I work for Pivotal, we're the majority donors of engineering to Cloud Foundry. (I'm on the buildpacks team as it happens)
Any custom script is just bash, and I can deploy to any OS.
I have a cluster of 5 or 6 instances running 20-30 services in this setup and it's been a dream. I tried to do something similar before, but Cloud66 dropped Joyent. It makes perfect sense because apps can be anywhere between traditionally deployed to fully containerized and still are managed with the same interface and config.
Since it's all open-source my deploy process is portable. But I've evaluated other vendors and processes and all seem more manual and less transparent.
The basic premise is that with standardization of hosting environment platforms an exchange is able to offer their customers a multitude of vendors who compete on price and will have the same features within their environments.
The most interesting opportunity with this is for resellers of cloud computing who can sell compute resources to customers at full price but only pay the exchange based on what they use. It won't last long but if it works resellers will make a fortune migrating customers from big names like Amazon to other big names like IBM through UCX.
It was almost as easy and quick to setup as Heroku. All the docker and AWS config was automatically done by Convox. The only issue I had was with Docker: an old Docker Toolbox installation left some environment variables preventing Docker for Mac from starting.
Granted, I have some previous experience managing deployment on AWS via AWS CLI and dashboard so maybe that's what made me quickly understand Convox concepts.
My only regret is I didn't setup with Convox from the start. :)
It's almost impossible to just drop into an existing workflow without a whole team of people managing it.
/bin/umount does an fstat before the umount syscall, so if an NFS mount is broken, it will probably hang forever instead of unmounting it! You can invoke the syscall directly to get the behavior you need.
About the only thing I can think of is Ceph, and it stands to reason there is a lot more NFS than Ceph expertise on the job market.
You're required to SFTP up a CSV file of what you've shipped to them that day, along with information about. They also have a creaky acknowledgement/acceptance procedure and the technical folks (seems outsourced) aren't very impressive. You have to pass a 'visual inspection' with your test files and it's the most ridiculous process you can imagine with box-checkers flagging you for the dumbest reasons.
So from this side of the transaction it always makes me wonder about articles like this, all their good tech must be internal only. To be fair to Wal-Mart the situation with other big store chains usually isn't any better.
I've also seen the Jenkins-to-Nexus thing a few times, never particularly happily. That said, I don't have particularly deep experience in Java shops, so it's possible that it works really well in some places.
Disclosure: I work for Pivotal, we're the majority donors of engineering to Cloud Foundry, a PaaS competing with OpenShift.
> As shown in the above diagram, Jenkins pipelines deliver application updates directly to the enterprise Nexus instance