Rolling your own servers with Kubernetes
gravitational.com
gravitational.com
Prior to AWS, I was in IT Operations at a large financial services company. I saw the writing on the wall that over time, companies would not want to manage this part of their IT infrastructure themselves. Keep in mind, I was someone who was responsible for keeping the lights on for a decent number of Linux severs.
For an individual company, there really isn't much value in having to maintain firmware levels on all your hardware, patch hypervisors (and try to coordinate all of the maintenance around a fixed pool of hardware), perform months-long evaluation of new hardware before purchasing, test and validate configurations on new hardware, etc. I used to do all of this. I don't miss it either.
Yes, the items above are important, but doing them right is really table-stakes for any reliable IT Operations department. You can choose to spend time getting these right, or delegate that responsibility to a service provider whose main job is to get that stuff right (and recoups that cost across a much larger customer base).
It's convenient for cloud vendors to have people believe the choice is between them or having to deal with hardware.
I also worked in ITOps at a medium-ish company and we were moving our colo to Azure, when I left.
Not saying at all that is how it SHOULD BE, but if you are planning on pulling back to onprem (or colo) it should be a concern as it is a hard to mitigate risk.
so while yes, there weren't any fixes for your onprem virtualizers -- there also wasn't any immediate danger as the attacker had to compromise one of your nodes before actually using these attack vectors...
First, having a full level of control can be desirable.
Second, you trade off maintaining firmware levels on hardware and software levels on hardware to managing the way cloud providers build networking and servers themselves (and the cost control as well).
Maintaining your own hardware and keeping it up to date really isn't as bad as people make it out to be. Its not hard to get right either, assuming you can hire any sort of decent system engineer.
If you are not certain, the answer is most likely “no”. The staggering growth of AWS happened for a reason.
Funny how for many decades companies and people were running their own servers. The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to afford their own infrastructure.
Who exactly is saying that no one can afford their own infra anymore?
What some people say is that cloud is usually cheaper than running your own infra. They may be wrong (I'm doubtful myself), but it's a wholly different argument.
That tends to lead to the typical old-style IT organization which is nowhere near as responsive to the needs of the business as it could be, and this is a competitive disadvantage.
The value proposition of cloud is much more than just technical. Engineers who don't understand why cloud is "winning" are focusing on the wrong factors. The technical aspect is only a part of the picture.
That conspiracy theory depends on not asking about the major costs areas you left out (e.g. ops, security, reliability) which are the most significant until you're at a fairly large scale. The places I saw switching to cloud providers did so after comparing the quality & cost of the options available, not because they were following the cool kids.
Places switched because they didn’t like having to forecast capacity so hourly billed resources turned on and off at the drop of a hat were super attractive.
They don't go away, but you have to spend considerably fewer resources on it because Amazon does a lot of it for you.
Edit: This effect is even more pronounced when you use Amazon's more specialized managed services, like S3, RDS, Lambda, etc. Then the things you mentioned almost completely go away, as you're letting Amazon completely maintain your infrastructure.
It's actually fairly difficult to make a bucket public without going through a few hoops even without that.
The only reason this is happening is sure laziness and lack of any true change control processes. It's not because it's hard.
S3 based URLs to get cheap web hosting for low traffic sites is exactly what leads to bad permissions as well. I’ve seen plenty of S3 objects that are made public so that they can be viewed from a web browser and are just a badly targeted script run away from being on the latest tech blog about how some other institution leaked PII.
Physical server management is in general a liability and cost center.
I'd argue that in many cases they do get significantly easier in a cloud environment especially for lightweight setups.
For example, if you run a managed sql database in aws backups are largely handled for you. Upgrades take a handful of clicks and ~5 mins. You dont have to think about patching the underlying operating system.
Broadly speaking, cloud providers give you lots of tools to make things that were fiddly much easier.
Beyond a certain level of scale you can find efficiencies which pay for all of that but many places aren't at the level where it's unquestioned win, and doing it yourself means you have to start paying the full cost immediately in the hopes that it'll become a win in the future.
Short-term budgets and time windows to meet deliverables shrank along with the costs. That's why so many are on the cloud.
Suddenly buying new equipment that we would use for years became like pulling teeth, no money they said. They couldn't explain what changed.
Then just as suddenly they were willing to approve spending more than our one time equipment purchase... per month in monthly cloud costs / some poorly thought out cloud deployments....
Result:
-1 Dedicated person to manage the lab.
+2 two new way more expensive dev ops guys.
-X Occasional new equipment costs / service contracts.
+Y A metric ton of reoccurring monthly costs.
"job added!"
This is awesome!
Edit: Your math seems off as well. Don't forget that you'd have to pay the SQL Server license regardless of where it's hosted, as well as the cost of dedicated hardware if you go that route, so the difference between the two options will be much smaller than $39k per instance. And I don't know if that salary number is real, but $150k including overhead seems really low for a good DBA (at least in the US)—I would expect at least $200k.
And obviously there are no absolutes—there are certain workloads that really do benefit more from dedicated hardware, even at smaller scales, and there are huge companies that are better off with cloud services (ex: Netflix).
The decision framework the executives use isn't just the "hardware+software" -- it's the whole "IT organization".
In other words, it's not "in-house cpu" vs "Amazon's cpu". It's in-house IT employees' speed of tech innovation vs Amazon's engineers'. An example of this disparity was Guardian's disaster with its in-house organization trying to use OpenStack.[1]
For many non-tech companies where IT computing is a cost center, their employees won't be able to match the iteration speed of Google's engineers constantly improving on GCP or Amazon's employees enhancing the features of AWS.
We've all heard the stories where a company's project submits a requisition to internal IT department for 2 development servers for their programmers -- and then the IT bureaucracy tells them that it will take 2 weeks. Over time, the internal IT dept treats the other departments as adversaries instead of customers. Executives get fed up with slow IT departments and get excited when a few clicks on AWS dashboard gets them servers spun up in 10 minutes. It's not just a cpu+hardware comparison.
Companies outsource to AWS/GCP/Azure because it's quicker turnaround with more datacenter features than their internal IT teams can deliver. Most companies are not like Facebook or Dropbox that can maintain an internal IT organization at a high level equivalent to AWS.
[1] https://www.computerworlduk.com/cloud-computing/guardian-goe...
Let's try this with different phrasing. In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. Forward 10-13 years. We have cheaper hardware. We have extra 10+ years of development in open-source software. And yet that list of self-hosting companies shrunk by a huge degree. Doesn't that seem interesting?
Of course it’s a trade-off.
If you’re large, with constant high resource needs and benefit from custom optimization, you may be better off with your own team/infrastructure.
If you’re small (2 it guys), fast changing, normal speed requirements / easy to scale out then the cloud should be interesting to you.
But with the cloud its a whole different domain. It may be easy too, but I don't think its as easy as hosting your own.
And your last two lines are great. It really comes down to the company about which tradeoff is appropriate.
But my point is that companies' IT departments did not maintain infrastructure just like Amazon did.
In ~2005 when companies were first experimenting with AWS cloud, they might start with dev & test servers. They click a few buttons and are amazed when new servers get spun up in minutes and their programmers are productive immediately. The natural question that company execs ask is, "why can't our own internal IT department spin up servers for us in 10 minutes?!? Why does it take them so damn long?!?"
They wouldn't have asked those hard questions if their internal IT capability was equivalent to AWS. Eventually, their improved experiences with AWS on Dev&Test&QA convinced them to migrate mission-critical Production workloads to AWS as well.
>We have cheaper hardware. We have (supposedly) better software.
You're still focusing on on hardware+software and not considering the IT employees' speed of execution in how company executives compare the situation.
Even Netflix as a tech company maintained their own datacenters for over 10 years but ended up migrating to AWS. Their "Guardian" moment was a big database corruption in 2008. They also had ambitious plans to expand into countries outside of USA. Those were some of the factors that convinced them they didn't want to invest anymore in their own datacenters and preferred AWS take care of it. Amazon employees iterated on datacenter capabilities faster than Netflix engineers could do it.
Unless things have changed since 2015, reports say Netflix eliminated their last datacenter already.[1]
The Netflix "edge appliances" for CDN streaming are located in others' datacenters owned by Verizon,Comcast,ISPs,etc.
[1] https://arstechnica.com/information-technology/2015/08/netfl...
The point isn't that they are still _in_ datacenters. Yes, of course, they are. Even the "cloud" ultimately resolves down to somebody's datacenter somewhere. The point is that Reed Hastings & Netflix wanted to get out of managing their own datacenters.
Putting their Netflix appliances inside of ISP owned datacenters still lets them avoid managing their own datacenters. Their critical user accounts signup, monthly billing, and analytics, etc workloads are at AWS. And as the article mentions, even updating the cache on the Netflix appliances is coordinated through AWS. The combination of those strategies keeps them out of the "datacenter business" and let's them stay focused on their core competency of "video content".
I’ve avoided my own datacenter by using Equinix, for example. I wouldn’t call that “cloud” though.
If you are a Comcast customer and your internet goes down, and Netflix is unavailable, who do you complain at? If you're smart enough to notice that both are down, the answer is almost certainly not Netflix. That does not make the Netflix workloads in Comcast data center any less critical, they are core business functions. But they are well aligned with Comcast, who also depends on the proper functioning of those datacenters.
It makes sense for Netflix appliances to be in Comcast datacenters then, especially given that Comcast cannot outsource their data centers any more than the Pentagon can reasonably do so.
Joe Company from off the street can outsource their data centers and derives no competitive advantage from maintaining their own private data centers. Netflix in that sense is closer to Joe Company than they are to Comcast, I guess. I'm not sure what all we can learn from this, but it's interesting.
AWS is used for control-plane / management-plane tasking - but the bits are pushed from locally-peered CDN boxes at all kinds of ISPs.
> It's in-house IT employees' speed of tech innovation vs Amazon's engineers'
The cloud shrunk the cost of IT team. Sure hardware is cheaper, but that cost is marginal versus employee cost.
I would question that equivalence. Speaking as someone who was trying to get what we now call devops going at the time, there were a _few_ companies starting big wins from automation and a ton of places which were content pouring huge amounts of human time into doing it the hard way and slow metrics for time to patch, provision new services, recover after failures, etc. When they had automation, it was known-bad practices like cloning VMs and had accordingly greater cost and lower benefits.
The companies in the latter groups were the ones who were faced with either spending a huge amount of time and money catching up or cutting a cloud contract and not having do a large percentage of that remediation work at all, while being able to deliver results immediately.
I can only think it was "interesting" if you've forgotten how difficult and expensive managing your own infrastructure can be.
So, there were a lot of, and still there a lot of corporations running their own stuff. Some of these are still running their own stuff well. Some got big, some stayed small in the past decade (some downsized).
It's not necessarily hard to run your own stuff either.
Of course, of course, managed platforms are easy (or easier) from a lot of aspects. (That's their value offering after all.) And the IT expertise market was never exceptionally great in a lot of places. Meaning it was hard to find good sysadmins, good ops people, good devops folks, good programmers, leads, tech PMs, POs, etc. who were able to work together and effectively run their shit, build their product/service, run with the flow (of open source or whatever vendor they used), and so on.
AWS/GCP/Azure is a big paradigm change. No longer do you need to argue about what and how, at best you pick one of the three, and that's it. No longer you need to think about hardware, colo, uplink, switches, peering, multi-homing, picking the right dedicated server provider, and so on. (Sure, you need network guys, who at least have some basic understanding of VPC - overlay networking, but the basics are easy, and the rest is YAGNI anyway.) And this is due to their size. They are big enough, that their price premium - compared even to whatever random small hosting provider you might find on the 'net - is irrelevant, because they are usually more efficient, more secure, more geo-available than the small ones.
But, ... of course, while generalizing all self-hosted companies (meaning that all self-hosted is a pain) leads to a falsehood, on the margin, especially during the IT consolidation of the past years, most companies moved away from self-hosting. And developers usually rejoiced. And similarly, generalizing the problem of self-hosting from your experiences again leads to a flawed conclusion (that it's necessarily hard and expensive to manage your infra), on average, outsourcing a cost-center in an ever increasing complexity world (that is potentially unbounded costs) and focusing on profit-centers is a sane decision.
Having had to interact with sysadmins and others, first at at a university and then in business, "work together" is too frequently overlooked.
The crusty, angry and rude stereotype proved right. I've never met a bigger group of downright bastards. Those who were nice people usually couldn't get the servers/software to work reliably. So you were forced back to group 1.
Otherwise, I’ve did some tests years ago and what I spent over 5 years would have gotten me only a few months on AWS. And I think I only walked into the data center once in those five years.
I worked at a decently large tech-ish company at that time. It took 3 months to provision a handful of dev servers in the data center. After filling out forms in triplicate. We asked for extras since we had almost zero ability to manage them (reboot, etc.) or get them re-built if something broke. No backups of any kind as far as I knew. We once had one break at night and apparently someone had to psychically drive there at 3am to reboot the thing after we filed a ticket. Used some sort of in-house distribution of linux that had it's own quirks.
I'll take the cloud any day of the week.
I can provision a handful of production servers in 5 minutes.
You're not in touch with reality. Try 2 months at a bare minimum.
Exactly this. As an ops person, this is exactly how I explain it. Sure we can build something fairly competitive on-prem or in a colo, but we won't have an entire team of top-flight experts bent on improving it as fast as possible. That per-GB cost is buying a lot more than just bandwidth and drive space.
> Over time, the internal IT dept treats the other departments as adversaries instead of customers. Executives get fed up with slow IT departments and get excited when a few clicks on AWS dashboard gets them servers spun up in 10 minutes.
This tends to be more of a problem with organizational mandates & processes. If those processes aren't addressed, then you'll end up with all the same problems. Perhaps they will have different labels on them, but underneath it will be the same issues, delays, outages, and recriminations
I have been working with them for the last few years and they've been great. They even roll out new hardware to existing customers for no additional cost.
Blows bad hosting companies out of the water. For example Media Temple who never upgrade their hardware for existing customers at no additional cost.
This is my view based on the work I've done. I like the ability to dispense with capacity planning or dealing with power supplies and fire suppression you get from cloud hosting, but when in any doubt I set up what I need using VMs (be those droplets or EC2 instances or whatever).
I like to imagine that I've avoided wasting weeks by wasting a few hours here and there.
It's not like the cloud is a magical place where no unexpected issues ever happen. Cloud providers can be surprisingly buggy, especially AWS, particularly at scale when you start hitting the implicit scaling limits their docs conveniently forgot to mention.
Each technology has their unique challenges.
An extra abstraction layer like k8s makes it a lot easier, which is exciting. It's also precisely why IBM bought Red Hat - a well-built k8s distribution like OpenShift is one of the very few real alternative to public clouds for many companies.
OpenShift is built on a prescriptive mentality. They choose everything for you from OS through pipeline and many deviations are unsupported. As you mentioned there are others, but depending on your environment OpenShift is either a very good fit or square-peg-round-hole.
It's not like I'm personally afraid of k8s or building docker images myself either, but anything to help my team be more productive without needing to hold hands or do gruntwork on their behalf improves the lives of everyone.
Running custom k8s in production is the 2019 equivalent of using Gentoo.
0: https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...
Yup. We've run into that repeatedly. The "we didn't notice anything on our side, please send more screenshots and logs" gets really old when working with "managed services".
Network packet losses/truncations, EKS control plane failures, cloudformation stacks getting stuck in really wierd states, inconsistent cloudformation implementations for new and existing services, ENI weirdness in containers...
Managed services just feel like they aren't - more and more every day. Amazon (or any other cloud) will never have the same investment in your availability and infrastructure as you will.
For a team with the goals of IAC, hands-off, and high-availability infrastructure, migrating is a pain.
It would definitely be an annoying migration, even if import works for your use-case, but honestly, dealing with CloudFormation is so irritating on a day-to-day basis for me that I'd consider it worth it.
The problem is that the reality has not lived up to the promise, and "first party support" means "only the first party can support".
After a few rounds of back-and-forth, I eventually gave up.
(I even gave them permission to copy test snapshot data, at their request, and sample queries to reproduce the issue. They could have done all the testing they wanted in-house. But that might have actually required them to spend time on their side to diagnose and fix something, I guess.)
It wasn't a critical issue, exactly, and we worked around it in code. I mostly just wanted to report a problem.
This is all operating on the assumption that those entities will care enough about your problem enough to do something meaningful to fix it. As others have stated here, at least in the context of "Big cloud providers", that frequently is not the case. Often you just get the runaround (continuous delays, requests for "more information", other attempts to stall in first level support).
Add a third party "managing" some piece of your company's infrastructure via a cloud provider into the mix and it often gets even worse.
It is true that there are real costs to having your own hardware/software onsite and people who know how to manage it. However, the promised reductions in cost/hassle of moving things offsite are frequently offset or even exceeded by the costs/hassles you get by not having control of things yourself.
All that assumes they even admit the issue is their hardware not your code, which is a mighty big assumption itself.
Maybe you simply haven't seen this kind of operation in action before?
Weeks or months later you're still learning about how you do the thing that will maybe lower costs and do cool things ....
Sadly instead of seeing more of these, most of these are now being outsourced to cloud providers, we have bought and drank the kool-aid that they can do it better and cheap. Which is not true, perhaps this is why it's so hard to find good unix and network admins these days.
I think you can argue that tech is always related to your core business and you only need a few servers to handle a million simultaneous users but those kinds of arguments aren't usually good enough to businessmen or entrepreneurs.
Developing IT policies for physical security, server security, update policies, purchasing, wiring and so on takes a huge amount of knowledge, and the possibility that a middle-of-the-road sysadmin is not only competent at, but excels, in all those areas is literally zero.
Cloud providers have world-class experts in each area whose sole responsibility is providing for their specific area. It's economies of scale at its best. It doesn't mean that it's always the best choice, but having to have someone do full-time server maintenance at no additional value to yourself or customers is just a huge drag.
I'm pretty sure that is true. However, the parent said "they can do it better and cheap", and cheap it isn't.
The few times I've done the analysis, cloud servers tend to cost about as much as self-hosting, when you factor in the cost of electricity.
There are reasons one may argue against cloud servers, on the basis of the internet's health, user freedom, etc. But for me and for many people they are at least both better and cheaper.
1) Do you really need to invest a million dollars in bin-packing containerized stateless microservices?
2) You don't have to use K8s to get the benefits of rolling your own servers. In fact, I'd argue you should do the latter well before you do the former.
3) Definitely hire someone who has done it before. You will save so much time and money your head will spin.
4) Do not just build a rack full of random commodity gear. Make sure it is suited specifically for your purposes, and then weigh the cost of service contracts and managed colo against a $100k+ cage monkey on call 24/7.
5) Do not fall for the "We've got <insert tech hype>, we don't need redundant hardware!" lie. The more parts of your system rely on lots of hardware, the more fragile your system becomes. Distributed decentralized services become a PITA when the underlying gear is flaky, and centralized services require it. Do not underestimate the shittiness of your colo; always design for the most redundancy you can get for what you have. If you can run dual power, do it. Dual network stacks, do it. Redundant disks, do it. Remote management, do it. Outside modem to a management port on the router, do it. Always be postponing entropy. Later, when you become a FAANG (yeah right) you'll have the time and money to automate away some of the redundancy issues.
6) It's sometimes harder to upgrade disk or bandwidth on an existing machine than it is to just buy another machine, but the more machines you have, the more problems (and overhead costs), and there is an upward limit on the scaling of most services without rebuilding them. So buy big on local disk and network, and buy new machines to expand cpu and memory. The redundancy and performance issues inherent to SAN/NAS often make local disks a better choice, as they are unlikely to impact your whole network at once, you can fix/upgrade them piecemeal, and they are less difficult to manage/operate.
7) Don't DIY critical components such as data replication; it's surprisingly hard. In fact, don't even believe vendors if they claim they can solve X for you with some magic software or hardware. Get them to show you a live demo on your own network.
8) Don't forget that in 3 years you'll be replacing it all.
Rolling Kubernetes on your own is quite hard though especially the networking part so there should be a market for a company helping out with provisioning.
Using public cloud can be very expensive for some work loads which were not written for the cloud due to higher resource usage. It is still quite hard to forecast cost of the cloud due to a lot of moving parts and micro billing for each item.
With traditional service providers it can be easier to budget for the service expenses.
https://penguindreams.org/blog/installing-mesosphere-dcos-on...
I remember the manual-ish server provisioning days of Autoyast and RedHat's Kickstart. It's not that difficult to bootstrap physical servers to get them to the point of being nodes in a Docker scheduling system. Rancher is another great tool for setting up K8s (which you can run against physical nodes once you get them past the initial install/bootstrap).
I worked at one place where they took a hybrid approach. They had DC/OS running in the local VMWare data center (which we were migrating to from AWS to save costs), but there were nodes that could scale to AWS as well and you could label your deployments for which data center you needed them to run in. A lot of high volume loads we were able to move back to our data center, but we also had a kick-ass platform team and our AWS bill was well over $150k/month.
The trouble is it's almost impossible to do a real cost comparison. There are just too many factors, and when you start going AWS, GCE or Azure, you start using all their other managed services that may not have self-hosted replacements, not to mention off-setting the costs of setting up master/slaves, backups, snapshots, redundancy and other operations tasks that come by default or with an additional price from a hosted solution.
Of course, it's not an easy task as a beginner to use nginx or traefik in its place, and to handle the complexity of that deploy.
- Kubermatic by loodse
- rancheros
- even mesosphere is kinda that now, except the first two are opensource, but definitely not the only solutions
I personally deploy kubespray here and there, but I would still recommend smartos for normal humans, unless they want to setup an elk + kafka cluster or whatever else you might want to do.
It also gives more flexibility on adding additional services around your monolith. Elk stack, kafka, that kind of stuff. Also gives you a standard api to deploy against. I don't think your metaphor holds up.
"If you consider what it takes to manage the end-to-end lifecycle of a single application, monolith or micro-service, you need a solution for the following items: deployments, application configuration, high availability, log and metrics aggregation, autoscaling, and load balancing across multiple application instances.
Kubernetes provides an opinionated way of doing all of those things. For example, Kubernetes leverages container images and declarative configs for packaging and deploying applications. For many people this approach is much simpler than what Puppet, Chef, and Ansible bring to the table in terms managing applications."
Using Kubernetes even with a monolith can be pretty nice, if (big if here) Kubernetes behaves.
If you consider what it takes to manage the end-to-end lifecycle of a single application, monolith or micro-service, you need a solution for the following items: deployments, application configuration, high availability, log and metrics aggregation, autoscaling, and load balancing across multiple application instances.
Kubernetes provides an opinionated way of doing all of those things. For example, Kubernetes leverages container images and declarative configs for packaging and deploying applications. For many people this approach is much simpler than what Puppet, Chef, and Ansible bring to the table in terms managing applications.
When it comes to high availability Kubernetes provides an orchestration layer across multiple machines, grouped in clusters, that deals with distributing applications based on resource requirements and automatically responding to node and application failures. When applications crash, Kubernetes restarts them. When nodes fail, Kubernetes reschedules the applications to healthy nodes, and avoids the failed nodes in the future.
Many of the patterns for managing applications, even monoliths across a handful of nodes, Kubernetes provides out of the box. In essence, Kubernetes is the sum of all the bash scripts and best practices that most system administrators would cobble together over time, presented as a single system behind a declarative set of APIs.
One other major caveat to all of this.
Just like I would not recommend standing up OpenStack from the ground up in order to deploy your monolithic application across a set of virtual machines, I don't recommend rolling your own Kubernetes cluster either. You should strongly consider leveraging a fully managed Kubernetes offering such as Google Kubernetes Engine, Digital Ocean's Managed Kubernetes, or Azure Kubernetes Service.
The rest seems reasonable, but I disagree strongly with that claim. If it's worth using the tool then it's also worth learning how it works.
Even just kubespraying a cluster myself helped me build a much stronger mental model of how Kubernetes works than trying to take over a colleague's black box Kops setup. GKE or another managed service would have been even worse.
Setting up a small cluster isn't that hard, and it will teach you a lot about the internals and how things can go south (and what to do when that inevitably happens).
[1] https://github.com/kelseyhightower/kubernetes-the-hard-way
[2] https://www.amazon.com/Kubernetes-Running-Dive-Future-Infras...
Working for one of those vendors. There is no cloud, just someone's else computer.
https://docs.google.com/presentation/d/105ZgwafitwXH6_sWevFH...
The setup is simple and it just works, but the maintenance and security aspects were more worrying. One time I found out that one of my nodes failed to automatically apply security patches from my distro. I couldn’t detect it sooner because my monitoring was lacking. The overhead escalates fast.
If I start an effort like this again today, I would make sure to get monitoring right from day 1. Possibly a combination of Prometheus with Grafana, or something along the lines of ElasticSearch APM.
https://github.com/gravitational/gravity
In all seriousness though, gravity is an open core toolkit to package and deliver complicated sets of micro services in air gapped and restricted environments as a virtual installable appliance.
It takes care of both application and Kubernetes lifecycle, software distribution and licensing workflows.
Thanks to these guys, they just lower the unemployment rate a little more!
Link?...
The servers cost me $1200-$2200 each and I would buy a new one every quarter until I got to where I am today. I haven't added new hardware for almost 2 years, but its more than capable of handling my workloads (mostly web apps, lots of servers satisfying sql,ldap,etc dependencies). I burst (and failover) to all the clouds today.
The R520 runs Proxmox and hosts whatever stuff I’m playing with at any given time along with Kubernetes, FreeIPA, Graylog2, probably my mailserver again soon, etc.
One R210 II runs Sophos XG to handle router+security duties, the other runs Windows Server Essentials 2016 for AD and NPS.
The CRS317 switch provides 10Gb networking for storage on a couple of servers, and everything else is connected to the EX2200.
Can't speak about the power given on their $400/mo cabinet deal.
You can build / assemble from open-source bits most of the classic IaaS bits (compute/storage/networkLB). It's doable, the open-source solutions are pretty good. But unless you have 10k compute nodes how can you bear the cost of having the expert knowledge necessary to debug this ?
Hardware sucks and is always on fire. Once you take this into account, the economic equation changes dramatically.
The complexity in our case is in the backend transmission network and unless we control everything (hardware, network) end-to-end as much as possible, we can't guarantee to getting close meet our SLA's for our customer. They wouldn't allow us to run our system in the cloud even if we wanted to.
- Application servers are stateless.
- All state lives in stateful databases.
- Those database handle high availability at the application layer by replication data, so you can use local storage and deploy them using StatefulSets. Has the extra benefit of avoiding the network storage latency penalty.
We're dealing with that exact issue right now. Our K8s cluster is not playing nice with a NetApp NAS. We are exploring other options such as local iscsi storage volumes ($$$) or an external state store db (slow).
[1] - https://rook.io/
I heard from someone at a large company, though, that it's not getting a lot of love from Red Hat nowadays, even if you're a large paying customer. Now I'm curious.
That announcement gave me the impression the project is making active progress.
My team is still very interested in Ceph and Rook, for the record.
These days I usually recommend that any data an app wants to persist be written to an object store or database.
And that any app that needs to write critical data to a disk should be provisioned with config management on bare metal (or pods statically assigned to machines with attached storage if K8s is a must).
This is mostly due to how tough the problem of storage is on K8s, and how much is at risk when it goes south.
I have a Brocade ICX 6450 with 4 10G ports in my homelab, replaced the fan with a whisper silent one. Cannot hear and is right next to me. Brocade ICX 6610 has 16 10G ports and 2 40G ports, not super quiet, but not that noisy either.
That said, I'm very much a networking noob, so maybe there's a way to make that work with some sort of copper to SFP bridge or something. Not sure if that would kill the advantages, even if it existed.
Good idea on replacing the fan rather than hunting for a quiet router!