The sad state of sysadmin in the age of containers (2015)
vitavonni.de
vitavonni.de
Believe me or not, most of the underlying infra does not run on the popular technology of the year. Far, far from it. That's why it works.
Modern devops, with its million tools that break backward compatibility every month sometimes becomes the running joke at lunch.
"Use what works" definitely presents a lot more choices these days and likely will forever more.
"Use what works well" is something different where "well" implies helpful, dependable, predictable, manageable and so on that will continue to scale with your needs. Only breaking things down the "old-school" way will lead towards success, stability, security and life outside the box.
Good devops is still, primarily, good engineers engineering good things, for themselves and others.
Granted the article is from 2015 but my impression is author is not just cranky, but scared.
They're being perverted into a role having a full load of pure operations with shit for processes (and, often, systems) and an expectation that you have time to automate and shore up all the shit and technical debt accumulated since.
Can most good or even extraordinary developers simultaneously be elbow deep on a dozen unrelated products and actually get reasonable traction? I can barely keep one glass castle together, myself.
Never having the time to properly finish a project that I was proud of delivering, turning those into services so we could leverage self-servicing was a dream that most of the time never happened, we were left with half-done systems requiring tons of manual intervention (lots of toil) while having to move fast to the next thing...
Whats a “devops person” for you and what keeps them busy? And why cant they have systems that hum along self-healingly automated?
So, not only is there the responsibility of creating this self-healing automated infrastructure, but keeping tools and resume up-to-date, as well.
Been there, done that, fought that shit the first time. And the second. And the third. (it's amazing how often I find myself solving what are essentially the same problems over and over.) It's one reason why you'll find we'll push back on the "ohh shiny". There are many wonderful and fascinating things coming out. Tech is an amazing field to be working in. But it's also ridiculously frustrating because no one pays any attention to _why_ things are done the way they are, or _why_ approaches haven't worked in the past (I'm all for re-introducing past failed approaches, as long as there's evidence those reasons have been investigated)
You'll find a common trend amongst us in that most of us sort of ended up in the role accidentally. Schools and college teach you to become developers. Few people tend to head to college with the view of specialising in the ops side of things.
Even speaking as a comparatively old-school sysadmin, my strengths come from being flexible and adaptable. What I do today is nothing like what I was doing 5 years ago, and what I did then is nothing like what I was doing 5 years before that, and so on down the line. The field is constantly in flux.
I just have the best part of two decades of experience to both anticipate the problems, and be able to get to diagnosis quicker when things do go wrong.
Even as the older sysadmins die off I'm fairly confident there will be newer ones to replace them, because people are going to continue to learn from the problems they run in to.
I'm even using ansible for ad-hoc stuff (tweaking a config, restarting a service) because it's easier to do that from a management server than log in to some remote host, get oriented as to the OS distribution and version, and run commends in the shell there.
Edit: The thing I really like about Ansible is how unsexy it is, it's just a nice sane way of doing largely what you could do yourself with ssh and bash but in a language that doesn't make you want to cry.
I've been around linux since the 90's and Ansible feels comfortable, predictable and stable - what you would want in a piece of software that can be mission critical in the most fundamental sense.
Want to guarantee your child's future employment? Don't just teach them to code (the machines will do that). Teach them how to build networks and truly understand network protocols.
You know like “ahh don’t call the messaging endpoint more than 800 mega-milli-times per mega-nano-second or it will cost you three bazillion CPU credits, but only on three and a half cores which will starve all your instances, issue an invoice and proceed to melt your credit card.”
I don't know how the situation is in the US, but in my country network engineering is actually quite a popular field of study. (we have college level education in network engineering).
The one thing that stands out though is that it's mostly done by youngsters who have either sysadmin experience, or worked in IT before that. Almost everyone who comes from high school goes into Software Engineering.
I think this is mainly because networking is quite an invisible field so to speak. Many people don't even know your job exists, and many young people only see the shiny hip side of it. (being Software Engineering).
Being good at network engineering is hard, especially once you get past entry level work and actually start being responsible for designing large-scale networks. Mainly because building a network is a major financial investment where garuanteeing performance is hard without either a ton of experience, or a shitton of lab time.
this kind of work pays very, very well though.
Building networks and understanding protocols is something machines can do TODAY. The entire internet was built to survive a nuclear war. It can reshape itself, and it most definitely understand network protocols. By definition, protocols are the language of machines over the network. And it's been like this for a few decades.
The only reason knowing low level network protocols in a world where machines can code anything (which makes them better analytical thinkers than humans) is to beg the machines for mercy in their ancient tongue.
It really, really can't! That quote about the Internet interpreting censorship as damage and routing around it? Or the one about information "wanting" to be free? Taken wildly out of context.
You lose and gain and you should always be mindful of what is coming. Humility is good. I love you young guys, your ideas keep coming and they are mostly good.
The trouble is making sure these plans actually work. This is why Netflix randomly execute some of their employees every month.
I certainly hope this practice doesn't start catching on with other employers.
Would that be their King Kong application?
The key is to pick smart people who are good at learning and find complex systems interesting. Then, of course, you need to have interesting projects for them to work on.
Some monkeys will just keep swinging branch to branch. Some monkeys check out the tree.
Of course, one can argue that that will just cause a new wave of openness and the cycle continues.
The risk is in treating anything as a black box, whether its a managed service or a container you pull from Dockerhub. It's something you'll get burned by eventually and need to learn from experience.
(As almost everyone else points out, the closed cloud vendor stuff is nowhere near flexible enough for most moderately complicated use cases unless you’re running at serious scale.)
This is a false crisis.
Touché. Remember, there's value in battletested and proven, e.g. https://www.quora.com/Why-do-satellites-use-old-processors
Puppet and Chef was the first generation. I wouldn't recommend. All the companies and people I know using Chef migrated away from it after many disasters. Nowadays, it's only mentioned in interviews to find out if candidates have real world fire fighting experiences.
Ansible is good. Used that for managing hundreds of machines at multiple jobs (some who migrated from Chef). It's been bought by RedHat, it's well maintained and I think it will have the brightest long term future.
Not sure about SaltStack. Never had the opportunity to try. I'd be a bit worried though on the long term prospect because I don't think they have much backing or user base.
saltstack is a well thoughout solution in my opinion. It makes more logical sense and is less of a mumbled mess then either chef or puppet and has miles better performance then ansible.
I know quite a few shops who use it. Its definitly smaller then ansible.
At the last gig, I wrapped salt deploys with a small Slack bot, so users would fire deploys from Slack; you could see what was going out and who was pushing. It was a very very nice, simple, fast solution that should scale to hundreds of machines easily.
Salt is not without warts but its definitely worth checking out.
Puppet and Chef is actually quite good and I still prefer it to Ansible for a number of reasons. I've certainly run it fine in environments of many thousands of servers, though I can understand that it can implode for some people at scale if they design their deployments in a certain way or structure their manifests/cookbooks a certain way. That said I've certainly seen Ansible fold on much smaller infrastructure, but that is also down to a number of factors that can be avoided or mitigated. Idempotency with Puppet is really strong which is something you want if not every single system in your environment is ephemeral, with Chef it's almost as good with that but not always with the first run, with Ansible you have to specifically consider and aim for it when writing code for it in your playbooks.
The fact that you get used to having Chef or Puppet run e.g. every half an hour is a good thing, where Ansible runs are more ad-hoc. This leads me to another thing that bothers me and that is where people think it's a situation of having to use e.g. Puppet or Ansible as if they're conflicting choices for the same tasks. They have a lot of things in common, but Puppet is more for managing and ensuring changes in an idempotent, non-conflicting way while Ansible is more for doing something a bit like that but more for ad-hoc or orchestration tasks. I think it's good to use both but also be sure what you use it for, since one can do a bit of what the other thing is good at but doesn't do it so well.
For example, I would consider using Ansible to do deployments and releases, rotate SSH keys, execute failovers, or even to install the Puppet agent for the first time. I would use Puppet to deploy and update monitoring agents and configuration, user access, ensure directory permissions, configure system things like rsyslog, logrotate, Postfix, ntp, etc.
That's mainly because Ansible folks advertise it as a configuration management tool, while in fact it's a deployment tool. The former needs asynchronous operations, especially because a node that is supposed to be reconfigured can be temporarily down. The latter needs to be executed synchronously, with reports being read as they come by an operator.
There are several other operation modes that are useful for a sysadmin, like running a predefined procedure with parameters supplied from the client, or running a one-off command everywhere (even on the servers that are currently down, as soon as they are up), but we don't have many tools to cover those cases.
A lot of folks I know have been bitten by Ansible's performance (Ansible has a central master that runs recipes on each node, rather than having nodes "pull" from a central master).
Later on you can run Ansible Tower, deploy Ansible agents everywhere, and basically use Ansible under the same client/server model like all the other tools.
Salt is eerily similar to Ansible, it's just geared towards client/server. Being experienced with Ansible, it was weird at first to use Salt because everything looked familiar, yet slightly different.
Ansible is not centralized. It configures servers with SSH and can operate from any user or host who has ssh access.
Other than that, my experiences line up with yours almost exactly.
Of all the tools, I first heard of Puppet first and so I'm assuming it was first on scene? From my limited experience, it seems Puppet is most widely used tool because of that reason. Not necessarily the best of the bunch, but first on the scene. Considering the effort required to roll it out, I am assuming whatever is deployed first will stay as the tool of choice.
I've tried out Puppet, SaltStack, and Ansible, in that order.
What I didn't like about Puppet is that once you deploy a change, the actual change can happen on the "client servers" at any point within next 20 minutes. I may be off on the exact duration but I remember that changes were deployed at any point within that range of time. To me that sounds like not a great idea. What if you want to switch over your web servers at a specific moment? And Puppet requires a dedicated command/control server.
Next I tried SaltStack. I liked it enough. Now that I think about it and hear someone else mention it, yah SaltStack is similar to Ansible. What drove me away from SaltStack was that you essentially need a dedicated command/control server from where all SaltStack commands are sent out to SaltStack "client servers". I did not want to dedicate resource (and money) for a server that is rarely used. And the personal web/lab servers I manage can grow small/large from 2 servers to 10 servers.
Next I tried Ansible. I think Ansible is the perfect choice for me. I only needed to 'devop' just a handful servers and also learn a tool that many businesses seemed to want on resume. So I picked Ansible and it's been great. Some operations are not as flexible as doing it with a shell script (and I assume same issue exists for other tools). But I've had good luck combining Ansible with little bits of shell script to get the result I need.
The best part of Ansible is that any Mac or Linux machine can be used as the "command server", provided that you have the SSH key pair on your Mac or Linux machine.
Lastly, some may not like the ad-hoc way of doing things on Ansible, but I prefer it that way.
CFEngine was first, it's based on a kind of maths called "promise theory" and it solved the problem of you had many different kinds of Unix owned by many different groups and had to have a consistent way of saying "all machines belonging to group X need to have user Y and package Z" and it would abstract away the slightly differing syntax between Solaris, SunOS, IRIX, OSF/1, Ultrix, yadda yadda. This is a problem that doesn't really exist anymore.
Chef I think came next, it was written by people who knew Ruby but didn't know maths so they used CFEngine terminology like "converging" but Chef doesn't really do that, it just runs Ruby scripts. If CFEngine was a scalpel, Chef is a mallet. Chef and Puppet are related somehow, same group of devs had a falling out and went their own ways, something like that. They are much of a muchness.
Ansible is cool because it recognises the reality of why CFEngine isn't relevant nowadays: most organisations are running just one particular Linux distro so you can do away with the abstraction and get all the benefits without the complexity.
Promise theory is not math, despite its name. It doesn't predict anything, it doesn't explain any phenomena. It's an architectural approach. Brilliant, led to a really great software (CFEngine), but it's not "maths".
See http://verticalsysadmin.com/blog/relative-origins-of-cfengin...
You don't. Configuration management is a wrong operation mode for a synchronous change. Still, you could order all your Puppet agents to run their scheduled operation now instead of leaving it waiting for its time.
Chef is a runner up. Love their community and Chef is pretty straightforward once you learn the lingo.
Puppet doesn’t really work for modern Git development workflows (Hiera and r10k are duct tape) and testing Puppet is kludgey. Also, most of the docs you’ll find for it stopped getting updated in 2015 or so.
Ansible is just about the easiest and most flexible thing going, but once you hit "very large scale" you're going to get bit by its performance and start worrying about when you actually update things. Ansible Tower starts to look good then, but it's not the well-walked path and brings you all sorts of other issues about how you distribute secrets to bootstrap things.
Chef is kind of nice when you don't have a lot of environments that you need to manage and about as flexible as you need it to be in those situations.
SaltStack shines when you really have a lot of heavy lifting to do and the Event System, Beacons and Reactors will honestly blow your mind with the complex things you can achieve in a way that's simple to reason about and maintain.
That said, there's really like 3-4 majorly different ways you can (or would want to) use Salt and understanding it and its documentation is a large cognitive investment. You will likely run into major pain at some point down the road if you choose to use it. I would only use it again if I had a really good reason to -- pretty much if there's no other alternative. I would not at all bother using it to try and do typical sysadmin automation tasks.
Strange side-note: The best managed Salt environments I've worked in or looked at were all masterless, whether at small or massive scale. It's my probably-wrong opinion that traditional master/minion SaltStack is always going to cause you enormous problems eventually when you need to either scale out or pivot on something.
heck. I have seem some guys do magic with CFengine and freeBSD.
(freebsd is a nice OS from an ops perspective anyhow, and has a lot more common sense and sane defaults in a lot of places).
So fucking true, bro
Ironically, modern devops and its million broken tools are a primary source of revenue for cloud providers, helping pay for your lunch in the first place.
Thing is, none of it matters. Bottom line matters.
They care to attract company decision makers. Decision makers are engineers in small to medium businesses, and managers in big ones. Sadly, its the big ones that matter for bottom line. So target is mid management, flashy power point presentations and 'conferences' that allow for justified travel and stay.
Good mid management, with tech background, exists, but is a minority.
Not all is lost, truth is out there (c) Mulder :))
Present capability vs. future catastrophic risk.
Technology is debt.
Do you disagree with what I said?
Tech-as-debt is a notion I'm playing with.
https://mastodon.social/@natecull/99318348047974414
https://plus.google.com/104092656004159577193/posts/jS1K9Mto...
Obvious antecedent: technical debt, Ward Cunningham.
Before, admins would try hard to prevent security holes, now they call themselves “devops” and happily introduce them to the network themselves!
1) The merging of devs into the sysadmin role was a product of: the work of sysadmins (particularly systems change control and security compliance) not being valued in our culture.
2) Devs delighted to be free of the shackles placed upon them by sysadmins who were encumbered by the concerns expressed in this article.
If you were a devop who resolved to fix the problems bemoaned in this article, my guess is you would turn around in 60 days to discover you'd become a sysadmin.
The stated goal of putting both systems administrators and software engineers on the same team is to reduce friction and increase communication. One of the worst, productivity-killing situations you can find yourself in when developing network software and services is caused by the traditional "old school" mentality of separating the two camps. When your software developers operate independently of your systems engineers and administrators they're forced to make assumptions about infrastructure, operations, and compliance goals. Both teams have the same goals so why are they not on the same team? I think some "old school" system administrators don't realize how costly such communication mistakes are. Getting 6 months into a development project to be told you cannot have a critical piece of infrastructure _for reasons_ is a costly, costly mistake.
Containers are a smart solution to the build problem. Don't build your containers from public, un-trusted images! Build your own images. Run your own, protected, registry. You still have all of the compliance and validation necessary and you don't end up debugging failed builds because one machine out of a thousand is running on some minor shared library version not supported by your software.
The author is complaining that you can't build these private trusted images. Software developers have got it in their head that containers are a way to package & distribute software. They're not, that's what the OS's package managers are for. If your software requires Docker as a build dependency, you have failed to properly package your software.
As a concrete example look at Ubiquiti's UNMS.[1] Their package consists of downloading & installing Docker binaries on your system, not tracked by the OS package manager, and then running a bunch of containers built from these public un-trusted images you just told me not to use.
They also conveniently ignore the fact that I already have a Redis server, I already have a PostgreSQL server, I already have an NGinx proxy. (Plus I guarantee my database servers are better tuned for my hardware than some random image from Docker's library.) It is not up to some random software developer where I should be drawing the isolation boundaries on my infrastructure. They also make the big assumption I want to use Docker to manage my containers in the first place. Perhaps my company already uses Solaris LX-branded zones, or LXC, etc.
Now imagine if instead of spinning up a PostgreSQL database container, it used MS SQL as it's database of choice. You think I'm going to let some random developer dictate whether or not I should spin up another SQL Server instance and pay MS for another round of cores / CALs?
Yes - you can build your own containers, and they're fantastic - if software developers properly package that software for ease of installation & configuration. Software developers should not be dictating what container/virtualization framework I use, what configuration management I use, etc.
[1]: https://help.ubnt.com/hc/en-us/articles/115012196527-UNMS-In...
To me, a Docker image seems like an ideal way to distribute some proprietary device management web software like Ubiquity UNMS, rather than requiring some obscure version of some database or whatever other dependency actually be installed and maintained by their clients. You can spin that image up on a server or group of servers, or on Amazon ECS or a bunch of other providers in a matter of minutes. With enough motivation, you could even export the image and manage the environment manually.
If you install their rpms or debs for any of their properties, you're almost always getting a copy of Mongo or some other dependent service... and it is probably going to be incompatible with whatever version your package manager has or you're already running (version-constraints-wise, not actual compatibility-wise).
This is an indictment of Ubiquiti, not containers in general. If their software were properly built, they'd be shipping you a docker compose setup or something with N different containers that you could substitute out (at a network level) for your own.
Every release was an Abbott-and-Costello "Who's on first?" routine. Do you have any idea how hard it is (especially in computing) to ask for something without being able to utter its name?
QA: "You left servername blank on this deployment document."
Dev: "I know; Ops won't tell me. Just ask them for where the service is currently."
QA: "Ops says there's 5 unrelated legacy services with that same project name, on different servers."
Dev: "5? I only knew about 3. You know, if I could query the schemas of the Prod DB, I could tell you in a jiffy which one it is."
Ops: "Pound sand. If you want look at databases that's what the dev DB server is for."
Dev: "Erm, OK well can I give you a listing of the Dev DB schema and you tell me if it looks like the one the Prod service is talking to?"
Ops: "Oh I see you want us to do your job for you now? You can compare the schemas."
Dev: "OK..."
Ops: "Just tell us which DB server you want the schema pulled for."
Dev: "But you won't tell me the server names."
Ops: "No."
My point is this is how bad communication can be when ops and dev are not on the same team.
The core problem is that there are "ops guys" and "dev guys". That creates conflicting incentives, even within the same team. It creates tension and a dynamic centered around bandying work around so that it's "the other guy's problem" in some situations, and hoarding logic onto the one segment so that there isn't an "obstruction" in getting things done in others. Moving the "segmented" guys directly into your team just makes these politics closer to the heart, which is not always an improvement.
Teams should be comprised of whole-platform "generalists" (in quotes because they really should be good at stuff, whereas "generalist" implies they aren't; here I just mean a competent non-specialist), where any single individual would be comfortable/capable performing any particular task that may come up. Of course, each member will have preferences and habits, little "skews", but it is important that these skews are controlled and used for mutual education, and not allowed to "flandersize" someone from "the guy who knows SQL better than the rest of us" to "full-fledged DBA who hasn't committed any C# for 3 years".
The right axis for separation is hardware v. software. If it's software-related, your dudes should be equally yoked, such that any SQL ticket would be assigned to any member of the team, or any "devops"/sysadmin/deployment ticket assigned to any member of the team.
These systems, from the OS up, are all part of the same thing, and they're all tightly integrated. Making the workload of the individual people on the team also tightly integrated is the only way to make sure that incentives align properly and that the most effective technical decisions are made, instead of decisions motivated, consciously or not, by offloading blame or other political/effort/convenience considerations that cause the overall system to suffer.
If you get into a sticky situation that requires specialized help from someone who has lived and breathed MySQL Server night and day, well, that's what consultants are for. Consultants would also be useful for inspections/sign-offs. But your core team can't tolerate being segmented out by component/implementation detail.
> Containers are a smart solution to the build problem.
Linux "containers" are a variety of things. True OS "containers" don't exist on Linux, but there are some rudimentary approximations. A Docker image is essentially a zip file, and sure, zip file-ish things may work fine for uploading artifacts to systems. Dockerfiles are unequivocally terrible, however.
I'm a security person who knows pretty good Python and simple database stuff (SQLite). I think I'm in the top 50% (humbly) of my field, probably higher.
But I don't know front-end, containers/CICD, or disrtibuted systems worth a damn.
I do believe parent, which is the idea that teams should have embedded resources. A "VM security team" operating firewalls and infrastructure and policy auditing should not only have security experts, but their own devops group that automates the crap out of everything, using 2018 best practices. Currently, my team's "dev" group is a separate team in another area whose work queue is fed by multiple, distinct teams. It makes learning and understanding our requirements really tough for them.
The thesis is NOT that a crew of superhumans can supersede all DBAs, security engineers, and infra people in the world.
It is rather that you can be a great software-side engineer, and that you can skew/focus on a few primary concerns, and develop and maintain a working knowledge in the others, sufficient to service your core project's needs.
Specialists can be called in as spot checkers, auditors, or short-term implementers, but they shouldn't be needed for the day-to-day of building, maintaining, and deploying your software.
In software, everything goes down to the same place: the system hardware. And these days at least, this is pretty much homogeneous between software segments. If you know how this functions, the differences are in the modes of expression and the conventions, not really the principles. We can learn the varying conventions well enough to be serviceable in all the elements that we send down to hardware -- not necessarily expert, but good enough for day-to-day work.
I'm not saying that everyone on the team should be better than the best DBA guy you've ever met. I'm saying that everyone on your team should be reasonably confident with SQL. Specialists have a place in your friendly local <security/database/whatever> consultancy.
And yet tolerate it will, because it is somewhat impossible to hire a team composed entirely of people who are each experienced and competent in writing and designing frontends, writing and architecting backends, deploying and maintaining whatever backing services you've using, build and release engineering, Linux, networking, etc. And what are junior developers supposed to do?
Any separation is ineffective except along this one particular completely arbitrary dividing line? If that were true we'd still be hunting and gathering and nothing else.
You have just lost all the speed to production advantages of containers.
"knowing exactly what you're running and being able to reproduce it" is meant to be the primary advantage of containers.
What you're basically saying is "if your container system admins do their job properly rather than throwing security and reliability out of the window, it can take a bit longer than not bothering". This is trivially true, but not really the point agentultra was making.
BTW the more I learn and use nix the more I see that this is the proper solution to what docker is currently marketed to fix.
Speed of production advantages are absolutely not due to pulling untrusted containers. If anything, it makes your life harder.
Hard to imagine any serious production setup not doing this... In most cases, you need to modify the containers anyway to suit your needs, and how else are you going to rebuild them all when the next OpenSSL update comes out?
Well, potentially unpopular opinion here, but an awful lot of sysadmins brought their looming obsolesence on themselves. I'm an app (as in "a program that runs on a computer", not an iOS add-on) developer, always have been. I get requirements from the business types, code it up in vi or Eclipse or whatever, get it working, and then they (the business) want to deploy the working app out to production so people can use it and the business can make money off of it. And, for decades, sysadmins have been a brick wall of pure hostility. They're not all like this, but a lot more are than aren't. Like, I get it - you're overworked and the demands on you are unreasonable. Yeah, me too. But I just work here, man. You're right, I don't know how to do your job, that's why I sent you an e-mail asking you what steps are needed to deploy an app into production since it's not documented anywhere. But rather than just tell me what you need so I can go gather that up, you're going to unload on me because you feel overworked and unappreciated, but you're sure as hell not going to unload on a manager or somebody with actual power, you're going to take it out on the developers who have no pull or voice.
Personally, I've always considered it a significant part of my job to make developers' jobs easier, especially with something like deployments and dependencies.
As such, I disagree that we've brought our own "obsolesence" on ourselves, but I do agree that those of use who have perhaps forgotten that ours is a service profession have hastened its demise.
Two points, immediately:
1) Can I autoscale the app, as in what is the data/files persistence model?
2) Did you write Readme.md with a) build steps b) networking requirements c) data sources d) authentication model e) database ACID requirements ?
It makes sense that these types naturally gravitated towards the devops models. I'm really not sure where this leaves the more compliance-minded systems folks though.
Working for profitable businesses where stability is valued over velocity.
You have to constantly fight the urge to just never touch anything.
I'm baffled how often I see an someone throw this sort of craziness - "go fetch this thing from some random third party" - into very important places, such as the startup procedures of a container. It's something I see in a culture of the two person startup just trying to get something out the door. It's definitely "technical debt", and frequently, it won't get removed. Thus, you try to scale up to meet load, and all these new instances go time out on the same external resource that's randomly having problems... boom! At the worst possible time. Never mind the potential huge security gaps.
But the specific _tools_ aren't the issue here. It's the culture of "ship something now we'll deal with fallout later". A lot of people start using Docker and won't ever look at the Dockerfile, or, will add a Maven dependency and won't even check licenses or security updates for _any_ of the transient dependencies.
Cloud technologies and containerization make everyone just think "we can do things so fast now" and never, ever pay attention to details that can come back to bite you.
On the flip side, it's a good time to be in cybersecurity; because this cultural problem will never, ever, get solved. :)
They like the idea of security and that's where it ends.
In many places if you try to "do things right" you will get fired in two months for being too slow/strict and they will happily replace you with a clueless easily trusting person who "goes and fetches things from random third parties".
Many times they get lucky enough to survive and they don't appreciate the risks that they took. That pace becomes the expected norm and sets the theme in the industry.
And when shit hits the fan the PR person writes a "we are oh so very sorry .. security is totally our number one priority" blog post. They blame and fire the poor bastard and replace him with another warm body.
When it comes to these "hidden" things like security companies do not reward and also punish "doing things right" so on average and over the long term we end up where we are today.
When the culture sufficiently shifts towards being sloppy you will get hammered down quick if you try to be the voice of reason because it ends up being you vs everyone else (the norm).
And, honestly, why should they? Security breaches have yet to hurt an actual company (they hurt users plenty, but not the organization that's actually responsible).
(I've seen similar claims in different ranges. Costs of breaches in the US are pretty high - over $7 million.)
Even Equifax probably wants the 3-4 billion in valuation it's lost since the breach.
The solution appears to be buying tools to avoid and respond to breaches quickly, instead of engaging and building in security awareness. (Microsoft's security development lifecycle comes to mind.)
IMO, both approaches are likely cost effective, though I have no numbers or research to back that up.
this is a very naive way of setting up a secure production enviroment.
Your perimeter security is worthless if you are loading non public images which have malware or even worse, unknown malicious code in them.
having a data breach or hack on your hands is something which could kill the company. That risk is not worth having a slightly faster productivity boost because you or your ops team is not able or willing to build a proper private repository setup.
> Consider for example Hadoop. Nobody seems to know how to build Hadoop from scratch. It’s an incredible mess of dependencies, version requirements and build tools.
And as the major introduction to the blog post:
> I’m not complaining about old-school sysadmins. They know how to keep systems running, manage update and upgrade paths.
Huh? Old-school sysadmins know how to keep systems running, manage updates and upgrades. At the same time nobody knows how to build Hadoop from scratch. At the same time, Hadoop build instructions themselves have curl|sh scripts or mirrors and the wiki page is outdated. And it uses Java (and thus maven/ivy). And that downloads the internet.
According to the blog, Hadoop, maven/ivy/sbt/any dependency manager, package managers, and everything is broken. But the tagline is:
> This rant is about containers, prebuilt VMs
What does any of this have to do with the "Age of containers" and pre-built VMs? Is the author just talking about Gentoo/LFS-style "compile the whole system from scratch"?
This feels like an incredibly rushed rant. I can only envision the author requiring to setup hadoop for the first time, breaking their head for a few days (it happens), and taking it out on everything.
Build systems just download and run random code from the internet without verifying that its the correct code, from the correct source.
Its a ticking time bomb.
Now, some dependencies are fraudolent (especially true in the Javascript world because it eventually targets a lot of user browsers), but nobody ever checked the sources anyway...
Think of a self-hosting programming language: you can't build it without a running installation of a previous version. (Anyone remembering "On Trusting Trust" at this point?) Or any application in an image-based language like Smalltalk. Development becomes path-dependent. It's inevitable to get into a situation where A and B cannot be made to work together, except in a derivative of a version that someone, somewhere made while holding their mouth the right way.
Pre-built containers and VMs are an admission that path-dependence is the way stuff is supposed to be.
A) work through building it yourself, or
B) get a container that claims to have a running Hadoop and hope it works for you?
If B wasn't on the table, what would happen?
You'll notice that the Debian Wiki users have given up on building it since 2010. That was three years before Docker even appeared. Almost nobody was using containers back then.
That's the whole problem with the article. It takes a problem (building Hadoop was bad), correlates it to a completely different tool (because we have docker, hadoop build scripts are bad), and goes on to rant about everything else.
Update: it was pointed out that this started way before Docker: »Docker is the new ‘curl | sudo bash‘«. That’s right, but it’s now pretty much mainstream to download and run untrusted software in your “datacenter”. That is bad, really bad. Before, admins would try hard to prevent security holes, now they call themselves “devops” and happily introduce them to the network themselves!
I mean there is a lot of cognitive load/disconnect we're talk about. As an ops guy, I can't look into every package. That's why I trust the package manager (apt-get, yum, whatever) and all the build maintainers who either volunteer or work on for Redhat/Canonical/SuSE/IBM/whoever.
Things get through. That's why we have all those security people out there who are digging around for bug bounties and find crap like the recent Ubuntu Snap package craziness.
Docker containers can be good. You can use an official Ubuntu or Alpine image, build your base, and create scripts to make sure your base containers don't go out of date. Most people don't do that. The official Docker containers are kinda a mess, but at least they're maintained. Grabbing some random container off Dockerhub? Yea that's not going to end well; unless you just use their source to build your own. Or if it's a container continually maintained but the person/company who wrote the service.
Docker containers do need better security introspection and that's going to be a big deal going forward. But this article is all rant and some, but not enough, substance.
Exactly!
And npm. And maven. And every damn package system for every damn programming language since package systems are now a requirement.
This is not my job (yet). Please tell me if I'm wrong, because I'll need to do it in the next few months.
I swear by the time I figure out my path problems I have 4 folders named /code/
And my working file system is /code/code/index.php
So when I run node.js (to compile my .scss) it messes up the folder pathing and ugh... why cant we do everything on production?
/rant
Is there anything more "get-off-my-lawn" than "These tools don't use the thing I like!"
Everyone repeat after me. Makefiles are not scary. I can write a shell script. Do I really need to learn grunt/gulp/webpack/npm/rake/fake/maven/gradle/ant and on and on and on?
Probably somebody has released another one in the time it's taken me to write this comment.
I use Rake (or Gulp, or whatever) because then I can use Ruby (or JavaScript, or whatever). Shell plumbing is fine for informal and small-scale stuff, and I make my code conform if somebody down the line (who may be me) wants to get out their duct tape, but the world is more complex than what /bin/sh can see. Shell is the lowest common denominator. Expecting everything to at all times be written in and for that lowest common denominator is not reasonable. We're a tool-using species and we refine tools over time to make them better. The profusion of tools happens because they iterate on each other to be better. If old tools were sufficient, people would use them because learning new ones is hard.
So, yes, you do need to learn those tools. Or invent a shell that isn't tooth-pullingly difficult to use with a JSON file (and do not say `jq`, I love `jq` as an inspector but it does not step to `JSON.parse` and a working subscript operator). Or change `make` so that a git checkout won't trigger a full rebuild. Lots of baseline, stump-simple things that `make` is just not going to do for you because it's built for a frankly outmoded method of development.
You can use Ruby in Make. I don't know Ruby but here's some Python:
SHELL = python
.SHELLFLAGS = -c
.ONESHELL:
.DELETE_ON_ERROR:
foo.txt: bar.json
import json
with open('$<') as input, open('$@', 'w') as output:
output.write(json.load(input)['the_text'])
You can even use different languages for different recipes with per-recipe variable settings.
Is it that hard to learn shell? Why is it so painful? What makes it the "lowest common denominator"? I use it all the time, but I admit at work I am one of the few.
> Expecting everything to at all times be written in and for that lowest common denominator is not reasonable. We're a tool-using species and we refine tools over time to make them better.
This is too vague. What makes a Ruby based build or JS based build a more "refined" tool? It sounds like familiarity is the real issue here.
> If old tools were sufficient, people would use them because learning new ones is hard.
How many people even know Makefiles these days anyway? The "modern" approach seems to be, learn a programming language and then try to do everything inside of it. Some languages are more interested in this cloistered philosophy than others (like JS).
If anything, I think the reason these build tools keep being proliferated is because nobody wants to learn anything more than the bare minimum to "be productive" (which, depending on what you're working on, can be anything from pushing out customer demos for a company that will never sell, to microservices operating at scale). Learning a language and never leaving its paradigms/comforts is easy.
That said, Maven is incredibly suck-tastic.
https://github.com/ingydotnet/json-bash
source json.bash
json='{"name":"Jason","friends":["Jimmy","Joe"]}'
JSON.load "$json"
joe=$(JSON.get /friends/1)
JSON.put /friends/2 Jeff
new_json=$(JSON.dump)
That is blatantly false, but I get the premise behind it. Writing and maintaining complex stuff in bourne or bash is not fun or easy.
I always use bourne until certain level of complexity or awkwardness is reached. It's pretty easy to write and dead simple to troubleshoot.
This is an anti-feature. When you need this, your build is too complicated.
Million+ line codebases got errm made with make. In my experience people most sneering at it have far lesser demands on their tool...
My challenge to you: I want a makefile that has 20 third party dependencies and can be built on osx, linux, and windows.
I can do this within an hour with gradle, ant, or maven. The ecosystem doesn't exist for this in make, and anything I could come up with to make it possible would end up being a tool that would look like automake and the monstrosity that it entails.
For something that isn't JVM-based?
and you've already made sure they are running the same JRE version, right?
And can it do it all incrementally in parallel too please because enterprise shops tend to have a ton of code
Oh and it would be really nice if you could make it so if someone else somewhere in the org has compiled that thing then could we just use their computed binary to save the time compiling locally.
THANKS!
;)
> Everyone repeat after me.
I don't mean to pick on you specifically here because this attitude comes up a lot. In short, a lot of people are doing a thing, a thing that you aren't familiar with, and your gut reaction is to say "everyone: stop doing that, and do what I say!".
Wait a second and consider that maybe there is a reason why this incredibly large number of developers are using these tools. That perhaps they evaluated various different options and decided that what they are using is more suitable than make. Maybe you could find out.
Here we have another fundemental "problem" between dev and ops.
The inherent friction because of different areas of concern.
Dev's want to build fast and create new features, but sadly (even with the whole devops notion) are usually not thinking about viability in production.
Ops people need to keep thing stable, something which is sadly undervalued a lot of times in companies.
Have you considered that perhaps the previous commentor is familiar with the other tools? Or perhaps that the large number of developers have streamlined their particular workflows for their particular use case and have not considered the flexibility needed for other cases? Or perhaps that there is a cost associated simply with having 47 different build systems?
If you're building in Elixir, I hope you're using Mix and not a Makefile. If you're building in Rust, I hope you're using Cargo, or some other Rust specific build tool.
And Makefiles do get stupid complicated when you need things portable and to work on Linux, Mac and FreeBSD; or allow them to have optional dependencies. That's why we have autoconf and all that ./configure && make && make install craziness.
For instance Make works based on timestamps and therefore works very poorly together with git. Switch to another branch and you can get weird effects based on what files were updated and not and often trigger needless rebuilds. And everyone uses git these days.
GNU Make, just using hashes instead of timestamps, would be a huge step forward.
Is ninja that tool?
Sounds like you're describing make on a system with ccashe installed. Hashing incurs a significant performance hit. The first build with ccashe is 20% slower than building without it[1]. Your modern make would likely be slower for people who just build something from source once and aren't doing incremental development.
There are modern tools that work for many things. Gradle supports native compilation these days as well as any JVM language. Bazel supports compiling many kinds of languages, and there's still scons.
Writing correct Makefiles is also really hard, and good luck debugging them. Make is simple and beautiful for small, self-contained projects without many dependencies, but it does not scale well (and even then it's tricky, see [0][1]).
Having 47 different build tools with their own flaws is certainly bad, but they exist because of Make's shortcomings. Just saying "make is fine, use it" won't fix anything.
A lot of the Rake tasks I've encountered in my career would have been easier to write in bash. Not a majority, but a sizable minority. I suspect that in many cases, the gain was that the authors were more comfortable in Ruby than in bash.
Writing good build systems is genuinely hard, and I think make is not a good build system.
https://gist.github.com/anonymous/d885c3bb66a319d22f9d60c1ef...
make is so bad you need automake to manage it.
there are much better tools. sadly, nothing LCD (least common denominator) so as to gain wide traction.
That said, for anyone distributing software, shame on them for not packaging their custom build so as to be runnable via ‘make all’ (just using make to drive everything else).
Autotools wasn't invented to overcome deficiencies in make, but deficiencies in C portability across Unix flavors.
Those deficiencies are greatly diminished today, both by POSIX standardization, and there being fewer viable surviving Unix variants that anyone cares to build for.
I've seen plenty of messes using scons and ant, more so than I've seen with make. Make is a solid tool.
New way: we have a dozen tools, they collectively do less than the old one and integrating them is a full time job for someone since any upgrade breaks something. But we all get to put the names of all of these things on our CVs! And that’s what’s important.
I wish I was kidding.
Devs see the world as one where velocity and progress is among the most important, while 'sysadmins' (a term no longer used by companies and recruiters, unfortunately) have to worry about keeping the applications actually up so they can be used.
DevOps just seemed to have swept the sysadmin under the rug under some pretty words about breaking down barriers. It feels more that devs broke down the wall sometimes. The amount of infrastructure chaos and mess and confusion I see today (as a contractor bouncing around different places) seems higher than traditional infrastructure shops of 10+ years ago.
Also, a ton of people use abstraction as an excuse to not learn a body of knowledge that contains fundementals (especially on the systems side).
In comparison to network engineering for instance, where having (quite) low level knowledge of how protocols work and interact with each other is vital.
This is just not true, otherwise everyone would need to know ASM or be able to easily contract someone who does.
Pretty much. I know several admins who appear to be joining a growing pool of luddites who rail against anything new. They're particularly butt-mad about anyone drawing more salary than them. "DevOps" is their favored totem to direct their ire at.
I used to try and convince them otherwise, but it turned out to be a completely futile waste of time. At the end of the day, persistent FUD just means a more lucrative job market for the engineers who are pragmatic and fearless enough to give an honest try at making the newer paradigms work for their employers.
Every build system is like Make, but more friendly for their language (IIRC Make was originally for compiling C and C++). Make just so happened to become generic enough to build damn nearly anything and also get bundled into most Linux distros.
I think the author is arguing that having to install a shit ton of dependencies to use some other Make-like build system is garbage. That’s true in some cases. But I wouldn’t want to use a Makefile for packaging Node; npm is great for that and understands how Node works.
There's a reason that Google has internal systems which can and do rebuild everything from source.
What will happen if public container images are no longer available for building your application?
Your in a for a world of hurt if a major dependency of your application doesn't work anymore because someone pulled the image.
You can also kind-of recreate a Dockerfile from an image using docker history: https://stackoverflow.com/questions/19104847/how-to-generate...
In the general case, it’s the same as a Ruby Gem or NodeJS module going away from rubygems or npmjs; once its gone, its gone
We still need to figure out how to best tackle the issue of online repositories being taken down or vanishing. For now we run apt-mirror.
Also, this is not at all endemic of containers, there's simply zero connection. Dockerfiles tend to be very simple and easy to reason about. The most complex application I have is around 50 lines of Dockerfile, and that's mostly just made more complicated to arrange things for the best layer caching.
I suppose we're supposed to believe that this is somehow worse than the days of debugging m4 macros and autotools just to get a build that doesn't work.
That being said, since Go binaries have no inherent dependencies, I have indeed made Docker images containing exactly one file: the Go binary. These containers are basically the same as fat binaries, with the benefits of Docker scheduling and networking at runtime.
To anyone reading this, you need some magic compiler flags in both Rust and GoLang to make sure it's a statically compiled binary (doesn't dynamically link against GNU lib-c).
But yes, this is super neat. I also like how it reads in the docker file:
FROM scratch ...
Apache has official mirrors that host repo files for various package managers so you can install using apt-get or whatever it is that replaced yum (dnf? dnf):
https://www.apache.org/dyn/closer.lua/bigtop/bigtop-1.2.1/re...
So precompiled binaries from official sources are certainly available.
"Unless you compile it yourself, you can't trust it."
Pragmatic solutions where possible. Like scanning containers, using OWASP tools on your repos etc
1. $ git clone git://anongit.freedesktop.org/libreoffice/core
2. $ apt-get build-dep libreoffice
3. $ ./autogen.sh && make
Just because something has a large code base doesn't mean we shouldn't be able to build it from source ourselves.
Personally, what I like is no longer having to setup arch specific build machines containing all of the build tools and dependencies for all binaries that I wish to self-compile. Instead, I either use the vendor's Dockerfile which already contains everything it needs to build from source or I simply write my own if there is not one available. Building and distributing these binaries in the form of Docker images is a breeze using Gitlab CI and container registry and is just as easy with a small VPS and Docker Hub.
Unless one can compile it oneself, how can one trust that a particular version of a binary release correspond to a particular version of a source release?
If the process is reproduced by another trusted-enough source and is identical to the official release, then I'd say one can go ahead and trust the binary release of either one.
Sadly, I don't think this is generally done, though perhaps ones own spot-checking of the official release is enough.
That's supposed to be the basis of modern science, too, though, of course, it's not generally done there, either.
What I'm trying to say is that the one doesn't exclude the other.
And we actually use make quite extensively.
I do however see the ops point, building from source hasn't gotten easier.
After the new updated base OS images for our systems after Meltdown and Spectre were out, it literally took us ten minutes of human and about an hour of machine time to recompile all of our containers, run the tests and deploy them to production on our Kubernetes cluster, replacing the old insecure ones.
At the scale of our systems, any grumpy sysadmin would have spent at least several days untangling the dependencies and carefully restarting all servers in the correct order after some manually `sudo apt-get`ing (and probably forgetting a few of the lesser used systems).
Sure, typing "FROM ubuntu" (which by the way is a trusted and cryptographically image, contrary to the OPs concerns) leaves me at the mercy of whoever I trusted compiled that image.
Then again, what difference does it make to trust whoever compiled that ubuntu-17.04.iso image I put on my CD?
Or, as Brian Tracy taught me to say in these situations: "You may be right."
if you are operating at suchs a large scale, your sysadmin should be automating things (not necesserly with docker mind you).
This automation has been possible on *nix for the better part of a decade by now.
As for any "tangle" of dependencies that may exist, I've never seen that be caused by any choice made by "Ops" but, rather, solely by those writing the appllication code being deployed. As such, it would apply just as much to a Docker image as to (what I view as) a traditional deployment.
I'm not sure what to do about it other than just not using NPM at all!
Docker is not production ready yet.
I sometimes feel like a crazy person.
Bull's eye!
the entire point of containers (and orchestration systems) is the ability to push updates without downtime.
Just update with a newer container....am I missing something?
Also about container security - a strict process internally can easily help counter that (I believe Shopify had a nice talk about it at the Google Cloud Platform event in Toronto - everything from using trusted images, running only signed images and going through a security check for each layer).
EDIT: To add to it, please don't patch containers - the entire point of an "image" is that if I run it locally and on my datacenter - it should behave the same - live patching them just voids this concept.
There aren't really easy answers here. You can't fix bad software with more tooling.
I've noticed over the last decade that a certain level of knowledge about building from source seems harder to find via search or engaging in a community forum/chat. The assumption is often that everyone will be using the package manager (I'd say that's especially true on macOS, but it might be an artifact of me spending more time there since it's been my most used dev machine), so if you have trouble building something, the answer will frequently be "just use homebrew -- whoever maintains the formula will already have solved your problems for you."
There's two problems with this: first, that's presumably true for the source build, and if you can hit a case that the source build doesn't work for, chances are pretty good you can hit a case that brew doesn't work for (my experience is that more often than not they go together). Second... I'm happier to use package managers for applications meant to live on my machine and never go live elsewhere (heaven knows there's lots of stuff I just want to install and get on with my life rather than fiddle with), but for applications I'm deploying, it seems to me it's generally wise if someone involved in the project has a picture of the build details and dependency graphs in their head.
Some of the automation we're throwing at operations is really convenient. It's not simple, though. And depending on how much forethought is put into it, I'm starting to think of it as ... maybe not technical debt, but a technical credit card which is super convenient at times, but also can easily become debt if you're not careful with it.
I'm a programmer, so I'll take heat for it .. but I don't see a need for them anymore.
The entire purpose of DevOps IMO was to close the gap between sysadmins and devs through code. Devs doing everything, including infrastructure, was and is the entire plan! Public cloud made this super duper easy.
The problem is devs don’t want to manage core infrastructure (VPCs, networking, modules for deploying lambdas and database clusters and container orchestration clusters, etc) and somebody has to do that stuff
Ideally, those would just be features like any other software team, as it’s all API calls at the end of the day. But lots of companies have issues with structuring their platform teams like software teams because its “not software” even though it is
This problem is more deeply entrenched at large companies with hundreds of millions of dollars of compute that they own that is owned by an old school IT function that can’t fathom the idea of either giving it up or making it accessible like cloud and would rather pay VMware tons for tools that make teams even slower than have their sysadmins become developers
Then there’s the whole protectionist “You’re taking my job” and “devs can’t possibly know this much about $infra” that isn’t dying off anytime soon
It’s complicated
Just because it's is implemented as API calls at the end of the day doesn't necessarily make it not "not software" (if you'll pardon the double negative), at least in the sense that I believe you mean.
To whit, I believe you're suggesting that if something can be expressed as code, it's all "software" and can therefore be designed, written, and maintained by the same kinds of experts, software developers.
I disagree, because the nature of the infrastructure-as-code code is too different from the application software code.
One could, similarly, express an FPGA configuration in code, but a software developer would not automatically be good at programming one. This is even likely to be true for less extreme examples, such as programming expertise not automatically transferring from general software (for lack of a betterm term) to code that works well on, say, GPUs.
In the case of IAC "software", a more mature design is more likely to resemble traditional sysadmin/network/security best practices than application software features. It could also have significant financial side effects if there's an error, assuming public cloud, which could require more stringent standards of control, review, and quality, especially if a company ends up in SOX territory.
>Then there’s the whole protectionist “You’re taking my job” and “devs can’t possibly know this much about $infra” that isn’t dying off anytime soon
I'm sure some of this exists, but my own experience is an attitude not that devs can't know a certain amount about infrastructure but that they simply don't, often because they actually don't want to.
Perhaps they fear that if they do end up knowing that much, they'll end up being the ones to manage that core infrastructure, which you indentified that they don't want to do!
It's just that they don't want to. They think it's boring, or perhaps even beneath them. It's a bit like tax preparation.
I don't think that's really changed all that much, even today, even if it's now programming against the AWS API, judging by the number of job postings for such a role.
Also, like with tax preparation, using a good expert instead of doing it oneself could save on a lot of money and/or future headaches, but it can come with "grumpy" old-fashioned-seeming nagging about procedures akin to annotating and saving receipts. Whether it's worth it depends on situation and, far more importantly temperament.
> It's just that they don't want to. They think it's boring, or perhaps even beneath them. It's a bit like tax preparation.
I'll half agree here, and half not. I'm coming at this as a programmer who has historically done ops as well, currently doing more ops than dev.
There's parts of it that programmers don't want to manage, like Apache configuration, provisioning and sizing VMs appropriately, etc. That's the boring stuff, and I totally get it. I don't like managing most of that stuff either, and have got everything set up with service discovery to handle a good chunk of the configuration, and Terraform for managing resources instead of having to click through the Azure Portal, and Nomad for scheduling tasks in the clusters. Groovy.
There's also the parts that the developers seem to have no clue about. They're very happy with all of these abstractions, but there's a lot of layers underneath. They think in terms of making HTTP requests; when those don't work, it's up to me to figure out that their library is keeping a pool of HTTP connections open, and the Azure load balancer silently drops those connections out of the NAT table after 4 minutes of idle time (without sending FIN or RST packets to either side, natch). When the app gets terminated because it's exceeded its cgroup memory limit, I'm the one helping them adjust their JVM heap size. And when their requests take too long, I'm there helping them look through their query logs to figure out what's happening.
All of these cases have happened in recent memory, and it's never been a matter of "this is boring and/or beneath me", it's been a matter of "HALP! I HAVE NO IDEA WTF IS HAPPENING"
Edit: so yes, I agree with you in the sense that I'm a developer who generally speaking doesn't need a sysadmin. I installed Linux for the first time when I was 12 years old, 22 years ago :), and I didn't have a sysadmin then. In those 22 years, I've seen a lot of weird shit happen, and most of the people I've worked with don't seem to be well equipped to dive 3 abstraction layers deeper than they're used to to figure out what's wrong.
I'm not sure you're even half disagreeing with me here, especially as you just admitted you may have the same bias as I do :)
> All of these cases have happened in recent memory, and it's never been a matter of "this is boring and/or beneath me", it's been a matter of "HALP! I HAVE NO IDEA WTF IS HAPPENING"
These two cases strike me as actually one case, with the former being the cause and the latter being the effect. At least that's my contention: they had no idea wtf is happening in those lower layers because learning them was beneath them (no pun intended). Nothing was really stopping them from learning about heap size versus process/cgroup limits.
On the other hand, you do bring up an important point about (mis)behavior of network infrastructure. That's not something the average programmer would necessarily encounter or have access to, especially at scale, and therefore wouldn't be expected to know. Still, my point still stands: there's nothing stopping programmers from gaining that knowledge if/when it becomes necessary, should they want to.
Now, I'm not trying to play dumb. I mostly don't want to read too much between the lines. Are you, in essence, saying that your half-disagreement is that sysadmins are needed, not because (average) programmers don't want to learn all those layers, but, instead, are incapable of learning them?
As a dev I'd like nothing more than k8s or similar to be the standard platform to run applications so that everthing is standardized and I dont ever have to require a sysadmin. I think already this is possible more secure than the "good old days", but in the future I expect it to be more so.
The last devops team I worked on had an obsession with shiny. Never mind that they couldn't bootstrap a new base database for their application anymore or automate the entire application deployment (even with a phased approach) on either VMware or AWS. They wanted to keep piling new tools (often with sub 1.0 version numbers) of new tools on top of an unstable foundation, and would just shrug when it fell apart in production (which it commonly did).
I tried pointing out to them that by giving commit access to their internal puppet git repository to every developer in the building, they had effectively given root access to them as well. All I received were shrugs and blank looks all around.
One thing from the article that doesn't seem to be discussed enough here in the comments is trend of pulling random Docker images from the internet and deploying on your infrastructure simply because it's easier to integrate random versions with feature X than maintain your own builds, or work with the vendor's provided packages to achieve the same result. The security implications of this in particular has been bugging me for years.
Isn't most of this being driven by society in general though? Everyone wants everything now... we're generally feeding this trend to do whatever it takes in the shortest amount of time.
While I still know these things, I haven’t had to employ them in a while. I will probably use those even less as I start getting into management. That’s a bit saddening.
But the new stuff I’ve learned over the years, namely, designing systems like I’d design software, is amazing. Applying TDD to infrastructure isn’t something I thought I’d be doing, but here we are, and we have “modern DevOps” tooling (and lots of other things) to thank for that.
Maybe, but knowing SElinux, kernel namespaces and syscalls for Docker seccomp profiles are crucial for security, and security is the Next Big Thing.
Sad you are getting into management. Lets push to undo that notion that manager is worth more salary than lead engineer. I'm pushing :D
> I start getting into management. That’s a bit saddening
Less one engineer in the world?
This rant is about containers, prebuilt VMs, and the incredible mess they cause because their concept lacks notions of “trust” and “upgrades”."
Fair enough, rant preemptively avoided.
https://github.com/NixOS/nixpkgs/blob/master/pkgs/applicatio...
For me in terms of interface, the strength of the Nix ecosystem is declarative system, ops and service configs in the same language used for package and build specification. The technical strength is striving towards reproducibe builds by hashing the dependency tree to build an immutable store. Yes sometimes this means getting binaries but you can always pin the package version or even have multiple versions in tandem. The practical upshot of this immutability is system level rollbacks, which are generally reliable although there are ways to break it. Yes there is garbage collection.
Nixpkgs has is quite an achievement, and yes it has its warts but we are working hard to make it better. If we manage to shape up the data science I will try it out at work too. Im very curious how it might scale.
I don't get why people still claim that this is "the Docker approach". This is not the docker approach, everyone hopefully knows this is an anti-pattern by now.
Because it is the Docker approach (as distinct from the container approach necessarily).
100% this. Learning Make a few years ago was one of the best decisions I’ve ever made. It’s simple (until it’s not), straightforward and available on just about any Linux and UNIX installation (version compatibility aside).
Trying to make the case for POSIX-compatible Bash scripts has been tough, though.
I also agree re: Docker. While you can secure a container image with USER statements in the Dockerfile and knowing how to give containers just the capabilities that they need to do their job, it is WAY too easy to run them as root and give them privileged access to things. It should’ve been the other way around. Also, every container orchestration platform seems like a really elaborate hack.
Wouldnt the approach for security updates be to replace the container?
However I do think OP made a mistake by using Hadoop as the example. Hadoop has much more in common with an OS at this point than a single application. The ability to download a single component of the eco system is mainly to support small scale testing and development.
You wouldn't build Debian by going and getting each piece of the linux kernel from its source building those building all the pieces of the middle ware up and so forth... that is the whole point of having Debian in the first place.
Do certain complex software eco systems need better support for fingerprinting their builds, Yes. Does all this mean the sky is falling... probably not.
Are these sysadmins actually looking through the open source code that they're compiling to make sure that there isn't a security flaw in plain sight?
With Docker you can get containers maintained by a trusted source, as a sysadmin you don't have to deal with all the hassle of upgrading things, you can just replace the container with the latest version. With Docker content trust the containers are signed.
There have always been cowboys out there and that has nothing to do with the tech stack.
Sometimes they get their unpatched servers encrypted by some ransomware, remember they don't have any backup, close shop and move on to the next business idea. I've seen that happen.
If you operating at large enough scale you can bring the "cloud service" in-house.
If "they should be delivered as cloud services" is some sort of k8s apologist stance for its sorry state of maturity, then we got issues. OTOH, if it's "You shouldn't run it in house unless you have an army of people to read every new commit", that's wrong, too.
On the other hand there's a level of complexity in distributed systems that is impossible to avoid even in stable infrastructure. You have a design choice of trying to make the system as easy as possible to operate (at the cost of other features) vs. finding operating models that make it less of an issue.
Personally I would rather spend time futzing around with my applications that run on kubernetes vs. trying to run kubernetes itself. It would be sufficient if Kubernetes services were portable across a marketplace of providers so I could pick a place to run my applications.
You can build images securely with Docker but it requires building them yourself, using private registries, checking carefully for vulnerabilities, and testing. If you don't want to do this, pay somebody else to do it right. There's no free lunch.
Good question, why don't we see exploits of all that implicit trust to the degree that, eg, the DOS shareware scene gave your PC visible virus infection, or the early internet gave us worms that would bog down the whole net?
My attempt at an answer: Because the black hats aren't hobbyists anymore. Visibility is for amateurs.
To a degree that's what I'm saying. Sentencing for "computer crimes" when perpetrated by non-corporate entities has reached epic levels. I'm sure qualified talent takes that into account.
https://ci.bigtop.apache.org/job/Docker-Sandbox/112/BUILD_EN...
Does anyone care?
Google isn't using Kubernetes with Docker on AWS (or even GCP). I'm quite confident that they're running their own software on custom bare metal, with custom virtualization/container abstraction layers and custom management software.
It's just moving the abstraction layer higher up the stack.
Also, who will design and build security for these systems? Or care about low-level performance?
A ton of dev's don't care about prod in my experience, and just want to ship shiny features.
Containers and VM's are not very different from a deployment or maintance standpoint. The deployment strategies used with containers could be done with (lightweight) VM's a decade ago. (heck, jails have been used for nearly 25 years)
Also, sysadmins have been automating stuff since forever. Without automation one inherently has a very unstable system.
With that in mind, I don't see sysadmins going anywhere soon. Every layer of abstraction is supposed to 'eliminate sysadmins', but inevitably it becomes yet another system that someone needs to audit and maintain.
Is is true that we had better security when we had to "compile everything from scratch" whatever that means? Did people compile the OS from scratch?
...yes. https://www.freebsd.org/doc/en/books/handbook/makeworld.html
P.S., make sure you have a spare couple days before running make buildworld on a 150 MHz Pentium.
Cloud infra today requires coding skills. Thats why so many web developers trying to do infra too. Without systems or infrastructure knowledge.
True cloud automation engineer, a.k.a DevOps, a.k.a SRE, is the sysadmin learned to code more than bash. Python, REST, classes, methods, objects.
Or web developer, learned Linux and networking in deep, down to userland to kernel system calls and routing protocols.
These people are rare, hence the ruckus and mayhem. You will be hacked.
"No one cares about verifiable security anymore."
The author defines a category of people and then attacks it for A) not verifying the integrity of its systems, and B) not doing things with make.
A is valid and something that needs attention. B is missing the point.
I've found this to be a great resource, hope you find it useful:
* http://gromnitsky.users.sourceforge.net/articles/notes-for-n...
> But then, everything got Windows-ized. “Apps” were the rage, which you download and run, without being concerned about security, or the ability to upgrade the application to the next version. Because “you only live once”.
These two a both very valid and great approaches for solving different problems. Sometimes you're just a regular user without any valuable data that just wants to do things in a quick and convenient way. And sometimes, you're a system administrator that needs to evaluate the whole build pipeline and plug all the holes for production deployment.
Both alternatives should exist, and one doesn't cancel the other.