Immutable Infrastructure: No SSH
boxfuse.com
boxfuse.com
Yes, that's going to prevent drift at first. No, it's not going to prevent people from doing silly stuff. At least ssh allows them to do silly stuff in a "normal" and visible way and doesn't stop you from debugging issues when you actually need it. If you enforce no-login without teaching people, they'll just reinvent it... badly - for example by making executable configurations, or pseudo-web-shells.
And there's one big issue with images which will cause a lot of friction in real teams - they're slow to build. And even slower to test. If you're doing it properly it means first testing locally on some approximate environment, then handing the package over to a system which will test the whole deployment, which takes ages to boot up all components. (time scales with system complexity - it's not ages for a single webserver of course) Anyone developing something related to integrating services will get additional frustration unfortunately.
> Enforcing immutability [...] You need to prevent log in.
No... If you need immutability, you make your system actually immutable and then audit logins when logins are needed. Preventing logins is just punishing people and slowing them down when they need it the most, because you don't trust them.
> Vulnerabilities like ShellShock simply vanish
Said a person who does not understand why ShellShock was an issue and why it doesn't matter if a user who already has local access uses shellshock.
There might have been a magic page in the software that allowed raw SQL queries...
I'm actually surprised at the naivety of whoever wrote this. I'm not an op at all and I understood enough about ShellShock to know that this isn't correct at all.
With this change, the team loses access to all the standard OS tools which are useful in the event of a service outage for debugging. What usually ends up happening then is that the standard tools are replaced with substandard and patchy implementations.
If your infrastructure is drifting then a better solution is to restart it more regularly. If you find staff are still making untracked changes then that's not a technology problem - it's a people problem.
SSH access would be more like "heroku run bash" where you don't log into an existing instance, you log into a fresh ephemeral instance that just runs sshd and gets destroyed on logout.
Debugging running instances is more interesting though, and I think is possible if each container has an SSH daemon with ephemeral keys, and a central authority they register with that you can hop through to get shell on any instance.
Ops is a really fun job to be in when you stop thinking of servers as "things that people log into".
It's simply "don't perform install, maintaince, or upgrade mechanisms over manual means ever".
Do NOT cut off your SSH channel. You'll be kicking yourself when you want to sftp a file off of a box pretty quickly. And you might need it in an emergency.
Focus on good automation - whether immutable systems or otherwise, and be sure you can rebuild your infrastructure completely from source control. Do this by refusing to make manual changes, and knowing any manual changes will be often clobbered by the system.
FTR, I'm in favor of restricting SSH/RDP, perhaps even disabling from the security groups (AWS), on a default basis, and enabling as needed for ops troubleshooting. If we have logging and monitoring in place cleanly enough, then we should be able to troubleshoot the majority of issues off instance. However that is almost NEVER the case (even if you have the tools, you still have too many edge cases). Case in point, I've been troubleshooting a Windows CPU usage issue recently. Our monitoring wasn't catching all of the disparate processes, so I couldn't see what was eating it up. Plus I would pull the node from the ELB prior to investigating, which then caused it to no longer peg CPU. Had I not been able to login while "live", I may not have ever seen what it was.
Obviously in order to perform such a function, one needs incredible reliability in your monitoring/logging tools. Sadly this is a more difficult function than it should be....
What? And disturb the tranquil immutability of boxfuse.com?
This works fairly well, but from experience we do need to keep an SSH setup on the machines. Just last week we had a load spike on a production server which was caused by a software bug, triggered by a usage pattern that was unexpected and not covered in our tests. If we did not have SSH (and access to the CLI tools on that machine) we would not have been able to debug this unexpected problem. I guess what I'm saying is that as long as software has bugs and hardware has glitches, we'll sometimes need access to low-level tools which can help us figure out the cause of these unexpected scenarios.
Are they suggesting that every time you need to apply a security patch, you'll have to build a new server and cut your traffic over to it? That sounds like a lot of DNS and Load Balancer config updates (mutable!) even if the newly patched server builds are 99% automated.
The idea of black-box appliances has been around a long time, and it has its place in the modern infrastructure, but I'm not sure it really solves the problems they are trying to solve (which sound more like change management issues).
It is an multi-AZ DB, which is very different to multi-region. (Availability Zones (AZs) are within a DC, regions are separate DCs)
See this Re:Invent 2014 talk about AWS Innovation At Scale http://mvdirona.com/jrh/talksandpapers/JamesHamiltonReInvent... => Slide 9: "Each AZ is 1 or more DC, No data center is in two AZs, Some AZs have as many as 6 DCs"
Also, just to clarify, you can do multi-region RDS read-replicas for MySQL: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_...
Until then, ignorance is bliss.
In my experience with RDS as a MySQL admin, Amazon was not using sane defaults for quite some time, which resulted in a remarkably non-performant product. Thankfully, they listened to feedback, so over the course of a few years the MySQL RDS instances started to become much more dependable and useful. I'd be happy to use one today, which is something I couldn't have said too long ago.
And even with RDS, having a DBA available (even just as a consultant) is still quite useful - it's unrealistic to expect your developers to write ideal SQL. You can get a long ways with RDS if your queries and schemas are well tuned.
You don't want to have to fight your way in in the middle of a disaster.
At Clarify.io we've considered having an on-login event fire when an admin uses SSH to log in to the box. This would schedule the server for termination within 24 hours. This gets you best of all worlds:
- Gives tools to debug and restore service when there is a failure
- Forces admins to not depend on SSH to bring a system up
- Replaces servers when they are potentially drifting
- Encourages a "cattle" mentality about the servers
Shellshock was not just from SSH, it was any program that used environment variables to pass data around.
Immutable infrastructure is a great ideal, but has various downsides. sometimes people need to access the raw system to do debugging. While shoot the node, and boot a new one has advantages, it does just move the problem down the line.
The BBC solution is a nice balance.
Now I can't login to the production server in order to strace the process to figure out why production is misbehaving.
And don't say you should always replicate the issue in preprod, sometimes you simply can't. Some issues only emerge at prod scale+load+latency that you don't have the financial resources or technical ability to be able to exactly replicate in a testbed.
You are drunk on too much koolaid, go home.
Sometimes it makes more sense just to kill that server and add a fresh one to the pool. Better yet if that process is automated. Good logging can actually be more valuable than mucking about in the live environment (or as I call it, panic mode).
You don't need to limit ssh to help handle state drift. Article writer doesn't seem to understand the difference between user access, and software deployment.
When a box dies in staging, it's nice to learn why.
This is an old-school ops team that learned about "devops" and "immutable infrastructure" and figured now they have all the necessary arguments to kick the devs off the servers once and for all. Not progress.
sudo apt-get install mypkg
(Or is there? Serious question)
My guess is that apt pinning/holding predates Gemfile.lock by quite a few years :)
The functionality of bundle install and bundle update are not replicated by apt pinning or holding. How do you roll back a bad update to your pinned packages? Put back the old Gemfile.lock and bundle (again)? Does that really work?
I guess I don't really use Ansible so I don't know, but it's very easy to roll back a bundle update if you keep your Gemfile.lock in revision control too. I've used apt pinning before though, and I'm quite sure rollback of a package update that has a dependency conflict with the old version is not as easy.
Aptitude has always handled that just fine for me, downgrading the dependencies as well if needed.
It is an issue with the package manager, like the python/virtualenv guy suggested, apt can't support keeping parallel versions installed at the same time and linking in the locked version into each separate project that needs a specific version. The conflicting versions do conflict. This is not a problem for bundler, since a given project is not likely to need two separate conflicting versions in a single bundler exec.
There is no "project" concept at all in dpkg. You just have to maintain wholly separate environments for those conflicting dependencies, on separate machines, if they show up in cases where you really need both versions.
Or work with containers instead, which is arguably not really a different solution than already proposed.
Oh, that's why I mentioned Ansible in my previous comment. I have a version controlled file with:
- apt: pkg=lib1 version=11.6
- apt: pkg=app version=33
...
As for multiple parallel versions installed, sure you can. All you need to do is packaging them using different prefixes. What you can't do is blindly use packages from the main repos; those are built to be used to support the OS programs, not yours. But even those are often made to have multiple parallel version: I have python 2.7 and python 3 installed, both from Debian's main repo.
Another issue with debian packages is that dpkg essentially removes the old package before installing the new one. If you're packaging assets served by nginx/apache then there is an availability issue in the mean-time.
That's what "apt-get autoremove" is for.
Config drift can be avoided by using the same process to create new VMs and delete old ones, which is also part of confirming that your redundancy setup really works.
I didn't realize they were production ready yet, but sounds pretty spiffy.
What's interesting is that I think Boxfuse is spinning what most people would call a weakness ("if stuff breaks, I can't SSH into the shell and fix it (because, not only is that a 'bad idea', there just isn't a Linux OS there to SSH into anyway)" and calling it a strength.
Nice spin. :-)
If I tried boxfuse, I'd probably look at embedding the Apache Java-based SSHD server, not for app setup, but debugging ... assuming its terminal session would provide useful information in the Boxfuse "there is no OS" environment.
(The boxfuse site uses the term "Secure Micro OS, Few MBs", so if I'm misinterpreting what their platform does, someone please correct me.)
What? No, they don't! ShellShock was about executing unintended code in a bash process where you control an environment variable but not stdin. If you control stdin, there's no need to sneak a command into an environment variable: you can just type it into the shell.
Apart from this, the article makes no argument for removing SSH: it only argues for restricting access to privileged userids. That is to say, the standard best practice for decades. Root should be reserved for special situations, not used by default.
Having everything build-oriented (basically building the builder on every deployment, than using it to build the service) is what prevents drift.
Running on a read-only file system is what guarantees immutability and prevents drift.
Not allowing SSH is plugging a single hole which guarantees nothing. Can the app-server write on its own disk? Can it modify configuration files? Maybe even code?
As a developer I want to be able to write applications that are not specific to any one deployment method. I want to be able to change my mind to host this on different kind of infrastructure. These kind of limits force you to create a strong adherence between your application and its tooling and the deployment method.
Of course, the Read/Write domain has to be explicit and limited, but this is good practice anyway. Running on a distributed storage grid based on something like CEPH can allow you to address this domain not only explicitly but also in a fault-tolerant and scalable way.
When you cut off SSH you are cutting off half of the tooling we have and you make everything overly complicated. I want to be able to tail any log on any host. I want to be able to scp for inspection any file to my dev machine.
At platform.sh (a PaaS running on a distributed grid of micro-containers) we run a build oriented immutable infrastructure so there is no, there can be no infrastructure drift.
We proxy SSH so security stays the responsibility of the orchestration layer not any single host or service. This means we can filter by role any connection to any service, but developers can still just use the tools that work.
So basically, all the machines you're running ansible on will have the latest version (if that's what you want, of course)
How do you deal with systems breaking? By monitoring for them. It also helps to make them 'disposable', but this is not a replacement for monitoring. Good monitoring will tell you when time is off, when disks are about to fill up, when permissions are bad and when packages aren't up to date.
Removing ssh does not give you monitoring. It does not prevent system state from changing. It really doesn't do anything but remove a very secure and simple method of communication and file transfer.
Minimal/disposable images? That's fine. It won't make your servers or services immutable. One day your hardware's going to catch on fire and your first clue won't be the smoke billowing into the neighboring rack, it will be the errors in your logs.
But that's a false dichotomy; having SSH in no way prevents you from designing and testing your software just as well.
The article seems to advocate putting yourself into that situation when you don't have to, which is silly.
And when something unexpected goes wrong anyway it's going to be great when a problem that might've taken an hour to diagnose instead takes two weeks, because SSH is a crutch.
I used to to do OSI international interconnect support an testing for my sins many moons ago.
The building scripts should be written in JavaScript, of course. Package it as a startup. Become millionaire.