New Year's Resolutions for SysAdmins
usenix.org
usenix.org
I'm a bit horrified to read this here. If you're a sysadmin and don't have both automated and manual testing of backups, it's hard to imagine what else was a more important use of your time.
There aren't many things that could bankrupt a healthy business overnight, but catastrophic data loss is certainly one of them.
An analogous entry for a lawyer might be to pay attention while reading contracts.
Keeping shit running now is always more important than keeping shit running in case of catastrophic failure.
Because a catastrophic failure is only catastrophic if you actually have something of value in the first place.
In the real world it's a balancing act in which there is no room for absolutism. Choosing which compromises to make is the hardest part of any job that comes with a level of responsibility.
But by the time you're hiring full time for the position, outside of a few edge cases where maybe you're SnapChat and you are growing 20% a week, it's probably time to settle down a bit and be sensible. At which point, if testing your backups to completion to avoid catastrophic data loss isn't #1 on the todo list, it's #2.
I'm not saying this is how it should be, but it is how it often becomes if the sysadmin or whomever taking on those responsibilities don't report to someone who also see the data integrity as priority #1 for the sysadmin.
I've worked in places where the CEO's e-mail client configuration is the #1 priority for the guy that should have been focusing on server backups, for example, and where prioritising the backups would be a bad career move for the person in question.
(Yes, that is a huge warning sign that it's best to find a different job)
Last place I worked with sysadmins, they were fire fighting like crazy, not enough of them, bad choice of suppliers, poor code bases meant things fell over and took far to long to run and a dev team begging to get off the internal network on to their own because of all the restrictions.
I needed a backup of the DB of one of the three systems to test a major setting change that no-one was 100% sure would work because no-one understood the bloody awful contractor code with the even more awfully designed DB.
Took them 2 weeks to get it to me.
Crazy place, quit after 2 months. Really thought I'd asked the right questions too.
more info: http://www.tty1.net/blog/2010/ifconfig-ip-comparison_en.html
Then I worked for an arbitrage desk at an investment bank. They used virtual addresses for different IPs, on top of vlans connected to different exchanges, redundantly (ie bonded).
Not a single IP-having interface appeared using ifconfig.
NOTE
This program is obsolete! For replacement check ip addr and ip link. For statistics use ip -s link.
http://www.openbsd.org/cgi-bin/man.cgi?query=ifconfig
Pedantry, but keep in mind that there isn't really "the" man page for this.
Ubuntu might want to deprecate ifconfig, but that doesn't mean everyone does.
And dnsip/dnsipq/dnsmx/dnsname/dnstxt: http://cr.yp.to/djbdns/tools.html
What the original post most likely referred to is to use ip instead of ifconfig, and to use dig instead of nslookup.
The Linux way isn't always the way the rest of Unix is going. In fact, most of our current OS's are a hodgepodge of different great things developed by different teams - for example, rsync from the Samba devs, sudo from OpenBSD, git from Linux, etc.
This is one of the great things about Unix is that it's a big sandbox to play in. The entire "lets replace init" movement from the last 10 years is pretty fascinating in it's own right. Spend some time looking at how the other crowds out there are doing things - you might find something interesting and helpful.
Edit: I'm also having a difficult time trying to figure out how to get up to speed with chef in the specific context of opsworks, since most of the chef-specific courseware I can find assumes I'm using a hosted chef server, and I'm not quite sure how that overlaps with the opsworks flavor. It's starting to feel like I'll be at this for another week before I can get even a simple stack running.
As well as the reasonably good ansible documentation, checkout the ansible-examples github repo. ( https://github.com/ansible/ansible-examples )
There's a few initial 'conventions' to figure out (roles, group vars, playbooks), but then it's so incredibly simple to get into.
My dad is playing with a raspberry pi at home, wanting to set up a web kiosk thing. I'd been doing something similar a few months ago for work, so I send him my ansible playbook. He could follow it as if it were a plain text checklist/tutorial, without needing ansible at all. If he wants to automate it, he can.
That, I think, is one of the benefits of ansible. It's not a lot more work than just writing an overview of what you need to do in the first place, but with the benefit of it being repeatable and automated.
It's intended for use with Chef Solo but potentially the early sections on writing custom chef cookbooks may be of use.
Cannot stress this enough! There is a sea change happening and sysadmins run the risk of being swept away unless they learn about automation (puppet/chef) software and cloud infrastructure services (openstack/aws). My new years resolution is to release screencasts on these very subjects. I have already released one on Learning Puppet with Vagrant @ http://sysadmincasts.com/episodes/8-learning-puppet-with-vag... if anyone is interested.
1) luckily backup seem to be reasonably automation friendly
2) "We do not support virtualization of any kind"
3) tied to hardware on one machine - f'n brilliant.
I personally haven't tried building environments on VMWare as I run pretty much everything in AWS. However, this presentation looks promising.
3.1 "Check that you can actually restore a file/recover a database"