On Configuration | Hacker News Reader

On Configuration

sstephenson.us

91 pointstelemachos53 comments

This is, I think, one of the classic splits between developers and sysadmins. Sysadmins are always logging in to unconfigured systems, new systems, foreign systems, and of course broken systems. Being fluent in sh, bash, vi (appreciating, but not depending on vim-isms) and knowing your way around every config file in /etc... those are basic tools for sysadmins.

Developers should have the most productive environments possible. If that means eleventy megabytes of dotfiles and the like stored in a git repo, so be it. Once you move in, nothing should get in the way of turning thoughts into code.

What this means for DevOps... is that you had better have a development system which is in no way experimental. Try not to do dawn surgery on a patient you're going to need to run a marathon in the afternoon.

Disagree sir. It all depends on how much you depend on your toolset to get anything done.

I don't use textmate because its not installed on servers. I used to use ksh instead of shimming in bash when on AIX systems because it was the default.

My toolchain: Bash, VIM, and... well, that's freaking it. No NERDTree, no craziness, nothing.

The only thing I need to configure my fully functional environment and be at full productivity:

set tabstop=2

set ruler

set expandtab

syntax on

filetype on

* formatting

No, what this means for DevOps is that you'd better create a foolproof method to insert your dotfiles into the distribution - and that it's more important than ever to argue for individual user accounts, not role accounts, on production machines. For security and logging, of course.

Love, DevOps

more important than ever to argue for individual user accounts, not role accounts, on production machines. For security and logging, of course.

That's a terrible idea on so many levels, I hope you're being sarcastic.

You're saying it's better for sysadmins to all log in as "prod" or "root"? How do you figure?

Firstly on properly automated hosts you very rarely have to login. And when you do then 'prod' and 'root' work just fine.

Scattering user-accounts across machines is a maintenance headache (trust graph, sudo passwords or passwordless sudo, filesystem permissions) and encourages bad practices ("i'll just run this small script, quickly").

If you're serious about auditing then wtmp and notoriously incomplete shell-history files are not your tools either.

Logging in a root is an exceedingly BAD idea, regardless of how properly configured the host is.

Let me know how your compliance audits go. I'm sure the pci dudes won't mind that "root" logged in from 20 different hosts.

Why would a human login at all?

In serious deployments people don't login to production hosts outside of extra-ordinary incidents. This entire discussion is based on a broken premise.

In single-server deployments, you're logging in because you're the only employee.

In ten-server deployments, you're logging in because every machine is trying to take on a new role or aspect and you're still learning how to automate that.

In thousand-server deployments you're logging in because the latest batch of Seagate drives has a statistically significant failure rate and you need to try some experimental firmware they wrote for you.

When do you get "serious"?

I don't always log into my production boxes... but when I do, I log in as 'jay'.

In ten-server deployments [...] still learning to automate

That's why your original comment tipped me off.

It's hard to recover from bad patterns like the one you proposed, it's easier when you try to get as many things right as possible from the start.

In thousand-server deployments you're logging in because

I take it you have not worked in such an environment before. I can assure you nobody manually logs into hundreds of hosts to "try some experimental firmware" - or to do anything really.

Beyond a couple dozen hosts pdsh and "knife exec" simply don't work anymore (rollback? what rollback?).

I take it you have not worked in such an environment before

Does AOL count? I built the mail system. It was kind of a big deal. We pushed 4,000 TPS through servers less powerful than an iPhone. I also spent time at Akamai, though I admittedly worked on a tiny, isolated test lab (150 servers or so, a fraction of a percent of the "real" server base).

I have worked with some amazing people, at all scales, and I have never seen a deployment that managed to completely avoid manual logins. No, you aren't logging into a few hundred hosts manually (well, there was this one shop... but let's not hold that up as exemplary). But you're probably logging into a few to test things out manually before you decide what your automated deployment script will be rolling out.

Even when you do deploy a fix, with a large sysadmin team it's good to know WHO exactly ran that one-off deployment script. When I said "auditing", I wasn't even thinking of PCI, or auditing against malicious actions of any kind - just simple troubleshooting forensics. But we have a great, robust way to see exactly who modified a file and when; it's the file system. Why reinvent it? (I just spent the last day trying to figure out who edited a script. Why, "postgres" did.. of course it did.)

Should you aim to do these manual config changes in the test lab first? Of course. Do you always succeed? No. Some things only get tested in the big lab. Reality intervenes. I try to plan for it.

Mind you, I'm arguing this partly to see if I can be talked out of it. I think I believe it, but I've been wrong about way bigger things. If you've achieved my automation nirvana, and you roll things out to a few thousand servers, never manually, and never wondering who did it, tell me how it works.

But we have a great, robust way to see exactly who modified a file and when; it's the file system. Why reinvent it?

That's a strange statement given your background.

Since when does editing a file change the owner or leave an audit-trail behind, on stock linux?

If you're running auditd then that's a different story but I've never seen that used in production for this purpose because it's hard to get right and because there are so many simpler options much better suited to the task (version control, puppet, chef, etc.).

Auditing or not, in my experience you inevitably end up in permission-hell when multiple users mess with files that need to be readable by a common gid (sticky bits, umask, something is always off).

If you really found a way to make this work at scale, a detailed write-up would be interesting (to me).

In my experience that way lies unmitigated madness. I've had to clean up multiple such "sudo-style" deployments in the past.

GAH! See, I just slipped up and showed my age. I'm thinking of the Stratus VOS filesystem, which did change the modifier when you edited a file. You're right; UNIX/Linux files have an owner, which doesn't change on modification. Which shoots a big blazing hole in my argument. You can back into who modified what if files were changed in a big batch, and they also created a few new files - but you could also back into that via login times.

Since those days, it's less that I've seen the individual-user system work at scale - it's that I've seen the single role account fail, repeatedly. I've managed to avoid permissions hell with a few well-chosen supplemental groups, but maybe I'm too optimistic that what used to work in one environment will work in another. Ask me in six months. :)

  > properly automated hosts

  > Scattering user-accounts across machines is a
  > maintenance headache

Seems to me that properly automated hosts could easily have user accounts automated too, no?

And when you do then 'prod' and 'root' work just fine.

They do until you have to show auditors who logged in and when.

Scattering user-accounts across machines is a maintenance headache

That's why God invented Active Directory.

They do until you have to show auditors who logged in and when.

SSH has logging. Also if you have audit-requirements then the login-log is normally the least of your worries.

That's why God invented Active Directory.

That must be one cruel god you have there...

I agree that SSH logging is sufficient. That was not the finding of our auditor, per best practices, blah blah. Thank SOX for that one.

That must be one cruel god you have there...

I don't have to _deal_ with it directly, just use it to authenticate logins on (most) of my unix servers.

Seen from afar, it's not that bad.

Now, she is a jealous god, as perceived by her acolytes.

Nobody really wants to hear about how, during last year's DR testing at IBM, my stuff was up and running waaay before AD was working.

If you're worried about logging who did what under the user, you can disable password access, and use ssh keys. IIRC, ssh can log which key was used to log in. I'm not sure how audit-able this would be (for example) in determining which user logged into 'prod' ran the malicious process.

Don't worry, it's a phase. You'll go back to configuring at some point, just more judiciously. And then you go back to "Vanilla install ONLY!". And then you start to tinker again, just even more carefully.

At least that's my experience over the last 20+ years. You need - for all the reasons outlined - the ability to work on stock systems, because you never know if you'll have all your favorite tools.

At the same time, some things improve productivity well enough that they're worth adding to your dotfiles, so you'll have them on the machines that you spend 90% of your time on. (I'm thinking "127.0.0.1 ycombinator.com" might be a great one ;)

Hey, you stole a line from my hosts file! :)

I no longer fall into deep pits of yak shaving, but I'm not sure I agree with the opposite extreme either ("use the absolute vanilla defaults").

But I would expect that there's a natural progression (sort of like the many "evolution of a <language> programmer" posts[1]). You start using these programmer's tools, and at first you struggle to do basic things with default settings. Then, later, you learn that everything can be configured. You spend days (weeks? months?) getting very little real work done, but configuring everything. Then, later again, you calm down, find a set of reasonable and productive configurations, automate getting them onto new machines when necessary and go back to work.

[1]: Example for Python: https://gist.github.com/289467.

It likely depends on what you're doing as well. For programmers who spend most of their time editing source code and working on a small number of development machines it makes sense to put some effort into customizations to optimize workflow. For sysadmins who have to be able to do work on thousands of machines across various environments it makes sense to know the stock settings inside and out.

This echoes what I've come to believe over time. I used to spend a LOT of time tweaking my environment, and searching for that magic productivity-enhancing utility that was going to make me super-productive and optimize my time.

I've gradually come to realize that most of that time was just a way of procrastinating, and that I was better off just taking what's there and getting on with it already. I'm amazingly more productive now that I've freed up all that extra time to actually work :)

My perspective on this whole thing is that I really enjoy configuring my tools and regardless of how incompatible my setup becomes and how much time I waste in the process, I don't see myself stopping anytime soon. Different people value their time differently.

It has become more of a hobby than some self-serving quest to improve productivity. Working on my config files or going through those of others in search of little gems is something I do for recreation - it's a very light mental activity. Configuring things is probably my favorite pastime in front of a computer :)

Still, highly configurable software should by all means come with smart defaults.

I don't mean to turn this into a book club, but this ordeal bears resemblance to Orr from Catch 22 fixing the thumb sized valve of the stove in Yossarian's tent. :)

       'If you want to work with something big, that's okay.
     But that valve is filled with tiny parts, and I just
     haven't got the patience right now to watch you  
     working so hard over things that are so goddam small 
     and unimportant.'
       'Just because they're small doesn't mean they're  
     unimportant.'
       'I don't care.'

This is one of the reasons I stopped messing around trying to find the perfect text editor. I focused on getting better with a stock VIM install.

I think the spirit of this post is that you should at least give the defaults a chance before you go crazy trying to make everything "better". However, I would add that you shouldn't totally ignore that voice that says "is this setting getting in my way?".

I think it's a balance just like anything else.

In high school, nothing was safe from my tinkering and everything was extremely fragile but also extremely tailored to fit my usage. In college, I found myself practicing a sort of configuration asceticism where all I needed to be productive was standard bash and vim. However, now that I'm a full-time software engineer I've found that by investing some time into configuration, I've optimized my common-case workflows tremendously and am much more productive as a result of it.

I think I'm still tremendously comfortable with just the bare necessities, and I agree that having this sort of habit makes context switching a lot easier, especially on foreign machines where you can't make many assumptions about what people have.

I think the decision point is centered on how much you find yourself context switching between setups. Personally, I'm only ever on one or two

free you from tweaking things and let you focus on getting the job done.

my ability to adapt to a foreign environment without frustration is more important than the benefits of configuring a local environment to suit my whims

This is overly simplistic. It certainly only applies to people spending a lot of time in foreign environments, which is not the norm unless you are doing tech support. I don't know if it's link bait or the penchant for techies to really like black and white simple rules for complicated things when they are even slightly outside their expertise.

I don't even believe that anyone thinks this is true.

I feel like everyone must know that you can get significant productivity gains by tailoring your environment to the things you do often but that obsessive tweaking of minor settings is a bad trap to fall into. Between those two extremes is where you want to be, where in that spectrum depends on what you do specifically. Welcome to the adult world of complex problems that don't have easy solutions. You are already used to this if you are a developer or sysadmin aren't you?

That answer makes for a crap blog post, but does anyone actually disagree that it's the right answer?

Apparently so

I think this is kind of an interesting phenomenon that extends to other areas as well. It's easy to become reliant on intellisense to the point of not being able to remember simple API functions, object names, etc, and it's tough when you find yourself in a situation where you have to work without it (say, for example, a job interview).

And programming isn't the only area that's being affected by advances in technology. Does anyone remember the study linking GPS to atrophy of the hippocampus? (http://articles.courant.com/2010-11-22/features/hc-weir-gps-...).

Yes, these things make our lives easier, but at the same time we become completely and totally dependent on them to function.

I've been sticking with the defaults pretty much since I started programming (around 1984) and always came across people doing a LOT of tweaking.

I can't help but wonder if that's a bit like guitar players that spend more time tuning their instrument than playing :)

Indeed.

Also, I generally don't like my tools enough to spend so much time on them. It's just a tool after all, I go to the hardware store, pick a hammer, and start pounding nails.

This is why choosing good default settings matter

I feel his pain to an extent. However, rather than learn to cope with an inferior but pervasive alternative, I just use a simple shell script[1] to configure zsh for me. I have it on my personal server with a short and mnemonic URL, so I only have to run one easy-to-remember cURL command, and my shell is configured. I can understand how this would get annoying if I did it for every application, but when it comes to the shell or editor, I am not willing to compromise.

1 - https://github.com/dbb/zsh-stuff/blob/master/newz.sh

I don't mind the rtomakyo dotfile style. (https://github.com/rtomayko/dotfiles)

I haven't implemented it myself - mostly relying on standard tools as the author does. Vanilla bash, vim, etc.

The other trick I like to employ (if I'm in control of the network) is have every machine on my network mount a central (NFS) home directory, with all my dotfiles in it. As long as that's backed up, and configured on every machine, it works very nicely.

I wish top 40 wasn't top 40. But if you want to sing along to the radio, you have to learn the words.

It sucks, but there is zero alternative unless you want to boil the ocean.

Ubuntu switched /bin/sh from bash to dash recently. Many distros are starting to abandon init.

It will get better. Until it does, our hands are tied.

My life became a lot easier after I settled on just using My Documents for music, pictures, etc.

Sort of OT: Does anyone know any good examples (on github or otherwise) of a dotfiles repo that can easily be installed on a machine? I've been wanting to do this for a while, but I'm sort of a noob on symlinking and such

Maybe not the best example, but here's my dotfiles. I use install.sh to link them. https://github.com/BenDoan/dotfiles

I've designed my dotfile repo to be easily installable on any machine I happen to be on with the help of a script that backs up any existing dotfiles that may conflict with those in my repo. This allows me to backup any existing configs, link my own temporarily, then restore the old ones and 'rm -rf <my-repo>' so it's as if I never even touched the original configuration.

If you want to try it out just run the following commands:

    git clone git://github.com/acx0/etc.git ~/etc
    cd ~/etc
    git submodule update --init     # my link script is in a separate repository so it's a submodule in my dotfile repo

    cd link.sh
    ./link.sh                       # this will show the status of any conflicting files
    ./link.sh -b                    # backup any conflicting files
    ./link.sh -wf                   # force write (symlink) my set of config files

And everything's set up. If you want to install the vim plugins I use, just run

    vim -c 'BundleInstall'

and let Vundle download the plugins (you have to restart vim to use the plugins).

Then when you're ready to restore your ~/ back to its original state, run

    ./link.sh -r                    # restore any configs that were backed up

which also takes care of deleting the configs you didn't have (so if you backed up with no conflicting files (backup dir empty), and then restore, it'll just remove all the existing symlinks)

If you didn't run the backup command, you can simply run

    ./link.sh -d

to delete the symlinks for all the files in the repo. Then just remove any traces of the repo with 'rm -rf ~/etc'.

Maybe one day it will be mundane to clone a tailored environment to a remote system. No changes outside $HOME and only in a controlled lifespan.

Yes. Why are our remote systems so difficult to use in the same way as our local systems - especially on the command line?

Not sure if this is a good idea, but it might help: https://github.com/pfraze/lugg

What I discovered is that in many cases, my ability to adapt to a foreign environment without frustration is more important than the benefits of configuring a local environment to suit my whims.

YES. That is exactly why endlessly messing around with text-editor configuration frippery is a waste of time.

I use a relatively vanilla bash shell, but I actually think the text editor is an exception here as long as you don't spend a lot of time yak-shaving.

It's pretty likely you'll be in a foreign bash environment on a semi-regular basis, it's pretty unlikely you'll sit down to code in a foreign environment.

My emacs env is trivially importable to any machine I happen to be on anyway.

I don't know. Whenever I use my friends Windows computers--and, reasonably, Windows is the default--I'm lost. And yet this is not a good case for switching to Windows!

Besides, these days, I work from my laptop. 99% of the time, I'm on my computer. Now, perhaps I'm remoting into some server, but if so I'm doing that from a local Emacs. When I need to use a new computer, assuming it's running Linux, I only need to copy a couple of configuration files over and it works exactly how I want.

So the only time I have a problem is if I'm using somebody else's computer for a short time. This does not happen often, and when it does it doesn't last long. Moreover, as I said above, chances are high that this computer is either running Windows or OS X, so I'd be at a disadvantage even if I didn't tinker with my setup.

Finally, fundamentally, I think configuring your system is worth doing if it makes you more productive. Most of the things I've configured in Emacs do make me more productive (as far as I can tell). Things like adding new keyboard commands for operations I often use and making Emacs automatically recognize when I'm doing work stuff as opposed to non-work stuff just make life easier for me.

Configuring any individual feature is an O(1) action. Assuming it brings a benefit, this benefit is going to be in O(n) to how much I use it. Given I use both my OS and my text editor a whole lot even a significant amount of time spent configuring everything to work well pays off.

I'm not wasting my time--I'm increasing my efficiency.

Just keep in mind that O(1) = O(1 + c). Depending on the relative size of n and c, you might want to stop and think :)

Happily, this gets amortized as n increases. So I just have to use Emacs more to make it worth it. A delicious proposition.

(Hey, I should totally write a mode for browsing HN from Emacs! Then I'll have no need to ever leave :P.)

try w3m-mode (http://www.emacswiki.org/emacs/emacs-w3m)

Try Conkeror. It's exactly what you want.

http://www.conkeror.org/

> Whenever I use my friends Windows computers--and, reasonably, Windows is the default--I'm lost. And yet this is not a good case for switching to Windows!

Actually, I think it is a good case for switching to Windows. I just don't think it's a sufficient case.

I've never been convinced by that! Procrastination is procrastination. If you weren't tweaking options, you'd be staring out of the window.

Agreed. The trick is to be away from your comfort zone periodically, so you can retain the mnemonics of vanilla bash/vim but take advantage of zsh's nice features as well. I typically don't have ZSH running on any remote server I work on (except my personal VPS) for precisely this reason, it helps me keep the vanilla shortcuts in memory.

But realistically, most my custom configuration is suited to working with large projects that have lots of files. I never open the whole project up in shell vim on the remote server, if I'm gonna be doing that I might as well just open it locally and push to Git. I feel that configuration should build on top of the editor/shell you're already using, not modify it so heavily that it renders the vanilla shell unrecognizable.

Reply on news.ycombinator.com