The Decline and Fall of BIND 10 [pdf]
ripe68.ripe.net
ripe68.ripe.net
The evolution of BIND had very similar sorts of challenges it seems. Earlier version worked new version was all different. So different that their customers (the system administrators) seem to have revolted. Ouch.
When you are building a part of the infrastructure that lots of people have to manage to keep the infrastructure running, its a special kind of challenge. Both in deployment, change management, and service evolution. Even after going through the process with NIS+ I'm not sure if I could even chart a path for a replacement BIND.
All too often devs get blinders on and don't realize that most people don't have their domain knowledge and therefore it's not quite so easy for them to see the forest for the giant stacked trees that are lying across the path.
If you can't avoid that morass, the new version better offer something so compelling you can't possibly skip it.
In my experience it was far more flexible than NIS which was both a good thing and a bad thing. The one big advantage it had for me was its support for SecureRPC. In time it was killed by LDAP.
Too many hours spent troubleshooting automounter / NFS failures through NIS and LDAP.
To say that it surprised Ed would be an understatement, and it set off a series of events that left me feeling pretty unsatisfied. The first thing it did was bring attention to the next version of the ONC product which was going to include NFS V3 and NIS+ and nominally just ship to customers. Except that now NIS+ was "valuable" SunSoft (the sub-company in Sun responsible for Solaris) didn't want to just _give_ it away (original plan) they could charge big bucks for it. So they rechristened it to ONC+ and wrapped some egregious license terms around this "new" product. That lead to a ferocious debate inside of the company, fractured relationships with long term ONC licensees (like SGI and HP), and an edict to not ship it on SunOS 4.1 to "encourage" adoption of Solaris 2.0. I would have left Sun at that point if James Gosling hadn't recruited me to come work on project Green, aka Java.
BIND is one of those programs that scares the living crap out of me, both from a security perspective and from a complexity perspective. (Gee, could those be related?)
Just take a look at the list of BIND vulnerabilities. We're still finding em, and I'm sure we'll continue to for years to come.
http://www.cvedetails.com/vulnerability-list/vendor_id-64/pr...
For a refreshingly simple and secure DNS serving experience, I highly recommend djbdns / tinydns / dnscache:
I couldn't agree more. I'm currently TAing a Bachelor Security course where I had to design some challenges for students to break into systems and one of them is DNS cache poisoning. I've spent the last few weeks digging into Bind's code, DNS implementation and old vulnerable versions and it's amazing how much stuff had gone ignored in the past and how many subtle vulnerabilities can hide for years in very complex software and protocols.
Kaminsky's DNS cache poisoning attack is fun.
I trust djbdns more than perhaps any other piece of C code I run, and recommend it.
This failure undoes much of the benefits of source port randomization. A patch is available from a third party, but it is not part of the standard source tree.
http://www.your.org/dnscache/
BIND, Unbound and other recursives have both SPR and duplicate query control. This fact explains things like this:
http://dnsreactions.tumblr.com/search/djbdns
Security is relative. From a software coding point of view, djbdns is smaller, has fewer features (e.g., ipv6 and dnssec lacking). I.e., there might be coding errors in large code bases.
But for me, an unmaintained code base with known, unpatched attacks is not a good choice.
BIND may have had a (really) bad run in the BIND 8 days, but BIND 9 was surely an improvement.
And if you really want to stay away from BIND, you can go for something like PowerDNS. Or a combination like PowerDNS+Unbound.
This is, for what it's worth, the original objection to qmail as well --- "it's not even maintained". It's not maintained because for the core job it does, it's _correct_. There's not a lot of mail software (at least, not software written in C) that can make a competing claim.
Has that addressed the patchiness blechery?
But bind is a fairly big and complex code base. Which is probably also tied to why people use it - it's got the features people need (and a _huge_ community and install base.)
People Fear & Hate Change
BIND 10 is quite different for administrators
– Lots of dependencies, slow build
– Lots of processes
– Tool to configure, not configuration files
People hate change.
Having lots of dependencies can actually be a huge pain. Yes, reusing existing software is wise. And most of the time, these days, on a modern unix, you're covered by the package manager. But not always. Someone is going to be building or installing on some freaky system, or some old version of something, and every additional dependency is going to be like a red-hot poker up the fundament. More dependencies is wrose.
Having lots of processes generally does make things harder to manage. Yes, there are advantages in terms of fault isolation and privilege separation. But it means that you can't just throw off a quick pgrep to see if the service is up, you have to worry about some parts of it being up. More processes is worse.
Using a tool to configure something instead of using a file ... nah, i've got nothing. There's nothing good about configuration tools. I have never come across a situation where i used a specific tool to configure something and didn't wish i was just editing a file. Tools for configuration is worse.
I'm sure there are lots of really awesome things about BIND 10. It clearly had a huge amount of care and attention given to it. But it does sound a bit like its developers were blind to the need of the system administrators who they hoped would ultimately install it.
I hadn't heard anything like this before, what is the reasoning here?
I work at Continuum using our Python distribution (Anaconda[1] and Miniconda[2]). We sell it as the Data Scientist's answer to python, but as a sysadmin, I really think it is the answer administrators have been looking for. It installs into one directory that can be blown away if needed. It provides virtualenv-like environments except it you can isolate different versions of python or any binary. You can "activate" your environment, just by setting your PATH to find the env's bin first.
The conda package manager installs binary packages, so you don't need to compile c-extensions on each machine. Conda allows you to build all of your packages for each arch. Binstar is our alpha package hosting service that lets you host packages and it has a multi-platform build service (super-alpha). And you aren't limited to Binstar if you want to host your own repos -- the commands are built into conda.
If you want to distribute non-python binaries, you can do that as well. I'm using conda to install node and node packages. I really like using conda for node binaries, because I can build them using npm and then I have a way to push versioned node binaries to production. (Internally, we've built most of the R packages, postgres, mysql, mongo, nginx as conda packages to provide non-root users a way to install normally root-only software).
[1] Anaconda: http://continuum.io/downloads [2] Just python: http://conda.pydata.org/miniconda.html#miniconda
And then you want to install a new Python side by side and you sort of do that but /usr/bin/python still points to the old version and you have to be explicit with /usr/bin/python2.6 in your scripts. But then some scripts will work and some still don't and you make /usr/bin/python point to /usr/bin/python2.6 and now you really broke yum.
And all you can do now is hate python because it gave you all these problems.
At least, with java, no one even attempts to use distro package tools to manage libraries, packages and dependencies. Can you see a sysadmin using a syslog replacement implemented in java, requiring java8, along with a webserver only running on java5?
With python, some tools (those big enough to be packaged) are no problem. Some random tool from pip -- not so much -- now you have to patch that python binary, those python dependencies etc... and not just track security.your-distro.org.
It is certainly possible to deploy your system in a similar way you deploy your apps (take on 100% of the burden of maintaining the whole thing) -- but normally you want the "system" to be a stable foundation on which to erect your rickety duct-taped mono-jvm-python-ruby app-stack with the help of puppet/chef/cfengine/salt/ansible... you don't want pip/hackage/gem/npm to manage your system. You might trust it to manage a single application.
You've still got to provision those, but, well, they're individual and isolated and tend not to go hammering into one another. Other than consuming all available system resources.
Oddly: IBM got this right with VM ... 40 years ago?
It used to be a major PITA with Perl. I've run into it with Ruby (and the proposed solutions are even more horrible: RPM).
And yes, Devs absolutely want teh neu hawtness.
Python doesn't really bring anything to an admin's life that isn't covered by those other two workhorses from the 90's toolbelt, except irritating change.
When a lot of organizations demand administrators keep a production net stable, with a staging net that bears only a faint resemblance to the production net, and a development net that is a caricature, is it any wonder the administrators despise change? I've been in some shops where the poor bastard administrators only have production to work with, and they are the ones yelled at and the quarterly reviews dinged when a change forced upon them, surprise, surprise, breaks a critical LOB process.
What kind of idiots tied their system to Python 2.4?
Nobody stops you for shipping system tools with Python 2.2 if that is what you want. But don't step on my toes and tell me that I can't install 2.6 and set it as the default one.
If you do that you give me a license to call you names.
[1]: http://fedoraproject.org/wiki/Python26#Available_Packages_fo...
Of course you know how easy it is to make your scrips use python2.4. Just make the shebang #!/usr/bin/python24 instead of #!/usr/bin/python. Then you can install another default version and if you keep the old version around you don't break anything.
Since other distributions are doing a pretty decent job at it (see gentoo), they don't have any excuse not to do it right.
Python has properties that are not congruent with the expectations of an inexperienced administrator. Its scent is unlike the other creatures that exist in our fold and it makes us feel inadequate that we cannot control it with the skills we use to control the rest.
If "using python" means intertwining the app into the system python / requiring sudo, etc then I would hate it too.
I'm sure it's really great on a developer's laptop... but the standards are and should be different. virtualenv is, to sysadmins, the symptom of a disease.
At a large enough place, that might be alright, since we might need to front-run or lag the distro anyway. But now I have to worry about n different Python versions and security patch versions, and it's a headache.
Give me a package of known-goodish stuff, not something which relies on lots of other fiddly stuff which I might need to build by hand, or maintain by hand.
And frankly significant whitespace still pisses me off: as a sysadmin, I have to make a lot of very small tweaks to random code work in various languages, and most of them have the decency to let me see where the blocks are with my eyes, instead of with my cursor counting tabs and spaces. I presume people who spend a lot of time with Python use editors that show spaces, but since everything else doesn't care (other than Makefiles, but Makefile whitespace is very simple), I don't know how to turn those options on, if they even exist in my editor. And I say this as someone who writes in erlang, with it's shitty line endings, when I get to take off my sysadmin hat.
I'm working on this: https://github.com/fenollp/kju/blob/master/examples/snippets...
"Don't worry about people stealing your ideas. If your ideas
are any good, you'll have to ram them down people's throats."
-- Howard AikenLooks so easy on surface. Made me want to write a toy server just to prove myself.
If you are writing it in C... things are more complex. All DNS records are variable length, so you have be damn sure you don't have off by one errors, etc. Also, proper DNS works over both TCP and UDP so you have to architect it such that you can work with both. Not knowing anything about the code of BIND, I can say this would take quite a bit of careful planning to not get wrong. I can see a whole slew of buffer overflows happening in all these places.
[1] https://github.com/ipartola/pulpdns/blob/master/main.py
My idea for this code was to have a DNS server that keeps track of which domains are the most requested and in the background keeps a thread running to always keep them in the cache. I noticed that even with a pretty fast connection my DNS resolver on my LAN can take 100-300 ms to return a name not in a cache.