Unscrewed: A Story About OpenBSD | Hacker News Reader

Unscrewed: A Story About OpenBSD

skeptech.org

186 pointsnetten64 comments

I'll never forget this time in college when I had set up an OpenBSD router in our apartment and came home one weekend to find that I was unable to log into it via SSH.

Actually, it was timing out.

So I hooked up a monitor and a keyboard to this old old P1/133 with 32MB of ram that had been taken out of a dumpster to be our router to see what was going on.

A few months prior to this I had been playing around with rrd and started generating traffic graphs so that I could get pretty pretty pictures of our traffic (and to figure out when one of my roommates was streaming his porn).

I had set the cron jobs to run every 10 minutes. Initially the data sets were so small that this wasn't an issue, but over time the database got larger and suddenly the cron job was taking 10 minutes and 10 seconds, and then 10 minutes and 20 seconds... on and on until finally cron jobs were running over cron jobs.

So after about 2 minutes waiting to get logged in, I finally get the load average for the system and it was somewhere around 146.20. After another couple of minutes, I was able to disable the cron jobs and kill off rrd graphs. In another 10 minutes, the load average was back to < .70.

All this time it was passing traffic without a hiccup and I believe it had another 100+ days of uptime after this before I moved out. What a box. I miss that router.

ah, nobody ever forgets their first lock file.

I've always (or, for many years) wondered why this wasn't more common. Every network I've ever built has been routed by Linux or a BSD, though I've maintained a few with Cisco routers. Partly because it's what I know, and partly because there's never enough in the budget to to everything the client and I wanted to do with Cisco or even Juniper gear.

Now, admittedly, I've never worked on big networks. But, I know what I like, and I know working on a Cisco feels like 1996, and that working on a full-blown modern UNIX/Linux system feels like home. And, fast network gear is available for Linux and the BSDs. 10Gb interfaces that work with Linux/BSD are not difficult to find, and 1Gb interfaces (sometimes several of them) have been standard on every server and barebones rackmount on the market for a decade.

I understand there are special purpose boxes that do things no off-the-shelf commodity box can do (I don't know enough about high end networking to know where to draw that line), but I honestly doubt 95% of business networks need them. And, yet, a huge percentage of business networks that have real routers (rather than little wifi routers from LinkSys/NetGear or whatever) have a Cisco of some sort, even when they don't have anyone on staff who knows how to work the thing. I've walked into several businesses that had expertise in house for everything except their router, which is also weird...why pick a Cisco router for your little ~100-1000 node network, if you've got Linux or BSD guys hanging out in the server room every day? They'd call someone and sit around waiting when their network was acting up.

I learned how to use Cisco routers just so I'd be able to replace them easily (enough knowledge to read the routing tables, firewall rules, etc., but not enough to do much useful), when I found situations like that.

Regarding first paragraph of the story (redundant pair or routers crashing at once), it worth mentioning one possible neat trick to avoid this: When having a redundant pair of devices (routers, etc.), make sure such pair is comprised of devices made by different vendors - thinking here being that if there is a bug that triggers router crash, it's unlikely that two different vendors would be vulnerable to the same one.

Or with a generous budget, you can do what BT have done with their 21CN project: For each macro-block (access, aggregation, core), have two redundant networks each one built by a different set of vendors.

No such thing as session synchronization for different vendors. You also need to configure everything for both of these vendors, which may or may not have equal feature sets.

It's hard to have independence without independence. You want separate, you're going to have to deal with separate.

Yeah but no sane organization is going to do that. It would create so much headache.

A sane organization will do it if the benefits outweigh the risks. An organization with sufficient risks will chose it. Most won't, because the risks (specifically, hard cash) won't be outweighed by the benefits.

But that's incidental to what was my real point, which is that if you do want a separate network, it has to be separate. Tie your two "separate" networks together with integration and you're returning single-points-of-failure back into the mix. Granted, practicality may dictate a couple of those... integrated authentication comes to mind... but you don't want the two separate networks to be all slickly and smoothly integrated for the most part.

It's not about avoiding bugs or vendor vulns when you talk about HA in a network config. It's about hardware failures or individual crashes.

Maybe for firewalls (synch'ing state), but for routers there's VRRP, at least.

In the context of hosting/cloud network what would probably be load balancer / reverse proxy which has the most state/sessions. And incidentally, LBs are probably the piece of dedicated network iron most ripe to be replaced with a bunch of commodity servers and some software.

Regardless, statefulness is bad of course, and less of it you have, the better you are off - we've seen this time and again: share nothing arch[1], Amdahl's law[2], locks-vs-RCU[3], memorizing tcp flows-vs-IP header hash[4,5], etc... Being able to make decision on current packet/memory access here and now, independently, without having to keep track of other flows/connections/sessions/threads beats alternative approaches every time [6].

[1] https://en.wikipedia.org/wiki/Shared_nothing_architecture

[2] https://en.wikipedia.org/wiki/Amdahl's_law

[3] http://www.rdrop.com/users/paulmck/RCU/hart_ipdps06.pdf

[4] https://tools.ietf.org/html/rfc2992

[5] http://www.juniper.net/techpubs/en_US/junos13.2/topics/conce...

[6] Intel's heroic effort to maintain cache coherency at all cost is a notable exception, although opinions vary wildly on whether this is actually the best thing to do, and how long they will be able to sustain it.

This topic is often discussed on nanog

The answers is the same as in software industry: Configuration management.

Plus it makes then harder to develop corporate cultures of deploying well known patterns that makes routers best used.

Plus it increases the surface of vulnerability for government backdoors.

In real life, there is a trend for going BSD box for cost/formation/stability/security reasons in the ISP world ... especially in non US countries that are getting kind of thinking the USA is becoming the new USSR in terms of systematic spying on every one.

These Europeans really don't get it, isn't it? They never were in the middle of cold war spying insanity where both power would puppet so called terrorists groups to make sure citizen would vote the right way under the fear.

What a bunch of stupid persons not believing that business companies are incorruptible and think they have incentive to betray them in exchange for gvt subsidies/regulations.

A couple of elephants in the room:

Why on earth were you running a hosting business with Cisco 2950s? Those are wiring closet desktop switches with tiny buffers. That alone probably caused a lot of pain. You didn't know about it, but they did.

Second, the Juniper J4350 is a software router based on BSD. There's no special hardware in there like ASICs. It's just a PC.

JunOS (historically) is based on BSD. They're moving to Linux now.

I love OpenBSD for small projects too, but let's admit that there was some bad logic in this decision making process. This could be called "how the mx10 saved my ass because I went with undocumented, unproven open source project that has been on github for 3 days" if the technologies were reversed.

> JunOS (historically) is based on BSD. They're moving to Linux now.

Do you have details on this? Given how active they are in the FreeBSD community, I would have thought to have heard about that already.

Depending on how you look at it some platforms are already running Linux.

QFX5100 runs a Linux KVM hypervisor with the traditional FreeBSD based Junos running as a guest. Same with the new x86 based SRX (SRX1500). There are more moving to this model.

The x86 based SRX is the most interesting with the forwarding daemons running on the hypervisor OS and the Junos based VM is just for management and protocols. It's not that hard to see that management and protocols can be easily ported.

There are more details that are locked behind an NDA.

well, he did mention the lack of $150k for the proper cisco hardware.

In economic terms, this would be known as "barrier to entry" if you don't have the money to enter the market. In business terms, it's known as "the cost of doing business." Not having the money to push packets is bad when you're in the packet pushing business. That's like trying to run a high frequency trading shop from Alaska and blaming your ISP instead of the speed of light.

That said, you can buy a ton of good gear cheap on the secondhand market. This is extremely popular in the lower end of the hosting market. It's cheap. It works. It's a known quantity. You can get secondary support from companies like NHR. You don't have to buy gear new.

Used gear was probably a much better decision than new gear here.

I've heard nothing about JunOS moving to Linux. That would be a truly bizarre move on their part, both in terms of technical quality of Linux vs FreeBSD, and licensing headaches with the GPL.

Do you have a source?

I found an article[0] from the end of 2014 that mentions they are using linux for some Open Compute project based switches (OCX models), as well as Yocto Linux as KVM hypervisor for JunOS on some models.

[0] http://www.enterprisenetworkingplanet.com/nethub/juniper-emb...

Ha, my first gig with a hosting company, 2950 was at the top of every. single. rack. We grew that company from a rack to about 40 racks using 2950s. Each one of those racks had hundreds of VPS customers. We didn't have issues with 2950s not doing what they needed to do, in fact, they were perfect for the job -- performance, price and design were just right & the success of the business kind of underscores that.

Later on, I was working a network overhaul at a private college dedicated to video game creation (DigiPen). Each student was provided a desktop with their own roaming profile. These profiles got huge because there were things like 3D Studio Max projects being transferred to whatever computer the student happened to be at during that period. So we're talking about hundreds of students signing on at about the same time & massive amounts of data being transferred in a very short time. The old network topology was melting down on a regular basis causing the entire campus to come to a halt.

We fixed that by putting in a pair of -- you guessed it -- OpenBSD/PF routers. Now this wasn't some "oh crap" kind of bandaid fix; we had planned this all out & implemented over the summer months when there were no students. It was all planned out (the only problem was that the original plan called for FreeBSD & PF, which crashed -- once we had OpenBSD running, it was 'set it & forget it.')

Maybe I'm just too dumb to know the difference, but those 2950s and OpenBSD routers helped me sleep a lot better at night... they were perfect for what they were intended to do.

And as more than a decade of experience since has proven, the right tool for the job is often the one that gets scoffed at as a "toy" or good "for small projects."

You monitored those ports for drops, buffer errors, and discards? You alerted when they occurred? Most don't watch the right counters or log messages. Even then, I guess you can get away with it if your customers aren't savvy enough to complain. Detecting microbursts in a VM would be impossible. Ignorance is indeed bliss.

One of my first bosses thought his Dell switches were awesome until I showed him all of the drops.

Also, roaming profiles were a great idea in principle and a horrible idea in practice. There's a good reason those things went away circa Windows 2000. ;)

What's the difference between OpenBSD and FreeBSD. I've heard great networking things about both, but never really seen them compared side by side.

Four letters. Oh, well I guess maybe just 3 if you squint.

According to Wikipedia [1], OpenBSD optimizes for maximum security, whereas FreeBSD optimizes for performance, and NetBSD optimizes on portability.

1: https://en.wikipedia.org/wiki/Comparison_of_BSD_operating_sy...

In addition to this, OpenBSD develops PacketFilter, aka PF, and so OpenBSD is the place to go for the latest and greatest from that project (and many other OpenBSD projects).

As might be expected from a description so terse, this is a bit of a generalization. I think my preferred explanation is that FreeBSD aims to be a more usable general purpose operating system, whereas OpenBSD is more conservative, preferring to grow at its own pace and in its own directions which may not necessarily be the state of the BSD art in terms of user friendliness. A great example of this is ZFS; no free or paid OS outside of Solaris supports ZFS to the extent that FreeBSD does, but if you want to use ZFS with OpenBSD, you're outta luck - it's not even on the roadmap. Last I checked, OpenBSD had no interest in moving from GCC to Clang/LLVM either.

That being said, both OSes are free, and so is VirtualBox, so why not spin up a couple virtual boxes and give them both a try when you have the time?

OpenBSD has a conspicuous philosophy which involves a strong preference for C over C++, a ruthless minimalist pruning of the packages and ports tree, and an allergy to prevailing fads (see the reasoning behind OpenBSD's use of comic sans serif and CVS for examples of this).

About... two decades ago I set up an OpenBSD box for my boss to use as a basic web host to pass stuff to clients.

The only time it failed -- the ONLY time -- was due to a full hard disk. Why was the hard disk full? Silly me, I forgot to implement some sort of logging discipline and a couple years' worth of logs of attempted exploits (mainly Windows NT exploits) glancing harmlessly off OpenBSD's hull had filled the disk.

From the article "So there I was, right out on the razors-edge, implementing production networks with an untested OS on brand new hardware. And oh how it failed."

Doing Enterprise IT Arch work, I turned down more than one hot new system and put others through significant lab work, to avoid this fun. It's hard, it's expensive and it's why I wish more hardware was open.

Can that hardware keep up with crypto bitrate and firewall rules, on multiple 10gb links?

Probably not. All of the big vendors use some sort of hardware acceleration to achieve >= 10gb.

There's some progress, though (like PF_RING in Linux and netmap on FreeBSD).

if by “multiple” you mean 2x, then yes ;)

still, you need some tweaking or things like dpdk/openonload/vma/etc...

Did you actually RTFA?

> The only reason I use “real” network gear anymore is ... for core routing 10gbps links.

Article is old; situation in 2016 might be different.

The Snabb Switch crew are starting to work on support for basic handling of packets on 100G NICs. However...

> With the new CPUs tending to increase the core count while the CPU frequency goes down, we'll get less and less "CPU cycles per packet". For example E5-2699v3 45M is 18 cores@2.3GHz. 10Gbps with 256 byte packets is achieved with 4.53Mpps, this makes ~500 clocks per packet on this CPU. Now make this 100Gbps and you get 50 clocks. You get this budget to receive, process and send the packet. 50 sounds pretty low to me.

Numbers refer to using only a single core. That particular CPU has 18 cores and 36 hardware threads.

https://groups.google.com/forum/m/#!topic/snabb-devel/pDd_uB...

Normally the solution is to have the NIC give you multiple receive queues and then tie each queue to a different core, meaning you get a packet on each core every time you clock them off the NIC. Increasing CPU cores gives you a massive win when you do that.

(On a hardware note, increasing cores and decreasing frequency makes the CPU look more like an ASIC :) )

I enjoyed the story, brother.

Commodity hardware and general-purpose systems win every time.

Honestly I wonder, who the hell buys these overpriced networking appliances? Similarly with expensive proprietary databases, or ESB systems. Are they just leftovers of a different time (note article is from 2007)? Is it a case of management buying from the salesman who bought them a nice dinner? Some place where these things generate actual value that I'm missing?

I think it's mostly about support and SLAs. Some people like to know that if something goes wrong there's a vendor they can shout at.

Sadly though, shouting at a vendor doesn't fix your issues within a couple of hours, and downtime can be a pain.

Running Linux/*BSD/Whatever on off-the-shelf boxen which you can pickup everywhere (and have cheap spares) and fix yourself will let you get back online quickly.

If it's not your money you're spending, then the support contract helps take the heat off you when there are problems. Also, the junior staff can do things when there are problems and you're not in the office.

And it is easier to get someone external, e.g. from an MSP, in to fix or rebuild something.

> Sadly though, shouting at a vendor doesn't fix your issues within a couple of hours

It does if your support contract is expensive enough.

> Honestly I wonder, who the hell buys these overpriced networking appliances?

people that need to switch more packets than what you buy off of newegg can handle.

edit: oh that and when you have a network of thousands of routers and switches that you want to centrally manage in a way that "mass ssh" doesn't scale to and you just want to buy the management software from someone that has written it before and not spend a while getting the software wrong before you get it right.

if you can count the number of routers you need in your head, you probably should just be using beige boxes with a commodity OS...

People who want to have support by a large vendor. If your DIY-setup fails you can't blame a vendor. In theory using the same kind of appliance as everybody else means you can easily find an networking guy to work with it, people who are good at both traditional networking gear and BSD/Linux are harder to find. If a dedicated box fails you can just have someone with some experience plop a new one in, reapply the configuration and be done. Correctly replicating a DIY setup is harder, especially if the person building it didn't do a perfect job at documenting it.

Appliances promise all sort of fancy features that might be hard to exactly replicate yourself (if you actually need them often is a different question, but often networking isn't exactly in a position to say "no" to such requests). Although there is more and more a trend of manufacturers also offering the software for virtual environments, many appliances are x86-servers anyways.

+ as the article mentions all those cases where you need specialised hardware, e.g. routing/switching at high speeds, or working in special environments.

Specialist hardware, supported by specialist network engineers should be the smart choice. It invariably leads to a monoculture.

If you buy Cisco and a Cisco specialist, you'll be buying Cisco for as long as the specialist is employed, regardless of any leaps or bounds Juniper et al. make.

Same deal with databases. Oracle is often a smart choice, but so is Postgres, SQL Server, Cassandra, Redis. If I've hired a DBA with 10 years of Oracle experience, the chances are that any move away from Oracle will rub against "Why hire an oracle specialist if you're not going to use them?".

My hope is always "Because they're smart, genuinely understand storage and retrieval and will learn the other systems faster than a smart generalist". But that isn't how it plays out.

I would hope that hiring a specialist means that, at some point, they come to you and say "So, I think I can save us money and increase uptime if we move to X". It happens in software all the time. Its hard to keep devs on any one particular platform before they get twitchy (also a problem).

In my mind buying Cisco is like using Heroku, Rails, Ember. All fine tools, that help you get a job done quickly whilst you figure out what your real problems are but shouldn't be seen as the final solution.

I've heard the "blame a vendor" argument again and again in my life, but I have yet to see this "blame" ever occur. Granted, I'm dealing with small and medium-sized companies, but I've never seen any kind of tangible financial outcome from "blaming" a vendor (other than that vendor not getting future sales).

"nobody gets fired for buying IBM"

https://en.wikipedia.org/wiki/Fear,_uncertainty_and_doubt

Everyone who says anything like "blame the vendor", have just drank too much marketing cool-aid.

Not tangible financial outcome for the vendor. But for the employee suggesting it, who doesn't have to explain why "his" "cobbled together thing" doesn't work, but can point to all kinds of outside validation for the choice, even if it doesn't work out.

> Commodity hardware and general-purpose systems win every time.

What's latency like on commodity hardware vs dedicated equipment? I've no experience but would expect the dedicated hardware to be better. BSD is a monolithic kernel, so you'd have the overhead of passing through the for each of the routed packets, right?

The packets never go out of the kernel, latency seems fine on my BSD router.

What word should be after "through the" in your post ?

Sorry. "Kernel". Through the kernel. But you were there already - I hadn't twigged that there's no need to jump out of userland in the first place.

> through the [?] for each

BSD pf is interpreted by a kernel module in kernel mode. There is no e.g. kernel / user transition involved.

Many proprietary databases provide significant scalability and/or feature benefits over their open source counterparts. Whether the customer in question actually needs those capabilities is of course often questionable, but they do provide real benefits.

Pointy Haired Bosses who need something to deflect blame onto in case things screw up. It's easier to assign that to an outside vendor than one's own subordinates that they are directly responsible for.

I've seen this many times and it will continue to happen.

The same PHB who gets wined and dined by the sales critters from said vendor down at the country club...

Medium sized stable businesses. Small shops can't afford them. Large shops (Google, Facebook) hire people and use open source software.

At the moment, network switching and routing with open source products + commodity servers can't scale like Cisco and Juniper...if you want to have a fairly standard network (OSPF, BGP, etc).

Google, for example, is able to use commodity hardware because what they are routing works with a non-traditional setup...specifically a Clos network. That works for them only because their app is designed in a way that it can served up like that. It would not work for the typical kinds of traffic that you see at a normal company.

At a certain scale, barring unusual exceptions, you really do have to go with someone like Juniper/Cisco, as the ASIC based acceleration and other features aren't there in the commodity world.

There's certainly some things in progress that may change all of that, like Intel's DPDK, QuickAssist, etc.

The networking hardware that Facebook builds uses ASICs: https://code.facebook.com/posts/717010588413497/introducing-...

For core routing on fast networks, I tend to agree. Outside of that, commodity hardware and open source work just fine.

I would add certain use cases of firewalls to that as well.

Facebook builds their own networking hardware (which runs open source software):

https://code.facebook.com/posts/843620439027582/facebook-ope...

Remember that "nobody ever got fired for buying IBM".

I once worked for a guy who wouldn't buy anything that didn't say Cisco on it, even when the networking guys made a solid case for something different. I often wondered if it was more than a "nice dinner" that he was getting.

Reply on news.ycombinator.com