FreeBSD 10.0 on Ubiquiti EdgeRouter Lite
rtfm.net
rtfm.net
Qualcomm-Atheros AR9331 based systems, with 64MB ram and 16MB flash (kind of hard to get in that config cheaply). You can get decent performance, potentially hang a 5.8/2.4ghz radio on it too, and power it off a single laptop USB port (200mA normal power use, and with stuff, still less than 500mA.) Can put second ethernet, wifi, 3G, etc. there. There's no current commercial hardware in this space which is perfect. That should change shortly.
Much beefier systems with "real" amounts of RAM and flash. I generally skip the Broadcom/RPi/etc. and go straight to an Atom or i3, but that's just for convenience, since I only need a small number of them. There's probably something more reasonable to do in the embedded space, but I'm indifferent on qty 1 devices between $500 and $200, especially if I already have most of the components for the $500 thing lying around, and it saves me an hour or two getting it working.
There are a lot of things missing from widely available commercial hardware in this space right now.
The Carambola 2 board is pretty cheap and it's quite awesome.
Missing a couple of things I want, though.
I've bricked (well, melted) 2 of them so far doing it myself.
Tell us more :)
It would be a bit higher cost, but could mean you could make a router like the EdgeRouter but with good performance without the proprietary accelerator (I believe that Intel's drivers are open source).
1. Data Plane Development Kit, which lets you skip the kernel IP stack (which takes thousands of CPU cycles to process) and do packet processing in userland taking just tens to hundreds of cycles per packet. http://dpdk.org/
DPDK supports Atom. This Atom has both virtualization and AES-NI, so you could do some interesting security things (the fireeye-style "run malware in a VM", and linerate crypto).
~$120 in parts means a retail price around $500. That doesn't seem at all unreasonable to me, but there's always the "sell it for $250 and make $5-10/mo MRC" option. I think Cisco/Linksys tried that but it was pretty weak.
I wish OS developers saw this as a problem. There is no reason kernel stacks should be so slow for tasks where all processing is done in the kernel. (For packets destined to userspace, you've got the syscall overhead to deal with.)
I recently tested the Linux network stack's PPS performance with an Intel X520 10GbE NIC. I used Debian testing, with the 3.12 kernel. My destination machine was an i7-3930K at stock speed. I wrote a simple kernel module adding an NF_IP_PRE_ROUTING hook returning NF_DROP with no processing, which would be the simplest possible code path. For a packet generator, I used another older machine with another X520, using the "pfsend" tool included with PF_RING, and the card in PF_RING DNA mode. That was easily able to saturate the link at line rate (14.8M PPS).
The result: the kernel was only able to sustain about 2.8M PPS.
I then loaded the DNA driver on the destination machine, used the included "pfcount" tool, and no packet drops - it was receiving the full 14M PPS.
I tested DPDK recently and had similar results.
I also modified the Linux ixgbe driver ixgbe_clean_rx_irq() function, and added a step in between the "fetch packets from RX ring" and "put packet in SKB and send to network stack" functions. Even when I added a bunch of useless comparisons for each packet, I was able to get ~12-13M PPS. I could get line rate by just dropping and not doing any processing.
It's possible the Solarflare NICs could be better at some metric like latency, by a matter of microseconds (only speculating here), but I can't see how they'd consider that worth the extra money for HTTP servers.
I must say: the ERL is terrible hardware.
You can get some performance out of it if you're using Ubiquiti's kernel, because it ships with a series of highly specialized proprietary modules that offload packet handling to hardware accelerated modules, for their custom proprietary network stack. But their kernel is old and shitty, and if you want to do anything interesting with the ERL, you have to build your own kernel.
So for any mildly interesting usage, it fails considerably hard.
And even if you do opt to use their closed-source binary blob kernel modules and software stack, as soon as you try to do any sort of advanced networking, you run up against the limits of their acceleration modules, and the OS is forced to fall back to standard non-accelerated mode, and performance drops. Try using PPP, any interesting iptables, MPTCP, OpenVPN, interesting routing tables, or anything cool: performance drops.
I would be very happy to see some small networking hardware with multiple independent Ethernet controllers pop up on the market for a decent price; every other month or so my disappointment with the ERL moves me to re-research this. Unfortunately, the EdgeRouter Lite is not the hardware anybody wants for interesting low-cost networking.
I'm hoping to really work on improving the documentation in the new Vyatta fork, VyOS, because poor documentation really let Vyatta down.
I want a secure (but end-user-openable) boot loader, good crypto performance (to do ~linerate AES, and some PKC operations, for things like opportunistic ipsec eventually), and flexible interfaces (ideally, N x Intel GigE and 2 x dual-band wifi radio chipsets).
Real RED/BLACK would be a huge feature, too, and potentially hardware mode switch, or some other technology to work with the trusted boot loader in a way that malware can't hurt. Watchdog. PoEable for the higher end, and in the low end products, absolutely 5v <2.1A (i.e. Apple iPad charger), and ideally <900mA (USB3) or <500mA (USB2).
I'm starting to get invloved in the fork that was recently taken, called VyOS [1]. Updating the base to a newer version of Debian is a high priority, but there's a fair bit of work in it.
Since Vyatta Core (the community version) is essentially dead, and there is at least one Ubiquiti guy involved in the fork, we hope that VyOS will eventually be the base for EdgeOS.
And as for the hardware acceleration - I wish the drivers were open source but even as it is, it's really a necessary evil to get very good performance on low cost hardware.
Presumably this will actually be released at some point soon.
Also the soekris hardware looks nice, but it's pretty expensive.
Unfortunately by the time it's released, the hardware will already be a bit old. But nonetheless, the specs are certainly sufficient for most cool things.
http://www.hwtools.net/Adapter/DB8605.html
This way you retain radio, but get two additional PCIE slots for SATA controller and another network adapter for example.
The fbsd image yields ~250mbps between two gigabit hosts.
I've been eyeing one of these for a while and putting fbsd on it interests me, but not if it means a huge peformance drop (even if I wouldn't really miss it).
The freebsd-mips mailing list already has a few threads discussing this; if you want to know why the router isn't achieving line-rate speeds "out of the box", you'll probably get the best answer there (and maybe even someone who knows how to fix the problem):
* http://lists.freebsd.org/pipermail/freebsd-mips/2014-January...
* http://lists.freebsd.org/pipermail/freebsd-mips/2014-Februar...
If you don't get a suitable answer there, you can also try freebsd-net:
I am running gentoo on mine at the minute, I did have FreeBSD on it for a bit.
I tried a larger flash drive but the driver doesn't like it and won't mount the root partition, need to find one it likes. Or use nfs/smb but that is a pain.
screen /dev/ttyUSB0 115200 # Linux-centric exampleI'm thinking about swapping it out for a Mirabox[1] I have laying around. Its an ARMv7 with dual Gigabit interfaces and 1GB of memory. With the 3.14 kernel the SoC is quite well supported. Now if only I could run pfSense on it.
[1] https://www.globalscaletechnologies.com/p-58-mirabox-develop...