Comment by luckydude | Hacker News Reader

Comment by luckydude | Hacker News Reader

Hi, I'm the guy who came up with the splice idea. It's based on what I learned doing this:

www.connectathon.org/talks96/bds.pdf

which was for the EIS (Earth Imaging System) project, a government effort to image the earth about 15 years ago. That project eventually had 200Mhz MIPS SMP boxes moving data through NFS at close to 1Gbyte/sec 24x7. So far as I know, nobody else has ever come close to that even with 10x faster CPUs.

Most of the people in this thread pretty clearly don't understand the issues involved, Rob included (sorry, Rob, go talk to Greg). Moving lots and lots of data very quickly precludes looking at each byte by the CPU. The only thing that should look at each byte is a DMA engine.

Sendfile(2) is a hack, that's true. It is a subset of what I imagined splice(2) could be (actually splice(3), the syscalls are pull(2) and push(2)). But it's a necessary hack.

Jens' splice() implementation was a start but wasn't really what I imagined for splice(), to really go there you need to rethink how the OS thinks about moving data. Unless the buyin is pervasive splice() is sort of a wart.

I'm assuming that the real objection to an interface like sendfile() is that you shouldn't need to copy_from_user() just to construct an SKB if the user address being referred to is pointing at a file backed VMA. If we could do a zero-copy send() from userspace, there would be no need for sendfile() in the first place.

IOW, { data = mmap(fd, ...); ...; send(fd, data + offset, length); } could be just as fast as sendfile() if we were smarter.

We're starting to build infrastructure to do zero-copy networking in userspace specifically for KVM. So far, it's not at the socket level but instead at the macvtap level but that's arguably another problem with the networking stack--macvtap/tap should just be an address family :-)

I'm not so sure. The problem is that when send() returns, the application is free to modify the buffer and expect that the modifications won't be visible to the other side of the link - the data has already been "sent" as far as the app is concerned.

So this implies that if you wished to send by DMAing directly from the page cache to the NIC, send() would have to block - not just until the data (and all previous buffered written data on the connection) has been passed to the NIC, but until the TCP ACK from the other side has been recieved (or RST). (Non-blocking sockets could obviously never use this method - they would have to perform the copy).

sendfile() obviously has this limitation already, but if you were to do it for send() you'd at least have to hide it behind a setsockopt(). Otherwise the weird behaviour is likely to upset existing applications (eg. select() says that write() will not block - but it does anyway, because the buffer you sent from was mmap()ed?)

No, TCP ACK has nothing to do with this. All that matters is when the NIC has completed DMA'ing the packet. At that point, userspace can be unblocked.

Non-blocking sockets only make sense when there's an intermediate buffer.

What happens if you need to retransmit a packet?

Exactly right. SOCK_STREAM sockets can't unblock the process until it knows that it won't need that buffer again for a retransmission.

> So far as I know, nobody else has ever come close to that even with 10x faster CPUs.

With 10-gigabit becoming commonplace in high-end server rooms, dual 10-gigabit cards hitting the market, and 100-gigabit on its way, I've got to ask, what you mean by no one has come close?

(Certainly, 1 GByte/sec is achievable; splice is a large part of that.)

The big print giveth, the fine print taketh away :)

Just because you have a 10 gigabit pipe (which is 1.2Gbyte/sec) that doesn't mean you can fill it. And filling it with a benchmark is different than filling it with NFS traffic.

So far as I know, nobody has come close to what SGI could do years ago, i.e, stream 100's of MB/sec of data off the disk and out the network and keep doing it.

I've tried with Linux to build a disk array that would do gigabit, not 10-gigabit, and couldn't come anywhere close. I think about 60MB/sec was where things started to fail.

I'd love to hear about a pair of boxes that could fill multiple gigabit pipes with the following benchmark:

$ tar cf - big_data | rsh server "tar xf -"

You can set any block size you want in tar, I don't care, the files can be reasonably large, etc.

If you manage to fill one pipe, then try it with 10-gigabit and let's see what that does.

I'd love to be wrong but so far as I can tell, we're nowhere near what SGI could do.

Serious question, why tar it? I bet you could get pretty sweet throughput with an array of SSDs and 10gig pipes.

In my case, it's because it's a pretty good emulator of what we do.

We're the oh-so-loved BitKeeper guys and we're working on tech for very large repositories. A bk clone is essentially the described benchmark. Or should be, if we're slower than that we're doing something wrong.

:)

Benchmarks, especially those from card manufactures are suspect. Just because you've got a network ping-pong test running at essentially line speed, doesn't mean I'll see any of that. I fully agree.

Filling any pipe with your 'rsh' benchmark is harder - on all the Linuxes I have on hand, rsh is aliased to ssh, and I've never seen a system deal with the overhead of ssh adequately. (There's high perf ssh if you really want; I've never tried it. http://www.psc.edu/networking/projects/hpn-ssh/) (Would you accept numbers over netcat?)

If I'm not getting 90MBytes/sec over Gigabit, then something, somewhere, is slowing it down.

We sell servers that sustain 300+ MBytes/s per client, via Samba for two clients. Those clients are very fussy about latency (video editing/DI). That's what we sell, so it's empirically that fast; ie not a benchmark. The server itself will actually do 900+ MBytes a second (to three clients, the same as page 14 on your linked pdf), but my small print is the market doesn't want that configuration.

These numbers are what I know, which doesn't include NFS. NFS-RDMA and pNFS are supposedly good for performance, but I don't have any first hand knowledge.

None of what I just stated contradicts you. But if I'm at 900 MBytes/sec with my level of hardware, I find it hard to believe someone with more budget isn't capable of a full Gigabyte/sec over Infiniband, arguably HIPPI's successor.

I, too, lament the demise of SGI.

Who is "we"? I might be another customer :)

I meant rsh, not ssh, wasn't trying to stick extra processing in there. Debian, bless 'em, still support rsh, I think it's rsh-clients and rsh-server.

For your 300Mbytes/sec, what's the network? And is the data in memory or on disk? If on disk, what sort of drives and what size files?

Thanks!

It's a product for a niche B2B market, but if you're buying, email's in my profile ;)

The network is 10-Gigabit ethernet, from server to a switch, then switch to client, all via CX-4 or SFP+. The data is actually coming off disk at that rate (in ram would be cheating :p ). It's backed by a 16-drive hardware raid using 2TB spinning disks (SATA) in a raid5. The performance we sell is actually 2 streams of read while simultaneously writing a stream to disk, so 600 out, 300 in.

The files are actually the hard part. This supports the dpx 'video' format. The format originated from film scanners, and so each frame is (basically) a 12.5 MByte jpeg. Multiply that by 24-30 fps, add in overhead for reading that many files/sec, and you get 300+ MBytes/sec. (This is also why I'm so sure there's more performance to be had if you don't have the same overhead.)

Reply on news.ycombinator.com