Comment by userbinator | Hacker News Reader

Comment by userbinator | Hacker News Reader

fleet-wide profiling revealed that 25-35% of all CPU time was spent just moving bytes around[...]If data movement were faster, more work could be done on the same processors.

Here's a different thought experiment: how much of that data movement is actually necessary? If data movement was taking up so much time, why not investigate why and if that's even necessary, instead of thinking it's the hardware that needs to be faster? As the old saying goes, "the fastest way to do something is to not do it at all." One style of code that seems to be particularly exacerbated by the widespread use of HLLs is making lots of unnecessary temporary copies of data. The biggest inefficiency isn't the hardware (unless your goal is to sell more hardware, but that's a different rant...) --- it's the insanely complex and bloated layers of software that gets layered on top of what is otherwise perfectly adequate hardware. From that perspective, faster hardware only leads to even more wasteful and inefficient software. Making memory copies faster will only cause software to make more unnecessary copies.

But after reading the whole thing, I can say that this is certainly a very strange article. It's written by someone who is presumably quite knowledgeable about hardware, yet seems completely oblivious to the fact that x86 has had a dedicated instruction for this since the 8086, and 16 bytes per cycle was already achievable with the P6 (1996) "fast strings" enhancement:

https://stackoverflow.com/questions/33902068/what-setup-does...

While you are right, we are where we are and there is tremendous leverage in making an extremely common (even if undesirable) thing more efficient. It kind of doesn't matter why millions of lines of code perform memcpys - they do, and we're unlikely to change majority of them anytime soon.

P.S. I work on the type of systems where we obsess over unneeded memory accesses and copies. Minimizing those is not as trivial as it may seem, especially if you care about memory safety.

> While you are right, we are where we are and there is tremendous leverage in making an extremely common (even if undesirable) thing more efficient.

Aren't you worried that this will just lead to another manifestation of the Jevons Paradox? I get that you already mention that everyone in your "field" understands that memory is the bottleneck to avoid, but still.

[0] https://en.wikipedia.org/wiki/Jevons_paradox

Memory gets moved for a lot of reasons. Any type of FFI usually requires memory copies. For instance, if you pass a C string from Swift to C, the C side needs to copy the memory to use it after the immediate call. Or if C returns a string to Swift, Swift needs to copy the string into its own format.

While the overall point is valid, this isn’t strictly true. A Swift string is frequently backed by a contiguous UTC8 buffer terminated with a \0, so passing to C is just a matter of passing the pointer and requires no copies.

But Swift has no way of signaling to C that the memory can be freed so I’m pretty sure it does a copy when using String’s cString constructor.

When you pass a Swift String to C you do so with a given lifetime. By default this is the duration of the call.

There is also the I/O, most interaction with disks or NICs are copying twice. We know efficient zero-copy I/O is achievable, it's just that it requires a lot of work for kernel and app developers with a very limited ROI.

This is my standpoint too.

Assuming Google uses gRPC and protobufs extensively, I can only imagine a chaotic polyglot environment where buffers are copied and transformed very often.

I work at a fairly large Python shop using libraries in C. Cost of FFI is non-trivial, memory management and moving data is the usual culprit.

Always two of them there are. Time and space. Spatial separation has the great benefit that it frees you from separation in time, which is technically and mentally more challenging. If you want or need that anything can happen at any time on either side, you end up copying (and ideally closing the door after). Very much like Erlang does with the Beam.

Cynical answer: because that's not sexy.

I have some coworkers trying to re-architect a big chunk of our application to selectively provide data because "only about 10% is used on each interaction". Nobody has bothered to check what % of the data is never used at all. If it's 50%, then you're complicating a whole lot of code and reuse semantics in order to get a 2-4x improvement in 'performance' that you're selling as a 10x improvement.

But to your bigger question, defensive copies are about keeping someone else from munging your data. Borrow Semantics prevent some of those scenarios, which at least moves some of that inefficiency to compile time, where the tax is paid by the perpetrator and their peers, instead of by the users.

A lot of the additional data movement is a.) defensive copies because too many data structures are mutable and b.) change of data format to internal software representations, e.g. wire->decode->IR1->...->IR1->encode->wire.

> change of data format to internal software representations

There is so much of this. Is there any hope for a world where we can ship working memory without costly XDR? Does anyone have data on the rate of change of time or energy spent on XDR translation?

I have thought a lot about this over the years, and I think one of the issues is that it is hard to specify fixed data formats in most programming languages (in a declarative way), leading to lots of code that decodes byte-by-byte or word-by-word. Since writing all that sucks, many programmers have turned to higher-level data formats such as JSON or protobufs that are decoded by libraries.

I have been working on a new feature in Virgil to make writing explicit byte-by-byte layouts easier and directly supported by the compiler. It's still a work in progress though.

If the heap were treated like a typed database, with transaction semantics, many of our problems with disappear. Layout should not be an implementation detail. I know Wasm won't solve this explicitly, but at least have a chance while everything is being redefined.

Wouldn’t that move the data transformation costs from at I/O time to all the time?

Not if the encoding can simply be consumed in place, needing only a validation pass if it's coming from an untrusted source. Cap'n Proto is pretty close to that for one example.

> a world where we can ship working memory

Yes, you can have zero serialization latency: https://capnproto.org/

It's a lot of work to reason about programs in order to reduce copies. A different, cheaper and more feasible approach is to reasoning about alignments and sizes.

I've seen some long running work on the rust compiler to use high level guarantees (lifetimes, ownership...) to improve codegen and one of the big tasks seems to be 'remove unnecessary memory copies'. Fun to watch, I hope there'll be a cool write up.

I'd imagine a lot of that was just same simple operation on a steam of data; there isn't that much to optimize there aside from finding a way to operate on more data at a time

Reply on news.ycombinator.com