Avoid character-by-character processing when performance matters
lemire.me
lemire.me
Most of the time if you see people doing this it's using vector registers and newer instruction sets, because that's where you get 2-10x speedups by doing it. But even on ARM and x86 without using vector registers, careful pipelining and vectorization can deliver pretty sizable speedups. It increases cache locality, cuts down on branches, and reduces register pressure.
I ended up working on a project a while ago where we had to port a game's software rasterizer to multiple targets, including a low-spec ARM device. Instead of trying to hand-code a NEON implementation (I don't remember for sure if it even had NEON...) I started by just vectorizing it to operate on groups of 4 pixels instead of one (because I could fit 4 pixels in a single regular-sized integer register) so that each iteration of the main loop was doing more work and spending less time waiting on memory. Just that alone ended up improving performance considerably at the kind of clock speeds I was dealing with, and on higher-spec x86 cores it was still a slight improvement for about a day of hacking and staring at disassembly.
Plugged it into compiler explorer and clang 10 does unroll the std::all_of version, doing four chars at a time.
The best of both worlds would be having the compiler transform the canonical std:: version into the fast one under the covers, but that's far from universal with current compiler tech and languages.
If anyone is interested, here is simpler version:
bool is_ascii_branchy(const std::string_view v) {
return std::all_of(v.begin(), v.end(), [](auto i) {
return uint8_t(i) < 128;});
} bool is_ascii_branchy(std::string_view v) {
for (auto c : v) if (uint8_t(c) < 128) return false;
return true;
}Rewrite it to always consume the full string, and it goes 8 bytes at a time: https://godbolt.org/z/xG7avr
In practice the best implementations will manually align and then unroll.
Wow!
If I use bool and keep the body branchless (z |= (string[i] < 0);), the vector instructions disappear and it goes back to doing only 4 bytes at a time. That is very odd.
EDIT: Oh I see now, it's an unrolled loop each iteration of which uses 8 256bit vector instructions for a total of 256 bytes.
If `size` was enormous and the first character "was negative"¹, then one extra byte of lookahead would never SIGSEGV; `size` is enormous, and by definition, that is presumably more than our lookahead.
If you mean, "the first character of the lookahead's block" then it still doesn't make sense. Even in the case where we exit later, we still can't access later bytes of the lookahead if we don't know them to be out of bounds.
In the example Godbolt you linked to, the compiler is doing 8 byte jumps with the index. (It's not doing a great job, as we're still reading a byte at a time. But the bounds checks and the index increment will get reduced.) But, what the compiler is doing in your example could also be done in the early exit example. It'd be basically the same code, but replace .LBB0_10 with something like,
cmpb $0, (%rdi,%rdx)
js do_exit
cmpb $0, 1(%rdi,%rdx)
js do_exit
# etc. for the remaining 6 reads
.do_exit
xorl %eax, %eax
retq
The cmp/js's are very likely not as efficient as the mov/or's, but that's not the point; the point is the unroll is possible.
It is not the early exit that prevents a loop unroll. Something else is stopping it, either it does not think it worth it, is not sufficiently intelligent, or some arcana of C prevents it.
(Your example alternates between mov and or, in a very interesting pattern, which makes me suspect it also knows something there, and perhaps more opportunity.)
¹This is an ugly example, since the implementation is free to have a char with range 0-255.
But `size` is just arbitrary number, while the code as written will not proceed past the first negative char. For example some jerk may write:
c_is_ascii_branchy("\xFF", a_gazillion);
> It is not the early exit that prevents a loop unroll
It really is!
I think that's the dispute: does an early exit prohibit the compiler from generating code with additional reads? Or is the actual prohibition about causing segfaults? Intuitively, I'd think that so long as it produces the correct answer and avoids visible side effects (like segfaults), any number of additional reads from anywhere would be legal, but I don't know the spec well enough to claim this is actually correct.
However, a compiler could align the memory accesses to avoid this problem on a particular platform, say based on its knowledge of the 4k protection granularity on x86. For example, if you 32-byte align your reads, and the source alogorithm reads the first byte in a 32-byte chunk, it's safe to read the remaining 31.
Actual implementations of things like strlen do exactly this, but these are generally hand-written: I haven't seen compilers do it during autovec. On problem is that it produce false-positives in tools like Valgrind.
Bonus fact: icc actually breaks the rules in similar scenarios and vectorizes (without any alignment) loops that gcc and clang don't, and may produce writes where none existed in the source, causing segfaults or concurrency bugs in correct programs. So any time you see icc successfully vectorize a loop that other compilers don't, you have to wonder whether it was smarter, or simply breaking the rules.
Yes, the question is about which "additional tricks" are legal according to the C standard. That is, what are the limits of what is technically allowed if you are wearing nothing but your language lawyer hat? Is the rule that anything goes as long as you don't segfault, or something more stringent? For that matter, is a segfault actually evidence that you have broken the rules? Note that this is different the rules for "undefined" behavior, as that pertains to the C source code rather than the generated assembly.
My impression is that the C standard assumes a hypothetical virtual machine that does not specify details like page sizes. I think this means that according to the standard, over-reads must be legal or illegal for a conforming compiler regardless of whether they cause a segfault on a particular real-world architecture. Or maybe the C standard doesn't make any claims as to whether the generated code will actually run on any particularly architecture?
> or simply breaking the rules
Which rule? My conclusion right now is that maybe there isn't one, and that the standard doesn't actually guarantee that a conforming compiler will generate code that runs to completion without segfault in the real world. There are simply too many differences in actual CPU's to make any firm guarantees. It's obviously fair to consider a compiler "buggy" if it does this, but I'm not sure it's noncompliant.
I probably should have been clearer, but it is my position the suggested trick (alignment) is definitely allowed by the standard. As you point out, the standard is working at a much higher level and is never going to say anything about memory protection, page size, etc. The program just has to produce the side effects required by the standard (output, etc), and whatever it wants to do to achieve that is fine. Even with my language lawyer hat firmly in place, I can't see any possible reason it would be disallowed: the standard doesn't even have the tools or language in place to even start talking about it.
> I think this means that according to the standard, over-reads must be legal or illegal for a conforming compiler regardless of whether they cause a segfault on a particular real-world architecture. Or maybe the C standard doesn't make any claims as to whether the generated code will actually run on any particularly architecture?
The standard makes no guarantees about any of it. It basically just says "these side effects must occur". How that is implemented is totally up to the compiler and runtime environment.
Note though that what I'm suggesting will "always work" in the sense that the a compiler targeting x86 would produce some x86-specific code that uses alignment and knowledge of the page boundary to avoid faults. This won't fail on some other architecture, because the code wouldn't be compiled in that way on a different architecture. It's not much different than the compiler using x86 specific instructions when compiling for x86: obviously it must do this and this can't work on some other uarch.
Added: I should be clear that this is a valid transformation for the compiler to make, but it is not conforming for a programmer to write a program that does this at a source level!
That is, the compiler can take a conforming program that doesn't make any over-reads at the source level, and compile it to assembly that makes "known safe" over-reads at the assembly level.
However, if you wrote the equivalent of the transformed algorithm yourself, in the source, it would not be a conforming program, because out of bound reads are definitely disallowed by the standard, regardless of what you "know" about the standard. This sort of asymmetry between what the compiler can do and what you can validly write in the source language is pervasive.
> Which rule?
I was a bit loose in my language. I mean the most important rule: that a conforming program must produce the side effects that would be produced by the abstract machine. Informally, that it "works at all rather than simply crashing".
You can write a simple, conforming, program in icc that produces "Segmentation fault" rather than "Hello world" or whatever, which is what other compilers produce and what the abstract machine will produce. This is breaking the rules (rule #1, basically).
Yes, and my example of an unrolled version of the early exit loop does stop at the first character, despite being unrolled.
(That is an interesting example, though, and I think is a good proof that generating extra reads is not permitted.)
Pages aren't 1 byte each, they're 4KiB. You can't just segfault at by crossing 1 byte at any arbitrary address. Some alignment checks to handle edge cases should let the compiler at least read things 8-byte aligned in the steady state and only fall back on edge cases.
Emitting OoB reads dependent on optimization level? Think about ASAN, debugger watchpoints...can't be worth it!
clang has a cool feature that highlights missed vectorization opportunities:
> clang++ -O3 -c test.cpp -Rpass-analysis=loop-vectorize
test.cpp:5:4: remark: loop not vectorized: loop control flow is not understood by vectorizer [-Rpass-analysis=loop-vectorize]
Do you really know the target system?
Most of the time, you want the compiled code to work not only on the current system, but also in future systems of the same lineage. What prevents a future version of the architecture from for instance lowering the translation granule from 4096 bytes to 512 bytes, or even lower?
You can minimize the risk of this by requiring that input addresses be aligned (and many ABIs/APIs do require this) but it's realistic to say that an extra byte of lookahead can and will SIGSEGV in unfortunate cases.
⇒ I think you should only start complaining if the compiler had pragmas to indicate that returning ‘false’ is extremely rare and that size can run in the kilobytes, and the program used them.
20 20 20 20 ff 20 20 20
^^
But, even with that, you can still unroll the loop to:
bounds check the next 8 bytes (is idx + 8 < size)
read 1 byte, abort if not ascii
read 1 byte, abort if not ascii
etc.
mov rsi, ptr
mov rcx, size
repz andb 80h ; while(rcx-- && !(al = *rsi++ & 0x80)) ;
setz al RESP uses prefixed lengths to transfer bulk data, so there is never a need to scan the payload for special characters like it happens for instance with JSON, nor to quote the payload that needs to be sent to the server.
The Bulk and Multi Bulk lengths can be processed with code that performs a single operation per character while at the same time scanning for the CR character, like the following C code:Sounds like it only does the character-by-character processing for a dozen or so characters (to read the the command, control chars, and the length). And I think the character-by-character processing is also converting the string "123" to an integer, so the trick of using &&/|| might not work.
ebuild: https://github.com/jakeogh/jakeogh/blob/master/dev-libs/raws...
return std::all_of(sv.begin(),sv.end(),[](uint8_t c){return c < 128;});
Second, even the slow case is 2 GB/s. While the principle is helpful, the extra complexity may not be worth it in the vast majority of real life code.
This is a standard technique, that is often used in String functions. You can see it also in implementations of strlen.
This is a nice gimmick to show off in an interview because the interviewer is definitely not going to get it, however it might backfire because the interviewer might realize a) you'll be out to get his job b) you might actually want to put stuff like this into production.
The idea is solid, don't get me wrong, and clever code is always clever code. Also like u/kevingadd mentions, the principle behind the thought is an important one, knowing when and how to handle data in bulk since you are looking for a needle in the haystack. For instance SQL and streams in Java are places where you need this kind of thinking.
But please, for the love of Gord, never put this bit of code in production.
There's nothing hacky about this — it's a very simple procedure.
> the interviewer is definitely not going to get it,
I sure hope any interviewer for a software engineering position would get it. But this has nothing to do with interviews.
In the real world, this kind of approach is critical to making effective use of compute resources. This is the kind of stuff you want in production.