Hacker News Reader

An ASCII SIMD wc would likely be much more than 4× faster. The 320 MB/s is speed TFA quotes is good for a byte-at-a-time solution, but pitiful as far as the capabilities of modern machines are concerned. A decent SSD is an order of magnitude faster than that. Around 1 GB/s (i.e. ~100 cycles/fetch) is table stakes for a task of this complexity.

(TFA almost nerd-sniped me into writing a SIMD implementation already, and you finished the job. End result: on an aging 2.5 GHz Broadwell, I’m seeing 300 MB/s for TFA’s C implementation and 3200 MB/s for my first attempt at an x86-64-v3 one. So more like a 10× speedup, and its highly likely one can do better. Admittedly I did spend around an hour on these 50 lines, and would need to spend more in reality to check for CPU features at runtime and so on.)

However, TFA calls going ASCII-only “cheating”, and I can see its point. I don’t know how difficult a Unicode SIMD wc would be or if it’s even possible to do well. (Bonus points: do you want to define a word boundary as a space/nonspace boundary the way[1] POSIX tells you to, or the more complex way[2] Unicode tells you to?)

[1] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/w...

[2] https://www.unicode.org/reports/tr29/#Word_Boundaries

View on HN

#if 0 /* shell polyglot */ exec ${CC:-cc} $CPPFLAGS -g -O2 -march=x86-64-v3 $CFLAGS -o "${0%.c}" "$0" || exit $? #endif /* SPDX-License-Identifier: CC0-1.0 */ #include <immintrin.h> #include <stdint.h> #include <stdio.h> #include <unistd.h> int main(int argc, char **argv) { const char *const progname = argc ? argv[0] : "<unknown>"; unsigned long long bytes = 0, words = 0, lines = 0; uint32_t prev = -1; for (;;) { static __m256i buf[1024 * 1024]; ssize_t k, n = 0; do { k = read(STDIN_FILENO, (char *)buf + n, sizeof buf - n); } while (k > 0 && (n += k) < sizeof buf); if (k < 0) { perror(progname); return 1; } if (!n) break; bytes += n; while (n % sizeof *buf) ((char *)buf)[n++] = ' '; n /= sizeof *buf; for (size_t i = 0; i < n; i++) { __m256i x = _mm256_load_si256(&buf[i]); __m256i nl = _mm256_cmpeq_epi8(x, _mm256_set1_epi8('\n')); lines += _mm_popcnt_u32(_mm256_movemask_epi8(nl)); const __m256i table = _mm256_broadcastsi128_si256(_mm_setr_epi8( /* 0 */ ' ', 0, 0, 0, 0, 0, 0, 0, /* 8 */ 0, '\t', '\n', '\v', '\f', '\r', 0, 0)); __m256i ws = _mm256_cmpeq_epi8(x, _mm256_shuffle_epi8(table, _mm256_and_si256(x, _mm256_set1_epi8(0x0F)))); uint64_t mask = _mm256_movemask_epi8(ws); words += _mm_popcnt_u32(mask ^ (mask << 32 | prev) >> 31); prev = mask; } } words += !(prev >> 31); printf("%llu %llu %llu\n", lines, words / 2, bytes); return 0; }