ashvardanian

Detailed release notes: https://github.com/pytorch/pytorch/releases/tag/v2.9.0

Will need some time to go through the details, but it’s increasingly rare to see teams consistently delivering meaningful improvements in the open. Impressive work!

Despite the promise of scalable vectors, I've struggled to find cases where it consistently beats NEON. Curious if others have found good use cases for SVE beyond AES-ing tiny string keys for hash-table lookups, scatter-gathers in small buffers, and histograms ;)

Those are extremely uniform latencies. Seems like on these CPUs most benefits from NUMA-aware thread-pools will be coming from reduced contention - mostly synchronizing small subsets of cores, rather than the actual memory affinity.

The second piece (uniform call syntax) looks convenient, though I don’t see a realistic way to integrate it into modern C++. The first (structural pattern matching) is, for me, more of a dividing line between low- and high-level languages. I tend to avoid it in my C++, just as I avoid inheritance, virtual functions, and exceptions… or `<functional>` header contents.

Still, it’s always fun to stumble on corners of the STL I’d never paid attention to, even if I won’t end up using them. Thought it was worth sharing :)

Wow, I didn’t realize someone had integrated it into their project before I even tried it in mine — thanks for the trust and for sharing!

I completely agree that tuning is needed for better CPU sleep scheduling. I’m hoping to look into it this October, ideally on some Alder Lake CPUs with mixed Performance/Efficiency cores and NUMA enabled.

Compared to Rayon or Taskflow, the biggest initial win is cutting out heap allocations for all the promise/result objects — those act like mutexes once the allocator gets hammered by many threads.

Hard to rank the rest without a proper breakdown. If I ever tried, I’d probably end up writing a paper — and I’d rather write code :)

I totally agree — most C/C++ developers with 10+ years of experience have built similar thread pools or allocators in their own codebases. I’d include myself and a few of my former colleagues on that list.

That said, closed-source solutions for local use aren’t quite the same as an open-source project with wider validation. With more third-party usage, reviews, and edge cases, you often discover issues you’d never hit in-house. Some of the most valuable improvements I’ve seen have come from external bug reports or occasional PRs from people using the code in very different environments.

That’s true — though in my benchmarks Tokio came out as one of the slower parallelism-enabling projects. The article still included a comparison:

  $ PARALLEL_REDUCTIONS_LENGTH=1536 cargo +nightly bench -- --output-format bencher
  
  test fork_union ... bench:  5,150 ns/iter (+/- 402)
  test rayon ... bench:      47,251 ns/iter (+/- 3,985)
  test smol ... bench:       54,931 ns/iter (+/- 10)
  test tokio ... bench:     240,707 ns/iter (+/- 921)

... but I now avoid comparing to Tokio since it doesn’t seem fair — fork-join style parallel processing isn’t really its primary use case.

Yes, I was planning a similar experiment with UCall (https://github.com/unum-cloud/ucall), leveraging the NUMA functionality introduced in v2 of Fork Union. I don’t currently have the right hardware to test it properly, but it would be very interesting to measure how pinning behaves on machines with multiple NUMA nodes, NICs, and a balanced PCIe topology.

As a rule of thumb, I find Zig projects often come across as higher quality than many C, C++, or Rust counterparts — it’s a surprisingly strong community. That said, I don’t write much in Zig myself, haven’t explored the projects you linked in detail, and wasn’t even considering Zig alternatives as I needed something for my C++ and Rust projects.

From a first glance, they seem to be tackling a slightly different problem: focusing on efficient handling of nested parallelism. Fork Union doesn’t support nested parallelism at all.

Yes, non-uniform workloads are supported! See `for_n_dynamic`.

The threads “busy-wait” by running an infinite loop in a lower energy state on modern CPUs.

And yes, there are more details in the actual implementation in the repository itself. This section, for example, describes the atomic variables needed to control all of the logic: https://github.com/ashvardanian/fork_union?tab=readme-ov-fil...

I was asked this a few months back but don’t have the measurements fresh anymore. In general, I think TBB is one of the more thorough and feature-rich parallelism libraries out there. That said, I just found a comparable usage example in my benchmarks, and it doesn’t look like TBB will have the same low-latency profile as Fork Union: https://github.com/ashvardanian/ParallelReductionsBenchmark/...

Implementing convenience APIs in Rust has been tricky, since a lot of the usual memory-safety semantics start to break once you push into fast concurrent code that shares buffers across threads. My early drafts were riddled with unsafe and barely functional, so I’d definitely welcome suggestions for quality-of-life improvements.

I likely posted this a few months ago, and it looks like it just got bumped by the platform. Since then the library has seen some improvements (especially if you run on a Linux NUMA box), and the GitHub README is probably the best place to start now: https://github.com/ashvardanian/fork_union

There are still a few blind spots I’m working on hardening, and if you have suggestions, I’d be glad to see them in issues or PRs :)

Ah, I see the article does mention "brute-force-like" — I must have skimmed past that. I'd be curious what exactly is meant by it in practice.

A small note: since the project seems to include @maratyszcza's fp16 library (MIT), it might be nice to add a line of attribution: https://github.com/maratyszcza/fp16

And if you ever feel like broadening the benchmarks, it could be interesting to compare with USearch. It has had an SQLite connector for a while and covers a wide set of hardware backends and similarity metrics, though it's not yet as straightforward to use on some OSes: https://unum-cloud.github.io/usearch/sqlite

My usual line of feedback would be to start with a more aggressive benchmark. Indexing 100K dense vector (100ish MB here) is not generally a good idea. Brute-force search at that scale is already trivial at 10 GB/s/core.

I’d strongly caution against many of those “performance tricks.” Spawning an asynchronous task on a separate thread, often with a heap-allocated handle, solely to deallocate a local object is a dubious pattern — especially given how typical allocators behave under the hood.

I frequently encounter use-cases akin to the “Sharded Vec Writer” idea, and I agree it can be valuable. But if performance is a genuine requirement, the implementation needs to be very different. I once attempted to build a general-purpose trait for performing parallel in-place updates of a Vec<T>, and found it extremely difficult to express cleanly in Rust without degenerating into unsafe or brittle abstractions.

Seems to be the first Arm CPU to hit 5 GHz. I couldn’t find the ISA details, and curious if they will support SME, like the M-series Apple chips?

Seems like it already does for some, assuming Pillow and FFMpeg are on the list.

Qwen models have historically been pretty good, but there seems to be no architectural novelty here, if I’m not missing it. Seems like another vision encoder, with a projection, and a large autoregressive model. Have there been any better ideas in the VLM space recently? I’ve been away for a couple of years :(

Thanks for the kind words! In this case, it isn’t tied to any programming language or locale-specific formatting. I just find commas less readable in long numbers, especially in running text across Western languages. Apostrophes feel clearer to me, so I usually stick with them.

Thanks! You are likely right! It took a lot of time to make sure that all 6 of ISA-specific versions of StringZilla (https://github.com/ashvardanian/StringZilla/blob/main/includ...) return the same output for both one-shot and incremental construction, and I’m not sure if it was a priority for other projects :)

Yes, at the scale of 128-bit registers NEON is mostly enough, except for a few categories of instructions missing in that ISA subset, like scatter/gather ops, that can yield 30% boost over serial memory accesses: https://github.com/ashvardanian/less_slow.cpp/releases/tag/v...

The consumer situation is changing. A few years ago, when I was working with a team on some closed source HPC stuff, we’ve got everyone Tiger Lake-based laptops to simplify AVX-512 R&D. Now, Zen4-based desktop CPUs also support it.

But its fair to say that I’m mostly focusing on the datacenter/supercomputing hardware, both on the x86 and Arm side.

Not that I’m aware of. Some commercial DBMS vendors are experimenting with integrations, but I haven’t really seen much in the Postgres ecosystem.

What excites me in this release is the quality of the new hash functions. I’ve built many over the years but never felt they were worth sharing until now. Having two included here was a personal milestone for me, since I’ve always admired how good xxHash and aHash are and wanted to build something of similar caliber.

The new hashes should be directly useful in databases, for example improving JOIN performance. And the fingerprinting interfaces based on 52-bit modulo math with double-precision FMA units open up another path. They aren’t easy to use and won’t apply everywhere, but on petabyte-scale retrieval tasks they can make a real impact.

This last wave of work was actually triggered by the industry over the last 2 years, as the volume of biological sequence data is growing rapidly and more BioTech and Pharma companies are rushing to scale computational pipelines.

Namely, if you look at DeepMind’s AlphaFold 1 and 2, bulk volume of compute time is spent outside of PyTorch - running sequence alignment. Historically, with BLAST. More recently, in other labs, with some of my code :)

Makes sense! I mostly focus on newer AVX-512 variants as opposed to older AVX2-only CPUs. As for aarch64, it is supported with both NEON, SVE, and SVE2 kernels for some tasks. The last two are rarely useful, unless you run on AWS Graviton 3 (previous gen) or some of the supercomputers with custom chips like Fujitsu A64FX.

It’s a very nice and detailed benchmark suite! Great effort! Can you please share the CPU model you are running on? I suspect it’s an x86 CPU without AVX-512 support.

I’m not an active RipGrep user, so can’t speak for all usage patterns. My guess: for plain substring searches, you probably won’t see much difference. Where StringZilla may potentially help is in character-set searches.