Rust is great but isn’t the core problem here using the wrong algorithm? It looks like this is ideally suited for a quad tree instead of a naive for loop. I would expect that to pulverise any current benchmark.
(e.g. it doesn't need to be "as fast as possible", just fast enough to no longer be a workflow bottleneck)