Probably doesn't matter much with only a few networks, but this is using the wrong data structure:

    if let Some(net) = ip_nets.load()
        .iter()
        .find(|net| net.contains(&addr.ip()))

ip_nets is a 'HashSet<IpNet>' but it should be a radix/patricia tree.

Something like https://lib.rs/crates/iprange