x86-64 has single instruction (MUL) that produces a 128-bit unsigned result by multiplying two unsigned 64-bit operands.
In addition to what jbapple said, if your hash map has more than 2^32 buckets, the latency of the division is the last of your problem.
Sure N (number of buckets) shouldn't be too big, but x (the input value) could be.
realistically any hash table bigger than the 32 bit boundary will be well outside any CPU caches and possibly even ram. At these sizes the memory latency will dwarf CPU speed