I wondered why you don't store the reciprocal w/ the hash table object. Obviously it wastes some space, but it wouldn't be any slower than your specific checks for 4 and 31, I think. (If most of the tables have size 4 or 31, then I'd use your code).
Also, "multiplying by the reciprocal" is a bit simplistic - there's some other instructions added in based on the specific divisor value. Adding more tests and branches for these likely would not pay off.
I only meant to store it in addition because the alternative, storing an index into the list of divisors (and thus reciprocals) might be slower due to an extra indirection.