>hash of supplied password does not match hash of stored password, access denied
Resulting from miscalc of either stored or supplied would indeed create DoS.
If the off by one is in a graphical driver that causes an overflow leading to no output, that's DoS.
If the off by one is in the memory mapping of input devices leading to no available input, that's DoS.
Simple math is kind of everywhere in the kernel and userland apps. If the math is wrong and results in memory mapping wrong such that kernel panics or is unuseable, that's a breaking bug regardless of simplicity.