One can object to this notion of customary international law, but if we're going to say that laws of the sea don't matter here, then we must also give up the legal notion that the strait is Iran's (and Oman's) territorial waters. There's no consistent legal framework under which the strait is Iran's territorial waters, but not an international waterway.
There's the practical reality that Iran can extort ships anyway, even with no legal pretense. But then it's essentially just piracy, and anyone with a few boats or mines can do that, unless they're stopped by force.