Only if they provide binaries, at that point it gets patent cooties, and they'd be liable for $25 million per year.
I think this is all left intentionally vague by patent owners, since they want you to just pay to be on the safe side, there's plenty of straightforward things you technically can't do with patented codecs, but which you'll never get sued for e.g. the person you pay to video your wedding has no licence to sell you the H.264 file the camera produced, and no way of getting one.
So I think it's at least partly based on what they haven't bothered to sue anyone over, though perhaps free speech comes into it in the USA.
Of course, you'd better not plan on using anything you learned from it until you negotiate licenses for your users (or the patent expires).