Not a dumb question!
There is a great video by the mathematician Richard Borcherds on this exact objection to current examples of quantum supremacy.
There is a great video by the mathematician Richard Borcherds on this exact objection to current examples of quantum supremacy.
https://scottaaronson.blog/?p=5460
He actually goes into significant details about your exact question and proposes some resolutions. It's quite long and I don't think I can do it justice in a TLDR so leaving the link to speak for itself.
Edit: I just read it and I don't feel like he really answered the question. He had a rebuttal about being able to freely specify parameters, but then his Facebook friend addressed his rebuttal, to which he replies "this is indeed more interesting"... which, well, it certainly is, and also doesn't answer the question!
(Well, I guess he also mentions we should expect speedup greater than Avogadro's number, which seems fair, but clearly that's not the bar anyone is claiming to meet, so it doesn't get us anywhere.)