Hardware specialization breaks the economics behind a CAPTCHA. To fight that you need to use a PoW that hasn't been ASIC'd yet, and be willing to change PoW functions at the drop of a hat. PoW functions that stress memory or cache are also helpful here, though you run the risk of browsers flagging you as a cryptominer (which is technically correct, even if economically wrong).
Seems like it should be the same cost (barring the friction of having a wallet, etc.) for both sets of people since Bitcoin is just a commodity and the value is the same to everyone, miner or not.
For example, a miner should value some fraction of a BTC the same way anyone else does, since they can sell or buy it at the same price a normal user can. The fact that they can profitably mine BTC just means they have a profitable business on the side, it doesn’t mean they should prefer to pay for services (or emails) in BTC.
So in order to moderately inconvenience said spammer, you have to make each and every ordinary user wait hours mining a few satoshis' worth of hashes in order to be let in. This is the exact opposite of what you want.
But even if you pick a novel function that doesn't have special-purpose hardware, spammers can still optimize their setups to lower the effective unit cost below what a legitimate user faces, by doing normal miner activities (picking hardware, scaling up, moving to where electricity is cheap, etc.).
Since your goal is to maximally discriminate between legitimate and spam use cases, you'd want spammers to face at least the same per-unit cost as legitimate users.
What's one way you can do that? Well, how about charging actual currency, whether Bitcoin or fiat? Money has the useful property of having the same nominal value for everyone, and not being amenable to further optimization.
In short, forcing users to actually run PoW themselves doesn't really make economic sense. Even if you're avoiding existing hash functions, it's mostly worse than just charging money because spammers have better ability to optimize against it.
And if charging money doesn't work, switching to local PoW is unlikely to be better.