To be ultra pedantic, you're paying American import tariffs to the American shipping carrier or American broker not CBP itself.
> refunds for it like Americans supposedly did
No one's getting those refunds, if they do get a portion back and not at the rate at which they increased prices on American consumers.
But more to your point, you can certainly pay American import tariffs on behalf of Americans, but your response is to raise prices on American consumers commensurate to the tariffs you're paying. Your margins are necessarily lower to sell to Americans, so you must raise prices for American consumers.
If your margins were high enough that you can absorb American import tariffs, then you're lucky the market hadn't provided a relevant market substitute.