This depends heavily on the country you’re in. In the UK banks are required to refund you within a single business day if they don’t have any evidence that the customer isn’t trying to defraud the bank themselves.

They try to squirm out of that of course. But in general getting your money back isn’t too tricky even with a debit card.