The account number/sort code combination alone is not usable by a fraudster to withdraw money in any way in the U.K., so perhaps this is where the disconnect is. You’d need to physically go to a branch and present the bank card and enter the PIN number, or present photo ID with matching details to the account.
There are “direct debits” that allow companies to draw money from an account, but they can only be set up by businesses, there are special guarantees, etc. etc. that mean the bank is liable to reimburse you immediately under the “Direct Debit Guarantee” if you report a fraudulent transaction.