(I'm genuinely surprised that the standard DELL and HP corporate workstation doesn't have its front USB ports deleted and its rear port access covered by a lockable metal cowl.)
So you'll have to replace all of your intentionally-broken computers with good ones, which will cost a fortune on top of all the employees' lost work time and no one will want to buy the broken computers from you so you'll have to pay to have them scrapped.
The person with the accounting data on a USB stick will get a formal reprimand for breaking the security policy.
At work, we keep a couple of power-socket-to-USB chargers around, if people want to charge their smartphone they can grab one. But simply disabling the USB ports on our users machines is not a realistic option at this point.