Gangster-style yakuza rarely bother normal people. It’s not worth the trouble for either party. There are some industries with a lot of yakuza-owned and operated companies, but their main goal is to make or lose legitimate money, so unless you are aware of some tells you’d never know that they are yakuza.