My understanding is that Yakuza are generally "Burakumin", which is basically a sub-ethnicity that has been discriminated against in Japan for hundreds of years. They have a strong cultural heritage of decorative tattoos, which is why people with tattoos aren't allowed in gyms or nice restaurants in Japan. It's illegal to discriminate against them in the workplace, but basically every company still does extensive family background checks looking for any sign of Burakumin heritage, and if discovered, they'll suddenly realize that you just aren't a good fit for the company.
This is why they're "born to lose". They can't get legitimate high paying jobs, so they build their own ecosystem that they can thrive in. Lots of criminal participation around the world comes directly as a result of systemic discrimination. These are often smart people making the best of a shitty situation that they were born into.