I'm surprised no one has pointed this out; you can report them for paid/spam links in your Google Search Console. It doesn't solve the problem but it will speed it up, especially if it's a manual PBN (private blog network) and not 'web 2.0s' as they say in the SEO world (links from Wordpress, Blogspot, and other already-hosted varieties). Google is pretty good at finding how the PBN is tied together - usually by looking at the hosting, and it will kill their rankings for a long time.
Also make sure to check your inbound links for negative SEO attacks. Since the most recent algorithm updates will penalize page by page for negative/spammy links it's possible that's happening to you as well. Google likes to pretend like that negative SEO doesn't exist, but it's probably the easiest way to climb to the top of rankings if the others are unaware of the intricacies of SEO. Search console is the only authoritative way to find those, but ahrefs will generally update faster and give you a head start. (I spend about 5 hours/week disavowing negative SEO links, and that's after I mostly-automated it).
In the meantime, you might want to put a little more effort into getting links pointing to you, if for no other reason than to not leave yourself in such a precarious position.
Feel free to email me and I'll look into it if you need more help. (I am over SEO at a YC startup trying to fix payday loans https://www.semrush.com/info/lendup.com+(by+organic), so as you can imagine we've seen it all... I hope).