You must study the code and document what you can. Reverse engineering it is sometimes the only option in your quest to understand it. Then you’ll need to make decisions about how to move forward given the constraints of the business whether it be time, resources, other priorities. There is no one size fits all answer but don’t be shy about vocalizing the risks and making sure the business knows the pain points. Sometimes you have to sell the problem you now have and get buy-in to really do something to fix the mess.
One thing I’ve realized is that when people don’t take the time to document and write clean code sometimes it boils down to their idea of job security.
Other times it turns out the problem is there is no single owner of this code base and it’s been hacked to death.
Lastly when something is ugly, enough and messy enough...then you can just call it proprietary technology...I kid I kid.