This seems like the sort of thing that happens when the people writing the code don't know the domain, and the domain experts can't express how the software needs to be tested.
This entire class of bugs should be caught via fuzz testing. Swap out the live back-end with one that fakes execution against a snapshot of prices, do whatever you can do, and verify position sizes are within leverage limits.
That probably would help. I think a properly functioning margin system would never result in negative balances on liquidation, as long as the fake executions were filled at exactly the trigger price. Doing crazy but valid shit and looking for negative balances would probably catch other issues too.