For example, there could be coverage requirements. Want to work on software that determines if a bridge will be safe or not? Then you need 100% branch coverage.
You gave as an example something from a building code. That's just a test case. There's no reason why for common types of software (CAD, fluid simulators) we could not have repositories of test cases that you must pass to within some tolerance. You could even make it adverserial and let people suggest new test cases.
Or we could decide on some reference implementation that you must either match, or exceed. So if you design CAD software, on any input that both your CAD software can handle that this reference implementation can too, you must match. That's more of a programmatic open ended test case.
If we're worried about discrimination, there's no reason why the the government couldn't publish a set of typical profiles (fake names, locations, credit scores, etc.) that your algorithm must treat equally to within some tolerance.
etc.
We can totally regulate software like we do other engineering disciplines. If we wanted to and thought that the resulting increase in cost and slowdown in development is important enough.