How does open source improve safety if we simply don't have the analytical tools to intuitively reason about LLMs?

You can't use this to prove that the model will always behave correctly (or desirably). At best, you can build test-suites to empirically check that it kinda-sorta appears to be doing the right thing most of the time. Which you can just as easily do with a black-box model.

It's not that I'm against openness. I just don't see how you can posit that it gets us close enough to safety.