Are they binaries? I haven't seen a binary in awhile tbh. Usually they're releasing both the raw architecture (i.e. code) and the weights of the models (i.e. what numbers go into what parts of the architecture). The latter is in a readable format that you can generally edit by hand if you wanted to. But even if it was a binary as long as you have the architecture you can always load into the model and decide if you want to probe it (extract values) or modify it by tuning (many methods to do this).
As far as I'm concerned, realistically the only issue here is the standard issue around the open source definition. Does it mean the source is open as available or open as "do what the fuck you want"? I mean it's not like OpenAI is claiming that GPT is open sourced. It's just that Meta did and their source is definitely visible. Fwiw, they are the only major company to do so. Google doesn't open source: they, like OpenAI, use private datasets and private models. I'm more upset at __Open__AI and Google than I am about Meta. To me people are barking up the wrong tree here. (It also feels weird that Meta is the "good guy" here... relatively at least)
Edit: I downloaded their checkpoint. It is the standard "pth" file. This is perfectly readable, it is just a pickle file. I like to use graftr to view checkpoints, but other tools exist (https://github.com/lmnt-com/graftr)