I never quite got why GNU decided to consider the freedom to access the source code and the freedom to modify it as a single bullet point. To me, they're fundamentally distinct concepts, and furthermore the former is VASTLY more important than the latter.
Without the ability to check a program's source code, and personally compile it from that source, I am putting my hardware totally in the hands of the developer. I must trust that his program will do what it claims to do, and if it doesn't I can do absolutely nothing about it - other than threatening to take my business elsewhere or sue, which are merely the generic last resorts available for every kind of abuse or fraud. And even those will not shield me as long the program's misbehaviour goes undiscovered or unproven. It's pretty straightforward why this is a very dangerous situation when handling data of any importance.
However, if I can view and compile the source code, but not modify it, then I wouldn't say the loss of freedom is remotely comparable. If I see that the program contains objectionable code, then it is unfortunate and perhaps frustrating that my only option is to reject the program in toto, but I cannot see how I can legimately consider myself wronged. I was offered a product with all the information required to make an informed decision, the product simply wasn't good enough.
A food analogy: closed source is like having to buy food without the ability to examine its ingredients. Open-but-non-free source is like having access to the original recipe but not being allowed to take out the parts you are allergic to. It's a shame, but your health isn't in any danger.