I agree with this completely; not being able to understand a patient's medical history often makes it much harder to treat them effectively. Especially in an emergency setting.
That said concerns about the sensitivity of information are absolutely valid and should be at the forefront of EMR systems.
In my experience (UK) access to records is reasonably well restricted to relevant staff. Access is audited in efforts to identify unauthorised access and this is taken extremely seriously; for example opening records for patients not under your direct care with no valid reason.
Further, for particularly sensitive information such as some psychiatric histories, or medical photography of sensitive areas, this is kept behind a secure area within the EMR systems, not generally accessible. Opening it requires signing a declaration that you have either discussed it with the patient and gained consent, or that you do not have consent and you are opening it in the patients best interests in an emergent situation as they are unable to consent (e.g. history of depression with paracetamol (acetaminophen) overdose brought in unconscious by ambulance.