Unless your state is open carry you do in fact need a permit to carry in public. In fact, some open carry states (ex Texas) still require a permit to do so.

I'm always frustrated but amused at how uninformed many of my fellow Americans are about the gun laws here (frequently it's the ones pushing for additional laws).

The key language from the courts is "reasonable regulation" on the use of firearms. That's a bit vague, but "we've created a process to let you concealed carry, but we won't actually issue a permit to anyone" pretty obviously fails the "reasonableness" test. If the restriction was that concealed carry is flat out illegal, that might actually be more legally tenable than this discretionary nonsense in places like the Bay Area. Of course, CCW regulations that are clearly limited to safety and competence with firearms and limit the discretion of issuing officials are almost certainly compatible with the Constitution.

Just for the sake of comparison, the rule that was struck down in DC v. Heller (which firmly established that the Constitution protects an individual right to keep and bear arms for defense, subject to reasonable regulation by the state) required gun owners to keep their weapons "unloaded and disassembled or bound by a trigger lock" at all times.

You definitely need a ccw to open carry a loaded firearm in many jurisdictions