Unless your state is open carry you do in fact need a permit to carry in public. In fact, some open carry states (ex Texas) still require a permit to do so.

I'm always frustrated but amused at how uninformed many of my fellow Americans are about the gun laws here (frequently it's the ones pushing for additional laws).