Without a reason, they shouldn't. But there's an ambiguous area in there.
For instance, my house in SF actually shares a foundation with the two neighboring houses. This was an issue when I wanted to remodel to get better ceiling heights. I'd say I'm not free to do whatever I want with a shared foundation.
Also, when you share a wall, you do have an interest in verifying that the electrical and firewall work done next to your house is up to code.
It can get out of hand, but there are a lot of reasons for permits and inspections. I don't think entering someone's house, scheduled well in advance, with a very limited agenda to simply verify that recently added wiring in a dense, crowded neighborhood with shared walls, and leaving immediately afterwards, would qualify as an unreasonable search and seizure.