London is a basically a bunch of villages with dense residential areas filling the gaps between. I've gone whole 6-month periods never going further west in the city than Holborn (Kingsway), and whole years never going south of the river east of Greenwich.
You're suggesting that downtown is Oxford & Regent Streets, i.e. essentially high-street fashion shopping. Thing is, almost every shop on those streets is also in both Westfield malls on either side of the city, and many of the bigger brands are also on dozens of high streets across the city. There's seldom much call for most Londoners to actually go to those streets (IMO).
In a business context, downtown in London is the City, but the tallest buildings aren't in the City, they're in Canary Wharf, a privately owned near-island with 24-hour guards at all entry points, and barriers that go up on the roads at night.