It's the UK term for Wall Street. It doesn't have as much of the geographic connotation these days - working in Canary Wharf or a Mayfair hedge fund can all come under the phrase of 'working the in the city'
The city is certainly a geographic area. It's the City of London, an entirely separate city with it's own government and everything.