I've always took "Wall Street" to mean all parties involved in the U.S. financial market, publicly traded companies in particular.
Everyone one else is considered "Main Street"...
They have their own language and culture that exhorts greed and profit above humanity and peace. ie. "Wolf of WallStreet"...