I just got back from my first trip to LA and my impression is the same. I stayed with a relative in Venice, met a friend in Silver Lake, and went downtown the next day. The amount of travel was pretty crazy. I'm from Chicago and traveling 15 miles is pretty much going from the far North Side to the far South Side, a trip that no one would make just for brunch.
An analogous LA day in Chicago feels like it would be brunch in Schaumburg, coffee with a friend in Gary, Indiana and dinner in Winnetka.
Maybe I'm way off, I was only there for three days, but that was what I took away from the trip.