I agree about the rush hour on the trains in the cities. I also live in a large city in Japan and have to battle to get on the train around 8am and 6:30pm. The guys that are working all the over time tend to work in the really big name companies (Toyota, Honda, Mitsui, etc.) which have their main working centers outside of the big cities. We have a lot of engineers who work at these locations and it seems that the minimum amount of paid overtime is 40hr/month for our employees (I have interviewed engineers here who have reported working 110hr+ of overtime in a month, which is illegal here).
The problem is, these are all reported paid overtime hours for contract employees. Patrick is talking about lifetime employees of the large firms, who do a lot of service overtime, which is unreported and un-paid.
When I was an English teacher I was often hired by these firms to do 6 month intensive courses for employees who were moving to an English speaking country for 3-5 years. These guys had an unbelievable amount of work for that six months and often slept at the office (which has showers and changing rooms) for their 3 or 4 hours of downtime.
So I think you're right, that not everyone is doing this, but there is a certain circle where it is highly expected.