I heard a saying that everyone lives on eight streets at any one point in there life. No matter where you go, if you live there long enough, you'll settle in and find a routine that you're used to.
The first month of living in Japan is bliss. You're learning a new language, playing with really cool public infrastructure, solving problems you didn't imagine you'd run into (is there a three-pronged outlet anywhere in this apartment?!), and seeing things that you'd consider "exotic" on a daily basis.
Once the novelty wears off, you start to understand that you don't quite fit into this culture quite as well as you thought you did. It turns out that growing up in America does infuse you with certain values, and they're not as universal as you think. When people say such-and-such is "part of the culture," that means that on an individual level, that thing is part of people's value system.
Going home before everyone else, for example, isn't just awkward; it's wrong. It's unfair that some of your coworkers have to stay late and some get to go home. What would your mother think? Why are you complaining that you can't use an ATM after 10pm? Why would you ever want to do that? Is it because you're irresponsible and bad at planning? Or are you just using it to stay out all night drinking?
Right now you might think you're alright with the working hours because, hey, Japan! But when your version of "Japan" is your apartment, your desk, the bar after work, and your train commute that you've cut down by 15 minutes due to your supreme knowledge of entrance positioning, you will wonder what happened to those dreams of castles, temples and teahouses. You could plan a trip somewhere, but you'd have to take a day off work when everyone's so busy, and that would be wrong.