Okay, so you travel from, say, Washington DC to NYC and then you have to set your clock back by... I think it's like 13 minutes?

Which is why railroads started instituting time zones in the late 19th century: because it turns out having to recalibrate your watch every hour or so is a recipe for creating train crashes. Railroad time didn't become standard time until the early 20th century, in large part because a lot of the smaller cities were pretty irate at having to adjust their clocks. (It also doesn't help that coincidentally, places like Chicago and New York City are very near the center meridian of their time zone).