I am interested in hearing the counter argument though.
Even if you did manage to avoid getting sick, there's a chance you could get stuck in quarantine in Japan, the US or both as countries become more aware of the virus' dormancy.
Anecdotally, I have several friends who have cancelled non-refundable trips. If it were me, the desire to go on the trip wouldn't outweigh the stress of worrying about being infected, being held in quarantine, or traveling while a global panic sets in.
That's the key point for me. My personal risk calculus has me staying in my own country: while I think it's extremely unlikely that international travel to non-hot spots would lead to infection, I'm not chancing entry / exit / in place quarantines.