That is absolutely true, and I fully admit that there isn't going to be a perfect way to do this unless you are very lucky. I'm only curious if you could use a method like this to make maps like shown above at
http://www.spiekermann-wegener.com/mod/time/time_e.htm where it's approximately correct in a somewhat useful way.
My gut says it's probably more realistic to do this for large scale maps (say between major cities, 100s of miles, where distance is pretty proportional to travel time). Once you get down to the city / street level, yeah it probably gets real dicey with high variation in travel speeds depending on the particular road and how routes are laid out, and you just aren't going to get pretty triangles fitting together nicely :-P.
In any case its a fun problem to think about!