I guess later in the game one doesn't mind randomly walking right by the castle just because it is the shortest path, just going to eat some time dealing with each guardian of course.
Same for swimming across a the river twice around a bend rather than just following the coastline.
Also there are 226 waypoints, the linked map is missing a lot...
I am not sure what it is trying to show. Was this the waypoints you had not done at the end of the game?