This is indeed frustrating, and it's the reason that when I play with newer players I strongly recommend to them not to place their 2 initial settlements on the same numbers!

If you put those initial settlements on 2 intersections that happen to restrict you to only rolls of 3, 6, and 9 you're in for a world of hurt even though 6 and 9 aren't "bad" numbers per se.