No, its not rude to use public roads to get between points connected by those roads, nor is it dangerous provided you are driving in a safe manner (and, if you are driving in an unsafe manner, that's a problem no matter why you are on those roads -- just as much if you are doing it to get to or from your home that is on the road as if you are using it to get between two points that the road happens to connect.)