Reframe this question in terms of a date: Is it rude to leave a date early? Most people would say no, provided that one uses some kind of plausible excuse. Hence all the mysterious phone calls / texts an hour into a first or second date.

I think the "time wasting" aspect is the important part: most people don't want their time wasted and, ideally don't want to waste the time of others—or to raise their expectations.