Your argument is that Rudin is hard for incidental reasons: he skips steps in his proofs. I personally did not feel this way when I read the book. But I am open to the idea that Rudin is imperfect. I personally admire Rudin's concision, but that may just be my personal aesthetic preference.
My point, and perhaps I did not convey it well, is simply that some subjects (in particular mathematics!) are inherently difficult. And if you want to learn them well, you have to face that difficult head on: you have to read hard books. Euclid's old saying comes to mind: "there is no royal road to mathematics!"