The Java plugin was a pain. The runtime took somewhere on the order of 10 seconds to load, and the browser was completely unresponsive during this time. This was also the era before CSS, so if you wanted animated buttons on your web site, you had to use Java. Imagine loading a page, and then seeing a column of gray rectangles that freeze your browser and eventually resolve into dithered (“web-safe” color) buttons. Not a great experience.
Real Media only played video and audio. QuickTime played video and audio and VR snapshots. There were also plugins for specific types of content like VRML.
I feel that Flash took off because Adobe spent lots of money pushing it everywhere, but also because it ran better than Java. The runtime was smaller to download than Java, it started up much faster than Java, and its vector animations played very smoothly compared to the alternatives of the time.