It very much depends on who's performing and what they're performing. When I used to go we'd start queuing maybe an hour before the doors opened but we were normally fairly near the front. The one time we struggled to get in was something like famous enough to get played on Classic FM (Grieg?).

If you're not sure get there a couple of hours early, have a look at the queue and if its non-existant pop down to the Science Museum for a bit, or The Natural History Museum, or the V&A, they're all within 5 minutes walk of the Albert Hall, all free, and all worth the trip by themselves.

I'm not really a classical music fan but I would certainly recommend going to the Proms at least once just for the experience.