I'd probably state it this way.
It's one thing if someone makes an offhand comment like "Oh yeah, Hitler wanted to be an artist!"
You'd be like "Ok, interesting tidbit but whatever".
It'd be a whole 'nother thing if one multiple different occasions you start talking about positive hilter attributes.
I think it's reasonable for people to be pretty uncomfortable with the number of times and frequency that RMS seems to want to talk about underage sex. That certainly does not make him a pedophile. It is, however, pretty off putting.
All this makes me feel very much like the Hans Reiser case. I was internet active at the time and remember all of his fans swear up and down about how innocent he was and how much he was a victim of the corrupt judicial system. Then that fateful day happened when Reiser was convicted and basically said "Ok, if you'll give me a lighter sentence I'll show you where the body is".
Stallman did a lot of good for the opensource community, which makes him a hero to many. I think, however, it's unhealthy to try and justify his statements on pedophilia. It's ok to talk about the nuance of the laws (Certainly there are major issues with the way CP laws work. It's a bit crazy that teens are getting convicted of it over normal teen behavior). Stallman has been really icky here. He talks not about age difference but instead how "voluntary" the acts are. Trying to read in "Romeo and Juliet" scenarios when he himself doesn't even mention that is trying to paint a more rosy picture of what stallman wrote.
All of this, frankly, can also be cleared up by clarifications from Stallman (which he's not published). He could say pretty much exactly what you've said here "When I was talking about X, this is what I meant". He's not done that and I think that's really telling. Maybe someone told him to STFU, but then again, he's not really been silent on his positions on sex throughout the years, so why start now?