It wasn’t that long ago that marijuana was an illegal drug in every state. And as a schedule 1 drug marijuana gets lumped in with crack and PCP.
I have no doubt updating security clearance requirements is a massive bureaucratic mess nobody is interested in doing.
So the requirements remain, but everyone just “massages” the answers to fit current norms.
The government has to remind you it's illegal because well "everybody" does it and it really doesn't seem like it actually is and it sure as hell doesn't feel like you're doing something wrong or harmful. So somebody has to tell you that this drug is really bad because you'll go to jail for it for a long time.
The main reason pot is bad is because it will get you a jail sentence. The main reason you'll get a jail sentence is because it's bad. Hope that clears it up.