> The reason everybody was scrambling for NATO membership was the threat of being taken over by USSR.
While it was a sign of being more comfortable with the US than the USSR, for many it was more a hedge against an eventual resurgence of Germany than a perceived imminent Soviet threat. Which, sure, looking back from after the Cold War seems a bit weird, but from the perspective of the immediate aftermath of the two World Wars with a short breathing space in between is less so.
> It should also be noted that Russia's claim to join NATO was rejected in 1954,
No, Russia’s suggestion that if appropriate changes to the structure and character of NATO were made it might be interested in discussing joining (not a “claim to join”), made as part of a multipronged strategy to disrupt the formation of European Defense Community, promote the demobilization of Western European militaries, and gradually push the US out of Europe (which everyone could see at the time, but we know for certain now because we have thr internal Soviet Foreign Ministry memos as the strategy developed) was given a hard pass.
> and Putin has stated that he has suggested Russia joining NATO to Clinton and was again rejected.
Russia was admitted to NATO’s Partnership for Peace, a membership on-ramp program, in 1994, and NATO has been pretty clear about the path forward if Russia wants full membership. Russia’s not interested in membership (though it may have been when it joined PfP), only disruption, so it hasn’t made any recent steps on that path, but that's Russia’s decision.
> but why was post-USSR crumbling and then Western-friendly Russia rejected?
Russia made a bit of progress toward membership during its brief Western-friendly phase, with the support of the Alliance. The 2000 discussions where Putin floated the suggestion of joining were after that period. It was a nonserious empty propaganda maneuver.
> I'll suggest that it was because US would then lose its favourite boogeyman.
Nah, by then Saddam Hussein’s Iraq was a bigger boogeyman than Russia; sure, the US had conflict—and almost, in the Balkans in the late 1990s, direct armed conflict—with Russia, but the US government publicly downplayed that conflict, rather than treating Russia as a boogeyman. Russia didn't really get back boogeyman status until it forced its way back to it with its invasions of Ukraine.