I've visited it and found it very worthwhile, but one needs to remember that at the same time they were building the pyramids in Egypt.
Skara Brae was occupied as early as 3180 BC. Egyptians didn't finish the first pyramids until ~2600 BC.
It is truly a wonder - especially given the different circumstances than a fertile river valley.
Of course 2600 BCE is about half way between the early Pre-dynastic period and the end of the Ptolemeic period. And only about a third of the way to present day.
Egypt is mind bogglingly long lived.
I don't think you really need to compare / remember that... we're talking history here, not some sort of building scorecard.
Different kinds of buildings though - I've been into one of the pyramids at Ghiza (Khafre's) and it actually doesn't look that interesting. I find the buildings at Skara Brae interesting because they are more human scale (beds and other furniture) and were where people actually lived for many hundreds of years - quite a bit before the pyramids at Giza at least.