My brother and I got the first two grey castles in the 1980s (the first castle yellow; grey was a fairly new colour back then). Those castles mainly relied on fairly large castle wall pieces. At first, even as a kid, I felt that was cheating; we should have gotten bricks to assemble all those walls. Later I discovered what a boon they were, because they allowed me to rapidly design new and different castles. Sure, I couldn't build a spaceship out of them, but just exploring the space of all the possible castles you could build was amazing.
Some of these specialised pieces inspire creativity. Especially with spaceships.
But I do agree that many modern sets just invite you to build this one thing out of them. We did actually turn out Millennium Falcon into an X-Wing and a TIE-fighter once, but we did eventually turn them back into the Falcon. It feels more wrong than it used to to build something else.