An arbitrary IPv6 address is indeed not practical to find by scanning. However, unless you're willing to type in that 128-bit value each time you need it (which maybe you are) you'll advertise this address somehow and if you do that your advertisements can be read by others.

For example suppose you put my-private-server.vanity-domain.example in DNS with an AAAA pointing to your private server - "passive DNS" service means big DNS providers will sell the answers they saw when anybody (say, yourself, on somebody else's computer) asks AAAA? my-private-server.vanity-domain.example. They don't reveal who asked, so this isn't personal information, but they do reveal what the question was and its answer.

A long time ago we used this to build target portfolios, if we're going to sell your company our product X, this is way we can see that you already have products A, B and C, but not D, E or F so we look a bit smarter coming into the sale.