"freedom of expression isn't freedom from consequences". Freedom of expression is about whether the government will use state powers to arrest or silence you. Being uninvited to a gathering is not that.
The very first hit searching "supreme court decisions white house press passes" yields the following, and I leave it to you to search further if it pleases you.
https://www.americanbar.org/news/abanews/publications/yourab...
I'm by no means a free speech absolutist, but there are certain things that are huge red flags for freedom of press and freedom of expression. The government moving against press that is critical of it is dangerous. We should always be wary when that happens.