It does seem that PRISM is building a list of who people associate with - through email and phone communications, or various chat programs. Through this list of who you associate with, you can certainly start to label people and deny them rights based on their membership in various groups.
Back in the 1950s, the State Department was asking people to promise they weren't communists before granting them a passport so they can travel abroad. The amount of information they can get on people has grown exponentially since then - and that doesn't mean we have to be less careful with it.
First amendment - freedom of assembly - grounds is a brilliant way to argue this.