Who are the people with yellow badges? The author never found out. And Google does not appear to have made the information public.
The important point here is that if temporary employees are working alongside full timers doing very similar work for long periods of time, the employer (Google, not the agency) runs the risk of a suit like http://en.wikipedia.org/wiki/Permatemp#Vizcaino_v._Microsoft.
Microsoft now (and I suspect this lies in part behind Google's various segregations of different classes of employees) makes an effort to restrict contractor access to various full-timer activities (like ski trips and such, as the author complains) and to ensure that contractors don't do full-timer work.
So this isn't some sort of evil conspiracy; it's an honest attempt to comply with US labor laws.
Is it fair? No. But Google's used that perception a lot in the past and their justifiably worried about losing that good reputation, given their increasing difficulties with retention and hiring.
A scandal about 3rd class workers with untouchable status is the last thing Google needs right now. They should have handled this better.