> Tech workers have a responsibility not just understand the technologies that they work with, but also educate themselves on the societal implications of those technologies.
I think this goes well beyond tech workers. I think it's time for society to legally recognize the balance between the value of ML systems and the privacy concerns of customers of ML.
Doctors and lawyers obviously should understand the value of privacy, but we, as a society, have also created legal rights and duties for them. Conversations with lawyers and doctors are legally privileged; at the same time, there are specific consequences for medical companies or lawyers who do not protect that information.
Companies like Google, Apple, Amazon, etc. certainly have the resources, intelligence, and sophistication to comply with a similar regulatory regime. IMO it should be possible to construct a law that allows companies to collect, store, and tag customer data for purposes of training ML systems, but sets serious duties, with consequences, on them to do it right.
Right now, what is to keep employees at these companies from abusing these systems to stalk, to surveil, to harass, or even just to feed their own curiosity? These data systems are core trade secrets for these companies, which means they are opaque to any kind of oversight from outside the company.
The free market can't create the necessary balance because customers need information to make decisions--information that they don't have. The result will be an increasingly chaotic "hero/shithead rollercoaster" as customers make snap judgments based on scanty or wrong information about what these companies are actually doing.
This is a classic case for regulation, which prevents a "race to the bottom" of sketchy practices for short term gain, while also protecting the ability of people and companies to use this technology to create value.
Doing this right will help data-leveraging companies in the long run, just like attorney-client privilege and HIPAA have helped lawyers and doctors build trust (and therefore value) in their customer relationships.