Open Source Is Throwing AI Policymakers for a Loop
spectrum.ieee.org
spectrum.ieee.org
“The scary part is how easy open source is to use”, and “a mistake in open source is much more difficult to correct then an algorithm from a - accountable - company.” Shows how little the author understands of this field. It won’t be tensorflow that’s biased against minorities in resume selection, more likely the data was biased.
That sounds like a great advertisement for FLOSS, really.
So far no-one responsible for the computer system or the prosecution has faced any repercussions.
Meanwhile in open source there could be many users raising issues if a bias is suspected. Many people working to fix it regardless of other priorities. And then an update will be pushed.
I don't know if the EU regulations require access to the data too, but that could be problematic in a practical sense ( e.g. do they deliver terabytes of raw data to the regulators?) and there may be privacy issues with companies sharing their data with regulators
This remark is absolutely spot on. This is why I don't like the language around 'algorithmic bias' and 'algorithmic accountability' -- it puts too much attention on The Algorithm, which lets big tech companies deflect us from scrutiny of their training data.
Your comment about the conflict with privacy is also spot on. Here's a paper (shameless plug!) about it: Show Us the Data: Privacy, Explainability, and Why the Law Can't Have Both [1].
What's interesting about the GDPR is that it's not purely about what regulators do: it's legislation that grants rights to people, rights that can be litigated in court. If I as a GDPR data subject demand an explanation of an ML decision about me, as I'm entitled to do, my lawyer could argue "the source code isn't an adequate explanation, we want to see the training data too". Then it'd be up to the court to make a ruling about my lawyer's request, not up to the regulator. The paper [1], co-authored by a data scientist (myself) and a lawyer, thinks through the privacy / explainability conflict as it might play out in court.
In practice, this will likely mean that using open source for these AI applications will become a lot more difficult in a legal sense, perhaps impossible, because there's no single entity that can speak to compliance of open source components (vetting each commit, etc.). Perhaps some orgs will emerge who are willing to monitor and oversee open source software and certify that it meets the requirements of the legislation, but it's unclear how that would work.
I'm not endorsing this model, just explaining the concern. I don't think the tech community really fully appreciates how misguided a lot of the current approaches to AI regulation among policymakers are right now. (Not just this, but initiatives like the current CAHAI work towards a binding international law instrument on AI.)
And yes, like the GDPR, the proposed EU AI regulation has an extraterritorial effect. Under the proposed AI regulation, if you're selling into the EU or to any EU customers but don't have an establishment in the EU, you have to actually appoint a representative to comply with these requirements.
Kubernetes Podcast 155: Software Supply Chain Security, with Priya Wadhwa
https://kubernetespodcast.com/episode/155-software-supply-ch...
I'm honestly worried that the greatest innovation in computing of the last decade seems to be inventing programs where literally no one understands how they work.
I think things will improve before the final act. I have contributed to some of the commentary towards the regulator on this, and a lot of the flaws mentioned are now well recognized.
Disclaimer: opinions above are my own.
No arguments presented. It's just that. Everyone can change opensource AI and add biases. Unlike companies, who are accountable and will easily fix it if something goes wrong. Tech is too powerful for simpletons to wield, we should restrict it.
> Opensource math is too dangerous and can't be left as it is. It's scary that everyone can just create math and we should make laws regarding this glaring issue. Proprietary math is more accountable, transparent and easier to fix than opensource math. Opensource math is more bias-prone by it's nature.
Plainly nonsense and I hope this worm doesn't get anywhere near a lawmaker's ear.
I remember seeing this misconception about Linux probably 15 years ago.
"How can you trust it? Anyone can change it!"
It's not like Wikipedia, dad. There are people deciding what ends up in the soup.
Wikipedia isn't even like Wikipedia - They enforce IP blocks, most of the pages that are vulnerable to vandalism are protected, and there are people watching pages they care about so they'll be notified if they change.
I don't see whats so scary about that, to be honest. Some random person who is messing with ML and doesn't know what they're doing probably isn't in a position to use this in a way that would be harmful to people.
The example of discrimination given in the article would only really apply to corporations and government, who would likely use proprietary datasets (and perhaps implementations.) So I don't understand why this piece takes aim at open source. Seems almost contradictory.
...which includes full-time ML people. So I'm not sure there's a lot of difference.
Makes perfect sense, right?
And while we're at it, the piece stinks of casual misinformation right from the first paragraph. AI isn't "curing disease", this is just nonsense hype. Why would I listen to the author if he can't get through his short introductory paragraph without spouting nonsense?
Par for the course for IEEE, which has always come off to me as a very self-aggrandizing, bullshitty organization.
Intuitively I would say policy should focus more on corporationsas open source is publicly auditable.
"Computer says no" is an incredible force multiplier for Authority, and anything that threatens the truthiness of the black box's output will be mobilized against.
Unaccountable security-through-obscurity is actually a bonus for Authority, too. Look at the whole Signal-Cellbrite [0] thing: Moxie found all these vulnerabilities, some going back years. What are the odds that an avatar of Authority('s technically-minded underling) found one or more of these Cellbrite exploits and used them as tools against enemies, or to exonerate friends? "Make sure you scan this cellphone first." Would any of that be possible if Cellbrite was legally mandated to be built from remote, publicly-viewable source every time they wanted to use it? Or, if you want to be less conspiratorial, how many vulnerabilities would Moxie have been able to find if the source had already been public (and subject to pull requests) for years?
The article just linked to the actual text of the bill but this summary seemed reasonable.
https://www.lawfareblog.com/artificial-intelligence-act-what...
Long story short, while I don't personally find this to be a particularly well-written article, the author's not entirely wrong. Open source machine learning software and models, in particular, are presenting a number of new challenges in a variety of areas, including regulation, that many of our existing systems are poorly equipped to grapple with.
[1] https://thegradient.pub/machine-learning-ethics-and-open-sou...
[2] https://thegradient.pub/machine-learning-ethics-and-open-sou...
Ok, maybe, but now we’re supposed to believe that people who know absolutely nothing about how computers work are in a better position to understand their limitations and regulate them?
The consequence of datasets being used for training (and trained models being incorporated into software) typically via a line of curl or wget in some setup code, is to leave the dependency, essentially, undeclared.
Yes, any biases in those models are more likely to be uncovered because of their public availability, and yes, the biases uncovered will probably be due to biases in the data, but right now we don't really have (or aren't using) processes, tooling, or infrastructure to do the necessary versioning and tracing of data or model dependencies.
In fact, when some researcher discovers a bias in a model and publishes it, most often no new version of the model or the data it is based on is released.
If by chance the biases are addressed and new versions of a particular model and dataset are released, all the other models based on the same data are left unaffected. And software that incorporates the biased models is unaffected by their re-release as well. Nothing will ever trigger those models to be marked as biased, retrained, and re-released, and typically nothing marks the dependent software as vulnerable to the bias either.
Both the companies implementing and using the system need to be responsible and accountable. Whether or not the software is open source. In their example of AI being used to prescribe sentencing, both the judicial district using the AI need to be responsible and accountable as well as the vendor selling, implementing, and operating the system.
The software being open source at least gives a chance for external eyes to evaluate a portion of the system (given that the data set might be private.) With proprietary software we have to rely on the honesty of companies and voluntary transparency or any whistle blowers which may come forward. In the example criminal sentencing example both the judicial district might be incompetent or biased and the vendor could be incompetent or greedy and not communicate system biases since it may hurt their brand or contract or growth.
It seems very clear to me that openness and transparency is key to combating bias or manipulation. And open source foists a minimum of openness into this.
There should be a name for that tension.
https://www.goodreads.com/book/show/34397551-the-chickenshit...
"If an open-source algorithm is flawed, it is harder to undo the damage than if the software came from one proprietary—and accountable—company."