Ask HN: Vulnerabilities in ML frameworks
Does anyone know of any vulnerabilities in machine learning frameworks, specially buffer overflows and arbitrary execution?
Arbitrary execution is more possible IMO. The way a lot of ML models are stored is by being a serialized file using a framework like pickle or Java's serialization. Theoretically, you could add code into a precompiled model that when someone loads would execute arbitrary code. This could be done using a simple technique like a code cave seen here https://en.wikipedia.org/wiki/Code_cave. I haven't had time to dig into this myself, but I honestly don't think it would be hard.
I think in the next couple years you will see more vulnerabilities pop up in these frameworks, but finding security vulnerabilities take time.
https://media.ccc.de/v/34c3-8860-deep_learning_blindspots
https://medium.com/@ageitgey/machine-learning-is-fun-part-8-...
I believe that most ML applications are fool-able by attacks of the same type as would fool a human doing the same task as the ML. The really big difference is the scale of attack required to fool ML vs a human.
For example, let's consider ML processing of camera+lidar data, similar to Google's self driving system, versus a cardboard cut-out of a puppy in the road. A human could be fooled by a really elaborate cardboard cut-out, or one viewed at a high speed. ML is likely to be fooled by a cut-out as well, but the elaborateness required of the cut-out would likely be lower (people would notice it doesn't look normal, while the ML has a much smaller dataset than the average person, and hasn't focussed on cute furry puppies as much). The ML would also likely be much better at dealing with high speeds than a human would be, because we don't have built in lidars and our brains work order of magnitudes slower; but taken to an extreme (ML can't complete processing in time to identify fake), the ML would not be able to tell the difference.
There are certainly other ways to go about it, but I think this is the most straight forward and general 'attack', in that false positives are unavoidable in ML (and in humans).