It requires a lot of force to override it. And remember, it requires you to keep the hands on the wheel. But you are not expecting to have to fight it, so there are some delays there. Then you're fighting a system that you don't know how much force it takes to override it. You don't want to apply too much force because you might be barreling down the freeway at 80mph. At those speeds you just nudge the steering wheel. But that's not the case when you need to override the car.
And it happens more then you expect. Every time the car mis-calculates the actual lines on the road it'll do it. Changes in paint where the old one is still there for example. Or just faded paint and all of the sudden it detects a diagonally going split between 2 slabs of concrete as the lane separator.
Or you're doing 50mph, and you're passing a couple of cyclists. If you don't turn on your turn signal and you weave to the left to pass them the car will literally steer back into the direction of the cyclists. Scary.