Do you really want to go there? That everything in the world would have a literal "caveat emptor" attached to it?
As for not wanting to force companies to release only "safe-for-children products", I do actually agree. However I consider it to be a matter of degree, and in this case for example, I think that if nothing else, Google should say the very least make the disclaimer a bit more prominent and maybe tweak the model so that it's not quite as confident in its claims in the AI Overview.
That would be nice, but as every effort to restrict kids from using software which are not safe-for-children keeps getting condemned for being invasive surveilence, and every effort to stop kids getting the hardware instead gets condemned because of how much of society is now built on assumption everyone has a phone…
Something has to give.
Dunno what, but something.
And yes, it is ok to remove choice if the existence of that choice violates other person’s rights.
Google can continue offering that choice if they make sure nobody is defamed.