Google APIs have always been difficult for me to use. I find their documentation super fragmented, and they always build the same thing twice with awkward overlaps. In this case, it's Vertex and GenAI - two different SDKs with isomorphic data structures that are seemingly incompatible, but only sometimes. I don't understand how these things happen - but as usual I blame PMs trying to mark their territory and pissing all over everything.
The new gemini models themselves though, are killer. The confusion is a small price to pay.