When I asked him to prove he was from Google, he didn't seem fazed at all and said he would send me an email from a google.com email to prove it, and gave me his name and "employee ID". We kept talking and he said the email should show up and it was sent from his side, but the email never came. I then said I'd call google support and ask to speak with him instead - he was still unfazed. I did call Google support (im on Google One for Gemini access so luckily I actually have access to a phone number I can call), and they said it was likely a phishing attempt. I did suspect scam from the start, but it did seem a tad more professional and polished than the usual scams - the person really sounded professional, good voice quality, there wasn't a whole lot of noise in the background, they weren't fazed by my attempts at verification and just tried to dodge them hoping I wouldn't notice instead, they didn't try any pressure/urgency tactics like scammers often do.
So this news is real.. as far as I can tell they were able to connect my email address to my phone number via a leak from Google. They were trying to escalate that into further access.