I've actually installed CalyxOS on an older Pixel 2, and it seemed to work fine for the things I need to do with my phone. I haven't made the leap to use it as my primary device for one reason: Supply-chain integrity.

I don't know whether the maintainers have the bandwidth, tooling, processes and procedures, and vetting to ensure that an image doesn't get backdoored. Given the Venn diagram of "people who want to keep stuff private" and "people who will go through the trouble of installing and using CalyxOS" probably has significant overlap, I would assume some well-funded adversaries, including state actors, are motivated to sabotage it.

At least with the standard Google Pixel images, I have some degree of assurance from Google's robust source management and security infrastructure. The bar for backdooring an official Google product is much higher than the bar for backdooring a hobbyist project.