I have both. I remember having to get the passport, and remember each renewal (getting pictures, collecting the documents, finding the forms, sending them in, waiting to get them back). I have Real ID as well and have no memory of actually getting it, which makes me think it wasn't very hard.
Having to carry a passport card within the US sounds harder than just carrying my license. Especially when the passport card has all sorts of warnings about keeping it in the paper sleeve it comes with. It acts like it needs to be treated with kid gloves, and not just thrown in a wallet, like a license.