I've been flying with a passport my entire adult life. In fact, I carry the passport with me everywhere and use it for other identification needs as well (e.g. buying beer, getting my tickets from will call, etc). I appreciate that the passport lists only my name, birth date, and place of birth whereas my driver's license includes my address, gender, height, weight, and eye color. Places that have technology to scan driver's licenses are almost never able to scan my passport, making for that many fewer databases my presence is indelibly recorded in. Ever since my state started using facial recognition technology on their IDs I've asked to have my driver's license made without a photo, so I can't even use it as an ID card if I wanted to.
Nowadays you can get a "passport card" which has the exact same form factor as a drivers license and can be used the same way as an "enhanced real id" for domestic flights or travel to/from Canada and Mexico.