There were a number of concerns, including for-profit projects he was pursuing despite his public insistence on OpenAI being non-profit as well as generally deceptive behaviour on his part. The last part at least is consistent with what others have said about Altman previously, including what allegedly led to his exit from YC, although they have kept those stories pretty quiet. But it seems like PG himself no longer has a lot of trust in Altman after he basically made him his heir apparent, and he has known him for a while now.
What's more, the driving force behind the board's move to remove him from OpenAI was reportedly Ilya Sutskever who was one of their key people and one of the handful of original founders. So it wasn't just a bunch of external people who wanted Altman gone, but at least one high level insider who has been around for the entire thing.
Altman himself was even once asked by journalists whether we could trust him to which he outright answered "no". But then he pointed to the fact that the board could remove him for any reason. Clearly, he was suggesting that there were strong safe guards in place to keep him in check, but it is now apparent that those safe guards never really existed while his answer still stands.