Looking back now it seems obvious but back then it just too overwhelming. With PHP there's this 'just get it done' attitude and futzing around with phpass wasn't something I could let myself do back then.
Of course I now regret it. I ended up using a different encryption library for password storage on my app. It isn't md5 but I know it could and should be stronger. So now I'm in a situation where I'm rewriting the codebase in a different language entirely and the cherry on top is that I need to migrate all my user's passwords and even some data from the original method to bcrypt, which I'm now using. It seems simple in theory but I have a feeling this is going to cause some headaches. However, its totally necessary and especially for the type of application I'm working on which puts a big focus on privacy and security.