I completely agree. Every time I remove boost from projects the build times drop dramatically and it usually doesn't take much work to swap the boost code for something else that isn't template fetish art.