There is one consideration however. One of the GPL license condition says "must license the entire work under GPL", which mean that in order to legally distribute the GPL part, the MIT licensed code will be under both GPL and MIT. As such the patent grant in GPL will cover the whole project for anyone distributing the GPL included version, and many interpret this condition as making the entire project GPL. The exception to that view is that a project can still continue license new code as MIT, and a distributor could simply remove the GPL parts when they want to use it in a MIT and BSD only/Proprietary/Patent enforcing situations.
While I have not seen many MIT projects do this in regard to GPL add-ons and patches, it is the standard model for open core MIT projects to do this in regard to proprietary add-ons. I would thus not call the existence of optional GPL patches or add-ons as "make your entire project GPL", unless the additional code is essential to run the program.