The Problem is not coupling charged matter fields to the EM field. You get a well defined set of coupled and (now) non-linear PDEs. The problems arise when you try to model point charges. Then the theory is plagued by infinities that originate in the infinite charge and current densities.
The infinities in QED are actually far less problematic than those in the classic theory. They just seem to be more problematic because you cant (approximately) ignore the backreaction of EM and matter fields.