I'm not sure this is the right comparison, in general, although I agree with your point (below). I suspect that one benefit of accepting domestic driving licences as ID, but not most foreign non-passport documents, is due to familiarity. That's probably as important a factor in spotting forgeries as the security features embedded in the document, which aren't very useful if the person checking isn't familiar with an authentic version of the document.
In practice, I tend to agree that someone is likely to not be familiar with many driving licences, such as (in the US/Canada) those from distant or low-population US/Canadian states, provinces, or territories, or (in much of Europe) a smaller European country's driving licence or national ID card, so a foreign passport is far from the main concern.