> That's even better, there are lots of ivory white houses out there to sue.
Only if those houses are in the same industry as your trademark. You can paint your house UPS-brown and they won't be able to sue unless you were using it as a place of business.
Case in point: Lake McDonald Lodge (https://www.yelp.com/biz/lake-mcdonald-lodge-west-glacier) can exist because its in a totally different business than McDonald's fast food.
Yeah, "McDonald" is trademarked, but trademark is very restrictive over what you can sue over.
----------
So your hypothetical "Ivory" trademark will only apply in the industry that you've registered the trademark under. You can't just sue private houses for using your color.
EDIT: Furthermore, to obtain a "Colormark" trademark, you need to prove that the color has secondary meaning to your particular business. Ex: Fedex-brown is a trademark because Fedex is using that particular brown to color their trucks and employee uniforms, and applies it consistently across their enterprise.
It would be unfair if I were to start my own rival shipping company using a similar color uniform as Fedex. I'd have to use blue-uniforms or green-uniforms instead.
--------
EDIT: As an example: General Mills was unable to trademark "Yellow" as a trademark on Cheerios cereal, because yellow was used on too many cereal boxes already and clearly wasn't a color that people associated with General Mills.