100 times yes. The mere presence of a single character from a different class does not increase entropy. It means that most common passwords move from 8 lowercase letters to 6 lowercase letters plus 1 number plus 1 symbol, which is even easier to search.

If you have a sophisticated entropy estimation tool, or library, great. Another simple thing is using a min length, plus restrict from a dictionary of the million most leaked passwords to outlaw those (can hold it in a bloom filter if size is a concern).