That's what I believed at first. Then I realized how concerned authorities are with grades. If you're looking to do work with some research group, people want to know your grades. If you want to get into certain graduate schools, you'll most likely end up competing with people with high grades. Grades aren't a super important determining factor but, of course, they are considered and they can be treated as a negative indicator if poor.
Nobody really cares that you decided to challenge yourself. If you don't at least try to optimize for grades, you're basically shooting yourself in the foot. If you don't optimize for grades, you'll be competing with people who do.