Well it obviously mattered enough for the Founding Fathers of the US to enshrine freedom of speech in the Bill of Rights, and in the last 200-ish years it also mattered enough for US courts not to overturn or politicians of various parties to change it.

Now where I'm from (Germany), not just "hate speech" is against the law, but it's also unlawful to insult another person. It's complicated, but for the latter it's mostly sufficient that they feel insulted by what you said to them personally.

Now while I don't go around insulting people in person or on the Internet, I personally think - for instance - that it should be allowed to call a person an asshole, if they behave like an asshole. Yet, if I did that here, or even online to another German person, they could go to the police and press charges. If the public prosecutor is sufficiently bored, this very low barrier could also be used to dox me in an otherwise reasonably anonymous setting, since the resulting lawsuit could result in my data getting subpoenaed from, say, Twitter and my ISP. This has happened to other people here in the past.

Now while I'm neither in favor of either hate speech nor randomly and viciously insulting people online, I consider the law in Germany as outlined unreasonable in an online setting. I think freedom of speech is more important fundamentally than another person's right to not feel hurt, or for some powers that be to silence or punish me because I said something inconvenient that they merely claim to meet some of the criteria for speech that is restricted here.

Mind you, this is the case all the while freedom of speech is enshrined in the German constitution as well. But I think it is a pretty good example of why I think freedom of speech should not be curtailed just in the name of another person's feelings about said speech. Even if a person, as you do, doesn't see a direct and tangible benefit in allowing that kind of speech, I would argue that a larger fraction of people are against disallowing it, because of the indirect consequences and where that line of lawmaking leads.

Another thing to consider is this: Say you're modestly happy with the current government wherever you live, and you'd be happy for them to have an "easy" way to curtail freedom of speech. Would you also be happy for the opposing political side to do the same thing? What if some extremists came to power?

This kind of reasoning is why free speech absolutists are so staunchly defending freedom of speech, even if it may be inconvenient or insulting to themselves or others.