Perhaps. Would you also say "Maybe it's better to keep the term 'sum' for combining the count of two sets, and to let the generalizations that aren't that (e.g., adding integers, adding vectors, adding polynomials, ...) be called something else"?
Just don't try to make the set theory version the "real" version, and then try to deny the use of the word in other places. Those other places were using it first; you don't have the right to hijack the word.
Regardless, the point still stands: why would you allow the word "sum" to fit all those uses (disparate, but with a web of family resemblances), but not grant the same to "derivative"?
It's not as though the similarity of terminology is chosen with intent to confuse; the intent is to illuminate. The name for the generalization is chosen to match its more familiar relative because it is often _useful_ to think in terms of the analogy, imperfect though it be. [It seems humans are such that we would never find our way to powerful abstractions without such overloading; the combinatorial explosion of names would be too great to comprehend.]
Frequently mathematics benefits a lot from abstracting to algebra because, at this point, it's purely about how to define elements and operations by their apparent behavior instead of by their metaphor or interaction with a larger idea (such as notions of space, continuity, rate, change... all of those require quite a lot of mechanics to get in place, while algebra is very light-weight).
As an example, there have been a lot of attempts to discretize calculus for computers. Usually, the goal here is to create a scheme of discretization which, in the limit, resembles the smooth computations we'd like to perform. This has been a successful program in practice, but it's known to be fraught with weird edge cases. It's easy to create discretized situations which violate intuition.
Much of the reason these failed is because they attempted to generalize from the notion of "rate of change".
There's also the idea of discrete calculus (not "discretized") which is what you get when you apply the algebraic laws alone to some very standard notions of discrete spaces (oriented simplicial complexes, in particular—the simplest discrete object which "has enough topology" to meaningfully have the algebraic laws of integration applied to it).
What you get in this case is a rich theory of discrete calculus which rederives half of manifold learning and graph theory as a special case. All of the laws follow precisely—and they must, as the entire construction was built to prevent such violations.
Finally, you can examine discrete calculus to find a notion of "rate of change" if you like. But it's alien from that which you might be familiar with from continuous domains. It would have been very difficult to arrive at this point trying to generalize that intuition.
But it's practically inevitable (not to say it's easy, just inevitable) to if you say that you want to take the algebraic structure of derivatives and integration and apply it to oriented simplicial complexes.