No, it can be an operator on any kind of (usually) ring or more general algebraic structure. What you refer to is the "usual" derivative of functions of one variable, just one of many derivatives one can define.
This is why, when we talk about rings and fields and such we say "multiplication-like" or "addition-like" operators. The operators defined for the algebraic structure may not be exactly like "standard" operators, but they still follow rules and you can still do cool things with them.
There are many consistent ways to define the derivative of a number.
The way we are all familiar with is to define a number as a zeroth-order polynomial.