That's like saying the pointer 0x123456 is immutable. Which is strictly speaking true, even in a language as lax as C. But it's clearly not what people mean when they talk about pointers (or references) being mutable. What people mean is that either variables are mutable, or memory is. So if you say:

    // suppose initially x == 0x123456 and x has type char *
    x++
    // x == 0x123457

this is really changing the value of a variable (neither of the pointers 0x123456 or 0x123457 have been changed in any way).

Or if you say:

    *x = 'a'

now you're modifying the memory at the address 0x123457 (again, the pointer itself has not changed).

The fact that the pointer itself, as in the address to memory (as opposed to the variable holding that address or the contents of the memory being referred to) is immutable is of approximately no value to anyone.

And frankly, it makes no difference (for these examples) whether this is a language with pointers or references. You can still set variables holding references to new values (if the language allows mutable variables), and you can still mutate the memory that references address (if the language allows mutation of the objects referred to by references), and that's plenty of rope to hang yourself on even without being able to do pointer arithmetic on the address values.