What is "right there"? If the computer should sort an array that is located in main memory, it needs to know their addresses (pointers) to even load them. There is no way around.
Lets say it's 4 8bit ints. It can be sequentially and directly in memory like:
stack: [0x01, 0x02, 0x03, 0x03]
stack: [0x07002412, 0x0700241A, 0x07002424, 0x070036A0]
heap (or stack, really):
0x07002412: 0x01
0x0700241A: 0x02
0x07002424: 0x03
0x070036A0: 0x04
That is what "void*" is doing. You're using a pointer instead of a value.
In C# this is what generics allow you to avoid.
// This is like my first memory layout above.
struct Foo<T> { public T MyField; }
// This is like the second layout above.
struct Foo { public object MyField; }