See also: https://en.wikipedia.org/wiki/Expansion_of_the_universe#Expa...
Start with a tiny sphere. Expand it. The sphere gets bigger, but no point on the 2D-like surface is the centre.
It's like that, but the surface is a 4D spacetime.
You can't use your imagination to visualise this. But you can use math to describe it.
Whether there's some kind of ultradimensional hypercentre is a different question. Even if there was, it wouldn't be accessible from this universe.
And don't forget we don't have a clue what spacetime is. Relativity has some nice descriptions of what it does, but there's no fundamental explanation of how the universe generates the phenomena we call position, time, and distance.
Perhaps some day I'll be able to wrap my head around your closing caveat. I suspect any question I ask on that would be another "where is the center" type question.
The expansion that's occurring is that the space between things is growing. But it's every space between everything all at once, everywhere.
There is no physical equivalent.
Due to the laws of general relativity, extrapolating that observation into the past, the implication is that the matter in the universe was ever denser the further you go into past. Furthermore, there is a point in time where it becomes infinitely dense. This is what we call the Big Bang. Because our equations break down at that point, we don’t know what happened at that point in time, or before, or whether there was a before. However, observational evidence like the cosmic microwave background strongly support the theory: https://en.wikipedia.org/wiki/Big_Bang#Observational_evidenc...
Note that “infinitely dense” does not imply that the universe shrinks to a single point at that time. Rather, it was infinitely dense at every point. (Again, this is like zooming out from an infinite grid infinitely far. The grid always remains infinitely large.) This means that the Big Bang happened everywhere at once. It’s a point in time, not a location.