I think the name "big bang" does the idea a disservice. It's not an explosion moving out from a central point.
My popular-science understanding of this goes like this:
So Einstein has this cool theory where space and time don't work like we think they do - they turn out to be a unified geometry: spacetime.
Since we are very small and move very very slowly[1], our intuition for how spacetime works is way off. It works a lot different from what our slow experience intuitions suggest. And the pictures we draw ourselves in our imagination when we imagine all of space are way off.
Space can have a curvature [2]. The curvature basically means that things moving in the straightest lines that they can move closer together or further apart. This turns out to describe gravity well.
The mathematics is thorny, but when we make the simplest possible models of the whole universe (sort of like an ant modelling the earth as a sphere because he knows it has a curved surface) we find that the entire space has a beginning and that the space part of it has been growing larger since then.
So it's like a 2d ant living on an inflating balloon. At some point the balloon was a small "point" and everything on the surface of the balloon is busy "moving away" from everything else as the balloon expands.
The big bang is what we call the small hot part near the beginning when space was very small (even in our terms of experience)
[1] Natural speeds are fractions of the speed of light and natural distances are distances related to how far light travels in a unit of time suited to our perception.
So, to the moon is a good kind of distance if you live at a second kind of level of perception, and around the earth about 8 times a second is a natural speed.
[2](without being embedded in a bigger flat space)