There are eight different causes that manifest in the observed "invisible wall" symptoms.
I find the level of detail given appropriate, and consider it as a valuable insight into game engine design as it was appropriate for the time.
This video does a wonderful job of illustrating how a handful of underlying issues can combine and manifest in so many different ways, and this was only the tip of the iceberg of 3D game physics glitches. And some of these issues are things games can still fall prey to today.
Essentially, due to the technique used for collision detection, tiny gaps in the geometry causes collision system to treat parts of the scene as having very thin walls that stretch to the top of the play space. Depending on if Mario's quarter steps align with this little gap means that a collision can be made despite it not being visible. Also explains why they impact Mario sometimes and not others.
Like throwing a marble through a mesh fence but you cannot see the fence. Odds are you will make it through but it is essentially chance.