What Feynman does (where this confusion comes from) is that you can look at discrete wave packets (i.e. photons) and the math comes out the right way for the primary wave if you assume that only some of these wave packets get phase-shifted, and add all elementary waves together afterwards.
But still, it's photons as "wave packets" that influence the whole system, not photons as independent particles that either bounce on something or don't.
I don't think that can be strictly true. No matter how dense the material is some photons have a chance to get through unimpeded through something like tunelling. Practically unlikely but mathematically possible.
But still I think saying "the primary wave gets absolutely and completely delayed" is not helpful. Using a wave description as in Feynman's lecture[1] is more enlightening: the incoming wave travels at "full speed" through the medium, but doing so it interacts with atoms such that they emit an additional wave, and the sum is a slower wave.
You can say it's the same since there's only one electric field in space and so the only "real wave" is the sum of all effects. But I find it quite helpful to think that one of the components in this sum is the original wave traveling at the speed of light in vacuum, also in the space occupied by the medium.