It's easier to understand by considering finite horizon problems. With finite horizon problem, the target value for the last time-step will exactly be the expected received reward. So your network learn to approximate it and use this approximation for computing the approximation at last time-step - 1. So you construct the value from the end (in a "dynamic programming" kind of way).

But the bellman equation works just the same with infinite horizon problem. The weights converges all simultaneously to make the network approximation solve the equation.

A great resource to understand RL is openAI's spinning up.