Yes, the initialization is important, but it'll eventually converge (well, maybe not guaranteed with networks). If you initialize too high then it'll err on the side of exploration, and if too low then it'll err on the side of exploitation. These are called optimistic and pessimistic initialization.

The basic (non-deep) Q-Learning approach, is to have a big table of every state-action and it's expected reward. The formula you've seen (the Bellman equation) is for updating these.

Using big tables to store Q values sucks though, so instead of using a table, you use a neural network.