Because there's only one particle that we can observe going through one slit. Cover up the other slit that nothing is going through and there's no interference.
Edit: I think I understand what you're saying. The field isn't separate from the particle - the particle is the quantized field itself.