First buy a breadboard (or more than one). Buy the resistors and wires you need in the near future. You'll need a crimper pretty soon to cut the wires.
A good power supply is nice, but you can make due with a battery holder and battery at first, say, a 9 volt.
Look at some simple projects you can build, and get the capacitors, LEDs, inductors etc. necessary to build them.
When you want to start soldering things together, get a soldering station, solder and stripboards. Also get solder flux . Soldering is much less of a pain when you have flux, depending on what you are doing.
As the comment section here attests - a soldering gun is hot. Solder is hot. You have to be careful.
Digikey.com and Mouser.com have many, many electronic parts. Adafruit.com has a lot of cool projects and guides. In fact, I would recommend you watch some Adafruit videos, read some of their guides etc.
After you learn some of the basics, you can take a step up and get an Arduino and muck with that.