If you want to learn to create sounds from scratch then maybe you should try a virtual modular synthesizer like VCVRack. It allows you to connect oscillators, filters, envelope generators together with patch cables.
If you want lower-level stuff and program a synth manually, then you can try PureData, but it might be too low-level for you and require dealing with math equations and algorithms. Also, its GUI is a little outdated.
If you want to know how certain sounds are created, you can look for tutorials or Youtube videos. Or you can take any virtual synthesizer, choose the preset you like and see how it's made. You can disable units one by one and see how it changes the sound. For example, you can take Dexed or Surge, both of which are free.
Of course, beforehands you have to learn the basics (what kind of waves are there, what is a spectrum, an envelope, how filters work and so on).
Regarding music theory, I think that tracker or 8-bit music uses standard notes and chords and you don't need any special music theory for it. Also if you can hear the notes then it will help you a lot. A good ear is important because it helps you to hear what is wrong with the sound and how to fix it.
Also, a desktop DAW like Ableton (which is recommended by neighbour comment) might be easier to use than a tracker.
Also, you can listen to the melodies you like and reverse-engineer them, if they are available in a tracker or MIDI format.