There’s plenty of free synth software. If you really want to get into it, but not be restricted by costs, check out something like VCV Rack

It definitely can, depending on what you are looking at. Something like a fully analog, polyphonic Jupiter-8 that still works 40 years after launch? Easily $20k+. A simple monophonic "learner" synth usually has one knob per function so it's easier to learn how synthesis works at a fundamental level, so a lot of people start there.

These are usually also priced for beginners, more or less, where a used Arturia Microbrute is about $200 and a used Korg Monologue is about $250. Then you can start getting into a lot of different design philosophies for everything in the middle. There are also a lot of digital hardware synths which tend to use frequency modulation so they can make some pretty crazy digital sounds.

If you go completely mad with the power, people love building eurorack modular synths one module at a time. It can end up being a really custom rig you can kind of "play" but of course, this is on the expensive end as well.

You can get a used midi keyboard for 20$ or an ok one for 100$ new. All the software is free. No better time than right now.

for hardware, check out Behringer. They have made cheap clones of classic synths along with a couple of their own creations.