Compared to software skills those are relatively easy to learn though and they have a longer best-before date than any kind of language/framework kind of knowledge.
Software is deterministic and quite easy to reason about. It either works, or it doesn't. Hardware relies on actual physics, and even minute changes can be the difference between working perfectly fine and not working at all.
A lot of hardware design is based on rules-of-thumb and institutional knowledge. Learning those as a hobbyist is incredibly difficult, and most of the time you essentially end up cargo culting what everyone else is doing - and there is no guarantee that everyone else is doing the right thing either! It is really easy to end up wasting hundreds if not thousands of dollars like this.
This is exactly why companies like Adafruit have become so big. They take care of all the hard part, and provide the hobbyists with essentially a bunch of lego bricks which neatly click together. The only thing you have to do yourself is... the software.
That's probably also why all software is 'bug free' ;)
But seriously: both software and hardware have their unique challenges. But those can be overcome and just like software hardware can be 'unit tested' by breaking down circuitry into manageable chunks. Adafruit is a success simply because they fill a need: the ability to create bespoke gadgets without investing lot of $ or learning a new skill. The market to programmers, not to hardware people, though I'm sure there is some overlap as well due to the convenience. But those skills are not substantially harder than software skills, they are just different.
I'm kind of lucky: I got into software through hardware rather than the other way around. To me software was an infinite parts budget (bounded by RAM limitations, usually). Hardware was a running expense, computing a one-time expense (or so I thought, hah!). So I simply got more mileage out of my pocket money and Saturday job earnings by saving for a computer rather than by spending it on various hardware components.
Good starterpoint: and FPGA evaluation board, such as Digilent's offerings. Those pack enormous power in a tiny setup and will teach you a ton of very valuable skills.
If that looks like a hit you can decide to deepen your knowledge.
Compared to software it is a costly hobby though, and it also occupies more space beyond just a laptop. And it can be quite messy.
As soon as there's a non-trivial analog element - anything frequency-dependent, resonant, exceptionally resistant to RF interference, or switching significant current - you absolutely do need that math.
You can model resonant filters with DSP, but you still need to understand z-plane digital models. It doesn't hurt to have some idea how they relate to s-plane analog models.
Cook-book tinkering is plenty fun, but you really can make things explode or burst into flames if your project is switching and/or carrying any significant load.
Stuff exploding or bursting into flames I've seen exactly once, on one of the most trivial circuits I ever built: a small boost converter for a windmill to charge batteries in low wind conditions. It worked extremely well. Until I disconnected the battery for service and then the boost converter kept on increasing its output voltage until the capacitors let out the magic smoke. Other than that stuff occasionally breaks. Oh, and if you do do RF stuff: beware of RF burns, that is a real risk, coils and capacitors in high power RF circuits should be treated with proper respect.
I'd be much more wary of Lithium-Ion batteries than analog stuff and buck-boost converters are cheaper to source as complete units than to build yourself (though you definitely can if you want). Your typical hobbyist isn't going to start off by building themselves an large inverter or a HVDC interconnect. They're going to build amplifiers, other audio gear and maybe some measuring kit or digital devices. Sound generators, function generators and so on.
By the time you reach the stage where you need to design a resonant LC circuit you'll have picked up a lot of working knowledge and some of that will tell you what bits to avoid and what bits you can probably handle.
I know plenty of HAMs that know enough math to be dangerous but they usually would not be able to do really complex stuff without access to tools (though I also know some HAMs that definitely would be able to do really complex stuff, they also have the corresponding higher level license).
Let's not pretend that everybody that builds electronics for hobby purposes is a math wizard, it just isn't true. Though it definitely doesn't hurt to have a basic understanding of RC and LC circuitry and to understand how to use op amps and other interesting components like that. Applying those is vastly different from designing them from scratch.
Also: quite a few people have a ton of fun just building kits and slowly expanding their knowledge and there is absolutely nothing wrong with that. At the highest levels you will need that math, but there is plenty of interesting stuff to be done lower on the ladder. HN is the last place where I would expect such gatekeeping.