What different routes are there to publish research besides academia? I would love to work on publications but it is not practical for me to return to an institution right now.
It's hard, there are a lot of i to dot and t to cross (and remember the j and f too).
The paper must have the correct length, a relevant introduction, conclusions and a lot of other form stuff, detailed but not too much detailed, the correct references, ...
And you must solve an interesting problem. The famous problems are too hard, a lot of people already tied to solve them. And "interesting" means what the community consider to be interesting, that sometimes it's what is interesting for the application and sometimes is something completely different.
And must be correct, in my university we got a few independent people (5?) claiming to have solved the Goldbach conjeture a few years ago. Unsurprisingly all had big mistakes.
My suggestion is to partner with someone already publishing math papers. Do you have a friend that is mathematician? Do you have a friend that has a friend that is mathematician? If your friends work in an uninteresting topic, you can ask for help to contact someone else that is working in something you like.
Be aware that there are abusive people, but most horror cases I heard are about experimental physics where a few extra hands are more helpful, mathematician usually don't lock people in the basement until they finish the experiments.
Once you understated how to publish, which journals are good, which journals are run by idiots, you can try to publish independently.
Research is not exclusive to academia.