Thrust was definitely also a problem, a glider is not particularly useful unless it has a huge lift-to-drag ratio, which is only possible with modern materials and a solid understanding of airfoil design, which is a whole other can of worms.
Even things that seem so basic that we don't even think about them, like high were not at all obvious: just look at Sir George Cayley's gliders.
It's like fire, super obvious in retrospective, and yet took thousands of years to become a technology.
... and they were right, because aerodynamics is not scale-invariant.
https://www.youtube.com/watch?v=lu4V9zn0wZ4
Also highly recommend his previous video about the design of the Rafale, it's basically a lecture in plane aerodynamics.
Though, they might have invented some kind of functional glider and used a steam powered winch. But, I'm guessing the aero-dynamics weren't that widely understood yet. And making stiff lightweight air-frames also took some time to happen. The contraption that the Wright brothers flew was a bit flimsy. And it barely flew. It was more of a proof of concept. Things like thermals and other updrafts that make gliders work weren't that well studied probably.
This video has a nice delineation of the collection of breakthroughs needed:
https://youtu.be/EkpQAGQiv4Q?t=391As mcarmichael said https://youtu.be/EkpQAGQiv4Q?t=391
Although the Wright brothers deservedly get a lot of credit, I think they made progress once they enlisted the help of Curtis due to his knowledge of lightweight internal combustion engines.
The book Bird Men is a god read on this subject.
Thanks for that recommendation, I'll add that to my book reading list!