I don't think it's hard to get into a good hedge fund, if you're in the right location (New York, possibly Boston or Chicago).

If you keep building your tech skills, you can get into a good hedge fund job after a few years. Just treat it the same as interviewing for a good tech job. Hedge funds more selective but also more meritocratic and predictable than Valley firms. The problem in the Valley is that, although it has as many smart people, pound for pound, as Wall Street... the Valley has more stupid people (and fewer smart people) in decision-making roles.

Startups aren't inherently damaging. The problem is that they're volatile and experience constant reorgs. Your job might be totally different in 6 months. Or it might not exist. This leads to a "job hopping" pattern that, from a more conservative finance perspective, makes you look bad. Banks and hedge funds are more stable so it's rare that someone is that unlucky as to have, say, 6 jobs in 5 years through no fault of their own. But it happens all the time in VC-funded startups.

I had a colleague in a past job that went from aeronautical engineering to a hedge fund. His career trajectory was MIT masters in aeronautical engineering -> ~5 years in aerospace -> 2 years at a financial software startup (where I worked) that was somewhat desperate for people -> hedge fund. Basically he could demonstrate a solid grasp of math from his past career, managed to find an entry-level position at a financial software startup (taking a pay cut), parlayed that into some knowledge of finance and industry contacts, and then jumped ship once the opportunity arose.