As one becomes more senior the hiring manager expects a candidate to do their due diligence and ask some questions beforehand rather than hopping right into the application process. For a referral, that has already happened somewhat as the candidate already presumably talked to the the referrer about working at that company. You don't have to be a referral but a hiring manager expects you to do this due diligence. You don't necessarily need to go through a friend and you don't have to work with your friends. However, having an acquaintance at the company even in a different team is a massive help.
Think of it from the company's perspective for a second. They don't want somebody who just "wants a job". Money is always a factor but if somebody takes a job solely for money then they'll jump ship the moment somebody offers more. Companies want somebody interested in their space who wants to work there. You mentioned that you purposefully pursued some roles that looked like they'd be good fit for you. That's smart! Make sure the hiring manager knows it. By just clicking apply on the website, that may not be communicated clearly.
Here's two concrete pieces of advice:
- Before going in the front door and applying for roles, reach out to the hiring manager and ask them some more details about the role and what working for the company/team is like. If you know somebody at the company who can put you in touch (even if it isn't the same team), that's better, but even if you don't that's ok. They will talk to you and if they don't you probably don't want to work there anyway. LinkedIn is useful for this.
- If you're applying for a local job, try to meet the hiring manager or somebody else at the company in person if possible. Don't stalk them but to give you an example, I know that a CEO for a local security startup runs the local OWASP meetup and I told a person interested in working in security to go to the meetup and talk to him if she was serious about working there. I've been a hiring manager before and I looked way more closely at any candidate I talked to in person. Don't sound desperate but show them that you're interested and serious enough to go out of your way to talk to them. I had a couple candidates who I didn't know reach out to me via Meetup.com for roles posted to the company website and I had no problem meeting them for coffee.
On a related tangent, I help organize a local meetup and I get asked the question "how do I land my first dev job" a lot. I've helped half a dozen people land their 1st or 2nd tech job. Assuming they already have tech talent, I essentially offer them the above advice. It works at that level but it works for senior roles too! As a hiring manager, you want to talk to qualified people about your open role.
Good luck!
Edit: formatting only