I suspect a lot of it has to do with the type of companies that tend to hire 3rd party recruiters. Namely, startups, and typically early stage startups who don't have any in-house recruiting staff. There's nothing at all wrong with early stage startup companies, except that my experience seems to indicate they don't actually know what they're even looking for. That's in addition to the likelihood that they just haven't been around long enough to develop a structured hiring process.
I don't know. What I will say is that your incentives and a 3rd party recruiter's are not the same. They want to place you quickly and get the commission, regardless if it's the best company or the most competitive offer. You're right that they do want to keep you at least somewhat happy, because if they get a reputation for treating candidates badly, that can be fatal in their profession. But, the people they really want to keep happy are the ones writing the checks. And, in this situation, that ain't you. You are quite literally the product here.
The thing about 3rd parties is they are really useful if you're already on the inside. If you have all the relevant CV points, they will get you interviews. This is the flip side of why people get ghosted by them: if they don't think you'll be easy to place, they won't bother with you, and in fact they think the employers will stop taking their calls if they push the wrong CVs.
Another thing about recruitment is that not everyone is good at it. There's a lot of young ones just out of uni who give it a shot, and suck at it. Badly organised, don't know the business, waste a lot time for a lot of people. I've got a bunch of LinkedIn contacts who are in various unrelated businesses because they started off in recruitment and couldn't bill enough to stay there. If you don't field a lot of calls the ones you do talk to might be these noobs who are looking for leads.
For recruiters within a company, try looking for them on LinkedIn, check out a recent post of theirs to verify if they still work for the company, and reach out with an application if they have an email address, or via LinkedIn messages.
How do you find promising companies in the first place?
But don't apply for the jobs on those portals, that is just a black hole. Just look at the ads, note down the names of the recruiters and firms they work for, and either phone them or message them on LinkedIn.
They'll have a chat with you to find out your situation, and then they will tell you what jobs they've got. Note that every recruiter in the whole world thinks they have a great relationship with Citadel, Millennium, and every other well known employer in the space, so those intros have no value at all to you, since anyone could be your gateway. They'll also have a bunch of less well known firms, so you need more than one recruiter since they don't all know everyone.
Of course LinkedIn is also a way for the recruiters to get in contact with you. Just set yourself to open and wait around, a fish comes every week or so.
That hasn't been my experience. In a startup people often wear multiple hats, so it's not uncommon for CTOs or CEOs to be directly involved in the hiring process. Startups usually list their open positions on their site and screen applications via email, or post on the HN Hiring and other niche job boards. Startups certainly don't want the cost and communication overhead of dealing with recruiters, especially because it costs them much more to make a wrong hire at an early stage.
In contrast, large and mostly technically out-of-touch companies love 3rd party recruiters. They give them a bunch of vague role requirements, and let recruiters do the dirty work for them. If there are problems with filling a role, they can always blame the recruiting agency.
My first job in 1996 was based on a return offer from an internship.
In 2008, I did the random submit my resume everywhere thing and in 2020 an internal recruiter from $BigTech reached out to me.
With remote work being more of a thing combined with my much stronger network and resume, I will probably lean much more on my network and resume the next time I’m looking.