basically a judgment call by the person in charge of a journal that the paper is not interesting or impactful enough to warrant going through with the rest of the review/publishing process
on the other hand you have the pay-to-publish journals that have a financial incentive to push as many papers through peer review -- these thrive on sub-par papers that are technically just barely 'good enough', but the upside is that the real good ones will also make it through. however, they inevitably face reviewer fatigue, and the most valuable ones will quit reviewing if they often send them low-quality papers. so basically once in a while they'll publish top notch research without being aware of it.
i'm not aware of any middle-ground solutions out there and it certainly feels like a tough problem to solve.
Imagine a world where all papers start their life as preprints. Researchers read these preprints (like arxiv) and comment on them (like PubPeer). Journal editors then search for the best papers (or hear via word of mouth within their field) and journals compete for the rights to publish the most interesting papers. When an agreement is reached, the journal organised reviewers and asks for any changes to the paper.
I may be missing something critical but I dream of a day when the actual people producing academic output have power over journals who produce nothing but fees.
Historically journal controlled peer review didn’t exist and science still progressed perfectly well.
I do suspect that this would strengthen the winner-takes-all dynamics in academic publishing. Like whoever has the biggest twitter following gets all the attention, then gets Nature and Science competing for their paper, which allows them to get funding to hire a publicist, who increases their profile.
In my utopia, publication success would be driven by interest within a field which means lesser known researchers who haven't had breaks in high impact journals are more likely to break through, because the community acts as the gatekeepers, not a small group of journal editors.
Like a recruiter rejecting screening you out without an interview.