The fundamental issue is that papers are often incremental, that is, they build upon some previous work, which is thereby referenced. The issue this creates is that the papers become hard to follow for anyone who is not already familiar with the previous work in the cited papers. Even if someone goes on to read the referenced papers, it often does not help as the problem is recursive.

This keeps on happening till the research becomes old and mature enough for someone to write a book on the topic which then is nearly self-contained (or assumes some prior background like mathematics, or undergrad-level courses, etc.). MOOCs are also helping the situation since the instructors create nearly self-contained courses which often also include very recent research results.

To solve the issue you mention, this basic notion may need to be broken, such that each paper explicitly explains its subject matter without relying too much on the references. Since much research does not go anywhere, this may be a premature optimization really though.

A thing that does bother me, and signifies the extent to which what you say is true is that "talking" to the authors of these papers nearly always delivers those good insights within a few minutes that you would not get after spending an hour on the paper they wrote.