All I can say is I hate when projects just lazily list every pull request for its change log / release notes. It makes it very difficult to discover what breaking changes there are.
I am also planning to add some PR templates so contributors include the context up front, which should make any release note generation more accurate.
Are you using any tooling to help with changelog curration? I know towncrier is all about fragments, so contributors must had write a brief summary of their contribution, which would be more in-line with your preference.
One thing I am trying to do is make the generator surface breaking/migration items explicitly, but I still think anything that requires human judgment (migration steps, caveats) should be hand-curated in a dedicated document like you suggested.