Original author here. This is indeed a bit confusing.

You are right for the case where Firefox's PDF.js is used (local or remote file in a tab or iframe). The XSS problem however is with web-applications that themselves use PDF.js. In that case, it does not run in a separate or special origin; that is a Firefox thing.

You are also right that the PDF format supports JavaScript, but that is something unrelated to this, and indeed highly sandboxed in all cases.