The problem was that NPAPI was an API for designing libraries that could be linked into the browser and run as if they were the browser's own code.

Fixing security issues when they stem from the browser running arbitrary machine code directly in its address space is several grad-student-years worth of computer science theory and may very well grind up against known-unsolvable problems like the halting problem. Purely hypothetically, you could approach it via solutions such as running a virtual machine inside the browser, but now your browser is actually a virtual machine that happens to connect to the internet. As if browsers weren't already heavyweight and complicated enough. ;) In practice, Firefox and the other browsers on the market didn't do that, and sometimes plug-in code would just crash and take your whole browser down with it. No fun, no fun at all.

In contrast, extensions run on top of the JavaScript sandbox and can only interface to the browser via the JavaScript-accessible API. The whole extensions framework piggybacks on the security work already done to allow untrusted JavaScript code from arbitrary websites to run in the browser.

Not an expert on this part, but I'd imagine that problem isn't having an API, it's more that running native executables is the problem if you can't sandbox them or limit their resource consumption. Also, relative to running Javascript, their base resource utilization per tab is significantly worse (especially for Java or .NET-based ones that have a base resource utilization from the VM)

Security issues is what everyone says is the reason to get rid of plugins. But the tinfoil hat wearer in me says this is just a means to remove more control from the user.

First they "removed" HTTP from most of the web so you couldn't trivially intercept text data flowing between the server and your browser window. Then they removed the ability to run code natively in the browser process that allowed the possibility to alter that data securely whilst maintaining HTTPS security. Next up, things like certificate pinning to prevent you as the user from tampering with the data they want your browser to show you. Then DRM will be added to the browser to start preventing you from intercepting the flow of data to your screen.

This is all happening in slow motion over decade long timescales. But some of us are starting to see how the "endgame" is aligning with the various bits they're pushing slowly over time in small increments.