Quake (idTech) is probably one of the earliest examples of a moddable game. All that it did was load your module (DLL/SO) via a well-defined export, passing it a structure with pointers to various important fields/functions. The SDK shipped with a few header files (or one, can't remember) which is all that you needed. No source code was required.

Making a moddable game doesn't require IP disclosure. All that it takes is a bit of discipline around impersonating the 3rd-party persona (dogfooding). It's not hard building a mod framework when compared to sticking to that framework when crunch-time hits. I'm guessing that's one of the reasons that games these days typically lack modability - during crunch time you really don't want developers "wasting time" making sure that the API is clean and used diligently.

From my experience, modding implies a lot of experimentation, you're often playing with unknown rules as there are very often big gaps in the documentation and you only have other mods (Which very often make their own errors) as examples. An api to an open source game must be much easier to understand.