Your open source work would fall under COTS with the expansion of "commercial" to cover open source works. It's something that is supplied from outside the government and not explicitly commissioned for the government.
GOTS might be something like a timekeeping solution that a government organization can opt to use (among various other options) that was produced for or by the government and now "sold" or distributed as a reusable system for government use.
Custom would be if they came to you to build something for their specific needs, but wasn't already a COTS/GOTS product (that isn't to say they won't take COTS/GOTS and customize it, though). Think of this as a "one-off" (or often starts that way). These might turn into COTS or GOTS products later on depending on who "owns" the system and its ability to be reused.
There are two types of commercial software: Open Source Software and Closed Source Software.
This is intentional. The US government wants to be able to use software that is available in the commercial (non-government) world.
For more info, see my paper "Open Source Software is Commercial": https://csiac.org/articles/open-source-software-is-commercia...