I only figured this out after building and tearing down my prototype and reassembling each of the modular bits, verifying they worked, and then noticing it didn't work after integration until I removed the LEDs. I have since learned I also could have used a buffer driver or LED driver, alkthough that adds more complexity to my simple prototype.
(in case you're wondering, yes, I know you're supposed to use a current limiting resistor, but I've also observed that my 5mm LEDs work just fine when given regulated 5V, they end up dropping 4.6V and consuming 40mA, which is about double the current they are rated for.)