At some point, though, I noticed that "something" [0] still managed to "get out".
After running some packet captures, it became clear what was going on. Although the device was using the network settings that I had manually configured, I had not specified a default gateway. The device decided it would use DHCP to discover the default gateway for the network and began automatically using it so it could get out to the Internet.
Since then, I've started specifying a default gateway for any devices that I don't want to get out. I give 'em an IP address that isn't in use on the network and, fortunately, I haven't ran into any other instances of crap like this happening.
[0]: I really wish I could remember what device this was but it's been a long time ago and I really have no idea, sorry.