First of all — no, it's an attention problem. I've used APC40 in a live performance once, but in live perfomance (where you play your own music in more or less pre-determined fashion), unlike the DJ set (where you play other people's music, changing the playlist dynamically according to the crowd), it's easier to manage. Apart from monitoring the crowd, searching the next track, marking it and pre-listening (because unless it's in the hundred or two hundreds of tracks you know by heart, you may not know what's the best point cue it it and how), playing with FX, letting the photographer park his backpack under your table, answering to some random question by the manager, you also should have some amount of your attention on reserve, so that when some unforeseeable shit happens, you won't butcher up the mix and people stay entertained.
Regardless, if you already want to group the "drummish" stems, then why would you want to deal with the separately in the first place?