I bet it's different when you're there. When you're at a fun event, the crowd has been warmed up, and everybody laughs, it's much easier to laugh then when you're home alone.
I only watch him on Youtube, but Colbert's audience comes across as quite genuine. Sometimes he doesn't get quite the reaction he was expecting and reacts to that. And he's genuinely funny, so the audience doesn't need much prodding.
I find it interesting to hear that he's quiet when the cameras are off. Trevor Noah also puts some "between the scenes" bits on Youtube where he banters with the audience. Not full of scripted jokes like the regular bits, but frequently funny, and he seems quite genuine.