It isn’t just caffeine though. I drink plenty of non-coffee caffeine, with little effect. As soon as I drink coffee, I have to go within 15 minutes.
If I had to guess, it has something to do with coffee's general acidity and individual differences in GI tract microbiomes.
Personal anecdote: For me coffee has a laxative effect which I don't get to the same degree when drinking strong tea or energy drinks with very high caffeine levels.