Could it be psychological?

The fact that caffeine doesn't seem to play a role, it looks like the act/ritual of drinking coffee and probably expecting some effect from it has a psychological impact?

You could always run a double-blind experiment on yourself and find some objective ways to report nervousness (purely a state of mind or is it translated into physical factors, like heart rate or blood pressure?).