Since Jetpack code is also what we run on WP.com (tens of billions of pageviews) it goes through a huge amount of performance tuning and optimization. The way modules work when you turn them off they don't have any overhead, similar to turning off a plugin. If you were using literally one thing it might feel like a lot, but as soon as you use 2-3+ things Jetpack does it's a lot more efficient than separate individual plugins to accomplish the same task.

It's not like features you don't use slow you down, or that the < 1 MB of code they take on disk is important.