You can do that by building it with only the modules you want: https://github.com/jquery/jquery/blob/master/README.md

There's also an official slim build without ajax and effect modules.