Nowadays, you will most likely never need to recompile your kernel. At least not if you are doing "normal" stuff.
Let me give you an example of what kind of stuff you will come head to head with:
Recently I was installing and configuring a Riak cluster. I wanted to configure it to use multi backend with a large partition ring. What these are is not essential to our discussion here. What is essential is that I hit the riak user's limit for simultaneously open files. This is a thing controlled by the kernel, but configurable in userspace. If you are interested, you can find how to raise the limit by using Google.
This is the kind of "kernel tweaking" one usually needs to do. 1. How to make it work 2. how to make it persistent after server restart 3. WTF?! it's not the same in debian/ubuntu and Red Hat (half of the solutions I found don't work)
But the harder part is not finding the solution. The harder part is finding out what is the problem. In order to do so, you need to have experience. Know where are the logs, know which logs to look at. Know how different systems behave under such conditions (in my case, Erlang VM). Be proficient with shell tools for diagnostics. Be able to use emacs or vi(m) on the spot (you have no graphical environment) etc.
And there is no easy way in obtaining this skills. The only way is to get your hands dirty. Very dirty. Build systems, break systems, fix systems. You can't be good by just following tutorials, nor by using a sandboxed environment like a VPS.