I'd also look into Buildroot [0]. I'm currently using Buildroot for a project and the whole distribution (including libraries and executables) is under 40MB. Since I am using a fixed dev board (i.e. peripherals are not going to change), I used lsmod to detect which drivers are needed and only build those, which really shrinks the kernel.
Buildroot also includes a cross compiler, so that you can rebuild the entire toolchain, kernel, and libraries in one go.