Just having your CI steps run scripts that install/build/deploy/??? already goes a long way to decoupling from your CI solution, and can be a lot faster than spinning up a custom image for your CI build. Custom images don't necessairly do much for fixing the last 10% either - I'll typically want CI-specific:
1. Build matricies. I was running windows builds on appveyor and linux builds on travis for a long time (faster builds / iteration / feedback.) Regression testing against multiple rustc versions can be done from within a script, but doing so via CI-configuration makes for easier to read CI results that flag individual builds as failed instead of an entire script that you have to go log diving into.
2. Custom runners. Be it custom hardware or licensing-burdened custom software, I frequently need to own the actual CI hardware, and using the native solutions for CI runners is typically easier than merely configuring whatever custom infrastructure I might come up with.
3. Images. Using prewarmed cloud-ci-specific images is often faster to spin up.