There's no conflict between using Docker for CI (and deployment) and not using Docker for development.
> (fragile?)
The Docker image isn't fragile, it's your software that risks becoming fragile if it's too strongly reliant on a specific environment.