Slow builds locally, but last I tried cloud build it was about the same.
Are you switching platforms between builds locally? (like building one for web / server / pc / mac etc). Platform switching essentially blows away the cache, making it take the full build time each time. There was an asset years back that would create folders for each platform's Library/ cache and switch between them when you switched build platforms.
In CI (eg using https://game.ci) you can prefix build cache keys with the platform to manage that automatically.
Unity Cloud Build is in general slower/more flakey/sometimes has cache issues compared to DIY CI, but it should speed up after a successful first build as long as it caches the Library/ folder, and they do separate that out by build platform properly. (Worth checking each build config setting is set to cache that)
Server Builds are super quick, it's the Unity WebGL Client builds that take forever even if I have cache. Although I have not migrated to Unity 6 yet, that's actually next on my to-dos which I believe there are lots of improvements for WebGL builds there. So I'm hopeful for better dev experience there.
I never knew about game.ci seems like it could be very helpful! Going to check that out... thank you!!