This is an okay and mostly correct introduction for anyone who has never heard of CI, but lacks any sort of depth. The author sounds like he just discovered the concept and wants to share it, but lacks practical experience.
The original poster is a cofounder of Semaphore.
E.g. one of our own builds took longer than 10mins at some point. You just assume that it's a function of size of the code and it's normal to rise over time. So my goal is to share that idea & why I think it matters, and get feedback. :)
It would be perfect if one could somehow keep the build quick using tools. A hard limit on a build might not be the best way. If the test is taking Limit minus one millisecond, and I add one new test that takes one millisecond, I break the build. But the culprit was really commit yesterday that added a test that takes 9 minutes. The worst thing one can do in CI is somehow flag the build as "not good enough" and point the blame the wrong way.
A tool that treated test quality/perf like any other asset and fails builds based on time limits etc would be perfect. It's really hard to do that especially on shared CI servers because of fluctuating performance.
I'm really not trying to crap on your decisions. I just want to know the rationale behind it. There are projects where everything is written in house with 10K lines of code. There are behemoths with 500k+ LoC and numerous dependencies. How does this factor into the decision?
But if we have a rule of thumb, we can know when it's time to start thinking about optimizations.
Code size is certainly a factor but it doesn't have to imply constant time increase. Speaking from experience in web apps, optimization of setup (whatever needs to happen before first test runs) + more parallelism + thinking more about decoupling is usually the right answer.
The point of these discussions is mostly to help people avoid ending up with a single 500k+ LoC behemoth.
And that was two years ago...