Comment by wyoung2 | Hacker News Reader

Comment by wyoung2 | Hacker News Reader

Parent Full thread

> tests that cannot be run locally

Fossil just got a "patch" command to help with cases like that: https://fossil-scm.org/home/doc/trunk/www/patchcmd.md

Previously, the standard solution was to commit work with potential cross-platform issues to a short-lived experimental branch, so you can check out the branch on all your test machines and try it there before merging it down to trunk.

> I'm not willing to sit through for the whole 1 hour the test suite runs for

Hour-long test suite runs are a problem in and of themselves. That sets the shortest period of time your development organization can react to a test failure. Look to any application of control theory — feedback systems, process control, OODA loops... — for what happens when you have a slow-reacting feedback loop.

One of those "guys who wrote the book" on Continuous Delivery covered the problem well here: https://www.davefarley.net/?p=218

You either need multiple levels of testing so you get some feedback quickly, or you need some sort of parallel build system that breaks the tests up so that the whole set completes in a reasonable amount of time. Without that, you're going to have people either sitting around waiting on test runners or skipping the tests entirely.

> One gets things done

Effort and resources spent on a massively parallel CD buildbot sounds like a good way to get things done to me. Computers are cheap; developer time is expensive.

Awesome, for some reason I couldn't think of the workaround of committing to a temporary branch.

> Hour-long test suite runs are a problem in and of themselves.

I'm not sure I agree. Let's say I have a data integration system that has 70 modules. Each module have some integration tests and some baseline benchmarks.

At the time of development I'm only concerned with the 1 or 2 modules I change whose tests can finish in < 5m easily.

But when I push the change I want to be sure nothing else broke. Maybe because the change was in a base module used as a library by some other modules. Maybe I forgot to run tests on 2 of the dependent modules. So running the entire suite on merges (and optionally on PRs if required) is a necessity in my opinion.

Thanks for the link though - I'll take a look.

> You either need multiple levels of testing so you get some feedback quickly, or you need some sort

> of parallel build system that breaks the tests up so that the whole set completes in a reasonable

> amount of time. Without that, you're going to have people either sitting around waiting on test

> runners or skipping the tests entirely.

100% agreed. With most of these points you can now ignore my previous paragraph. By a 1 hour test suite I mean the wall-clock time of a single build-test cycle on CI runners. i.e. to run ~130 test suites in parallel against various containerised databases and some DBaaS (like maybe Redshift, BigQuery etc.) takes 1 hour where each individual module takes 5 to 10 mins. Some may take longer depending on the depth of things you are testing.

> Effort and resources spent on a massively parallel CD buildbot sounds like a good

> way to get things done to me. Computers are cheap; developer time is expensive.

100% agreed there again. In my opinion the organisations that treat CI improvements as technical debt and not as first class citizens of the software development flow are doomed to move slowly and have a hard time getting things done.

Reply on news.ycombinator.com