Hacker News Reader

Comment by dan353hehe | Hacker News Reader

Agree with this so much. One of the main things that I try to do is ensure that the turn around time between testing and writing code is as short as possible.

As an example I inherited a codebase about 2 years ago that would take 15 minutes to run the test suite. It was painful. It was just long enough that it would waste an entire day just trying to implement a simple feature. The project was behind schedule and no one wanted to work on it because it just took too long to do anything.

So I spent a few weeks running benchmarks on the tests and figuring out where the slow down was. I did a bunch of different things, reusing SQL tables, only truncating tables ones that were used, refactored how the tests worked, removed some code so that docker was not needed to do the testing, etc.

I got those tests running in 3 seconds down from 15 minutes. And that changed the whole velocity of the project.

Always sharpen your ax.

View on HN

Absolutely. What I find astonishing is how undervalued test feedback time is.

I don't know what drives that, but in most jobs I've had to fight to do the right thing.

Just recently I've made some optimizations to get a functional test suite from 65 minutes down to 3.6. Parallelism is my favorite lever, as it scales so well, and core count is still going up!

Probably something to do with that XKCD about waiting for compiling, "The #1 programmer excuse for legitimately slacking off": https://xkcd.com/303/

Running tests is the same type of thing!

Another much under-utilized way to speed up test suites: delete tests

I understand the underlying message but I prefer the equivalent following phrasing: "check test suite coherence".

It moves the deletion as a means towards the coherence rather than as an end in and of itself, and also includes refactoring as an option.

I hope it helps the less experienced developers grab the intention more explicitly.

I think the core point isn't about coherence, but tradeoffs. Some tests are not worth holding onto: ones that are slow, flaky, and test code that is of marginal importance. In those cases, the ideal tradeoff can be to delete the test. Perhaps to rewrite a better one, or perhaps to just 'eat' the risk incurred by removing it in favor of improved iteration times and the other long term benefits of not having to maintain it.

As always, mentioning that Your Mileage May Vary (YMMV) puts the emphasis/decision on the people facing this or that situation rather than blindly applying advice found on the Internet.

And rightfully so :)

This is being down-voted, but there is merit to this. Tests should be audited for continued need.

Can one configure tests to run in “full”, “actual” and “scope:<modname>” modes?

Also just deleting dead code and tests that go along with it

Tend to get a boost in application boot/startup time, too

A few more things I'd add

The OP mentioned using tests instead of the entire game. You could also consider splitting the tests. The code base I'm on now has 9 test suites. It would be much slower if all 9 were merged into 1. So, if you have a monolithic test suite consider breaking it apart.

Our team also has a CI/CQ so I never run all the tests myself. I run the tests I think my change affects. When it passes I upload to the CQ and let it run all the tests. Then I go work on something else and check later if it all passd.

In games there is often no need to run on the target hardware for 95% of tasks so choose the hardware that's fastest. In other words, if you're making a game that can run on PC, PS5, XBox then develop on PC and only switch to PS5,XBox when you have to (platform specific features, checking perf, etc.). For VR for example I'd test via PC on a link cable or on a Vive/Index/Rift and only test on standalone Quest when I absolutely had to. So much time saved.

The same is true in other places. I work a very large project that run on many platforms (Windows, Mac, Linux, Android). Building and testing on Linux is 10x faster than Windows, 40x faster than Mac. Once I found that out I switched to Linux for my day to day work and only pull out the other devices if the stuff I'm working on is platform specific.

How much time did removing docker shave off?

Docker helped me to significantly increase unit test speed on one project. Each test was recreating database, run dozens of DDL scripts over and over (to ensure clean environment). I reimplemented in in a way that DDL scripts were run once, then container with database was commited to an image and that image was re-used for every test. Also multiple containers were run at once, so tests could be run simultaneously. Docker is a miracle technology sometimes.

Normally you'd open a transaction, run the unit test, then roll back the transaction.

That makes each test start at the same, clean state.

That's not normal, because the test is now not testing the code in its eventual context (which is : the transaction commits). In my experience unfortunately this is a very common pattern (tests are far removed from "reality").

Only if your database lacks the tools to do it effectively. "The transaction commits" isn't exclusive with "and you can roll back to before it started".

Would it be possible to cp /var/lib/your-database /var/lib/your-database-clean after it's been set up, then after each test just overwrite /var/lib/your-database again? I'm sure that the database will need to be properly shut down and restarted, but that is far less expensive than rebuilding a large database each time.

Disk is cheap. CPU and memory are the expensive assets.

1. That wouldn't work if code under test uses multiple transactions.

2. Rolling back a transaction is not exactly free.

3. Does not allow running multiple test simultaneously.

> 3. Does not allow running multiple test simultaneously.

It does with almost every form of transaction isolation mode. Read uncommitted in Postgres still behaves as read committed, so in Postgres you're never missing the mark here.

> 1. That wouldn't work if code under test uses multiple transactions.

A lot of abstractions interpret nested transactions as save points for this reason.

It's not the only valid pattern, but it is a consistently behaving one that's straightforward to implement and gets you most of the UX you want from integration test design.

I still think that tests would be flaky without complete isolation in a different database instances. Transaction isolation is nice, but for serial isolation you would get errors when you can't commit that transaction and you'll need to repeat the test (which might cause further issues) and for lesser isolation levels you would get some phantom reads which might manifest as a rare test failures which is a terrible thing, because you can't reproduce it.

That's the best part - tests never commit data. Everything that happens is within the transaction, and only visible to that test.

It doesn't let you test interactions of parallelized code doing independent/interacting database workloads, but most web/CRUD apps don't do that.

For some reason not many people like the "maintenance" type of work.

Maintenance type work tends to be chasing issues you don't personally care about in increasingly arcane edge cases or trying to intuit why someone else, who has moved on to something that sounds way more exciting than this, came up with a specific solution.

Even more so in a lot of orgs maintenance work gets hit by the "cost center" mentality and management only rewards not hearing about it.

Not full time, but documenting interfaces, speeding up tests, generating stubs and harnesses, profiling pain points - all these I find to be meditative tasks, and I do them when I'm not 100%.

I can do about 4-5 hours a day of "difficult" programming, but then I can easily do another 4-5 hours of this maintenance type work, and the cumulative effects of repeated maintenance time keeps everything running super smoothly.

I'd add that the cumulative effects of always knowing that the maintenance backlog is under control and that you always have time for the maintenance makes it much easier to consistently deliver the 4-5 hours of "difficult" programming.

The reason is simple. Maintenance work doesn't progress your career.

Why bother?

That's not the reason. The work itself is not fun.

Even if the whole project has no career purpose, it's more fun to write new code.

It can be fun, it just depends on who you are. Sadly most developers prefer to follow PM and churn in features instead of this precious work.

I actually enjoyed a maintenance project I was on. There was less time pressure. Finding and fixing a bug is a logic puzzle in its own right.

So things can be largely a question of perspective.

I actually love chasing down other people's bugs. I often get to fix other developers new shiny features shortly after they've hit production. And I love the challenge.

It's also nice that bug chasing has less of an expectation of giving good times estimates, which the software industry is famously poor at anyway.

Well, if it were fun, it would lose its shine if you're constantly being told it doesn't matter to the business.

Reply on news.ycombinator.com