Hacker News Reader

I wouldn't call your second case unit testing, but rather integration testing.

Part of the point of unit testing is that you keep the 'unit' under test to be as small as possible so you don't have these sorts of problems and can run tests as frequently as is useful. It doesn't remove the need for other types of testing suites, but they can be isolated and run at separate times.

I think the real point is that "testing in production," more commonly known as "monitoring," has to happen anyway. So early on, if you have a choice between adding better, more complete monitoring and adding tests, even integration ones, the monitoring will prove more helpful once you deploy than the tests were. Don't get me wrong, it sucks having to manually perform integration tests but when a product resembles a website and is semi-automatically deployed (e.g. has staging servers with automatic deployment, rapid rollback deploys or a canary-server deployment model, a few minutes of caching and few to no expectations that the app operates perfectly as long as it can be fixed within half a day, well, that's a very different project. In that environment, the piece of mind monitoring provides (no client errors in production) largely outweighs having no monitoring, just tests (did I test everything? What about (obscure government agency) using (outdated version of IE) who is reporting that bug?). Of course, as code is more complex than a website, or where you begin to have significant JavaScript, having at minimum tests for success-paths will help prevent regressions for application-wide changes, e.g. adding PJAX to a site and missing an onload. It's just harder to justify tests until you really need them on some assignments.

>Now all of a sudden you're administrating a pool of VMs with different OSes, writing web-scrapers, launching web servers every test cycle, waiting for tests of pages you haven't changed, running n tests * m pages * o server configurations * p clients, and all of a sudden you're spending more time keeping your tests working, or changing your software so tests run faster, and you're hardly spending any time working on features users want.

You're right - it eats a lot of time and it shouldn't. It's not as easy as unit testing and it should be.

I actually wrote an open source testing framework to handle this type of thing precisely because I was getting so frustrated with having to rebuild the boilerplate code to do this everywhere I went.

This is a simple example test I made with it - it uses a browser (w. selenium), libfaketime, and a mock SMTP server to test a django/celery/postgres/redis app:

https://github.com/crdoconnor/django-remindme/blob/tests/tes...

And this is the harness - it runs 5 services and a browser in 50 lines of code:

https://github.com/crdoconnor/django-remindme/blob/tests/tes...

The framework is currently in pre-alpha now, but it should be a bit more stable in a few weeks. It should be usable to test virtually any kind of software that runs on UNIX, also (not just python apps).

Easily handling the "m pages * o server configurations * p clients" problem is on the roadmap. Different browsers and server configurations will be treated as "just another kind of fixture", and it will be easy to feed fixture groups so that you can run the same automated test under multiple scenarios.

Your second case there is still relatively easy to test!

Consider the problem of intersecting two arbitrary 3D surfaces to get a curve. There are an infinite number of possible inputs. Except for rare simple cases, for each pair of inputs there are an infinite number of valid approximate solution curves AND an infinite number of ways to express each of those curves. Also, you want solutions which are well-behaved -- no kinks, no singularities, etc -- and as simple as reasonably possible (but no simpler).

And even that's still better than something like MP3 encoding or digital musical instrument UI testing, where the quality of the result is arguably subjective...