Why No One Uses Your Test Framework | Hacker News Reader

Why No One Uses Your Test Framework

slideshare.net

38 pointsraymondh23 comments

My pet reason: Your framework doesn't solve any of the genuinely hard problems that annoy us when we're trying to test stuff.

Lots of people have written variations on assertions and mocks. But where's the full-stack test harness for a Rails app with heavy JS usage, that runs at a reasonable speed, gives us useful traces on the client and server, and doesn't break when I slightly rework my HTML?

Or, where's the tool for testing integration with REST services outside your control, that doesn't depend on the real service most of the time, but can be told to run against the real service as a sanity check when you want to? (Like Ruby's VCR, but better.)

Or, how about a test framework that accurately emulates mobile browsers and automatically?

I'm not getting on anyone's case for not having written this stuff. Because hey, I'm just as guilty of not solving these problems as is the rest of the community. But if you want to make a testing tool that people will absolutely love, solve some of these problems really elegantly. (I know, easier said than done. And yes, some tools have solved some of these problems to some extent. But I'd consider each of my examples to be an ongoing pain point for developers.)

> But where's the full-stack test harness for a Rails app with heavy JS usage, that runs at a reasonable speed, gives us useful traces on the client and server, and doesn't break when I slightly rework my HTML?

If you're talking Python, Lettuce does this is Python, there also is Splinter. You can run selenium with a headless firefox quite easily with Xvfb.

If you're talking rails, Capybara has drivers for Webkit (QI), selenium and even PhantomJS. Their speed is variable, but it's not too bad.

On the "HTML changes break my tests," I'd argue that A) you should write your front end tests not tied to the HTML if at all possible. Refer to elements by semantic names, not XPath or CSS selectors. And be smart about your steps. Make your submit_form() method actually look for a submit button in more than one way. B) you shouldn't have more than a handful of full stack frond end tests. They're hard to write, slow, and don't give much help telling you WHAT went wrong. Most of the time the ROI is only there for verifying a few user paths, but not much more than that.

The Page Object Pattern can help reduce the brittleness of automated UI tests.

http://code.google.com/p/selenium/wiki/PageObjects

So, while I agree that it isn't right to "get on anyone's case for not having written this stuff", I do think it is reasonable to get on peoples' cases when they do release things if they didn't solve the "real problems": my message to people about to release a test framework would be "do you solve a real problem? if not, just keep it to yourself: please do not pollute the keyword space"; the corollary to this is that if you spent a weekend on it, it almost certainly does not solve a "real problem" ;P.

Let a hundred flowers bloom.

If only we could then still find the rare ones. :( This same attitude is what leads to hundreds of "articles" that are just "I read an article on some other website and summarized it on my website", making it impossible to find any of the actually useful or new content on any new subject. :(

This is pretty much my impression every time I try to test something in Python. I've never liked xunit, and I especially dislike it when it's a direct port from Java. Yes, it's familiar to Java programmers. But when I'm writing Python, I want something that looks like Python. (XML DOM libraries also like to think they're Java. That makes Common Lisp my favorite language for munging XML, since it's the only language bold enough to come up with some other API. But I digress...)

Also, I don't understand mocks in Python. Say I have a class Foo that has a method "bar" that I want to override. OK:

   class FakeFoo(Foo):
      def bar(self):
          print "OH HAI"

Done. It's built in to the freakin' language :)

Now make some other code use your "mock" without editing its source. Now assert on the calls made to "bar()," including arguments. Now make it raise an exception without writing a second class. And make it return something else without writing a third. Now patch a built-in. And a context manager. And a dictionary. When you're done with that, mock a class without writing a new class at all.

If you honestly think that sub-classing is equivalent to mocking/stubbing, you don't understand the purpose of the activity.

I know libraries can do all those things, but I've never preferred it over the real thing. As an example, I might do something like this (not Python):

   my $bar = 0;
   class FakeFoo {
      sub bar { return $bar }
   }

   my $thing = ThingToTest->new( foo => FakeFoo->new );
   $foo = 42;
   is $thing->make_bar_two_bigger, 44;
   $foo = -10;
   is $thing->make_bar_two_bigger, -12;

Mock enthusiasts would write:

   my $fake_foo = Mock->new( Foo->class )->instance;
   $fake_foo->return_sequence( bar => [42, -10] );
   my $thing = ThingToTest->new( foo => $fake_foo );
   is $fake_foo->make_bar_two_bigger, 44;
   is $fake_foo->make_bar_two_bigger, -12;
   ok $fake_foo->ensure_exhausted('bar');

Yes, it's less lines, but I think it's harder to read because the values are being set up long before they're used.

Well, this was typo-y. $foo in the first codeblock should be $bar, and $fake_foo->make_bar_two_bigger should be $thing->make_bar_two_bigger. More coffee is required :)

If you want more pleasant test-writing in Python, nose is commonly used, high quality, and compatible with the standard library's unittest package:

http://readthedocs.org/docs/nose/en/latest/

Well, here's Michael Foord's (@voidspace, referenced in the slides) Mock library so you can see what extra stuff it gives you.

http://www.voidspace.org.uk/python/mock/

FWIW, Michael Foord's mock library has been approved for inclusion in the Python's standard library.

See http://www.python.org/dev/peps/pep-0417/ and http://www.gossamer-threads.com/lists/python/dev/973274

Well there are very good efforts to make this situation better like Konira (RSpec like Python DSL) and Lettuce (BDD but you can also use it for unit tests). But nobody really pays attention to it (well maybe just Yipit and me). And its authors are really cool and active, you can even tweet with them about problems you have, I even had conversations about testing theory (and nomenclature) with them.

Python's unittest.py was a direct port from Java's JUnit.

More recently, a Google contribution to unittest.py added a huge number of self.assertThisThatOrTheOther() methods. The new methods were designed to have better diagnostic messages for test failures (i.e. diffs between expected and actual).

Much needed test discovery was added by Michael Foord.

Third-party testing modules like py.test and nose.py are popular substitutes for unittest.py. They use a light-weight syntax and are joy to use.

There seems to be a lot of subtext here that I'd love to see more about. I'm only an occasional python hacker and I'd love to know more about this.

One question, is there a web testing framework for python that they do like/recommend?

For a general purpose testing framework, Nose is great. For web testing, combine it with Selenium.

It's hard to get the full context of this presentation just from the slides, but I'm assuming the author is talking about Python test frameworks. I've been using nose in combination with coverage (http://pypi.python.org/pypi/coverage) for a year and have been very happy with it.

Thanks for the flashbacks from the TiP BoF. I thought we'd never get to the end, but man that was a fun and nerdy time.

I really liked the shot about web testing. It's a crying shame that a lot of web testing frameworks don't bother supporting javascript.

No-one talk to me about unit testing unless you're already using strong typing.

Python (the subject of the OP) is a strongly-typed, dynamically-typed language.

I think it's pretty clear from context that the OP meant static typing.

Since he's being a dick about it, I very much understand your desire to pedantically point out his error of terminology. Unfortunately, "strong" typing is a very overloaded term, which has never had a widely agreed meaning amongst either academics or practitioners. There are generally two camps: "strong" === "static", and "strong" === "no dynamic coercions", and neither is technically incorrect.

Reply on news.ycombinator.com