Hacker News Reader

Comment by encoderer | Hacker News Reader

I'll write it up when I finally have this whipped, but we have a unit test stumper ourselves. In our Django project, the tests have always run slow. Once we got above 50 tests I finally decided to hunt for the cause. It creates a test DB from fixtures for each test case but i couldn't understand how that could possibly take several seconds per test.

Then I was coding on the train home one night and in the tube my hotspot lost connection. Then I happened to notice... with wifi off the tests ran 10x quicker. Pull out netstat.... I see http requests to akamai essentially as each test setUp() is run. It's SSL so I only see a hostname, not the full path, and they just look like this, but the hostname is dynamic: a23-34-132-181.deploy.static.akamaitechnologies.com

I spent 30 mins reviewing our code, grepping my code and packages directories, and stepping thru in a debugger. No luck.

View on HN

Could be a remote XSD referenced from a local XML file? I've had that happen a couple of times...

The worst was having a Map keyed on URLs in Java -- turns out URL.equals() and URL.hashCode() do DNS lookups... Easiest 50% test speedup ever!

Oh, this is an interesting lead.... I feel pretty certain the request isn't coming from our Python after stepping-thru the code.

The other suggestions -- using a SSL proxy -- has been my plan of attack.

If it takes several seconds to set up a test, I would give this a try:

  - start a test from the IDE,
  - wait a second or so,
  - manually pause all threads,
  - look at the stacks of the threads.

Might take two or three attempts, but likely should learn you a lot. Alternatively, use a real profiler to find the call(s) that take lots of time.

mocking/replacing the http/urllibs? save a traceback when a connection is made?

Try an strace?

    strace -e trace=network ./run_my_unit_tests

Also -f so it follows forking (I always forget that the first time, swear, and have to re-run strace with it)

What OS are you running? It's pretty straightforward to get going on OSX in my experience.

Is the URL class in Java special, versus just a Uri structure?. Because relying on DNS for equivalency sounds totally wrong for many cases. At least such a compare should have a special name to indicate resolution will be attempted.

The URL class in Java is special, as in especially broken. It's very old code that probably wasn't a good idea even back then, and it's never been changed for backwards compat reasons.

Use java.net.Uri, or e.g. the equivalent from Jersey or whatever your local framework/HTTP client/... brings with it.

There is a couple of other places where this fails pretty hard. If you compare to URL's on the same shared host, they will be considered equal and equality of URI's can change as the network goes up and down. I would imagine at some point (if it hasn't already) become deprecated.

Joshua Bloch used this as an example in one of the brilliant Java Puzzler video series with the net advise of don't depend on non-local state for equality.

Thats a good idea. Java and how it handles XML deserve a special place in hell, the motions to install a local XSD resolver are... not intuitive.

Try running it through an SSL-capable proxy like mitmproxy[0]. You should be able to get the full request/response details that way.

[0]: https://mitmproxy.org/, https://mitmproxy.org/doc/ssl.html

Ned Batchelder had an interesting post recently about finding temp file creators through monkey-patching[0]. I could see a similar approach being helpful here by patching the socket module to find all call stacks.

[0]: http://nedbatchelder.com/blog/201503/finding_temp_file_creat...

Take a tcpdump and open it in Wireshark - you can't see the content of the requests, but the first packet will probably have an SNI header that tells you what hostname you're trying to connect to.

I haven't had an opportunity to try this myself, but rather than setting up some kind of MITM proxy, maybe you could try this:

https://jimshaver.net/2015/02/11/decrypting-tls-browser-traf...

That's a weird one. I've got some advice about your testing methods though. Fixtures are sloooooooow. They also run before every single test method, not just once per TestCase class, unless you're using Django 1.8 (which introduced setUpTestData which is run once per class rather than once per method). You'll get some additional speed up by converting your fixtures into python.

If that was with Java, I would do something to make network terrible slow or even unresponsive and try to dump stacktrace to catch the bugger. Not sure if you can print stacktrace in Python in the middle of something.

Can you attach the JDK sources and put a breakpoint in the HttpRequest constructor? Then you should be able to find the culprit right?

dns timeout

Reply on news.ycombinator.com