vancan1ty

Algorithmic run time is "time complexity." There are many different types of complexity which we can measure in some sense, including space complexity, energy complexity, circuit complexity, and architectural complexity.

http://en.wikipedia.org/wiki/Computational_complexity_theory

I agree with you that there seems to be no foolproof way to measure the overall "software complexity" of a piece of software. There are a lot of methods proposed, but none is provably optimal or even all that useful it seems. Search "Evaluating Software Complexity Measures" by Elaine Weyuker for a somewhat dated paper discussing the topic. I would be very interested if anyone had any recommended resources on the current state-of-the-art regarding software complexity measures.

Seems to me the only way to measure complexity absolutely would be some sort of extension of the minimum description length principle -- what is the shortest way we could write the specification for the entire machine + software that runs on it. Of course this seems completely infeasible for any real world task.

Is the method you described not another form of inferential statistics? (Definitely not hypothesis testing however).

It seems that this is a combination of several techniques:

1. First, we either explicitly or implicity choose a model to relate deaths and RTOR laws (perhaps a linear relationship, i.e. [deaths w/ RTOR] = a*[deaths w/o RTOR]).

2. Then we perform point estimation to estimate the parameter "a".

3. Then we compute a confidence interval.

With respect to the RTOR example in the paper, it seems to me that it WOULD be incorrect to reject the null hypothesis that the change in crash numbers arises from random chance for ANY INDIVIDUAL STUDY. In this case it seems that you must figure out a way to transfer information between studies to establish this idea of "statistical significance." Perhaps a survey of studies or usage of bayesian techniques would have resolved the difficulty.

People often have trouble applying statistical methods correctly, and perhaps often can manipulate statistics to tell a given story. And indeed, P=0.03 on its own is meaningless without an understanding of how a study is set up and a plausible hypothesis.

But inferential statistics is grounded in sound theory, and used correctly and with appropriate assumptions is a powerful tool to reason about data. Without it, how are you supposed to reason about (for example) study results? Appeal to intuition?

It seems to me that significance tests are not all-powerful, or foolproof, but are still a very valuable tool.