Comment by kragen | Hacker News Reader

Comment by kragen | Hacker News Reader

Parent Full thread

it doesn't sound like hypothesis that unable to handles large data sets in this case, though that is indeed not its forte; it sounds like you were rejecting a large proportion of the shrunk instances, so hypothesis would try shrinking by setting a generated integer to zero, in order to see if the bug still existed for zero, and your test would reject it because it had a zero in it. not fail, but reject. for small instances this was just inefficient, but for larger ones it got to the point that hypothesis gave up

someone in that thread suggested that you use a different instance generation strategy that can't generate zeroes, instead of rejecting the instances hypothesis's shrinker most loves to generate, once they are generated. did you try that?

how does clojure.spec.alpha handle this differently?

in mjaniczek's comment at https://news.ycombinator.com/item?id=40876437, they call out your case as a disadvantage of hypothesis's approach:

> The disadvantage is that the generators are now parsers from the lists of bytes that can fail (introducing some inefficiency) and user can make a crazy generator that the internal shrinker will not be able to shrink perfectly. Nevertheless, it's the best DX of the three approaches, ...

though presumably you wouldn't agree that your test was written in a 'crazy' way

> hypothesis that unable

this should read "hypothesis was unable". we regret the error. those responsible have been sacked

If hypothesis can’t shrink a result further, shouldn’t I expect it to just return the failing result?

There’s usually always a point where further shrinking is not possible.

yes, but if it's too hard to generate an instance your test will accept as a valid instance (not passing, valid), i'd instead expect it to tell you to fix your test, so you can get properly shrunk results, and that's what it did. did you try the different instance generation strategy suggested in the thread? how does clojure.spec.alpha handle this differently?

On the surface, clojure.spec.alpha seems to handle things more or less similarly (you get random results skewing heavily towards small/trivial/edge values), but the main difference is that it can be used and abused without problem. As long as you can write predicates and compose specs, it handles it for you and “just works”.

did you try the different instance generation strategy suggested in the thread?

you make it sound like clojure.spec.alpha doesn't do shrinking at all. possibly you don't know what shrinking is? shrinking keeps your results from looking "random" by simplifying them until all the simpler versions of the test case pass, so it finds the simplest failing case instead of a "random" one. this also enables hypothesis to be a lot more aggressive at looking for failing cases, for example by generating much larger test data sets than you would want to pore over, because if it finds one that fails, it will shrink it before showing it to you

if i've misunderstood you and clojure.spec.alpha does do shrinking, what does it do in the case where your test case is written to reject all but an exponentially small fraction of shrunk cases? does it just take an exponentially long time to run your test suite? is that what you mean by 'just works'?

i guess you didn't try the different instance generation strategy suggested in the thread, since i asked three times and you didn't answer three times

Reply on news.ycombinator.com