https://www.quora.com/What-statistical-test-should-be-used-f...
(Of course, even better would be a Bayesian decision analysis which takes into account the full posterior distribution, costs, and benefits, allowing one to decide whether the evidence is enough to justify the more expensive box or whether one should extend the experiment further to collect more data.)
* quick, if conversions are 0.25%, what does a 0.1SD mean difference between arms of an A/B test mean? 'Uh...' Exactly.
I'm not sure what you mean. If all you have is binary data, there's not much you can do but a chi-squared test or binomial regression. As I understand it, a chi-squared is considered nonparametric: https://en.wikipedia.org/wiki/Chi-squared_test https://en.wikipedia.org/wiki/Fisher's_exact_test