Hacker News Reader

>> The HN title is misleading: the terms do not forbid benchmarking, just public dissemination of the benchmarks.

If a tree falls in the forest, does it make a sound?

The title is not misleading. Performing 69,000 benchmarks but being unable to publish them is de facto banning benchmarks, period. Hiding behind the idea that benchmarking is hard therefore it should be banned because it might be fake news is ludicrous.

It is also not standard software EULA licensing, unless you think Oracle's practices are somehow industry standard and good for everyone.

This way they can make you spend a ton of time bringing up a trial version on your internal systems to do the benchmarking. Effectively forcing lock in and sunk costs fallacy thinking then put you into tons of meetings that are really just attempts to upsell.

Honestly, if I had widely used software with an enforceable EULA term which allowed me to benchmark but not publicly disclose bad results I found, it would make for even worse PR for the company: I'd be able to go to a tech industry reporter saying "I ran benchmarks on this software, and I think many people would be interested in the results, but the company forbade me to release them publicly. I will still share them non-publicly with any interested parties under NDA." Or if private dissemination were also forbidden, I'd change the wording accordingly.

The better way for a company to handle this concern, if they feel it's important, is to proactively run and release benchmarks including commentary on the results, together with everything necessary for anyone to reproduce their results. Even better if they fund a trustworthy neutral third party to do this instead, with proper disclosure of the funding.

They can then respond very effectively to bad PR about badly done benchmarks. Unless their performance is actually bad, or course.

That's certainly how things should be. But this clause was one of Oracle's early innovations (https://danluu.com/anon-benchmark/) and they did pretty well. Do we need to understand how they got away with it to have a good chance of changing the norm?

Question for any lawyers - can they gag any benchmarks that are published against the terms? I'd imagine the license terminates - what kind of damages can companies realistically extract when you break your license agreement?

> It's not that there's anything to hide

Clauses like these make me think you think there is. Which is perhaps a bigger red flag than the attempted censorship in itself.

> Spreading wrong performance information can hurt a business.

Firstly, I'd suggest you should perhaps focus on trying to educate your users on how to make better performance tests - if they are bad at making benchmarks then they are likely bad at running your server as well.

Secondly, boohoo. Not spreading unflattering but correct performance information might not hurt your business but will hurt your customers.

Lastly, you are curtailing speech which isn't ethical and due to that I'm pretty ambivalent if any hurt visited your business. Imagine if everyone did that about everything.

> complex

> requires expertise

I'm sure every despot anywhere used a variation of your argument, something about economics being complex offshoot of mathematics that requires expertise to handle so you best not share any overly rushed opinions.

> Apache-built binaries

Which people can benchmark to their heart's content and I'd hope the clause would irritate many people into publishing their own benchmarks..

I think you should also include restrictions in your EULA to prevent people from publishing statements about your code quality too. Very few are qualified to measure it, and spreading potentially false information can hurt a business. I can think of several other items that have limited expertise in evaluating and could cause negative PR harm if done falsely, so you probably should just enumerate those in the EULA too.

Also, I gather from your comment in support of these restrictions, they are more than boilerplate and you will pursue offenders.

I think car companies should prevent people from publishing safety or performance analysis of their vehicles. And restaurants should disallow public reviews of their business. (Very few people have the culinary knowledge to properly and objectively assess a restaurant's quality.)

Nope. You don't need to protect us from people who do a bad job benchmarking we can do that ourselves. If we read their methodology and disagree we can make up our own minds. Anyway my experience is terrible with Atlassian's services so without benchmarks I can just continue to assume that is true forever and always recommend against them when we're making infrastructure decisions. I guess I can thank you for a big bundle of products less to consider.

I get your point, but I also think you're wrong about it only being harmful.

Even if the review is done incorrectly, it's a data-point on a misconception your users have, and it gives you a chance to respond accordingly.

The company I work for has to deal with performance complaints all the time, some very from very public and loud entities in our Market. A company deciding to move away from our product signals very clearly across the market and it shows in our renewals numbers -- yet, we've never considered saying "don't benchmark us" in our contract.

Every single time these complaints kick up, we treat it as a chance to prove that we know our stuff, we reach out and offer to assist with re-evaluating, and explain our position, and most of the time, it works. It also shows a big public commitment to helping instead of just hiding behind our Legal Team, and it shows we know what we're doing.

We're not a huge company by any means in terms of actual persons able to act, but we still make it work without the need to put out such terms. If your customers are frequently doing something to make your product less than ideal, then you have an education issue that needs to be resolved.

> Spreading wrong performance information can hurt a business.

Spreading correct performance information can hurt Atlassian's business. That's why they don't want us to.

I'm surprised and disappointed by this stance. Other people in this thread have mentioned that such a clause isn't even legally enforceable, so I'm curious if you have any perspective on that.

Ultimately, though I appreciate you sharing your perspective, I personally will be steering clear of caddy and your other work.

Have you ever invoke that clause to demand that someone take down benchmarks of Caddy?

Do you believe that clause in the Caddy EULA is enforceable in the United States under the Consumer Review Fairness Act?

For me if benchmarks are forbidden, it means that software is significantly worse than its competitors. So this clause by itself is worse than any benchmarks. There might be exceptions like Oracle, but not everyone is Oracle.

> Benchmarking requires expertise that, it turns out, very few people have. I don't think I even have enough skills to do it correctly and meaningfully.

Very important and often overlooked point.

But I wonder, why not forbid public dissemination of inaccurate, non-reproducible benchmarks?

> Spreading wrong performance information can hurt a business.

Wouldn't that be libel? (IANAL)

The problem is when the benchmark is accurate, reproducible, and based on a completely nonsensical scenario. It's not libel to say "Jira is 50% slower than Bugzilla" when your benchmark serially creates 25000 tickets, but it's not a fair claim either.

That's not to say that anti-benchmarking licenses are the right solution, of course. I can just sympathize with why Atlassian wants one.

Expertise so rare it seems Atlassian themselves don't have it.

Suppose you state that a product performed poorly on measurement Y of benchmark X. The problem is not reproducibility or accuracy, it's that either Y or X are always really stupid in some way or another.