Because?
Is the correlation 1:1? No. Is it still relevant when making a decision about what technology to use? Absolutely.
That's the idea of benchmarks, but it's not actually true of real benchmarks, because invariably some factors which can improve a particular benchmarks beaean inverse relation to performance for some other use cases of the technology.
I'm not having a problem.
> is that everybody already knows that,
Certainly, some people act like they don't, at least in the context of specific benchmarks.
And some people outright claim the opposite of what you say everyone understands, e.g., by claiming that benchmarks inherently correlate with every possible use of a technology.
> and plenty of folks can still extract value out of a benchmark anyway.
Understanding both the general issue and, ideally, the specific areas of potential concern in relation to particular benchmarks and your intended use is a big part of being able to effectively extract value from a benchmark.and, yes, lots of people do recognize those facts and extract value from benchmarks.
Others don't, and still apply benchmarks in decision-making, but it's less clear that they are extracting value. Confusing the measurement most readily available with the measurement most relevant to need is a common problem (and not just with benchmarks.)
You're throwing up your hands and saying, "TOO DIFFERENT FROM REALITY!"
Other folks don't do that, and while not 1:1, they are able to correlate the performance of a benchmark with the performance of their own use case.
Try not to get wrapped around the axle on "everyone", by the way, it's not literal.
You can correlate lots of things. Poor countries increase penis size. Ice cream leads to murder. Cheese kills people by tangling them in their bedsheets. These things are strongly correlated. But they have no causative relationship.
To put it another way, a single benchmark test used find a solution is scientifically equivalent to finding one "average" man, making him run a series of athletic tests, and making his performance the benchmark for men's athletics.
Like I've been saying, benchmark results are plenty useful if you know how to use them.
Again, I'm not having a problem.
> is you can't figure out how to use benchmarks to understand how a system works
No, I have no partucular problem evaluating whether a benchmark is useful to a decision and if so, how.
Nor have I said anything indicating any such problem.
> You're throwing up your hands and saying, "TOO DIFFERENT FROM REALITY!"
No, I'm saying he naive statement upthread that benchmark performance correlates with every possible use of a technology is nonsense.
That's it.
> Other folks don't do that
Actually, some do, but that's neither here nor there.
> and while not 1:1, they are able to correlate the performance of a benchmark with the performance of their own use case.
Once again, yes, lots of people have the skill to figure out whether and in what way particular benchmarks have utility for their usecases. I explicitly said that in the post you are responding to.
That's very different than your claim that I reacted against, which is that any benchmark inherently correlates with every use case, which is—again—complete nonsense.
Oh. That's what you thought I said? Right. I didn't mean to say that, and if somehow I did say that I withdraw. Benchmarks are valuable if you know how to use them, but of course I agree with you, a benchmark isn't always relevant to every use case.
Side note, quoting many small portions of a person's comment and replying exclusively to the quoted bits is inferior to replying to them with full sentences/paragraphs. I've only ever seen what you're doing done by folks who are super interested in arguing and completely uninterested in having a conversation.
It definitely comes across like you do have a problem, which you have repeatedly stated you do not (and I believe you)! It just muddies the water, what you're doing here.
We didn't intend the benchmarks to be a sales pitch. We are proud of the performance of MemSQL and what it has been able to achieve in our customers' workloads. We wanted a way to show what we are capable of with concrete numbers. We chose the standard benchmarks because they are well understood, not because they were necessarily representative of any given customer.
In general, benchmarks are useful to understand the strengths and weaknesses of a product at a basic level and how it compares to its peers, but we strongly encourage anyone evaluating their options to do a proper POC comparison on their actual use cases.