It's a strong signal of unknown quality. Caveat emptor. Closed-source makes that extra difficult.
We are used to this, hence why folks here point it out.
We are used to this, hence why folks here point it out.
https://github.com/l-labs/master-benchmark
Which can also be run with minimal modifications on ngn and others.
For database the h2o.ai for duckDB really is a solid hard bench IMHO:
https://github.com/l-labs/rust_ipc/blob/master/tests/ipc_tes...
Which is ~6k tests (Rust is actually what I use to drive release testing)
Unknown quality is the absence of information, you can’t have a signal for it.
If you are using “unknown” as a euphemism for “poor”, which is the most obvious thing that makes the signal meaningful in the direction that seems apparent from context, why?.
That's also why "closed-source" was mentioned in the same sentence. Two missing signals.
That's also why the author keeps pointing to the benchmarks and test suite, which is the signal they'd like us to judge by. Seems entirely reasonable to me!