Demand the impossible: Rigorous database benchmarking
erthalion.info
erthalion.info
LDBC creates TPC-style application-level database benchmarks which can be used for system-to-system comparison. We provide detailed specifications, data generators, benchmark frameworks, and multiple reference implementations. The benchmarks are implemented by vendors for their database products, and the implementations submitted to be run by independent third-party auditors to ensure their correctness and reproducibility.
We have found that there is a market for audits for graph processing systems, albeit it is quite small: over the last 4 years, we have published 34 audited results, see [2], [3], and [4].
A major problem we face is that process of implementing the benchmark for a system and getting an audited result is long (and therefore expensive). Vendors spend months implementing and tuning the benchmarks for their systems. It is also typical for the auditor to spend 50+ hours on the auditing process, which includes a lengthy code review step, setting up the system, running the experiments, testing ACID properties, writing a report, etc. The length of the process is exacerbated by the lack of widely-adopted standard graph query languages. This potentially necessitates the auditor to learn a new language for every new audit.
We have tried to mitigate this problem by improving our documentation, creating more reference implementation, distributing pre-generated data sets. There are new standard graph query languages (SQL/PGQ, GQL) but their adoption is currently very limited. Overall, the auditing process is still quite long, which is mainly caused by the essential complexity of the problem: implementing an application-level benchmark and getting reliable results is very difficult.
[1] https://ldbcouncil.org/introduction/
[2] https://ldbcouncil.org/benchmarks/snb-interactive
It is quite typical that vendors find performance issues in their systems when they first adopt the benchmark. This then leads to improvements in their operators, optimizer, storage, etc. These improvements will be included in a new release of the system – which is great, as a main point of the benchmark is to drive innovation and raise the general level of performance in the technology space.
It's important that the audits can only be conducted on systems that are generally available. We don't allow audits on some one-off version of a system that has special features just to perform well on the benchmark (these are known as "benchmark specials" and are also illegal in TPC benchmarks).
"Tuning the benchmark" is probably "tuning your system to score high on the benchmark". There are always caveats of course. You, as a developer, may be able to find some implicit assumptions in the test and exploit it to "game" the test. Suppose, for example, the test assumes that acknowledging deletion means deletion completed, but your code simply schedules a file for deletion instead of actually deleting it before acknowledge is sent.
You'd have to argue somehow that your system is behaving in such a way that scheduling and deleting are essentially the same thing which would allow you to pass the test, but that's about the level of "cheating" that you can do on these tests.
Also, by the nature of things, the way these systems are sold, you don't really want to pull these numbers out of nowhere. This kind of unsophisticated fraud will get you disqualified very quickly, but the investment into building such systems is usually on the scale of upwards from five or more years until it even gets into the stage where you can show something to the customer. So, it's a large investment that you don't want to risk.
- open/closed load generation strategies
- nature of the systems-under-test being almost always that of a networks of queues
- behaviour of systems-under-test tending to be non-normal and modal
let's take the max_wal_size parameter for ex.
for most systems it's much simpler to set the max_wal_size to a value high enough that its value does not put pressure for checkpoints under load.
so get enough free disk space, allocate maybe 5%. look at the logs, increase the value if checkpoints are required.
(the downside is that crash recovery will take longer since it will read the wal)
for each setting, we need to remember trade-offs
and in this particular case, the trade-off is increased recovery time after potential crashes
it is possible to conduct another benchmark that will measure this time, showing how it increases (in "avg" case -- "normal" TPS, and in "worst"* case -- increased TPS, e.g. massive UPDATE with random IO/access), and collect another set of interesting data and then combine two sets to support decisions
in result, we will choose something like: we decided to use 32 GiB for max_wal_size and 15 min checkpoint_timeout and we know that if we crash in the worst* case, DB will need up to 10 minutes to recover; but benefit is that we have much less disk IO stress during massive writes.
___
*) "worst" here has two levels that show off when writes are massive with random IO pattern of block writes:
- excessive writes from frequent checkpoints
- additionally, more WAL needs to be written due to (if) full_page_writes=on
?