For Redis - The benchmark client and remote server (ie AWS ElastiCache) were in the same AWS availability zone (us-west 2a) to minimize network latency as much as possible.
It might make more sense to write a harness that allows your software to be used with standard Redis wire protocol so it can be properly benchmarked and compared against the dozens of existing solutions in the space.
Also it seems like you accidentally discovered the Latency Numbers Everyone Should Know [2]. Local operations are faster than network operations.
1. https://redis.io/docs/management/optimization/benchmarks/
2. https://static.googleusercontent.com/media/sre.google/en//st...
Most DB tooling out there only works for a client-server model.
And implementing the Redis protocol would imply changing our architecture significantly and negatively affect performance (ex. a producer-consumer queue to serve requests, ser-deserialization costs).
Yes, you’d be doing a fair and honest benchmark if you want to compare yourself to Redis.
Hope this addresses your original question about why we wrote a custom benchmarking client.
https://github.com/inlinedio/ikv-java-client/blob/master/src...
It is quite simple and is available here. There is nothing malicious in there to make IKV appear faster. Although ff you do see a bug, I am happy to fix and republish results.
> We use a single benchmarking client to drive traffic to - (1) embedded IKV instance on the same machine as the client (2) A multi-sharded Redis cluster (built using AWS ElastiCache for Redis) in the same data-center (AWS availability zone). Using multiple clients is complex since we’re testing an embedded database (other benchmarking tools which only test standalone DB services can drive more load using multiple clients). The maximum amount of load that can be created depends on the number of parallel threads in the client and the average response time of the underlying database.
This is a ridiculous comparison: "embedded IKV instance" vs a cloud service.