Nice. I am curious if you would seek to maintain this with some kind of a foundation(apache)
based funding or keep it internal to your work?
> I used AI a lot, but I was the one making the design decisions based on the tests that claude picked from my history of troubleshooting sessions as part of my job.
Cool, this seemed like a good learning experience, learning from testing it in a real world. I wonder if you plan to write up some of the experience in a blog or something?
Just adding my benchmarking repo, that contains terraform code and scripts using which I performed the benchmarks, helpful for re-evaluating and verifying the numbers published on our website. And possibly to learn where we might have gone wrong.
I haven’t tested against it, that will be next, i will run the benchmarks on the same method and publish the results, or if you want to, you can as well. Even if it performs poorly, I will report that too like we did with dragonfly.
That's true, however the intention to run the benchmark would be against most of the widely used commands and see if there is any command or use case where kividb might be a better fit than kvrocks. strictly speaking from hardware performance, it would certainly beat kvrocks in terms of sheer throughput, but in terms of volume where does it stand would be something interesting to experiment.
Pipelined throughput that high usually means the loadgen is the thing you're measuring. Past ~5M ops/sec we needed 3 separate memtier hosts to keep the client off the critical path. Any p99 numbers alongside it?
Yes, for building test cases, for code refactoring, and commenting, because it writes cleaner code, but yeah design decisions were mine. The benchmarks I ran were what dragonfly advertises, so scripts for those were written with AI. For redis compatibility, we run redis’ own unit test and e2e test suite with every change to make sure I am not breaking anything each change and you can review it in the repo.
Nice. I am curious if you would seek to maintain this with some kind of a foundation(apache) based funding or keep it internal to your work?
> I used AI a lot, but I was the one making the design decisions based on the tests that claude picked from my history of troubleshooting sessions as part of my job.
Cool, this seemed like a good learning experience, learning from testing it in a real world. I wonder if you plan to write up some of the experience in a blog or something?
How did overall memory footprint look during ingestion vs Redis? Have any tradeoff graphs as the dataset gets larger and with mixed workloads?
Could you add Valkey to the benchmark? Curious about different key-types too, like list and hash. Also, big values (2KB) and small values (<256B)
Just adding my benchmarking repo, that contains terraform code and scripts using which I performed the benchmarks, helpful for re-evaluating and verifying the numbers published on our website. And possibly to learn where we might have gone wrong.
https://github.com/kividbio/kivi-benchmark
How does this compare and benchmark with https://github.com/apache/kvrocks
I haven’t tested against it, that will be next, i will run the benchmarks on the same method and publish the results, or if you want to, you can as well. Even if it performs poorly, I will report that too like we did with dragonfly.
Well Kvrocks is on disk, not in memory.
That's true, however the intention to run the benchmark would be against most of the widely used commands and see if there is any command or use case where kividb might be a better fit than kvrocks. strictly speaking from hardware performance, it would certainly beat kvrocks in terms of sheer throughput, but in terms of volume where does it stand would be something interesting to experiment.
Pipelined throughput that high usually means the loadgen is the thing you're measuring. Past ~5M ops/sec we needed 3 separate memtier hosts to keep the client off the critical path. Any p99 numbers alongside it?
Has there been heavy usage of AI in building this?
Yes, for building test cases, for code refactoring, and commenting, because it writes cleaner code, but yeah design decisions were mine. The benchmarks I ran were what dragonfly advertises, so scripts for those were written with AI. For redis compatibility, we run redis’ own unit test and e2e test suite with every change to make sure I am not breaking anything each change and you can review it in the repo.
there's a .claude/ in the gitignore
Yes correct