NB: You're trading non-negligible space efficiency here, which gets worse as you lower the false positive rate. Partitioning means uneven occupancy rates across partitions, and either you accept low occupancy (increased filter size overall) or you're pretty much guaranteed to have hot blocks (increased false positive rate for specific partitions).
I hadn’t seen that one hash paper before. This is a really cool optimization over Miztenmacher. Do you have benchmark numbers with different input sizes you can share?
I’ll have to read the paper and code more, but it seems like this would be amenable to supporting filter merges as well (as long as the new size doesn’t exceed the filters’ capacity).
NB: You're trading non-negligible space efficiency here, which gets worse as you lower the false positive rate. Partitioning means uneven occupancy rates across partitions, and either you accept low occupancy (increased filter size overall) or you're pretty much guaranteed to have hot blocks (increased false positive rate for specific partitions).
I hadn’t seen that one hash paper before. This is a really cool optimization over Miztenmacher. Do you have benchmark numbers with different input sizes you can share?
I’ll have to read the paper and code more, but it seems like this would be amenable to supporting filter merges as well (as long as the new size doesn’t exceed the filters’ capacity).