I really wish they turned it into a dependency or database framework where users could define their own business logic to swap out the double entry accounting, while reusing all the system architecture and networking features, consensus etc.
Sort of like a new paradigm where opinionated custom databases could be created with arbitrary entry logic built on this stack.
This was always the plan, and if you look closer at VSR and the state machine interface you’ll see it’s already pluggable. We just haven’t packaged it. (We’re dogfooding our first few internal “CustomBeetles” before we package and document.)
Hey, very inspiring article, Redis engineer here.
How do you work with static allocation on variable query structure, and row counts that can explode depending on the data shape?
And isn't there a benefit for small allocations on advanced memory allocations that you can't leverage if all is working in big page allocations? Do you implement memory allocations from scratch or leveraging existing allocator on top of these memory blocks strategy somehow?
For example, if you take a look at our LSM compaction, regardless of the table size, we compact at the 512 KiB block granularity, and everything is streaming.
The same principle applies everywhere.
In our experience writing TigerStyle (and for all our internal code and tooling, not only TB as DBMS), we’ve never had a scenario where static allocation was not applicable or didn’t produce a better design.
You also tend to become more memory efficient, not less. Again, since you’re streaming. (You’re not allocating a massive buffer, just because a file is multi-GiB.)
I really wish they turned it into a dependency or database framework where users could define their own business logic to swap out the double entry accounting, while reusing all the system architecture and networking features, consensus etc.
Sort of like a new paradigm where opinionated custom databases could be created with arbitrary entry logic built on this stack.
This was always the plan, and if you look closer at VSR and the state machine interface you’ll see it’s already pluggable. We just haven’t packaged it. (We’re dogfooding our first few internal “CustomBeetles” before we package and document.)
That's cool! I will be then probably taking a closer look!
Thanks! Watch IronBeetle too on Twitch if you’d like to go really deep.
llvm for databases?
I'm pretty sure they manage to get that level of performance and reliability, since they have a very limited schema.
I'm pretty sure you can't just do a precise 128 byte align, if one of the element is an image blob or varchar(1000).
No, the LSM in TB is generalizable at comptime to any combination of power of two sized key/value tuples.
Joran from TigerBeetle here! I created TB. Happy to answer questions!
Hey, very inspiring article, Redis engineer here. How do you work with static allocation on variable query structure, and row counts that can explode depending on the data shape?
And isn't there a benefit for small allocations on advanced memory allocations that you can't leverage if all is working in big page allocations? Do you implement memory allocations from scratch or leveraging existing allocator on top of these memory blocks strategy somehow?
Thanks! We use streaming data structures.
For example, if you take a look at our LSM compaction, regardless of the table size, we compact at the 512 KiB block granularity, and everything is streaming.
The same principle applies everywhere.
In our experience writing TigerStyle (and for all our internal code and tooling, not only TB as DBMS), we’ve never had a scenario where static allocation was not applicable or didn’t produce a better design.
You also tend to become more memory efficient, not less. Again, since you’re streaming. (You’re not allocating a massive buffer, just because a file is multi-GiB.)
Their simulation is fantastic:
https://sim.tigerbeetle.com