> ParticleDB’s home page advertised that ParticleDB was “formally proven” with a TLA+ proof. However, during our collaboration no TLA+ specification was available. It is unclear what properties might be proven, or whether they described ParticleDB as built.
> We present 96 issues in ParticleDB, ranging from the pedestrian (crashes, deadlocks, data loss, etc.) to the piquant (partially-invisible rows which could not be deleted, WHERE 4 = 5, dropped tables re-emerging in distorted forms, confused columns, strings turning into integers, and so on).
Definitely seems worse than usual (looking back most databases seem to have <10 issues total, and the majority seem "mild") - since it's a database for AI was it also likely vibe-coded or just new? Then again even if new, I wouldn't expect so many issues in a stable release
It’s wild how bad it is. Not surprising for a vibecoded database, but why did they hire Kyle for a test, instead of first doing ANY testing themselves?
> ParticleDB 1.1.4 claimed to be a Postgres-compatible database ... Instead, we found a library of confusion
> exhibited frequent, profound transactional anomalies in essentially every way we could measure. We routinely observed the loss of committed writes, internal consistency violations, G1a (Aborted Read), G1b (Intermediate Read), and P4 (Lost Update). In the append workload, successfully appended values would routinely disappear from later reads of a given key. Our test suite inferred every kind of cyclic anomaly including G0, G1c, G-Single, and G2-item, but many of these can be attributed to more basic issues, like write loss. These issues occurred with a single server and without faults, and affected every transaction mode, interface, and WAL mode tested
these were tests done on pre release builds and some experimental concurrency modes as well.... The engine has since been redesigned and all of these issues have been fixed..
Its also faster than Redis for KV, also numbers show this should be top on clickbench, official submission is in progress
Why should anyone believe that? I mean it might be true, I don't know. But if such a report turns out many fundamental issues it doesn't quite sound plausible that a redesign would suddenly not have any issues anymore. It might have fewer issues, it likely will have different issues as well though.
Bro you've got to be kidding. pre-release builds? Aphyr directly addresses this in this analysis. Furthermore, given the absolutely ridiculous nature of some of these bugs, and the low confidence exhibited by the authors in the fixes, how can anyone assume that suddenly things are correct? The database was so janky that it sounds like most of the time was spent on basic bugs that a decent set of unit-tests would've caught. Who knows what wasn't uncovered because the db was so broken to begin with?
I mean did you see the "primary key equality" thing? Holy shit.
Good on them for being good sports, but after reading this I would never touch a piece of software built by anyone involved with this company. The sheer hubris of their claims combined with the specific reality of their broken garbage software (with the contradicted marketing claims and hallucinated documentation) is literally breathtaking.
It must have been amusing to see all these defect tickets coming in, only for them to frantically shovel each one to increasingly overworked LLM agents to apply pig-lipstick while the humans in charge oh-so—slowly start to realize their software quality is not quite what their glazing agents had been telling them. What a world we live in, huh?
I can 100% guarantee their codebase is not human-navigable.
And you still have to make sure all the real constraints are in. I had to fix a proven correct distributed algorithm that just happened to believe updates were done atomically and with no latency.
Since the library claims to be postgres compatible, I wonder if its developers ran the existing postgres jepsen suite against it (possibly with some adjustments). Or why not.
Especially now, where you could probably just ask AI to adapt the existing suites to the cosnistency claims this database makes, and get something useful out of it. While certainly not as good as a professional test, I'd expect it to at least find some low hanging fruits before going for the expensive and public official Jepsen test.
"Orkes said they ran Jepsen’s existing Postgres test harness to find and resolve issues in ParticleDB prior to this work. However, that same test harness failed immediately when run against ParticleDB 1.1.4. Orkes also stated they tested ParticleDB with SQLancer, which generates far more complex queries than the ones we tested. It is therefore perplexing that nearly every feature we tested exhibited glaring errors, and that our findings seemed to come as a surprise."
Is/was ParticleDB open source? There is a Github link on their website, however the only public repo in their org seems to be the Golang client library.
> ParticleDB’s home page advertised that ParticleDB was “formally proven” with a TLA+ proof. However, during our collaboration no TLA+ specification was available. It is unclear what properties might be proven, or whether they described ParticleDB as built.
> We present 96 issues in ParticleDB, ranging from the pedestrian (crashes, deadlocks, data loss, etc.) to the piquant (partially-invisible rows which could not be deleted, WHERE 4 = 5, dropped tables re-emerging in distorted forms, confused columns, strings turning into integers, and so on).
Definitely seems worse than usual (looking back most databases seem to have <10 issues total, and the majority seem "mild") - since it's a database for AI was it also likely vibe-coded or just new? Then again even if new, I wouldn't expect so many issues in a stable release
The discussion section of the report is very damning.
[dead]
It’s wild how bad it is. Not surprising for a vibecoded database, but why did they hire Kyle for a test, instead of first doing ANY testing themselves?
Brutal!
> ParticleDB 1.1.4 claimed to be a Postgres-compatible database ... Instead, we found a library of confusion
> exhibited frequent, profound transactional anomalies in essentially every way we could measure. We routinely observed the loss of committed writes, internal consistency violations, G1a (Aborted Read), G1b (Intermediate Read), and P4 (Lost Update). In the append workload, successfully appended values would routinely disappear from later reads of a given key. Our test suite inferred every kind of cyclic anomaly including G0, G1c, G-Single, and G2-item, but many of these can be attributed to more basic issues, like write loss. These issues occurred with a single server and without faults, and affected every transaction mode, interface, and WAL mode tested
these were tests done on pre release builds and some experimental concurrency modes as well.... The engine has since been redesigned and all of these issues have been fixed..
Its also faster than Redis for KV, also numbers show this should be top on clickbench, official submission is in progress
Why should anyone believe that? I mean it might be true, I don't know. But if such a report turns out many fundamental issues it doesn't quite sound plausible that a redesign would suddenly not have any issues anymore. It might have fewer issues, it likely will have different issues as well though.
I look forward to the second report from Jepsen for the new implementation then.
> and all of these issues have been fixed..
You can't be sure until they test it again
Bro you've got to be kidding. pre-release builds? Aphyr directly addresses this in this analysis. Furthermore, given the absolutely ridiculous nature of some of these bugs, and the low confidence exhibited by the authors in the fixes, how can anyone assume that suddenly things are correct? The database was so janky that it sounds like most of the time was spent on basic bugs that a decent set of unit-tests would've caught. Who knows what wasn't uncovered because the db was so broken to begin with?
I mean did you see the "primary key equality" thing? Holy shit.
Good on them for being good sports, but after reading this I would never touch a piece of software built by anyone involved with this company. The sheer hubris of their claims combined with the specific reality of their broken garbage software (with the contradicted marketing claims and hallucinated documentation) is literally breathtaking.
It must have been amusing to see all these defect tickets coming in, only for them to frantically shovel each one to increasingly overworked LLM agents to apply pig-lipstick while the humans in charge oh-so—slowly start to realize their software quality is not quite what their glazing agents had been telling them. What a world we live in, huh?
I can 100% guarantee their codebase is not human-navigable.
Great to implement a DB just like human memory, unstable, and untrustworthy.
> ParticleDB’s home page advertised that ParticleDB was “formally proven” with a TLA+ proof.
Perhaps they meant formerly, probably just a speech to text problem
TLA+ doesnt prove the code. if they never read the proof, then how do we k ow that it proved the right things?
And you still have to make sure all the real constraints are in. I had to fix a proven correct distributed algorithm that just happened to believe updates were done atomically and with no latency.
lol, reminds me of the common misreading "provably safe" / "probably safe"
Since the library claims to be postgres compatible, I wonder if its developers ran the existing postgres jepsen suite against it (possibly with some adjustments). Or why not.
Especially now, where you could probably just ask AI to adapt the existing suites to the cosnistency claims this database makes, and get something useful out of it. While certainly not as good as a professional test, I'd expect it to at least find some low hanging fruits before going for the expensive and public official Jepsen test.
The company says they did. From the post
"Orkes said they ran Jepsen’s existing Postgres test harness to find and resolve issues in ParticleDB prior to this work. However, that same test harness failed immediately when run against ParticleDB 1.1.4. Orkes also stated they tested ParticleDB with SQLancer, which generates far more complex queries than the ones we tested. It is therefore perplexing that nearly every feature we tested exhibited glaring errors, and that our findings seemed to come as a surprise."
Seems to me that “we ran the Jepsen test” is just something an LLM thought would sound good on a DB’s website.
Aphyr does great work, Jepsen always kicks it out of the park :)
Is/was ParticleDB open source? There is a Github link on their website, however the only public repo in their org seems to be the Golang client library.
Fatality
Slop DB written in Rust. They probably changed the landing page after the bad results, you can still see the old one: https://web.archive.org/web/20260518203012/https://particled...
"Written in Rust · Formally Verified · PostgreSQL Compatible"
-https://github.com/jepsen-io/particledb is currently 404 (probably still private?)-
EDIT: It's up now
Sorry about that--yes, it's still private. Hoping to have that resolved shortly.
Looks like it's up now. I'll put that link in the toptext as well. Thanks!