I had the exact same thing with the Quickbooks api recently. You cannot trust the responses or webhooks at all.
On create a user or invoice for example sometimes it will return an error, yet it actually created the entity. This means you have to check manually after creating everything to know if its created properly.
Then you have the issue that sometimes quickbooks takes a while to update, and locks the company file while it does some background magic. This means you cannot immediately do the existence check, and also sometimes the check errors or times out which essentially means you need to keep checking forever until you can properly reconcile your db against theirs. But with hundreds/thousands of transactions per minute this state is never reached. You perpetually live in a state of trying to catch up but never managing it.
When I brought it up with Quickbooks dev support their response was literally "Its your job to make sure things are created properly in our system".
How did we get to this place where we started putting up with systems that cannot ever be trusted?
If you have two or more services that need to agree about state, and you have some set of rules that govern what state changes are valid, and you don't want to mess around with any of this "what do we do when we miss and update vs when we get two of the same update" nonsense, and the services aren't in a position to query the same database, then you should really consider a permissioned blockchain. Consensus hard, but it's harder if you're not using tools that understand that what they're going for is consensus.
Webhooks are simple and ubiquitous, and that's both a weakness and a strength. It's also why they are used for a lot of things, even things they are not great for (state sync).
These weaknesses are why we[1] added FIFO endpoints, Polling Endpoints, and what we call "Svix Stream" as ways to do ordered state synchronization (each with its own tradeoffs). This lets people consume the events in the way that best fits their use-case. We are working on more things to make the state sync even easier. I'd love to hear about more challenges people are facing with webhooks, as we want to make these things better.
OP: I'd love to hear more about your thoughts there, and will send you an email in a moment.
P.S, if you're unfamiliar, please check out Standard Webhooks[2]. It's a spec we created to help with signature verification that has been adopted by OpenAI, Anthropic, Google, and many others. We are chipping at one webhook challenge at a time. :)
1: I'm the founder of Svix (mentioned in the post), we do webhooks infrastructure as a service.
With webhooks, consumers get to asynchronously respond to updates from a provider. If no data has changed, a provider will not send any updates.
With SCROLL, consumers are responsible for choosing when to ask a provider for updates. Without a mechanism for knowing when data has changed, consumers will be forced to be pessimistic and poll providers for new data on some cadence.
I see two issues with the proposal: (1) SCROLL will lead to an increase in unnecessary network traffic for both the consumer and provider, and (2) because a consumer cannot know when data has changed, the lag between a consumer's local model and the provider's data model will be larger when with Webhooks.
Assuming you're not using the proposed streaming option, I suppose you could always send a webhook for that fact alone? In other words, an empty notification, with semantics of "something has probably changed, better poll the SCROLL if you aren't already".
Webhooks are a painful problem. To clarify, Stripe's events API definitely ships a cursor and polling it has been the method preferred by large consumers for a long time.
Stripe events API is one of the examples of how to do things properly. And SCROLL is just trying to create a common spec so that everyone offers a stripe-like event polling api.
This is really a great article from implementation perspective and the challenges associated with it. The OP already captured the reasons. Even though the reason 1 has been mentioned it did not go deep into it and somehow focussed a lot more on 2
> 1. Trigger a side effect: send the receipt, start the build, ping the channel.
> 2. Keep a copy of the provider’s data correct:
On high level
1. System can either be PUSH or PULL, webhooks are essentially push and towards the end the OP is exploring the possibility with PULL. The caveat is that OP already iterated the PUSH mechanisms thrice and is aware of all the hardships and is somehow hoping that PULL would solve them. Unfortunately the grass is same on other side too.
a) The availability of the server can always be questionable in PULL mechanisms and its a lot of load on servers to support this kind of data at scale in bulk to multiple customers. You are essentially getting into database table scans. Its becomes a lot more costly with NOSQL databases.
b) The customer would end up making way too many calls to server even if data is not available or there would be additional latency when data was updated and when it was queried. This is one of the reasons why servers prefer to push instead of pull if they can find a listener available on other side.
c) CRLs (Certificate revocation lists) are good example which are available for PULL, same for all clients and yet rarely anyone does it correctly or does it at all even though its in security domain. In fact they are simple files on webservers in most of the implementations.
2. The primary use case for Webhook is for triggering the side effect and allowing the customers to choose if they want to subscribe for that event. A customer subscribing for everything even if its non-actionable should just treat it as logging data.
3. Logging data can and always have gaps, it should never be treated as source of truth. I might question the need for deduplication, usually there is a unique identifier and almost all databases support insert ignore kind of clause. Logging the event data just provides you with better availability and latency, the source of truth is still with the provider if a next step needs to happen.
4. If user cancelled the subscription in stripe and the event never arrived then its a system design issue or system availability issue on the client side. The complete data checksum or bulk imports at night are attempt to fix the problem in a hammerhead way . I understand it exists in lot of places, however it defeats the whole purpose.
> its a system design issue or system availability issue on the client side
If all events need to arrive, then the problem is not "notification" (which would be solved by webhooks) but "database replication": subscribe to new events, fetch the full snapshot, fetch the updates in range, have the monotonic value to establish the "range" in the first place. Reach the eventual consistency.
The proposed SCROLL handles half of these, which limits its use-cases.
Nice write-up of the webhook data consistency problem - I've been bitten by some of those Stripe issues myself. I agree that webhooks should come hand-in-hand with a "records updated since" endpoint that allows high-level discrepancy checks and API endpoints to pull full logs for any record. There needs to be some sort of reconciliation process that is both fully verifiable and not a firehose of data.
The end here reminds me of "The Log: Real-time data's unifying abstraction" [0], which has unfortunately had a bit of link-rot since 2013.
One complication in this approach involves access-windows: What if my system is only supposed to be seeing stuff that happened during two separate weeks in the year, because those are the spans when it was subscribed or authorized?
So the data-host would need to maintain a concept of "connection history" for other services, and also use that to filter/modify its real event stream, inserting artificial "initial state" roll-ups of events that happened in dark periods.
That is completely fine. In the spec cursors just need to be lexicographically comparable. They could easily be timestamps if the provider chooses to. In many of my example they are just alphanumeric dictionaries.
This topic always surprises me. I do not understand the sequence of logic that leads people to build synchronisation mechanisms based only on webhooks.
Webhooks aren't at-least-once, nor at-most-once, nor are they guaranteed in-order. Some people build systems to make them more reliable, but if you really care about the data you need to think of a webhook delivery as best-effort, a bit like UDP.
That's before you get into all the extra complexities around these systems being owned by different people. For example, either or both system might have to roll back their database. Or either side might have a long-term bug in how they process webhooks, and now you have months of broken data.
My view is that the only reasonable thing is to start with the process that gets things back into sync if everything is broken. That almost certainly involves a poll or query of at least the upstream side, and maybe both sides.
I find that if you put a decent bit of engineering effort into that "disaster recovery" synchronisation, it can often act as the main or only synchronisation process for quite a lot of systems.
Stepping up from that, it's often useful to introduce webhooks as notifications only; that is, to provide a signal that some or all of the data is stale. You have to do a bit of consolidation, but this approach is usually enough to get completely reasonable latency for the kind of applications the author is describing.
Only if that wasn't enough for speed/scale reasons would I reach for a truly "push-driven" fast path. But you always have to be able to disaster recovery assuming the stream is wildly out of sync.
Some bits of the author's idea seem reasonable: certainly, I would love for there to be a standard protocol to request new data since some cursor or since some timestamp, ideally with some webhook notifications to give hints on when to poll.
The problem I have with the author's idea is that it is very strongly event-based, but the desired outcome isn't event-based. The desired outcome is almost always "the state over here looks like the state over there". Relying too strongly events ends up at the same kind of problem another level down: the "disaster recovery" script ends up wanting to compare the states anyway to figure out whether the events are broken.
Going fully event-sourced can work (although, I think, less often than advertised), but it really relies on everyone collectively agreeing on the same event stream being the source of truth. Once you start doing work across multiple organisations then that coordination is relatively rare.
What really surprises me is the variation in maturity on this topic. There seem to be people at all experience levels who are both doing this well and doing it badly. I have worked with people with decades of experience whose whole design just collapses if you ask "but what if X?" for some really banal values of X like "we have an outage for more than five minutes" or "we have to restore the DB to yesterday" or "someone, one time, accidentally merges a bug into master".
As an aside, I do find the obvious LLM-ness of the blog post and the proposal a bit disheartening. These are problems that require diligence and precision of thought. LLMs may be able to achieve those things, but that level of quality just isn't expressible in "Claudish".
> I find that if you put a decent bit of engineering effort into that "disaster recovery" synchronisation, it can often act as the main or only synchronisation process for quite a lot of systems.
Another benefit is that you get to exercise those disaster recovery mechanisms regularly as part of the normal functioning of the system, rather than a specialized path that is only rarely exercised (and thus may be broken when you need it the most).
I have seen it expressed here a few times, but these LLM blog post tech proposal things really give off the vibe (pun intended) of someone being gaslit by claude into thinking they are onto something really important or innovative. It feels really weird reading it. A bit like reading someone's somewhat self aggrandizing journal entries, or walking in on someone... Doin' stuff. I don't hate LLMs, they have made some aspects of my job easier for sure, but I think some people are still pretty off base on where and how it's acceptable to offload to an LLM.
Maybe the LLM has written maximum 50 words of the article by just being directed to switch things around and improve grammar and or internal consistency
"And here’s the absurd part: that history exists."
"It’s a jigsaw puzzle where the manufacturer had the original picture, cut it up, mailed me the pieces one at a time, lost a few in the post, mailed some twice, and printed nothing on the box."
"and that’s the entire problem: nothing announces a gap."
"None of this is any provider’s bug. Their webhooks work exactly as documented. The problem is what a webhook is: a notification, “something happened, here’s a POST about it.”"
Almost this entire section is clearly written by an LLM
I had the exact same thing with the Quickbooks api recently. You cannot trust the responses or webhooks at all.
On create a user or invoice for example sometimes it will return an error, yet it actually created the entity. This means you have to check manually after creating everything to know if its created properly.
Then you have the issue that sometimes quickbooks takes a while to update, and locks the company file while it does some background magic. This means you cannot immediately do the existence check, and also sometimes the check errors or times out which essentially means you need to keep checking forever until you can properly reconcile your db against theirs. But with hundreds/thousands of transactions per minute this state is never reached. You perpetually live in a state of trying to catch up but never managing it.
When I brought it up with Quickbooks dev support their response was literally "Its your job to make sure things are created properly in our system".
How did we get to this place where we started putting up with systems that cannot ever be trusted?
If you have two or more services that need to agree about state, and you have some set of rules that govern what state changes are valid, and you don't want to mess around with any of this "what do we do when we miss and update vs when we get two of the same update" nonsense, and the services aren't in a position to query the same database, then you should really consider a permissioned blockchain. Consensus hard, but it's harder if you're not using tools that understand that what they're going for is consensus.
Webhooks are simple and ubiquitous, and that's both a weakness and a strength. It's also why they are used for a lot of things, even things they are not great for (state sync).
These weaknesses are why we[1] added FIFO endpoints, Polling Endpoints, and what we call "Svix Stream" as ways to do ordered state synchronization (each with its own tradeoffs). This lets people consume the events in the way that best fits their use-case. We are working on more things to make the state sync even easier. I'd love to hear about more challenges people are facing with webhooks, as we want to make these things better.
OP: I'd love to hear more about your thoughts there, and will send you an email in a moment.
P.S, if you're unfamiliar, please check out Standard Webhooks[2]. It's a spec we created to help with signature verification that has been adopted by OpenAI, Anthropic, Google, and many others. We are chipping at one webhook challenge at a time. :)
1: I'm the founder of Svix (mentioned in the post), we do webhooks infrastructure as a service.
2: https://www.standardwebhooks.com/
With webhooks, consumers get to asynchronously respond to updates from a provider. If no data has changed, a provider will not send any updates.
With SCROLL, consumers are responsible for choosing when to ask a provider for updates. Without a mechanism for knowing when data has changed, consumers will be forced to be pessimistic and poll providers for new data on some cadence.
I see two issues with the proposal: (1) SCROLL will lead to an increase in unnecessary network traffic for both the consumer and provider, and (2) because a consumer cannot know when data has changed, the lag between a consumer's local model and the provider's data model will be larger when with Webhooks.
Assuming you're not using the proposed streaming option, I suppose you could always send a webhook for that fact alone? In other words, an empty notification, with semantics of "something has probably changed, better poll the SCROLL if you aren't already".
Doesn't doing that just reinvent eTags on hypertext resources from the RESTful wars and XML Web Services days 20 years ago?
1. Long Poll the cursor to pull down the latest events
2. Trigger a long-poll even if in exponential backoff because they shot you a webhook saying 'eTag changed!'
If your API is just wrapping Kafka, it can long-poll
Webhooks are a painful problem. To clarify, Stripe's events API definitely ships a cursor and polling it has been the method preferred by large consumers for a long time.
Stripe events API is one of the examples of how to do things properly. And SCROLL is just trying to create a common spec so that everyone offers a stripe-like event polling api.
This is really a great article from implementation perspective and the challenges associated with it. The OP already captured the reasons. Even though the reason 1 has been mentioned it did not go deep into it and somehow focussed a lot more on 2
> 1. Trigger a side effect: send the receipt, start the build, ping the channel. > 2. Keep a copy of the provider’s data correct:
On high level
1. System can either be PUSH or PULL, webhooks are essentially push and towards the end the OP is exploring the possibility with PULL. The caveat is that OP already iterated the PUSH mechanisms thrice and is aware of all the hardships and is somehow hoping that PULL would solve them. Unfortunately the grass is same on other side too.
2. The primary use case for Webhook is for triggering the side effect and allowing the customers to choose if they want to subscribe for that event. A customer subscribing for everything even if its non-actionable should just treat it as logging data.3. Logging data can and always have gaps, it should never be treated as source of truth. I might question the need for deduplication, usually there is a unique identifier and almost all databases support insert ignore kind of clause. Logging the event data just provides you with better availability and latency, the source of truth is still with the provider if a next step needs to happen.
4. If user cancelled the subscription in stripe and the event never arrived then its a system design issue or system availability issue on the client side. The complete data checksum or bulk imports at night are attempt to fix the problem in a hammerhead way . I understand it exists in lot of places, however it defeats the whole purpose.
> its a system design issue or system availability issue on the client side
If all events need to arrive, then the problem is not "notification" (which would be solved by webhooks) but "database replication": subscribe to new events, fetch the full snapshot, fetch the updates in range, have the monotonic value to establish the "range" in the first place. Reach the eventual consistency.
The proposed SCROLL handles half of these, which limits its use-cases.
Nice write-up of the webhook data consistency problem - I've been bitten by some of those Stripe issues myself. I agree that webhooks should come hand-in-hand with a "records updated since" endpoint that allows high-level discrepancy checks and API endpoints to pull full logs for any record. There needs to be some sort of reconciliation process that is both fully verifiable and not a firehose of data.
The proposed “feed” solution is functionally indistinguishable from incremental reconciliation.
The article presents a good framing and is well-written, but doesn’t really propose anything new.
The end here reminds me of "The Log: Real-time data's unifying abstraction" [0], which has unfortunately had a bit of link-rot since 2013.
One complication in this approach involves access-windows: What if my system is only supposed to be seeing stuff that happened during two separate weeks in the year, because those are the spans when it was subscribed or authorized?
So the data-host would need to maintain a concept of "connection history" for other services, and also use that to filter/modify its real event stream, inserting artificial "initial state" roll-ups of events that happened in dark periods.
[0] https://news.ycombinator.com/item?id=6916557
> Ask with a cursor and you resume where you left off.
How does the provider know what event the cursor you provided refers to?
Sounds like external state that needs to be managed ("cursor" -> timestamp)
That is completely fine. In the spec cursors just need to be lexicographically comparable. They could easily be timestamps if the provider chooses to. In many of my example they are just alphanumeric dictionaries.
This topic always surprises me. I do not understand the sequence of logic that leads people to build synchronisation mechanisms based only on webhooks.
Webhooks aren't at-least-once, nor at-most-once, nor are they guaranteed in-order. Some people build systems to make them more reliable, but if you really care about the data you need to think of a webhook delivery as best-effort, a bit like UDP.
That's before you get into all the extra complexities around these systems being owned by different people. For example, either or both system might have to roll back their database. Or either side might have a long-term bug in how they process webhooks, and now you have months of broken data.
My view is that the only reasonable thing is to start with the process that gets things back into sync if everything is broken. That almost certainly involves a poll or query of at least the upstream side, and maybe both sides.
I find that if you put a decent bit of engineering effort into that "disaster recovery" synchronisation, it can often act as the main or only synchronisation process for quite a lot of systems.
Stepping up from that, it's often useful to introduce webhooks as notifications only; that is, to provide a signal that some or all of the data is stale. You have to do a bit of consolidation, but this approach is usually enough to get completely reasonable latency for the kind of applications the author is describing.
Only if that wasn't enough for speed/scale reasons would I reach for a truly "push-driven" fast path. But you always have to be able to disaster recovery assuming the stream is wildly out of sync.
Some bits of the author's idea seem reasonable: certainly, I would love for there to be a standard protocol to request new data since some cursor or since some timestamp, ideally with some webhook notifications to give hints on when to poll.
The problem I have with the author's idea is that it is very strongly event-based, but the desired outcome isn't event-based. The desired outcome is almost always "the state over here looks like the state over there". Relying too strongly events ends up at the same kind of problem another level down: the "disaster recovery" script ends up wanting to compare the states anyway to figure out whether the events are broken.
Going fully event-sourced can work (although, I think, less often than advertised), but it really relies on everyone collectively agreeing on the same event stream being the source of truth. Once you start doing work across multiple organisations then that coordination is relatively rare.
What really surprises me is the variation in maturity on this topic. There seem to be people at all experience levels who are both doing this well and doing it badly. I have worked with people with decades of experience whose whole design just collapses if you ask "but what if X?" for some really banal values of X like "we have an outage for more than five minutes" or "we have to restore the DB to yesterday" or "someone, one time, accidentally merges a bug into master".
As an aside, I do find the obvious LLM-ness of the blog post and the proposal a bit disheartening. These are problems that require diligence and precision of thought. LLMs may be able to achieve those things, but that level of quality just isn't expressible in "Claudish".
> I find that if you put a decent bit of engineering effort into that "disaster recovery" synchronisation, it can often act as the main or only synchronisation process for quite a lot of systems.
Another benefit is that you get to exercise those disaster recovery mechanisms regularly as part of the normal functioning of the system, rather than a specialized path that is only rarely exercised (and thus may be broken when you need it the most).
I have seen it expressed here a few times, but these LLM blog post tech proposal things really give off the vibe (pun intended) of someone being gaslit by claude into thinking they are onto something really important or innovative. It feels really weird reading it. A bit like reading someone's somewhat self aggrandizing journal entries, or walking in on someone... Doin' stuff. I don't hate LLMs, they have made some aspects of my job easier for sure, but I think some people are still pretty off base on where and how it's acceptable to offload to an LLM.
Maybe the LLM has written maximum 50 words of the article by just being directed to switch things around and improve grammar and or internal consistency
"And here’s the absurd part: that history exists."
"It’s a jigsaw puzzle where the manufacturer had the original picture, cut it up, mailed me the pieces one at a time, lost a few in the post, mailed some twice, and printed nothing on the box."
"and that’s the entire problem: nothing announces a gap."
"None of this is any provider’s bug. Their webhooks work exactly as documented. The problem is what a webhook is: a notification, “something happened, here’s a POST about it.”"
Almost this entire section is clearly written by an LLM
Dude is not wrong ... but good luck convincing the Internet to switch to a sane system, when everyone already thinks web hooks are a "solved problem".