The Scuba example is convincing for debugging. What the post doesn’t cover is cost. If you keep raw wide events instead of pre-aggregating, how do you handle storage and query cost at scale? Is sampling the main lever, or do teams still keep cheap metrics for alerting and use events only for investigation?
The whole wide events thing is Honeycomb’s marketing pitch. It’s essentially offering developer velocity, ie you don’t have to think about what should be a metric in advance. They have an open source project called Refinery which is a smart sampling proxy that samples successful requests but always forwards errors, slow queries, etc.
You can do anything you like with any approach but if they are mostly about ad hoc queries, dumping everything into BigQuery as a wide event can be the most economical. However if you want to run alerting, SLOs, etc with frequent queries over aggregated metrics, the best bet is using OTel Collector and metrics store of your choice like Prometheus or Mimir.
Most companies could save a bundle on observability if they were to move to a hybrid that includes both of those approaches instead of cloud provider logging, log-based metrics, and alerting.
Potential PII nightmare. You're now potentially circumventing access controls from sensitive data.
(2024)
prev. submission - https://news.ycombinator.com/item?id=39529775
Why is 77rushi77's comment flagged + dead? Seemed like a good question.
Edit: Looks like it's been restored.
Looks like it's because of:
> Don't post generated text or AI-edited text. HN is for conversation between humans.
But who can tell, you could ask the mods (email found in the guidelines)
Doesn't seem particularly robot-generated to me, nor to the classifier I made for a related purpose: https://xkqr.org/aicomment/#lang=none&prior=50&c=Pc_BbcMwEAT...
Then that check accounts history and tell me if you see a pattern or not, this one is really obvious.
The Scuba example is convincing for debugging. What the post doesn’t cover is cost. If you keep raw wide events instead of pre-aggregating, how do you handle storage and query cost at scale? Is sampling the main lever, or do teams still keep cheap metrics for alerting and use events only for investigation?
The whole wide events thing is Honeycomb’s marketing pitch. It’s essentially offering developer velocity, ie you don’t have to think about what should be a metric in advance. They have an open source project called Refinery which is a smart sampling proxy that samples successful requests but always forwards errors, slow queries, etc.
You can do anything you like with any approach but if they are mostly about ad hoc queries, dumping everything into BigQuery as a wide event can be the most economical. However if you want to run alerting, SLOs, etc with frequent queries over aggregated metrics, the best bet is using OTel Collector and metrics store of your choice like Prometheus or Mimir.
Most companies could save a bundle on observability if they were to move to a hybrid that includes both of those approaches instead of cloud provider logging, log-based metrics, and alerting.