At a past job I owned both the migration to and running of a centralized alert aggregator. The idea was that any team at the firm could use a standard logging library or Prometheus style metrics to send alerts etc.
We were migrating from a "witches brew" of custom libraries, random alerts, alerts from different teams etc.
It was, as you can imagine, a total mess of alert metadata.
Some teams would just jam an entire alert into one field. Others would have 50+ metadata label alerts. And so on.
We eventually got everything into a standard but it was a lot of work. We even resorted to embedding SREs on each team to help clean things up as "alert tag cleanup" is never high on anyone's to do list.
It gave me a new appreciation of how if you let each team organically create their own alert plan while also never doing any alert gardening, a small problem can rapidly become a big problem.
Putting your writing in version control turns out to buy you a lot of metadata for free. These days it's an Obsidian vault in git, but I used to just keep plain markdown files for notes/writing/etc
At a past job I owned both the migration to and running of a centralized alert aggregator. The idea was that any team at the firm could use a standard logging library or Prometheus style metrics to send alerts etc.
We were migrating from a "witches brew" of custom libraries, random alerts, alerts from different teams etc.
It was, as you can imagine, a total mess of alert metadata.
Some teams would just jam an entire alert into one field. Others would have 50+ metadata label alerts. And so on.
We eventually got everything into a standard but it was a lot of work. We even resorted to embedding SREs on each team to help clean things up as "alert tag cleanup" is never high on anyone's to do list.
It gave me a new appreciation of how if you let each team organically create their own alert plan while also never doing any alert gardening, a small problem can rapidly become a big problem.
Putting your writing in version control turns out to buy you a lot of metadata for free. These days it's an Obsidian vault in git, but I used to just keep plain markdown files for notes/writing/etc