Finally, a tool that tackles this problem without forcing you to adopt an entire ecosystem. I appreciate how this leans into the Unix philosophy by just focusing on the decomposition step and doing it well. Really appreciate the modular architecture here.
Thank you and we do really appreciate it. The Unix philosophy was certainly one of the influences upon us, we wanted CtrlB Decompose to carry out one task well, converting noisy logs into structured and compact representations which are easier for both humans and LLMs to handle.
A few days ago I made something to use in addition to head/tail: Venetianblinds.js shows equally spaced chunks of a file https://github.com/firasd/venetianblinds
So I can kinda see these as being part of the same workflow like VBlinds for 'what does this file even look like' before calling the decompose
I really like the idea of this. What I'd be looking for is to see how you convert those into OpenTelemetry metrics/logs or even traces.
if you find correlations between multiple log rows, those could be put into logs and mark with a trace/span id according to OpenTelemetry spec. Then even more tools could be used to actually analyze, alert, etc on those logs.
Also, extracting attributes, group logs, aggregate, etc. Those could be made searchable before being retrieved of a tool like sluicio, grafana, datadog etc so less data is being shipped (thereby keeping cost low).
Thereby, going from million of log rows no-one will read, to valuable metrics, shipping to a tool for visualization.
Thanks a lot, I truly do appreciate the thoughtful feedback.
We agree that there are a great many interesting downstream possibilities concerning OpenTelemetry, correlation, derived metrics, and more sophisticated telemetry pipelines once you are able to reliably extract structure from noisy logs.
To us, CtrlB Decompose represents the initial step. Although we launched it as a standalone tool for producing logs that are friendly to LLMs, it also feeds into part of CtrlB's special log compression pipeline. The identical structured representation aids in reducing both storage space and token usage without losing the information required for search, investigation, and AI-assisted debugging.
The idea you've advanced regarding enriching the logs with trace context and generating higher-level signals before sending them on downstream is indeed a very interesting one, and we are currently considering it as the project develops. Thank you for bringing it to our attention!
Finally, a tool that tackles this problem without forcing you to adopt an entire ecosystem. I appreciate how this leans into the Unix philosophy by just focusing on the decomposition step and doing it well. Really appreciate the modular architecture here.
Thank you and we do really appreciate it. The Unix philosophy was certainly one of the influences upon us, we wanted CtrlB Decompose to carry out one task well, converting noisy logs into structured and compact representations which are easier for both humans and LLMs to handle.
Very interesting.
A few days ago I made something to use in addition to head/tail: Venetianblinds.js shows equally spaced chunks of a file https://github.com/firasd/venetianblinds
So I can kinda see these as being part of the same workflow like VBlinds for 'what does this file even look like' before calling the decompose
I really like the idea of this. What I'd be looking for is to see how you convert those into OpenTelemetry metrics/logs or even traces.
if you find correlations between multiple log rows, those could be put into logs and mark with a trace/span id according to OpenTelemetry spec. Then even more tools could be used to actually analyze, alert, etc on those logs. Also, extracting attributes, group logs, aggregate, etc. Those could be made searchable before being retrieved of a tool like sluicio, grafana, datadog etc so less data is being shipped (thereby keeping cost low).
Thereby, going from million of log rows no-one will read, to valuable metrics, shipping to a tool for visualization.
good work!
Thanks a lot, I truly do appreciate the thoughtful feedback.
We agree that there are a great many interesting downstream possibilities concerning OpenTelemetry, correlation, derived metrics, and more sophisticated telemetry pipelines once you are able to reliably extract structure from noisy logs.
To us, CtrlB Decompose represents the initial step. Although we launched it as a standalone tool for producing logs that are friendly to LLMs, it also feeds into part of CtrlB's special log compression pipeline. The identical structured representation aids in reducing both storage space and token usage without losing the information required for search, investigation, and AI-assisted debugging.
The idea you've advanced regarding enriching the logs with trace context and generating higher-level signals before sending them on downstream is indeed a very interesting one, and we are currently considering it as the project develops. Thank you for bringing it to our attention!
Skill written, problem solved with my log feeding. Thanks!
Thanks! Happy to hear it solved a real problem for you.