> The big argument against kafka-delta-ingest was Apache Kafka. If an organization has Kafka for other reasons, then kafka-delta-ingest can be a useful “sidecar” process to persist data flowing through Kafka. If however the organization is running Kafka just for ingestion, there are cheaper options available. As the organization evolved, the other consumers of Kafka drifted away, driving the value proposition of kafka-delta-ingest lower and lower.
It's great to see my name mentioned here, cheered me up as I'm not having a good weekend. It feels like a lifetime ago since I got to work on this and parquet-rs.
Yeah, I agree with the conclusion on Kafka here. Kafka is a very resource-heavy application that's worth replacing with leaner options.
Oxbow seems very interesting, I presume it's designed to forward data that trickles into some S3 bucket as parquet data?
> The big argument against kafka-delta-ingest was Apache Kafka. If an organization has Kafka for other reasons, then kafka-delta-ingest can be a useful “sidecar” process to persist data flowing through Kafka. If however the organization is running Kafka just for ingestion, there are cheaper options available. As the organization evolved, the other consumers of Kafka drifted away, driving the value proposition of kafka-delta-ingest lower and lower.
Sounds like how most of Kafka stories end.
It's great to see my name mentioned here, cheered me up as I'm not having a good weekend. It feels like a lifetime ago since I got to work on this and parquet-rs.
Yeah, I agree with the conclusion on Kafka here. Kafka is a very resource-heavy application that's worth replacing with leaner options.
Oxbow seems very interesting, I presume it's designed to forward data that trickles into some S3 bucket as parquet data?
If only for the initiative driving the work behind delta-rs, I’d say it was well worth it!
What about WarpStream? Isn't that purpose built for lower Kafka ingest costs?