I used to be extremely into Dataflow/Apache Beam and literally had the paper printed on my desk AND the book. I agree they got Event time versus processing time and Never rely on completeness right, and I love they go deeper into why that was just a hard thing to accept. My head still hurts thinking about unbounded stream triggers and I'm happy we got around to a table-centric model. I still think taking some ideas from Spanner, databases as message bus or consistency information per row, basically in-database CQRS would have been nice, though. Great paper.
I worked in the streaming area for a decade, doing research and development (see: https://scholar.google.com/citations?user=Rdf5OIYAAAAJ&hl=en). After moving on from streaming specifically and moving into the general problems in large data warehouses, I also concluded: just default to SQL for all analytics and the database lens is the best way to think about streaming for analytics.
I still do think that stream programming models are extremely interesting and powerful. But I used to think they would eventually become more mainstream as a way to elegantly program for high throughput, low latency massively parallel systems. That has not been the case, and I no longer think that it will be. People get by with the existing programming languages and models, that seems to be fine.
I've similarly spent a decade in the streaming space, started a stream processing startup, built three streaming platforms at various large companies... and I basically agree.
Streaming always felt like it was about to happen, and it kept not happening, in a year of the linux desktop sort of way. I (and others, who also optimistically started stream processing startups) thought it was a technology problem but ultimately it's a demand problem: very few companies actually need low latency processing. And continuously running, stateful systems are inherently more complex to operate and evolve compared to batch systems.
Now that relatively low-latency ingest into warehouses and datalakes is easily achieved, it's really hard to make the argument to invest in complicated streaming systems.
That said, it still feels like we've landed in a suboptimal spot. Stream processors (in particular, those following the dataflow model) solve two problems that are hard in batch systems:
1. Determining and signaling completeness for a time period (when have I likely received all of the data for 12:01pm, such that I can now safely process it)
2. Avoiding expensive recomputations for periodic queries
I do think long term we'll see some amount of streaming influence moving back into the batch systems, particularly as object storage gives us the ability to run stateful workloads with less operational headache.
(If anyone else finds these problems interesting, I'm hiring for my stream processing team at Cloudflare. Email in my profile)
> People get by with the existing programming languages and models, that seems to be fine.
People like what they like, and they have trouble with anything other than imperative structured programming. It’s been a pain over the last decade plus to get people just to try functional programming, and they already know what functions are!
It’s so hard I think it’s a losing battle. No matter how good a paradigm people will always stick to a) what they know and b) what they’re paid to know.
However, I still think there’s a future for paradigms like dataflow because LLMs use random esoteric languages with great ease, and without complaint. They will use the tool for the job, even if they’re unfamiliar with it at first. “Mainstream” doesn’t really matter anymore.
> That has not been the case, and I no longer think that it will be. People get by with the existing programming languages and models, that seems to be fine.
.. as long as the processing they do fits in a map().
I still deeply believe that a new class of programs can be unlocked by using reduce() or all_gather() in a massive concurrent stream.
In a stadium, why can't a uuid be embedded as an acoustic chirp emitted by all the smartphones, then triangulation-reduced in a 3D map of all devices, enabling a stadium-wide sound system?
Why don't we have MMORPGs that can deal with 100k users in a single game instance?
I used to be extremely into Dataflow/Apache Beam and literally had the paper printed on my desk AND the book. I agree they got Event time versus processing time and Never rely on completeness right, and I love they go deeper into why that was just a hard thing to accept. My head still hurts thinking about unbounded stream triggers and I'm happy we got around to a table-centric model. I still think taking some ideas from Spanner, databases as message bus or consistency information per row, basically in-database CQRS would have been nice, though. Great paper.
I worked in the streaming area for a decade, doing research and development (see: https://scholar.google.com/citations?user=Rdf5OIYAAAAJ&hl=en). After moving on from streaming specifically and moving into the general problems in large data warehouses, I also concluded: just default to SQL for all analytics and the database lens is the best way to think about streaming for analytics.
I still do think that stream programming models are extremely interesting and powerful. But I used to think they would eventually become more mainstream as a way to elegantly program for high throughput, low latency massively parallel systems. That has not been the case, and I no longer think that it will be. People get by with the existing programming languages and models, that seems to be fine.
I've similarly spent a decade in the streaming space, started a stream processing startup, built three streaming platforms at various large companies... and I basically agree.
Streaming always felt like it was about to happen, and it kept not happening, in a year of the linux desktop sort of way. I (and others, who also optimistically started stream processing startups) thought it was a technology problem but ultimately it's a demand problem: very few companies actually need low latency processing. And continuously running, stateful systems are inherently more complex to operate and evolve compared to batch systems.
Now that relatively low-latency ingest into warehouses and datalakes is easily achieved, it's really hard to make the argument to invest in complicated streaming systems.
That said, it still feels like we've landed in a suboptimal spot. Stream processors (in particular, those following the dataflow model) solve two problems that are hard in batch systems:
1. Determining and signaling completeness for a time period (when have I likely received all of the data for 12:01pm, such that I can now safely process it)
2. Avoiding expensive recomputations for periodic queries
I do think long term we'll see some amount of streaming influence moving back into the batch systems, particularly as object storage gives us the ability to run stateful workloads with less operational headache.
(If anyone else finds these problems interesting, I'm hiring for my stream processing team at Cloudflare. Email in my profile)
> People get by with the existing programming languages and models, that seems to be fine.
People like what they like, and they have trouble with anything other than imperative structured programming. It’s been a pain over the last decade plus to get people just to try functional programming, and they already know what functions are!
It’s so hard I think it’s a losing battle. No matter how good a paradigm people will always stick to a) what they know and b) what they’re paid to know.
However, I still think there’s a future for paradigms like dataflow because LLMs use random esoteric languages with great ease, and without complaint. They will use the tool for the job, even if they’re unfamiliar with it at first. “Mainstream” doesn’t really matter anymore.
> That has not been the case, and I no longer think that it will be. People get by with the existing programming languages and models, that seems to be fine.
.. as long as the processing they do fits in a map().
I still deeply believe that a new class of programs can be unlocked by using reduce() or all_gather() in a massive concurrent stream.
In a stadium, why can't a uuid be embedded as an acoustic chirp emitted by all the smartphones, then triangulation-reduced in a 3D map of all devices, enabling a stadium-wide sound system?
Why don't we have MMORPGs that can deal with 100k users in a single game instance?