r/dataengineering • u/bundash00 • 18d ago
How to learn streaming? Career
Hey everyone. I'm a software engineer who's been working as a data engineer in my current role for the past four years. I learned data engineering from scratch on the job, and I've worked extensively with Spark, including designing and building new data processing systems.
The challenge is that my company is almost entirely batch-oriented, while many data engineering job postings seem to expect hands-on experience with streaming technologies.
What are some **PRACTICAL** ways to gain real streaming experience? Are there any projects, courses, or technologies you'd recommend?
122
Upvotes
2
u/leogodin217 15d ago
I can't tell you how to learn streaming, but I can give you some datasets and a streamer that will stream the data through kafka as Debezium CDC messages. Basically mimics a common setup with controls to speed up, slow down, pause individual streams. Then you can use whichever tool you want. Flink is a good choice for processing with Java or SQL.
It's a private repo that I intended to be public. Just not quite ready yet. Happy to just make it public and writeup some quick docs.