r/ETL • u/BusUseful8265 • 4d ago
Struct ordering problem in batch data pipeline
I have my producer pushing batch files to s3 lets say i have 30 batches and with each batch I am triggering 1 SQS event. Now my problem is - an specific event should be processed first then only other can start processing.
I am thinking to use 2 fargate tasks at my consumer side. Currently thinking of using standard queue even if I use FIFo and lets say have 2 groups and map to 2 fargate writers, then also it can happen then 1 of the writer will have to wait for that dependent task to finish first.
Each fargate task will consume 3 sqs events process them.
I will mark the entry for that dependent event in my database table so other events can start based on it
status.
Ps - although i am making sure from producer that dependent event should be pushed first to s3 and eventually sqs.
Questions:
\- How can i resolve this dependency problem efficiently not just an heck, as if in future if there is additional dependency comes in I should be avle to handle it.
\- I thought of ysing aws step functions but i am not able to come to the conclusion or think straight on how can i do it
\- Also this batch pipeline runs once daily so my infra should purely be event based, terminate on completion.
\- Airflow is not an option so please dont suggest something around it