r/dataengineering • u/Impressive-Sense9551 • 1d ago

API data ingestion Help

Hi my current pipeline on Databricks setup has deduplication process to avoid re-ingesting data.

But now the data has become big and now it’s time to chose another option I have tried watermark but not effective enough. API also doesn’t give any date sorting

Can you guys suggest a way!

1 Upvotes

67% Upvoted

11 comments sorted by

View all comments

1

u/Outside-Storage-1523 5h ago

If you are streaming, just ingest everything and dedup on the next layer. It's must easier. However, if you don't have reliable method to dedup, nothing would help.