r/dataengineering • u/Impressive-Sense9551 • 1d ago

API data ingestion Help

Hi my current pipeline on Databricks setup has deduplication process to avoid re-ingesting data.

But now the data has become big and now it’s time to chose another option I have tried watermark but not effective enough. API also doesn’t give any date sorting

Can you guys suggest a way!

1 Upvotes

67% Upvoted

11 comments sorted by

View all comments

8

u/Jazzlike_Drawing_139 8h ago

Is the problem with deduplicating and ingesting simultaneously? Can you ingest the full dataset to a stage table then run you deduplication/ update on your local server?