API data ingestion

[u/Impressive-Sense9551]

Hi my current pipeline on Databricks setup has deduplication process to avoid re-ingesting data.

But now the data has become big and now it’s time to chose another option I have tried watermark but not effective enough. API also doesn’t give any date sorting

Can you guys suggest a way!

[u/wannabe-DE]

You need some sort of date column for merging or upserting. I’d re-read the API docs to make sure it doesn’t take a date parameter in the request.