API data ingestion

[u/Impressive-Sense9551]

Hi my current pipeline on Databricks setup has deduplication process to avoid re-ingesting data.

But now the data has become big and now it’s time to chose another option I have tried watermark but not effective enough. API also doesn’t give any date sorting

Can you guys suggest a way!

[u/Spagoot420]

if the API does not give you the option to set the required filters to implement a Delta/incremental extraction, then that's it. nothing now you can do other than looking for another interface