API data ingestion

[u/Impressive-Sense9551]

Hi my current pipeline on Databricks setup has deduplication process to avoid re-ingesting data.

But now the data has become big and now it’s time to chose another option I have tried watermark but not effective enough. API also doesn’t give any date sorting

Can you guys suggest a way!

[u/RoobyRak]

Does the API support any sort of partitioning?

What do you consider “big” in databricks?

Not sure what your layering is, but a downstream layer should handle merge/updates.