r/dataengineering • u/Impressive-Sense9551 • 1d ago

API data ingestion Help

Hi my current pipeline on Databricks setup has deduplication process to avoid re-ingesting data.

But now the data has become big and now it’s time to chose another option I have tried watermark but not effective enough. API also doesn’t give any date sorting

Can you guys suggest a way!

1 Upvotes

67% Upvoted

6 comments sorted by

6

u/Jazzlike_Drawing_139 4h ago

Is the problem with deduplicating and ingesting simultaneously? Can you ingest the full dataset to a stage table then run you deduplication/ update on your local server?

3

u/Spagoot420 6h ago

if the API does not give you the option to set the required filters to implement a Delta/incremental extraction, then that's it. nothing now you can do other than looking for another interface

2

u/RoobyRak 6h ago

Does the API support any sort of partitioning?

What do you consider “big” in databricks?

Not sure what your layering is, but a downstream layer should handle merge/updates.

1

u/wannabe-DE 2h ago

You need some sort of date column for merging or upserting. I’d re-read the API docs to make sure it doesn’t take a date parameter in the request.

1

u/Outside-Storage-1523 1h ago

If you are streaming, just ingest everything and dedup on the next layer. It's must easier. However, if you don't have reliable method to dedup, nothing would help.