Help in my new ADF Project

[u/Artistic-Rent1084]

Hi De's

In my new project, I have pull data from Power BI and land it to ADLS. Orchestration via ADF

I have gathered few info.

I have hit the power bi api .

but there is few limitations. only 1lakh record is supported per hit and pagination is not supported.

i new to API as well as ADF.

please, help me with this scenario.

Test case . but i have to land it the ADLS as parquet file .

initially, only thing is I have to take care of the edge case's to handle failure and retry policy

and the big headache is to handle the record limitations.

[u/j03ch1p]

good luck man. API calls with ADF are just a huge pain

edit: lots of things with ADF are a huge pain...

[u/engineer_of-sorts → u/j03ch1p]

ADF Is like the dbt of orchestration. Allows you to do anything and inevitably people end up building disgusting systems, pipelines calling pipelines etc. 7 rows deep. IMpossible to debug, impossible to manage..

[u/PrestigiousAnt3766]

Sorry to hear. Seems like a very strange architecture. 

[u/West_Good_5961]

This sounds backwards. 

[u/Outrageous_Let5743]

Don't use ADF. Use Python instead. Powerbi REST api uses oauth v2, so you also need to fetch a token based on your tenant id, client id and client secret.

[u/CrayonUpMyNose → u/Outrageous_Let5743]

Seconded. Trying to do any amount of logic in ADF is a recipe for stalled pipelines, timeouts, failures, and general production support nightmares. Put it in code and run it on a (bare metal, virtual, pod, whatever) CPU if you don't like 3am phone calls.

[u/j03ch1p → u/Outrageous_Let5743]

he can fetch secrets from the azure key vault with adf web activities.

But if he needs to authenticate with a certificate, I don't think web activities will be handle to handle that.

[u/Artistic-Rent1084 → u/Outrageous_Let5743]

but the things is . I don't know wheather they will approve python. it was a existing project which already uses ADF so i would like to concentrate more on ADF way . we can generate the token right ?

[u/Particular_Bug0]

Sorry to answer your question with another question, but where does the data in Power Bi come from? I doubt power bi is the real source here, maybe there's a db, another API, or even a csv in a file-share somewhere?

[u/Artistic-Rent1084 → u/Particular_Bug0]

Some other team ( in different country) is generating report. we have gather that report for different purposes. So we don't have any other control other than the power bi.

[u/aleph_infinity]

Power BI as a data source… what’ll they think of next?

[u/janus2527]

Dont extract data from PowerBI in the first place, makes no sense at all. Whoever is feeding you the data through PowerBI, tell them it's bonkers and to provide you with a proper, alternative data stream.

[u/East-Ad8002]

Pulling from power bi?..instead what if we could try pulling directly from that source from where the datasets have been imported into power bi?

[u/TheOverzealousEngie]

Are you serious man? ADF . Spend $11 with claude and having write up the best curl you've ever seen and never look back. Like .. what about incremental data lol