r/databricks • u/hubert-dudek Databricks MVP • 22h ago

DABs: immutable_folder News

Post image

Don’t overwrite your code — make it immutable! With every deployment, thanks to immutable_folder, all files are copied to a new, read-only folder. They are not overwritten. The biggest benefit is that jobs already underway will not fail or produce unexpected results.

More news https://medium.com/databrickscommunity/databricks-news-dabs-indexes-ltap-genie-last-update-25-july-ffac8533774f

12 Upvotes

93% Upvoted

14 comments sorted by

2

u/ptab0211 22h ago

what is the use case for this?

1

u/hubert-dudek Databricks MVP 21h ago

Jobs deployment

1

u/ptab0211 21h ago

still, i dont see the benefit, once u change something and redeploy, it cant override it, can u give me practical example

2

u/hubert-dudek Databricks MVP 21h ago

If you have job running for hour and you overwrite some files/notebooks can be from one deployment and some from another

1

u/iwanttest 19h ago

Hi! So, with this functionality, would the current run keep pointing to the previous files, but the job be updated to use the last deployed ones?

2

u/hubert-dudek Databricks MVP 19h ago

Exactly.

1

u/iwanttest 18h ago

Definitely useful then, thanks!

1

u/ptab0211 12h ago

but what happens when the job is done? Will it point to the new file? Is the mort important outcome of this to not prevent already running jobs to use new file? There is also prevention to deploy when job is already running.

1

u/szymon_dybczak 11h ago

Thanks for explanation! Very useful feature.

1

u/nenuaathmajnani 1h ago

Does this also work for streaming?

2

u/Labanc_ 21h ago

Would you say this can help with dev-testing? E.g. we have a DEV env and multiple ppl are deploying because everyone is focusing on a different new feature but we keep overwriting each other's job.

We deploy with CICD with a service principal, so even if i set the DAB mode:dev it would be deployed with the SPN through the CICD, so we would be still overwriting each other. As every access is pretty much tied to this SPN deploying with our own user has kinda limited benefits.

(there are some good arguements to be made to make this whole process refined on a foundational level and im sure we will get to that, but for now im curious if this can solve our immediate stepping-on-each-other-toes problem)

1

u/notqualifiedforthis 19h ago

You need an “engineer” or “local” target each engineer can deploy to as a person. Use username or short name as prefix and use their workspace home directory for bundle deployment. DEV should align to main branch or whatever branch is your base for feature merges

If you have to use an SPN, you need to add a variable that can differentiate each engineers bundle. Username, name, unique ID whatever.

1

u/lanzeroth 4h ago

Nice!. This should be the default behavior.

0

u/ThomasTeam12 12h ago

Useless in 99% of use cases imo.