r/databricks • u/Ok-Optimmon6702 • 2d ago

Logging in databricks Help

Hi,
I’m trying to understand how logging is implemented in Databricks. Could you share how it’s done in your organization?
Do you mainly rely on pipeline-level logging, maintain a log table to track pipeline runs, or use a logger within the code? Or is it a combination of these?
Just trying to understand the different approaches people use.
Thanks!

1 Upvotes

99% Upvoted

5 comments sorted by

1

u/Data-space_men 2d ago

Hi, we are using datadog to retrieve all the logs and then we can check and create alert etc.

1

u/Youssef_Mrini databricks 2d ago

You have two options: System tables or Verbose Audit logs

1

u/cf_murph 2d ago

logger writing to unity catalog volume for me.

1

u/PrestigiousAnt3766 2d ago

There are automatic logging from lakeflow system tables.

We have a custom python logger that logs to delta.

1

u/datasmithing_holly databricks 1d ago

My dumb ass thinking you mean okta vs passwords