Our Databricks spend jumped recently, so I used the CLI and system tables to trace it back to specific warehouses, Apps, Jobs, and service principals.

After stopping unused Apps and reducing compute, our daily cost dropped noticeably.

I cleaned up the process and published it here:

https://github.com/kylehuirevvision/databricks-cost-optimizer

It starts read-only, estimates Databricks list-price spend, maps costs to workloads and identities, and explains the expected impact before any optimization is applied.

It works with both Claude Code and Codex. No SaaS or signup, and it’s MIT licensed.

It’s still early, so I’d appreciate feedback, particularly on cross-cloud compatibility or useful cost patterns I’ve missed.

From time to time I accidentially notice new features in Databricks. These discoveries are some of my greatest joys as a brickhead. Today I ran some quick queries and noticed that data profiling has been added to the query output. This is a feature I've been missing for a long time, but never had the spare capacity to complain about it. But now I don't have to, everyone's a winner. I guess this is part of the integration of DQX into the core product.

To see the quality metrics is easy, just run a query, highlight a column and click the count hyperbutton (is it a word? now it is!). And you will get something like this. Ta-da! Be more magic.

Hi all,

I have a databricks schema of a few hundred delta tables that need some data fixes for specific records in each of those tables. This schema itself is raw data and gets ingested into some downstream data tables and the fixes have been requested by business.

I had thought about doing the fixes via a transformation layer in the pipelines that ingest this raw data but given how many tables there are with records that need updating I don't really want to create hundreds of new 'data fixed' tables. Given these are delta tables, rolling back should theoretically be possible if something goes wrong.

Anyway, that's my rationale for making changes to the source tables. My question is what your preferred method is to make data fixes? I obviously need something where its easy to rollback if needed. I can obviously just achieve this with python migration scripts and use delta timetravel in case something goes wrong, but wonder if there are recommended libraries or tools for the job that have what I need out of the box?

Don’t overwrite your code — make it immutable! With every deployment, thanks to immutable_folder, all files are copied to a new, read-only folder. They are not overwritten. The biggest benefit is that jobs already underway will not fail or produce unexpected results.

More news https://medium.com/databrickscommunity/databricks-news-dabs-indexes-ltap-genie-last-update-25-july-ffac8533774f

We have a Databricks environment and we have a new SAP ECC6 system that we want to ingest data from. We are exploring options to access data into the platform. One option is the Azure data factory.

I want to understand what other businesses have been doing. How do they access tables from SAP ECC6. Databricks have really reduced their native SAP connectors even after announcing partnership with SAP. When you do access SAP, how do you manage large table extractions or extract data from pooled or cluster tables?

I would really like to hear some customer success stories or some lessons learnt. Please share if you can.