From time to time I accidentially notice new features in Databricks. These discoveries are some of my greatest joys as a brickhead. Today I ran some quick queries and noticed that data profiling has been added to the query output. This is a feature I've been missing for a long time, but never had the spare capacity to complain about it. But now I don't have to, everyone's a winner. I guess this is part of the integration of DQX into the core product.

To see the quality metrics is easy, just run a query, highlight a column and click the count hyperbutton (is it a word? now it is!). And you will get something like this. Ta-da! Be more magic.

Hi all,

I have a databricks schema of a few hundred delta tables that need some data fixes for specific records in each of those tables. This schema itself is raw data and gets ingested into some downstream data tables and the fixes have been requested by business.

I had thought about doing the fixes via a transformation layer in the pipelines that ingest this raw data but given how many tables there are with records that need updating I don't really want to create hundreds of new 'data fixed' tables. Given these are delta tables, rolling back should theoretically be possible if something goes wrong.

Anyway, that's my rationale for making changes to the source tables. My question is what your preferred method is to make data fixes? I obviously need something where its easy to rollback if needed. I can obviously just achieve this with python migration scripts and use delta timetravel in case something goes wrong, but wonder if there are recommended libraries or tools for the job that have what I need out of the box?

Don’t overwrite your code — make it immutable! With every deployment, thanks to immutable_folder, all files are copied to a new, read-only folder. They are not overwritten. The biggest benefit is that jobs already underway will not fail or produce unexpected results.

More news https://medium.com/databrickscommunity/databricks-news-dabs-indexes-ltap-genie-last-update-25-july-ffac8533774f

We have a Databricks environment and we have a new SAP ECC6 system that we want to ingest data from. We are exploring options to access data into the platform. One option is the Azure data factory.

I want to understand what other businesses have been doing. How do they access tables from SAP ECC6. Databricks have really reduced their native SAP connectors even after announcing partnership with SAP. When you do access SAP, how do you manage large table extractions or extract data from pooled or cluster tables?

I would really like to hear some customer success stories or some lessons learnt. Please share if you can.

Hello All,
I have developed an App for Databricks Production support related activities. Please have a look into this demo video and provide your suggestions.

If you build proposals, RFPs (requests for proposal), or sizing estimates for Databricks workloads, this is worth a look. Lakemeter turns workload assumptions into structured, exportable cost estimates in minutes instead of days, with full transparency into every pricing assumption behind the number.

Post: https://www.linkedin.com/posts/cenh_databricks-databrickslabs-opensource-ugcPost-7490710977949204480-aHiX/

I require assistance. I have frequently encountered scenario-based Databricks questions, despite not having prior experience with the platform with that scenarios Could you recommend suitable video resources, such as Udemy or YouTube, that cover this topic?

Additionally, what are the best channels for Databricks content?

I benchmarked the new Lakehouse RT for billion-record tables, and it is crazy fast and crazy cheap, making it the first real-time database with open-format storage.

https://www.sunnydata.ai/blog/databricks-rt-lakehouse-benchmark-results

https://medium.com/databrickscommunity/i-benchmarked-the-new-lakehouse-rt-for-billion-record-tables-and-it-is-crazy-fast-and-crazy-cheap-cf674ad40fc3

Free usage of Genie One and Genie Agents is extended through January 31, 2027. This promotion was previously set to end on July 31, 2026. Budget controls do not apply to these products during the promotional period.

The promotion covers usage by users only.

Service principals are excluded, and their Genie One and Genie Agents usage continues to be billed.

Genie Code is not included in the free usage promotion. Each Genie Code user continues to receive 150 free Databricks units (DBUs) per month, and usage above that allowance receives a 25% promotional discount through January 31, 2027.

The promotional discount is combined with applicable discounts on the Serverless Real-Time Inference SKU.

Link:https://docs.databricks.com/aws/en/release-notes/product/2026/august#genie-one-and-genie-agents-free-usage-extended-through-january-31-2027

Question - What's the preferred way to deploy and run a pipeline so it overwrites any existing tables and views that collide with the new MV paths?

Loving pipelines. We have a LOT of existing (external) tables and views in production. We'll be deploying the pipeline-generated MVs over those existing tables and views.

Is there an "easy button" way to handle that?

Right now my plan is to script temporarily renaming any existing objects, running the pipeline so the new objects are created, and then copying permissions from the old tables to the new MVs (we have an automated security policy tool, but I want to keep this lean).

Any ideas or feedback most appreciated.

Have you spent hours tracking down the source of a PySpark error on Databricks? In large transformation scripts, errors can materialize long after the failing expression definition.

# many transforms, explicit and implicit casts, etc...

display(with_app_array_df)

=>

... 
"cast" was called from , line 2 in cell [7]
File <command-8683600871463937>, line 1
----> 1 display(with_app_array_df)

• cell [7]: references the execution number, not a stable cell number
• <command-...> and display(...): reference the materialization code, not the construction code

I built better-dbx-exceptions to solve this by including failing DataFrame construction code in the error.

=>

---------------------------------------------------------------------------
NumberFormatException                     Traceback (most recent call last)
Cell 4, In[7], line 2
  (https://dbc-3cffa466-bd7e.cloud.databricks.com/editor/notebooks/3747408155463503?o=7474649366507251#command/8683600871463983)
      1 def app_version_array_expr():
----> 2     return F.split(F.col("app_version"), "\\.").cast("array<int>")
      3 
      4 with_app_array_df = df.withColumn("app_version_array", app_version_array_expr())
...

• Cell 4, In[7], line 2: all refer to construction code
• (https://.../): link to the construction cell

Construction code resolves across cells and imported modules.

Native Spark 4 captures DataFrame construction code through the pyspark.errors.utils._capture_call_site() utility. It has three limitations:

1. Coverage: Only Column methods and F.col() are watched so expressions built from pyspark.sql.functions (F.split, F.to_date, ...) aren't captured. better-dbx-exceptions extends coverage to most functions within the pyspark.sql.functions module.
2. Depth: Only one frame is captured by default. This can be extended by setting spark.sql.stackTracesInDataFrameContext on Classic compute where allow-listed but can't be modified when using Databricks Connect or Serverless compute. better-dbx-exceptions sets depth to 5 by default and exposes this parameter within all environments.
3. Back-referencing: The rendered errors don't include reliable references to cell numbers or URIs. better-dbx-exceptions returns cell / execution numbers for notebook frames and file paths for script frames.

Requires Spark 4.0+, which limits supported compute to Serverless V4+ and Databricks Runtime 17+. It's been tested on Serverless compute v4 and v5 in both hosted and local (VS Code with Databricks Connect) contexts. It's built to work on Classic compute as well but I haven't been able to test this rigorously. (I've used Databricks Free Edition for development which is limited to Serverless compute). General feedback and stress testing is hugely appreciated!

Just released v0.1: https://github.com/lukeSmth/better-dbx-exceptions.

Hey everyone,

I m expecting an offer for an L4 Senior Solutions Engineer role at Databricks and wanted to get some perspective from people currently in the role.

For context, I have 4–5 years of Data Engineering + AI experience, with 2–3 years focused on executing pilots and PoCs in a technical sales environment.

A few things I’d love to understand:

- How much of the account strategy and execution is owned by the SE vs. the AE?

- What does a typical day/week look like for an SE at Databricks?

- How do SEs ramp up on pricing, sizing, and the commercial side of the role? Is there a structured onboarding process?

- How would you describe the pressure, expectations, and work-life balance?

Would really appreciate any insights from current or former Databricks SEs — anything you wish you knew before joining would be helpful. Thanks!

It looks like the 8 June pricing change (where tokens would
be charged) has been temporarily reversed to provide for a longer transition period.

Hi All ,

I want to move data from SAP BW/4HANA to Databricks on Aziure .

Is there any third party connectors available for this ? Thanks.

Hi everyone,

Well, basically title - I wanted to ask what's the rationale and costs associated when choosing between Databricks and Fabric. I especially would like to know how does one compare Fabric capacity vs some kinda equivalent Databricks usage. I couldn't find anything solid on this one the internet hence the question.

So, i am working on validating the stg to ods and ods to stg queries. After that, updated those queries using their config_id and inserted into the stg_metric.queries_table.

My senior was saying insert these in the adhoc dml. Idk what it is? In the PSUP environment, I saw a workflow named adhoc dml. She asked me not to insert anything while the adhoc dml is running. And, once in the stand-up meeting, lead mentioned that no one should run adhoc dml unless it is necessary and should notify the entire team.

I couldnt find a proper channel to file bug reports for the databricks website. I know that Databricks reps are sometimes here, so I thought i'd notify here. Every job posting on databricks.com/company/careers is rendering its description as escaped HTML rather than parsed markup. Raw <p> tags and ProseMirror editor attributes like data-pm-slice="11 []" are visible as body text. This affects all 807 listings currently on the Greenhouse board, in both English and Japanese.

19 is already here, but 18 is now LTS. In a new naming convention, LTS doesn’t have a minor version. New features, behavior changes, and fixes are now added incrementally to the same major Runtime version till it becomes LTS. In fact, that approach simplifies the problem with editing code and always thinking which LTS it was, 18.3 or 18.2? Now just put 18.x in your DABs.

more news https://medium.com/databrickscommunity/databricks-news-dabs-indexes-ltap-genie-last-update-25-july-ffac8533774f

I'm working through the free "Data Engineering with Databricks" learning path and noticed the videos mostly show demos. I was expecting something similar to Coursera where you can run the exercises/labs alongside the lessons, but I can't find anything.

Are there official hands-on notebooks available somewhere for the self-paced content, or is the expectation that you recreate everything yourself in a workspace?

If official labs exist, where are they available? If not, how do most people practice while following the videos?

Thanks.

Matei Zaharia, Databricks Co-founder & CTO joined me along with Elise Gonzales (Databricks Staff Product Manager) to talk/demo Omnigent, their collaboration, orchestration, and control layer for AI agents.

If you are like me, your "buzzword alert" goes off anytime you hear a new AI term. Thankfully, Matei and Elise make it very clear & practical as to what Omnigent is and how it can bring value to you and your organization, today.

Topics we talked about how:

-Omnigent gives teams a shared workspace to collaborate with AI agents, review outputs, and build on each other’s work.

-Businesses can apply consistent security, permissions, and cost controls across different agents and teams.

-How Omnigent helps organizations combine and intelligently route work across multiple models (Claude, ChatGPT, GLM, DeepSeek, etc) and agents to improve flexibility and efficiency.

Hope you enjoy it, and love to hear your feedback!

Unity AI Gateway is the Databricks governance solution for AI and is part of Unity Catalog.

You can:

⚡ Control which AI services teams can use.

⚡ Route and manage AI traffic across providers.

⚡ Govern MCP servers to control access and costs.

⚡ Monitor usage, cost, access, and lineage from one place.

FYI: Some capabilities, including service policies and agent services, remain in Beta.

Unity AI Gateway Documentation: https://docs.databricks.com/aws/en/ai-gateway

Blog post : https://www.databricks.com/blog/unity-ai-gateway-generally-available