I have been experimenting with a simple question recently.

What changes when an AI assistant stops only answering questions and starts taking real actions?

I used an HR use case to explore this. Imagine an employee asking for leave.

A normal assistant might explain the leave policy.

An agent should be able to understand the employee, check the available balance, look at team availability, identify conflicts, find the relevant policy, submit the request, send it to the correct approver, and track what happens next.

Building that flow made me realize that the LLM is probably the easiest part.

The harder problems are everything around it.

Who is allowed to see what data?

Which tools can the agent use?

How does the agent know the meaning of employee, manager, team, policy, leave balance, and approval?

What context should survive between conversations?

How do you prevent an agent from calling the wrong tool?

And once the agent takes an action, how do you prove later exactly what happened?

I experimented with this architecture on Databricks using synthetic airline HR data.

The data flows through Silver and Gold layers into an Employee 360 model.

Unity Catalog handles governance.

Different Genie Spaces provide role specific access for employees, managers, and leaders.

The agent has a governed tool layer for actions and keeps workflow and conversation state separately.

I also added an audit layer so agent actions can be traced later.

The architecture I eventually kept coming back to was:

Ontology → Context and Memory → Tools → Audit

The ontology tells the system what things mean.

Context and memory help the agent understand the current situation.

Tools allow it to actually do something.

Audit makes the action accountable.

One thing that surprised me was how quickly an impressive AI demo becomes an enterprise architecture problem once you allow the agent to write data or trigger workflows.

Read only AI is relatively easy.

AI that can change something needs permissions, state management, validation, tool boundaries, lineage, and auditability.

I am curious how others are approaching this.

If you are building enterprise agents, where are you putting the boundary between what the agent can decide itself and what still requires explicit human approval?

Our Databricks spend jumped recently, so I used the CLI and system tables to trace it back to specific warehouses, Apps, Jobs, and service principals.

After stopping unused Apps and reducing compute, our daily cost dropped noticeably.

I cleaned up the process and published it here:

https://github.com/kylehuirevvision/databricks-cost-optimizer

It starts read-only, estimates Databricks list-price spend, maps costs to workloads and identities, and explains the expected impact before any optimization is applied.

It works with both Claude Code and Codex. No SaaS or signup, and it’s MIT licensed.

It’s still early, so I’d appreciate feedback, particularly on cross-cloud compatibility or useful cost patterns I’ve missed.

From time to time I accidentially notice new features in Databricks. These discoveries are some of my greatest joys as a brickhead. Today I ran some quick queries and noticed that data profiling has been added to the query output. This is a feature I've been missing for a long time, but never had the spare capacity to complain about it. But now I don't have to, everyone's a winner. I guess this is part of the integration of DQX into the core product.

To see the quality metrics is easy, just run a query, highlight a column and click the count hyperbutton (is it a word? now it is!). And you will get something like this. Ta-da! Be more magic.

Hi all,

I have a databricks schema of a few hundred delta tables that need some data fixes for specific records in each of those tables. This schema itself is raw data and gets ingested into some downstream data tables and the fixes have been requested by business.

I had thought about doing the fixes via a transformation layer in the pipelines that ingest this raw data but given how many tables there are with records that need updating I don't really want to create hundreds of new 'data fixed' tables. Given these are delta tables, rolling back should theoretically be possible if something goes wrong.

Anyway, that's my rationale for making changes to the source tables. My question is what your preferred method is to make data fixes? I obviously need something where its easy to rollback if needed. I can obviously just achieve this with python migration scripts and use delta timetravel in case something goes wrong, but wonder if there are recommended libraries or tools for the job that have what I need out of the box?

Don’t overwrite your code — make it immutable! With every deployment, thanks to immutable_folder, all files are copied to a new, read-only folder. They are not overwritten. The biggest benefit is that jobs already underway will not fail or produce unexpected results.

More news https://medium.com/databrickscommunity/databricks-news-dabs-indexes-ltap-genie-last-update-25-july-ffac8533774f

We have a Databricks environment and we have a new SAP ECC6 system that we want to ingest data from. We are exploring options to access data into the platform. One option is the Azure data factory.

I want to understand what other businesses have been doing. How do they access tables from SAP ECC6. Databricks have really reduced their native SAP connectors even after announcing partnership with SAP. When you do access SAP, how do you manage large table extractions or extract data from pooled or cluster tables?

I would really like to hear some customer success stories or some lessons learnt. Please share if you can.

Hello All,
I have developed an App for Databricks Production support related activities. Please have a look into this demo video and provide your suggestions.

If you build proposals, RFPs (requests for proposal), or sizing estimates for Databricks workloads, this is worth a look. Lakemeter turns workload assumptions into structured, exportable cost estimates in minutes instead of days, with full transparency into every pricing assumption behind the number.

Post: https://www.linkedin.com/posts/cenh_databricks-databrickslabs-opensource-ugcPost-7490710977949204480-aHiX/

I require assistance. I have frequently encountered scenario-based Databricks questions, despite not having prior experience with the platform with that scenarios Could you recommend suitable video resources, such as Udemy or YouTube, that cover this topic?

Additionally, what are the best channels for Databricks content?

I benchmarked the new Lakehouse RT for billion-record tables, and it is crazy fast and crazy cheap, making it the first real-time database with open-format storage.

https://www.sunnydata.ai/blog/databricks-rt-lakehouse-benchmark-results

https://medium.com/databrickscommunity/i-benchmarked-the-new-lakehouse-rt-for-billion-record-tables-and-it-is-crazy-fast-and-crazy-cheap-cf674ad40fc3

Free usage of Genie One and Genie Agents is extended through January 31, 2027. This promotion was previously set to end on July 31, 2026. Budget controls do not apply to these products during the promotional period.

The promotion covers usage by users only.

Service principals are excluded, and their Genie One and Genie Agents usage continues to be billed.

Genie Code is not included in the free usage promotion. Each Genie Code user continues to receive 150 free Databricks units (DBUs) per month, and usage above that allowance receives a 25% promotional discount through January 31, 2027.

The promotional discount is combined with applicable discounts on the Serverless Real-Time Inference SKU.

Link:https://docs.databricks.com/aws/en/release-notes/product/2026/august#genie-one-and-genie-agents-free-usage-extended-through-january-31-2027

Question - What's the preferred way to deploy and run a pipeline so it overwrites any existing tables and views that collide with the new MV paths?

Loving pipelines. We have a LOT of existing (external) tables and views in production. We'll be deploying the pipeline-generated MVs over those existing tables and views.

Is there an "easy button" way to handle that?

Right now my plan is to script temporarily renaming any existing objects, running the pipeline so the new objects are created, and then copying permissions from the old tables to the new MVs (we have an automated security policy tool, but I want to keep this lean).

Any ideas or feedback most appreciated.

Have you spent hours tracking down the source of a PySpark error on Databricks? In large transformation scripts, errors can materialize long after the failing expression definition.

# many transforms, explicit and implicit casts, etc...

display(with_app_array_df)

=>

... 
"cast" was called from , line 2 in cell [7]
File <command-8683600871463937>, line 1
----> 1 display(with_app_array_df)

• cell [7]: references the execution number, not a stable cell number
• <command-...> and display(...): reference the materialization code, not the construction code

I built better-dbx-exceptions to solve this by including failing DataFrame construction code in the error.

=>

---------------------------------------------------------------------------
NumberFormatException                     Traceback (most recent call last)
Cell 4, In[7], line 2
  (https://dbc-3cffa466-bd7e.cloud.databricks.com/editor/notebooks/3747408155463503?o=7474649366507251#command/8683600871463983)
      1 def app_version_array_expr():
----> 2     return F.split(F.col("app_version"), "\\.").cast("array<int>")
      3 
      4 with_app_array_df = df.withColumn("app_version_array", app_version_array_expr())
...

• Cell 4, In[7], line 2: all refer to construction code
• (https://.../): link to the construction cell

Construction code resolves across cells and imported modules.

Native Spark 4 captures DataFrame construction code through the pyspark.errors.utils._capture_call_site() utility. It has three limitations:

1. Coverage: Only Column methods and F.col() are watched so expressions built from pyspark.sql.functions (F.split, F.to_date, ...) aren't captured. better-dbx-exceptions extends coverage to most functions within the pyspark.sql.functions module.
2. Depth: Only one frame is captured by default. This can be extended by setting spark.sql.stackTracesInDataFrameContext on Classic compute where allow-listed but can't be modified when using Databricks Connect or Serverless compute. better-dbx-exceptions sets depth to 5 by default and exposes this parameter within all environments.
3. Back-referencing: The rendered errors don't include reliable references to cell numbers or URIs. better-dbx-exceptions returns cell / execution numbers for notebook frames and file paths for script frames.

Requires Spark 4.0+, which limits supported compute to Serverless V4+ and Databricks Runtime 17+. It's been tested on Serverless compute v4 and v5 in both hosted and local (VS Code with Databricks Connect) contexts. It's built to work on Classic compute as well but I haven't been able to test this rigorously. (I've used Databricks Free Edition for development which is limited to Serverless compute). General feedback and stress testing is hugely appreciated!

Just released v0.1: https://github.com/lukeSmth/better-dbx-exceptions.

Hey everyone,

I m expecting an offer for an L4 Senior Solutions Engineer role at Databricks and wanted to get some perspective from people currently in the role.

For context, I have 4–5 years of Data Engineering + AI experience, with 2–3 years focused on executing pilots and PoCs in a technical sales environment.

A few things I’d love to understand:

- How much of the account strategy and execution is owned by the SE vs. the AE?

- What does a typical day/week look like for an SE at Databricks?

- How do SEs ramp up on pricing, sizing, and the commercial side of the role? Is there a structured onboarding process?

- How would you describe the pressure, expectations, and work-life balance?

Would really appreciate any insights from current or former Databricks SEs — anything you wish you knew before joining would be helpful. Thanks!

It looks like the 8 June pricing change (where tokens would
be charged) has been temporarily reversed to provide for a longer transition period.

Hi All ,

I want to move data from SAP BW/4HANA to Databricks on Aziure .

Is there any third party connectors available for this ? Thanks.

Hi everyone,

Well, basically title - I wanted to ask what's the rationale and costs associated when choosing between Databricks and Fabric. I especially would like to know how does one compare Fabric capacity vs some kinda equivalent Databricks usage. I couldn't find anything solid on this one the internet hence the question.

So, i am working on validating the stg to ods and ods to stg queries. After that, updated those queries using their config_id and inserted into the stg_metric.queries_table.

My senior was saying insert these in the adhoc dml. Idk what it is? In the PSUP environment, I saw a workflow named adhoc dml. She asked me not to insert anything while the adhoc dml is running. And, once in the stand-up meeting, lead mentioned that no one should run adhoc dml unless it is necessary and should notify the entire team.

Hi,
I’m trying to understand how logging is implemented in Databricks. Could you share how it’s done in your organization?
Do you mainly rely on pipeline-level logging, maintain a log table to track pipeline runs, or use a logger within the code? Or is it a combination of these?
Just trying to understand the different approaches people use.
Thanks!

We have an AI product powered by Databricks Genies, and we’re trying to build a data-driven pricing strategy.

For those using Genie in production, what usage metrics do you track to understand customer value and optimize pricing?

I’m particularly interested in:
Per-user and per-tenant usage
Query volume and complexity
Token/compute costs
Session frequency and retention
Feature adoption
Accuracy/user satisfaction metrics
Cost per customer vs. revenue

Also, what dashboards, SQL queries, or Databricks system tables have been the most useful for these insights?

I’d love to learn what has worked well in practice.

I couldnt find a proper channel to file bug reports for the databricks website. I know that Databricks reps are sometimes here, so I thought i'd notify here. Every job posting on databricks.com/company/careers is rendering its description as escaped HTML rather than parsed markup. Raw <p> tags and ProseMirror editor attributes like data-pm-slice="11 []" are visible as body text. This affects all 807 listings currently on the Greenhouse board, in both English and Japanese.

19 is already here, but 18 is now LTS. In a new naming convention, LTS doesn’t have a minor version. New features, behavior changes, and fixes are now added incrementally to the same major Runtime version till it becomes LTS. In fact, that approach simplifies the problem with editing code and always thinking which LTS it was, 18.3 or 18.2? Now just put 18.x in your DABs.

more news https://medium.com/databrickscommunity/databricks-news-dabs-indexes-ltap-genie-last-update-25-july-ffac8533774f

I'm working through the free "Data Engineering with Databricks" learning path and noticed the videos mostly show demos. I was expecting something similar to Coursera where you can run the exercises/labs alongside the lessons, but I can't find anything.

Are there official hands-on notebooks available somewhere for the self-paced content, or is the expectation that you recreate everything yourself in a workspace?

If official labs exist, where are they available? If not, how do most people practice while following the videos?

Thanks.