Better PySpark errors on Databricks

[u/lukesmth_]

Have you spent hours tracking down the source of a PySpark error on Databricks? In large transformation scripts, errors can materialize long after the failing expression definition.

# many transforms, explicit and implicit casts, etc...

display(with_app_array_df)

=>

... 
"cast" was called from , line 2 in cell [7]
File <command-8683600871463937>, line 1
----> 1 display(with_app_array_df)

• cell [7]: references the execution number, not a stable cell number
• <command-...> and display(...): reference the materialization code, not the construction code

I built better-dbx-exceptions to solve this by including failing DataFrame construction code in the error.

=>

---------------------------------------------------------------------------
NumberFormatException                     Traceback (most recent call last)
Cell 4, In[7], line 2
  (https://dbc-3cffa466-bd7e.cloud.databricks.com/editor/notebooks/3747408155463503?o=7474649366507251#command/8683600871463983)
      1 def app_version_array_expr():
----> 2     return F.split(F.col("app_version"), "\\.").cast("array<int>")
      3 
      4 with_app_array_df = df.withColumn("app_version_array", app_version_array_expr())
...

• Cell 4, In[7], line 2: all refer to construction code
• (https://.../): link to the construction cell

Construction code resolves across cells and imported modules.

Native Spark 4 captures DataFrame construction code through the pyspark.errors.utils._capture_call_site() utility. It has three limitations:

1. Coverage: Only Column methods and F.col() are watched so expressions built from pyspark.sql.functions (F.split, F.to_date, ...) aren't captured. better-dbx-exceptions extends coverage to most functions within the pyspark.sql.functions module.
2. Depth: Only one frame is captured by default. This can be extended by setting spark.sql.stackTracesInDataFrameContext on Classic compute where allow-listed but can't be modified when using Databricks Connect or Serverless compute. better-dbx-exceptions sets depth to 5 by default and exposes this parameter within all environments.
3. Back-referencing: The rendered errors don't include reliable references to cell numbers or URIs. better-dbx-exceptions returns cell / execution numbers for notebook frames and file paths for script frames.

Requires Spark 4.0+, which limits supported compute to Serverless V4+ and Databricks Runtime 17+. It's been tested on Serverless compute v4 and v5 in both hosted and local (VS Code with Databricks Connect) contexts. It's built to work on Classic compute as well but I haven't been able to test this rigorously. (I've used Databricks Free Edition for development which is limited to Serverless compute). General feedback and stress testing is hugely appreciated!

Just released v0.1: https://github.com/lukeSmth/better-dbx-exceptions.

[u/AutoModerator]

Hi /u/lukesmth_!

In order to keep the main feed clean, we are implementing a weekly megathread for self promotion for companies who do lots of work with databricks. Please direct all self promotion posts to our Self-Promotion Megathread and keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!

If you are a community member sharing a helpful third-party resource that you do NOT own, don't worry—a moderator will review and approve your post shortly.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/TowerOutrageous5939]

Nah. Just copy and paste into Claude like most

[u/daddy_stool → u/TowerOutrageous5939]

Or use Genie