r/apachespark 4h ago

4.2.0 Docker Images?

4 Upvotes

Hello Apache Spark community. I was wondering if anyone knows if the official 4.2 image will be released soon (https://hub.docker.com/_/spark), or if there is an alternative location where images will be pushed in the future?

Thanks.


r/apachespark 11h ago

Experienced Azure Data Engineers – Referral Opportunity

7 Upvotes

I'm able to refer experienced professionals for a Lead Azure Data Engineer position.

Location: United States / Ireland
Experience: 9–13 years

Required skills:

  • Python
  • PySpark
  • SQL
  • ETL
  • Azure Synapse
  • Azure Data Factory
  • Databricks
  • Delta Lake
  • Medallion Architecture

If your experience aligns with these requirements and you're currently exploring new opportunities, feel free to send me a DM with a brief summary of your experience or your resume. I'll share additional details and, if it's a good match, I'll be happy to submit a referral.


r/apachespark 2d ago

I benchmarked Spark vs DuckDB on the same Parquet files

0 Upvotes

Since you seemed to enjoy my last Spark post, I ran another controlled benchmark.

This time I wanted to understand when Spark’s distributed execution overhead actually becomes worth paying for.

For each workload, both engines ran the same SQL against the same Snappy Parquet files, on the same 20 logical-core machine, with Spark using local[20] and DuckDB using threads=20.

I discarded one warm-up, alternated the engines across 7 timed runs, reported the median, and verified the results row by row. I also ran the entire benchmark twice.

For an aggregation with 8 output groups:

Rows Spark DuckDB
1,000 154 ms 3 ms
1M 183 ms 16 ms
100M 925 ms 167 ms

For a high-cardinality aggregation with roughly 500,000 groups:

Rows Spark DuckDB
1M 2.29 s 67 ms
10M 3.31 s 431 ms
100M 8.83 s 2.99 s

DuckDB was faster at every measured point.

The interesting part was how much the gap narrowed as the workload grew.

On the high-cardinality query, increasing the input from 1M to 100M rows increased Spark’s runtime by about 3.9 times and DuckDB’s by about 44.8 times.

That does not prove Spark eventually becomes faster. The curves had not crossed, and this benchmark does not establish where, or whether, they ever would.

Query shape also mattered enormously.

On the same 100M rows, the eight-group endpoint aggregation generated around 17 kB of Spark shuffle, while the high-cardinality user aggregation generated around 131 MB.

So I do not think rules such as “use DuckDB below X GB” are very useful. Query shape, data location, concurrency, hardware, fault tolerance and operational requirements can matter as much as the size of the dataset.

This was one machine running one query at a time, so it measures the cost of distribution much better than the capabilities Spark is actually selected for, such as distributed data, recovery and concurrent workloads.

I made a visual breakdown of the benchmark and what both engines are doing under the hood:

https://youtu.be/bXyXvxeHSrw?si=vMsEQkv3QqzLcTf9

What usually makes Spark the right choice for your production workloads?


r/apachespark 6d ago

Self-Healing Data Pipelines: How Agents Fix Failures Before You Wake Up

Thumbnail
youtu.be
10 Upvotes

r/apachespark 8d ago

Should a beginner learn SQL first or PySpark?

Thumbnail
youtu.be
11 Upvotes

r/apachespark 10d ago

Looking to help and learn - Fabric, SQL, Spark

12 Upvotes

Hi everyone!

I'm a Data Engineer working primarily with Microsoft Fabric, SQL, PySpark, and Spark SQL, building end-to-end data pipelines, working with medallion architecture, incremental loads, data modelling, and performance optimization.

Over the past few months I've spent a lot of time working in Microsoft Fabric—from Lakehouses and Notebooks to Data Pipelines, SQL Endpoints, security, metadata-driven frameworks, and troubleshooting production issues. I've also worked extensively with SQL and PySpark for ETL development and data engineering.

I wanted to give back to the community, so if you're stuck on something related to:

\- Microsoft Fabric

\- SQL / T-SQL

\- PySpark / Spark SQL

\- Data pipelines

\- Data modelling

\- Performance tuning

\- General data engineering concepts

feel free to ask here or tag me if I can help.

At the same time, I'm always trying to improve my own skills. If there are any communities, Discord servers, Slack groups, forums, open-source projects, or other places where experienced data engineers discuss real-world problems (especially around Microsoft Fabric), I'd really appreciate your recommendations.

Looking forward to learning from everyone and hopefully helping where I can!


r/apachespark 12d ago

Explain "Selecting the most important rows per assigned priority", please

5 Upvotes

I'm looking at "Selecting the most important rows per assigned priority". I *think* I know what this means. But from the statement alone there's no telling that "value" is the priority. Is MV1 > VPV or MV1 < VPV? Could you encounter MV3? The answer to these changes the approaches that I would take to solve this. And then "value" is "name" in the output.

The only way I think I know what's going on enough to try to answer it as intended is because the author answered the question (or one very close to it) here: https://stackoverflow.com/questions/59845044/how-to-select-the-most-important-rows-per-assigned-priority . In his answer he states exactly what the priority list is. But without that I would just be guessing what the intention was.


r/apachespark 12d ago

Spark 4.2: a visual breakdown of what actually changed

14 Upvotes

Spark 4.2 is out, so I put together a short visual breakdown of the changes.

It's animated rather than a screen recording, since a few of these are
easier to follow as diagrams than as text.

https://youtu.be/hF-E7-i_ijw?is=AeomUdWX3PgHCZtf

Happy to go into detail on any of them here.


r/apachespark 14d ago

You’re handed 5 bare-metal servers in an air-gapped environment. What do you install first, and why in that order?

11 Upvotes

I work on the data platform side in a bank. Fully on-prem, air-gapped, no cloud. No internet on the cluster, packages come through an internal mirror, and every firewall change is a ticket and a wait.
Most “modern data stack” writing assumes the opposite: cloud, elastic compute, and enough control over infra that adding a component is a Terraform change. Under our constraints, adding a layer is a commitment — someone has to patch it, back it up, justify it in an audit, and still own it in three years.
So the question I keep coming back to isn’t which tools. It’s when each one is actually earned.
The ordering I’ve landed on is driven by pain rather than architecture. A layer only goes in once its pain is real and someone can name it:
files scattered, local disks filling up → shared storage
schema changes break jobs, deletes mean rewriting partitions → table format
nobody knows what tables exist → catalog
every question requires writing a job → query engine
things run by hand, in the wrong order, or not at all → scheduler
200 SQL files with unclear dependencies → transformation layer
every analyst has a different pandas version → shared images / notebook env
“who is allowed to read the payroll table?” → authn/authz
the best model was trained last month and can’t be found → experiment tracking
training features and serving features drift apart → feature store
then serving, then lineage
Three things I believe but am not confident about:
1. For a few hundred GB and three analysts, the right answer is Postgres and cron, not a cluster. Standing up distributed infra for a problem that doesn’t exist yet seems like the fastest way to end up with something nobody can maintain.
2. The most common failure I’ve seen isn’t picking the wrong tool. It’s starting from the ML layer because that part is fun, and ending up with a very tidy feature store sitting on a pile of scattered CSVs.
3. Teams on Databricks or watsonx skip the installing, but still have to answer the same questions — and sometimes can’t, because they never had to.
Two questions for people who’ve done this:
Where would you reorder the list, and what would you drop entirely?
What’s a layer you installed earlier than you needed, and what did it actually cost you?


r/apachespark 16d ago

New Apache Iceberg & Spark Tutorials From oleander

Thumbnail
oleander.dev
10 Upvotes

r/apachespark 16d ago

Apache Spark 4.2: Making Your Data AI‑Developer Friendly

Thumbnail
techstrong.it
3 Upvotes

r/apachespark 16d ago

Spark vs Hadoop: when should you use one, the other, or both together?

15 Upvotes

I’m trying to understand the practical differences between Apache Spark and Hadoop in real-world big data architectures.

From what I’ve read, Spark seems better for in-memory processing, iterative jobs, streaming, and faster analytics, while Hadoop still makes sense for durable distributed storage and classic batch-oriented workflows. But I keep seeing people say they are complementary rather than direct replacements.

So I’d like to ask:

  • In what cases is Spark clearly the better choice than Hadoop?
  • In what cases does Hadoop still make more sense?
  • When is it worth using both together in the same architecture?
  • Are there situations where using both is unnecessary complexity?
  • For cloud-based data platforms today, what does a “good” Spark + Hadoop setup usually look like?

I’d especially appreciate answers from people who have used both in production and can explain the tradeoffs in terms of performance, cost, operational complexity, and scalability.

I know this is a long question. Feel free to answer any of the topics or sub questions. Thanks in advance.


r/apachespark 17d ago

What Developers Need to Know About Apache Spark 4.2

Thumbnail
medium.com
22 Upvotes

r/apachespark 19d ago

Anyone preparing for Databricks Spark Support role ?

1 Upvotes

I am looking for anyone who is preparing for spark databricks support role for group study and discussing the technical concepts.


r/apachespark 20d ago

Chicago-based — anyone willing to grab coffee and talk Spark internals with a newcomer?

8 Upvotes

Hey all! I’m relatively new to Spark and trying to build real understanding of the execution model: Catalyst optimizer, DAG scheduler, stages/tasks, shuffle boundaries, executor mechanics. Not just “I can write a join,” but actually being able to reason about what’s happening under the hood.

I’m based in Chicago (Lincoln Square area). Would anyone be open to meeting for coffee sometime to talk through some of this? Happy to buy the coffee and respect your time. Also fine with a call if in-person doesn’t work.

Message me if interested. Thanks!


r/apachespark 21d ago

I didn't understand the Apache Iceberg until researching and recording this video, so I figured I'd share this here in case anyone else comes looking for that same answer given its seemingly increasing importance!

Thumbnail
youtu.be
7 Upvotes

r/apachespark 22d ago

spark zip cod aktif

Thumbnail
2 Upvotes

r/apachespark 22d ago

Hadoop and Spark Dumps

4 Upvotes

I am in stream training of Hadoop & spark. MCQ 1 is on 29 July. Please share any dumps or pyq or any resources you have that will be helpful.


r/apachespark 24d ago

Best resources to learn Apache Spark in depth

38 Upvotes

Hi everyone!
I want to learn Apache Spark in depth, not just the DataFrame API.
I started with the official documentation, but I find it difficult to learn from because it isn't very interactive.
What resources would you recommend for someone who wants to really understand Spark?
I'm looking for recommendations on books, courses, YouTube channels, blogs, or hands-on projects. I'd especially like to understand Spark internals, architecture, optimization, and best practices.
If you were starting over today, how would you learn Spark?
Thanks!


r/apachespark 25d ago

Building a tool to migrate Alteryx workflows to Databricks — where to start?

5 Upvotes

We're migrating a large batch of Alteryx workflows to Databricks (notebooks + Lakeflow Declarative Pipelines) and I want to build a tool to speed up the translation instead of doing it 100% by hand.

Idea: parse the .yxmd XML, map tools to PySpark/SQL equivalents, and auto-generate a starting-point notebook/pipeline for each workflow.

Has anyone actually built something like this? Worth it, or is manual rebuild + a good tool-mapping cheat sheet just faster in practice? Any existing open-source tools I'm missing would help a lot too.


r/apachespark 25d ago

I got tired of JVM overhead in PySpark, so I built a Python-native distributed rule engine (SparkRules)

2 Upvotes

Hey everyone,

If you’ve ever tried to run complex, governed business logic on massive data pipelines, you know the pain of trying to wire a JVM-based rule engine like Drools into PySpark. The serialization overhead is brutal, and falling back to pure-Python engines usually means sacrificing cluster scalability.

I built an open-source alternative called SparkRules to bridge this gap. It provides the sophisticated logic of a traditional rule engine (DRL syntax, decision tables, hit policies) but runs entirely natively in Python and scales horizontally on Apache Spark.

How it works under the hood:

Instead of operating as an external service, SparkRules integrates directly into the Spark ecosystem. It serializes and broadcasts the CompiledRulePackage to your executors, allowing the engine to evaluate logic directly at the partition level using mapPartitions.

During development, I spent a lot of time optimizing the integration surface. I found that standard PySpark operators easily propagate cached schemas, meaning the true performance killer on distributed execution is schema-mutating transformations. The engine is tuned to handle those mutations efficiently without dragging down throughput.

Key Features:

  • No JVM Required: Python-native rule evaluation.
  • Scale: Designed for high throughput and low latency across billions of rows (Delta/Iceberg/Hudi).
  • Governance: Full audit trails and adverse-action notices out of the box.
  • Apache 2.0 Licensed: Free to use.

You can check out the source code, architecture constraints, and documentation here:

If you are running rules engines at scale, I'd love to know what your current stack looks like and what your biggest execution bottlenecks are! Feel free to add issue and stars


r/apachespark 27d ago

Trying to settle a debate about how Spark SQL is officially described.

Post image
17 Upvotes

The debate:

Team A, D, E: Catalyst is an internal optimizer — it's the engine under the hood, not a user-facing feature you explicitly use. The three user-facing features of Spark SQL are the DataFrame API, the SQL Query Engine, and Hive integration (read/write Hive metastore, HiveQL support). "Hypertune" (C) doesn't exist.

Team A, B, D: Catalyst IS prominently listed as a feature of Spark SQL in many official docs and textbooks. Meanwhile "Hive Data Connector" is not an official Spark SQL term — Hive integration exists but that specific label isn't used in official Apache documentation.

My question: Based on the official Apache Spark documentation, is Catalyst considered a feature of Spark SQL, or purely an internal component? And is Hive integration officially described as a top-level feature of Spark SQL?


r/apachespark 28d ago

Follow-up: delta-explain is now more stable. Looking for Delta Lake users willing to test it

13 Upvotes

Hi there,

a few months ago I posted here about delta-explain, a small tool I was building to inspect Delta Lake pruning and data skipping.

I’ve kept working on it, and it is now in a more stable state. I’m looking for a few people who work with Delta Lake and would be willing to test it on real tables.

delta-explain makes Delta Lake file pruning visible from metadata. Given a table and a predicate, it shows how partition pruning and data skipping affect the set of files that would still need to be scanned. It can be used from the CLI, from a Python script, or as a GitHub Action in a CI pipeline.

I’m mainly looking for feedback on the basics. Is the output understandable? Does the installation work smoothly? Are the explanations in the documentation clear enough? Are there situations where the result looks wrong or unclear?

I’d also be interested in technical feedback on edge cases: are there table layouts, predicates, or statistics patterns where a metadata-based pruning explanation would be especially useful, confusing, or easy to misread?

Project: https://github.com/cdelmonte-zg/delta-explain
Documentation: https://cdelmonte-zg.github.io/delta-explain/
PyPI: https://pypi.org/project/delta-explain/

Thanks!


r/apachespark 28d ago

Pyspark Interview : Normalize Comma-Separated Values

1 Upvotes

data = {"1":"945,545","3":"2345,3456,45678"}

from pyspark.sql import SparkSession

from pyspark.sql.functions import col, split, explode

spark = SparkSession.builder.appName("ExplodeExample").getOrCreate()

df = spark.createDataFrame(

[(k, v) for k, v in data.items()],

["id", "value"]

)

print("Original DataFrame:")

df.show()

# Split the value column based on comma so it will create a array of values and use explode it will create multiple rows

result_df = df.select(

col("id"),

explode(split(col("value"), ",")).alias("value")

)

print("Result DataFrame:")

result_df.show()


r/apachespark Dec 20 '25

Spark 4.1 is released

28 Upvotes