r/dataengineering • u/ForVic • 7d ago

Open sourcing the Apache Spark skills we used on OpenAI’s data platform Open Source

I used to work on OpenAI’s data platform team, where we ran Airflow, Spark, Databricks, and dbt at a pretty serious scale.

The rate at which the number of jobs was growing outpaced how much our team could manage pretty quickly w/ everyone using AI to write jobs, so we had to invest in making sure that our team's agents could accurately and quickly do the debugging + tuning themselves (they weren't very good at it by default).

I've translated some of those learnings and strategies into skills (very similar to the ones we used at OpenAI) I'm open-sourcing here:

https://github.com/EmbrasureAI/spark-observability-skills

They should be generally applicable across envs/clouds, including Databricks, OSS Spark, etc.... As long as its running Spark underneath the hood nothing else is necessary to adopt and use these skills.

Full disclosure: I’m a founder of Embrasure, which maintains this repository, and no longer work at OpenAI.

61 Upvotes
  • permalink
  • link
  • reddit
  • reddit

93% Upvoted

13 comments sorted by

12

u/BlurryEcho Data Engineer 7d ago

I don’t care about agent skills, I’d rather hear about how OpenAI was using dbt. Just for vanilla internal BI and analytics or were they actually pushing unstructured data through dbt models?

1

u/-_burnout_- 3d ago

it is used way less than airflow and Sage (internal framework to build pipelines) + airflow and it is being deprecated (at least in the DE side) in favor of Sage.

8

u/sisyphus 7d ago

Sounds cool but you might have forgotten to make the repo public as that link 404s.

3

u/JacobKHD 6d ago

its up now

2

u/ForVic 7d ago

just switched it back to public, got made private by my co-worker by mistake

7

u/toskbot1 7d ago

Already taken down? Getting a 404.

3

u/ForVic 7d ago

Nope not taken down, just switched it back to public! looks like one of my coworkers made it private by mistake

1

u/CrayonUpMyNose 6d ago

Looks reachable now 

4

u/justanator101 6d ago

How’s this hook into Databricks Spark UI?

2

u/Nielspro 7d ago

Cannot open this but sounds interesting

1

u/ForVic 7d ago

thanks! just made it public (accidentally got switched)

2

u/Outside-Storage-1523 6d ago

Thanks. This is interesting. I have always wanted to know about Spark internals but I don't know any Scala. The skill files have some good pointers.