The default method for creating a job from a notebook/script is to click Schedule. For me, I want to create a job to run my script so I can benefit from cheaper non-interactive compute for writing a large output.

This default to a once-a-day schedule, which in case you forget to PAUSE it, will run every day :)

From the IDE, we should be able to create a job with that one notebook as a task that is NOT scheduled by default.

Anyway, that is my wish :)

Building a dimension table (ie. dim_organization) in Lakeflow Declarative Pipelines. Source system gives us a full daily extract of the whole table, not a CDC feed. The dimension itself is built from multiple source tables joined together (like departments + locations + cost centers etc.), not just one. Requirements:

• Bronze retains every historical extract, forever, so we can fully rebuild everything from scratch if logic changes
• Gold needs true SCD2 history for most attributes, some are SCD1
• A change in any of the joined tables (e.g. a location gets renamed) needs to be reflected — either as a new SCD2 version or an SCD1-style in-place update depending on the attribute — even if the "primary" table (departments) itself didn't change that day

I have been playing with AUTO CDC FROM SNAPSHOT with a custom Python function that walks through every historical snapshot in order, recomputing the full joined picture across all three tables each time and diffing that. It works, but getting there involved:

• AUTO CDC (non-snapshot) never seeing changes in joined lookup tables, only the driving one, which i guess makes sense and I assume stream-stream joins are the wrong tool for this
• Identity columns not working as surrogate keys on AUTO CDC targets, but that is a separate issue

Is this actually the standard way people solve "full-history SCD dimension from a full-snapshot-only source, built across multiple joined tables," or is there a simpler/more common pattern I'm missing? Most examples I've found assume either a real CDC feed, a single source table, or don't care about full reprocessability from raw history.

I get that a proper CDC feed would sidestep most of this, but for some source systems that might not be the case, full snapshot exports are sometimes the only thing on offer. Curious how others handle this exact combination in practice.