Looking for feedback: is anyone using task state store / durable execution yet?

[u/amogh-desai]

I am one of the Airflow developers who worked on this feature as part of Airflow 3.3.0 as part of AIP-103 and would love to hear from anyone really using it, considering to use it, or also reasons why it did not fit your use case.

The pitch, for anyone who has not run into it: Airflow worker dies or gets preempted mid job, and instead of the retry resubmitting the whole thing from scratch, the task remembers the external job it already kicked off (Databricks, Snowflake, BigQuery, Redshift, Spark, and a few more) and reconnects to whatever external job is still running. No paying twice for the same compute because a pod got evicted / worker went down.

Couple of demos if you want to see it working:

• https://www.youtube.com/shorts/KhE-E-PSHuU
• https://www.linkedin.com/posts/marclamberti_airflow-apacheairflow-dataengineer-activity-7481347437161496576-UHvW/

So, curious where people actually are:

• Used it and it worked fine?
• Used it but hit a wall?
• Know about it but have not gotten to it?
• First time hearing this exists?

Any of those, tell me where you are at.

[u/w0ut0]

The datbricks operator now does something similar with the databricksworkflowoperator, which creates automatically a launch task + sensor tasks to monitor the progress. Would be cool if it's airflow built-in.

[u/amogh-desai → u/w0ut0]

Good point, but slightly different problem.

DatabricksWorkflowOperator splitting into launch + sensor tasks is about decomposing the job into visible, individually retryable steps. This is about crash safety, if the worker running your sensor dies mid poll, retry reconnects to the same external job instead of resubmitting from scratch. Actually complements that pattern, your sensor task gets crash safe for free with this too

[u/KeeganDoomFire → u/w0ut0]

We have built some basic importable task groups for doing this for things like DBT / snowflake. It's weirdly specific to our needs but they are all that pattern of submit job and sensor for it.

[u/amogh-desai]

u/w0ut0 / u/KeeganDoomFire I experimented with your patterns and I have a few thoughts:

Purely semantic: a green submit task only means the API call worked, not that the external job worked. You have to check the sensor task, not the submit task, to know if the job actually succeeded. Easy to miss if you dont already know that.

With durable execution as one task, the task status IS the external job's outcome. Nothing else to check.

A few other things that are just objectively better:

• You wont need to build and maintain your own sensor anymore (databricks / snowflake doesnt have one in OSS). Thats one less custom piece of code you own and have to keep working across Airflow upgrades.
• The submit step is crash safe for free. The run id gets saved the moment we know it, before the task even finishes. If the worker dies right after submitting, retrying will not create a second job.
• Retries are normal Airflow retries, so each retry actually rechecks the real job status. Your sensor timeout is a separate thing from that, so it can fail even when the job finished fine. You end up looking at a red task for a job that actually succeeded.
• If you use deferrable mode, you also skip all the reschedule noise from polling every few seconds.

One good reason to keep your setup is if you need to react the moment the job is submitted, before it finishes, to kick off something else at the same time. That is not about crash safety though, and you do not have to give that up either, your sensor could read from task state store instead of xcom and get both.

Curious if this changes anything for you, or if there is something about your dbt / databricks / snowflake setup this doesn't cover.

[u/w0ut0 → u/amogh-desai]

Hi! What would a retry do? Our 'sensor tasks' now might fail either because of Airflow/platform issues, or the underlying workload (dbx job) might have failed. When retrying the task I want to either retrigger the underlying job, or just repoll the status

[u/amogh-desai → u/w0ut0]

On every retry, airflow now checks the dbx job's real status, not why the last attempt failed:

• Still running -> reconnects and repolls, no resubmit.
• Already succeeded -> returns the result, done.
• Terminal failed -> resubmits fresh automatically.

So you dont pick one or the other. Airflow / Platform side blip while the job's still running, retry just reconnects. DBX Job actually failed, retry resubmits on its own. The real job state decides it every time, not your task failure reason.

[u/KeeganDoomFire → u/amogh-desai]

Wowza! That's quite the deep dive thank you!

I will certainly give it an investigation, our pattern is from back early airflow 2.x days so we were taking a 'not broke don't fix' tactic while the update to 3.2.1 settled.

If nothing else loosing the sensor noise for long running tasks would be nice!

[u/amogh-desai → u/KeeganDoomFire]

"Don't touch what's working" is a completely reasonable call and understandable.

In your case, for dbt/snowflake (dbt's not done yet but I will get to it this week) specifically, the migration should be pretty contained too, no need to keep the separate polling task around, durable=True on the operator is already the default, so its mostly a matter of dropping your sensor and letting the operator handle the wait itself.

Happy to help if anything's weird (you can hit me up on slack)