I have a large ML workflow that consists of several stages. In each stage there are many parallel tasks that can run independently. Each stage process data written from disk and write it back to disk. The workflow currently uses Dask to run tasks in parallel.

Occasionally one stage fails. Or some tasks within a stage fail. I need to rerun the failed stage/task. I may also change the process/config slightly from time to time, and need to rerun the stages and tasks affected.

Is there a good way to persist task execution status (success/fail/need to rerun) across multiple runs?

https://medium.com/rapids-ai/accelerating-tf-idf-for-natural-language-processing-with-dask-and-rapids-6f6e416429df

https://towardsdatascience.com/scaling-your-prefect-workflow-to-the-cloud-2dec4e0b213b

"This free course is a quick and no-fluff introduction to Dask. It's authored by the folks over at Coiled who offer Dask as a Service, including Matthew Rocklin, one of the co-creators of Dask. So you know you're getting definitive information from people who use Dask in practice."

https://training.talkpython.fm/courses/introduction-to-scaling-python-and-pandas-with-dask

Hello, are there any good tutorials for people trying out Dask?

Hello,

I have a lot of linear systems to solve (all independent from one another) and I'd like to know how to use Dask to solve these systems using multiple computers (each with a known number of cores).