Scaling Out

Looking for the latest Prefect 2 release? Prefect 2 and Prefect Cloud 2 have been released for General Availability. See https://docs.prefect.io/ for details.

Follow along in the Terminal

cd examples/tutorial
python 06_parallel_execution.py

Let's adjust our Flow to distribute its Tasks onto a Dask Cluster, parallelizing its execution. This may sound involved, but will actually be our simplest adjustment yet:

from prefect.executors import DaskExecutor

# ...task definitions...

# ...flow definition...
if __name__=="__main__":
    flow.run(executor=DaskExecutor())

This will spin up a Local Dask Cluster on your system to parallelize the tasks. If you already have a Dask Cluster deployed elsewhere, you can leverage that cluster by specifying the address in the DaskExecutor constructor:

flow.run(
    executor=DaskExecutor(
        address='some-ip:port/to-your-dask-scheduler'
    )
)

Furthermore, you can implement your own Executor for use with any Prefect Flow, as long as the object provided satisfies the Executor interface (i.e. appropriate submit, map, and wait functions, similar to Python's concurrent.futures.Executor interface). In this way, the sky is the limit!

Up Next!

What else can Prefect do?...