Overview

Looking for the latest Prefect 2 release? Prefect 2 and Prefect Cloud 2 have been released for General Availability. See https://docs.prefect.io/ for details.

The Prefect task library is a constantly growing list of pre-defined tasks that provide off-the-shelf functionality for working with a wide range of tools anywhere from shell script execution to kubernetes job management to sending tweets. A majority of the task library is community supported and thus opens the door for users who want to contribute tasks for working with any number of tools. Tasks in the task library are created with a specific goal in mind such as creating a Kubernetes job with CreateNamespacedJob or invoking an AWS lambda function with LambdaInvoke.

Below is a table showcasing some of the tasks that have been contributed to the task library for interfacing with varying tools and services that users have deemed useful. For a full list of tasks in the library and more information on how to use them visit the API reference documentation for the prefect.tasks module.

Tasks marked with the Verified badge have been reviewed and approved by Prefect.

Task library in action

Just like any other Prefect task, tasks in the task library can be used by importing and adding them to your flow.

from prefect import task, Flow
from prefect.tasks.shell import ShellTask

ls_task = ShellTask(command="ls", return_all=True)

@task
def show_output(std_out):
    print(std_out)

with Flow("list_files") as flow:
    ls = ls_task()
    show_output(ls)

from prefect import Task, Flow
from prefect.tasks.shell import ShellTask

class ShowOutput(Task):
    def run(self, std_out):
        print(std_out)

ls_task = ShellTask(command="ls", return_all=True)
show_output = ShowOutput()

flow = Flow("list_files")
show_output.set_upstream(ls_task, key="std_out", flow=flow)

Keyword arguments for tasks imported from the task library can either be set at initialization for reuse purposed or optionally set and overwritten when defining the flow.

from prefect import task, Flow
from prefect.tasks.shell import ShellTask

# Will only return the listed files
ls_task = ShellTask(command="ls", return_all=True)

@task
def show_output(std_out):
    print(std_out)

with Flow("count_files") as flow:
    ls = ls_task()
    show_output(ls)

    # Override command to count listed files
    ls_count = ls_task(command="ls | wc -l")
    show_output(ls_count)

from prefect import Task, Flow
from prefect.tasks.shell import ShellTask

class ShowOutput(Task):
    def run(self, std_out):
        print(std_out)

ls_task = ShellTask(command="ls", return_all=True)
show_output = ShowOutput()

ls_count = ShellTask(command="ls | wc -l", return_all=True)
show_output2 = ShowOutput()

flow = Flow("count_files")
show_output.set_upstream(ls_task, key="std_out", flow=flow)
show_output2.set_upstream(ls_count, key="std_out", flow=flow)

For more information on the tasks available for use in the task library visit the API reference documentation for the prefect.tasks and if you are interested in contributing to the task library visit the contributing page!