# Docker Tasks


Collection of tasks for orchestrating Docker images and containers.

Note: If running these tasks from inside of a docker container itself there are some extra requirements needed for that to work. The container needs to be able to talk to a Docker server. There are a few ways to accomplish this:

  1. Use a base image that has Docker installed and running (e.g. https://hub.docker.com/_/docker)
  2. Installing the Docker CLI in the base image
  3. Talking to an outside (but accessible) Docker API and providing it to the tasks' docker_server_url parameter

It may also help to run your container (which will run the Prefect Docker tasks) with extra privileges. (e.g. --privileged=true) and then installing Docker in the container.

# BuildImage

class

prefect.tasks.docker.images.BuildImage

(path=None, tag=None, nocache=False, rm=True, forcerm=False, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, stream_logs=False, **kwargs)[source]

Task for building a Docker image. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

  • path (str, optional): The path to the directory containing the Dockerfile
  • tag (str, optional): The tag to give the final image
  • nocache (bool, optional): Don't use cache when set to True
  • rm (bool, optional): Remove intermediate containers; defaults to True
  • forcerm (bool, optional): Always remove intermediate containers, even after unsuccessful builds; defaults to False
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method build). See https://docker-py.readthedocs.io/en/stable/api.html for more details
  • stream_logs (bool, optional): Adds debug logs with build image status info from Docker
  • **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.docker.images.BuildImage.run

(path=None, tag=None, nocache=False, rm=True, forcerm=False, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, stream_logs=False)[source]

Task run method.

Args:

  • path (str, optional): The path to the directory containing the Dockerfile
  • tag (str, optional): The tag to give the final image
  • nocache (bool, optional): Don't use cache when set to True
  • rm (bool, optional): Remove intermediate containers; defaults to True
  • forcerm (bool, optional): Always remove intermediate containers, even after unsuccessful builds; defaults to False
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method build). See https://docker-py.readthedocs.io/en/stable/api.html for more details
  • stream_logs (bool, optional): Adds debug logs with build image status info from Docker
Returns:
  • List[dict]: a cleaned dictionary of the output of client.build
Raises:
  • ValueError: if either path is None



# ListImages

class

prefect.tasks.docker.images.ListImages

(repository_name=None, all_layers=False, filters=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, **kwargs)[source]

Task for listing Docker images. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

  • repository_name (str, optional): Only show images belonging to this repository; if not provided then it will list all images from the local Docker server
  • all_layers (bool, optional): Show intermediate image layers
  • filters (dict, optional): Filter the results. See https://docker-py.readthedocs.io/en/stable/images.html for more details
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method images). See https://docker-py.readthedocs.io/en/stable/api.html for more details
  • **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.docker.images.ListImages.run

(repository_name=None, all_layers=False, filters=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None)[source]

Task run method.

Args:

  • repository_name (str, optional): Only show images belonging to this repository; if not provided then it will list all images from the local Docker server
  • all_layers (bool, optional): Show intermediate image layers
  • filters (dict, optional): Filter the results. See https://docker-py.readthedocs.io/en/stable/images.html for more details
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method images). See https://docker-py.readthedocs.io/en/stable/api.html for more details
Returns:
  • list: A list of dictionaries containing information about the images found



# PullImage

class

prefect.tasks.docker.images.PullImage

(repository=None, tag=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, stream_logs=False, **kwargs)[source]

Task for pulling a Docker image. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

  • repository (str, optional): The repository to pull the image from
  • tag (str, optional): The tag of the image to pull; if not specified then the latest tag will be pulled
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method pull). See https://docker-py.readthedocs.io/en/stable/api.html for more details
  • stream_logs (bool, optional): Adds debug logs with pull status info from Docker
  • **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.docker.images.PullImage.run

(repository=None, tag=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, stream_logs=False)[source]

Task run method.

Args:

  • repository (str, optional): The repository to pull the image from
  • tag (str, optional): The tag of the image to pull; if not specified then the latest tag will be pulled
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method pull). See https://docker-py.readthedocs.io/en/stable/api.html for more details
  • stream_logs (bool, optional): Adds debug logs with pull status info from Docker
Returns:
  • str: The output from Docker for pulling the image
Raises:
  • ValueError: if repository is None
  • HTTPError: if image doesn't exist or provided repository/tag is not correct



# PushImage

class

prefect.tasks.docker.images.PushImage

(repository=None, tag=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, **kwargs)[source]

Task for pushing a Docker image. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

  • repository (str, optional): The repository to push the image to
  • tag (str, optional): The tag for the image to push; if not specified then the latest tag will be pushed
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method push). See https://docker-py.readthedocs.io/en/stable/api.html for more details
  • **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.docker.images.PushImage.run

(repository=None, tag=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None)[source]

Task run method.

Args:

  • repository (str, optional): The repository to push the image to
  • tag (str, optional): The tag for the image to push; if not specified then the latest tag will be pushed
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method push). See https://docker-py.readthedocs.io/en/stable/api.html for more details
Returns:
  • str: The output from Docker for pushing the image
Raises:
  • ValueError: if repository is None



# RemoveImage

class

prefect.tasks.docker.images.RemoveImage

(image=None, force=False, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, **kwargs)[source]

Task for removing a Docker image. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

  • image (str, optional): The image to remove
  • force (bool, optional): Force removal of the image
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method remove_image). See https://docker-py.readthedocs.io/en/stable/api.html for more details
  • **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.docker.images.RemoveImage.run

(image=None, force=False, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None)[source]

Task run method.

Args:

  • image (str, optional): The image to remove
  • force (bool, optional): Force removal of the image
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method remove_image). See https://docker-py.readthedocs.io/en/stable/api.html for more details
Raises:
  • ValueError: if image is None



# TagImage

class

prefect.tasks.docker.images.TagImage

(image=None, repository=None, tag=None, force=False, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, **kwargs)[source]

Task for tagging a Docker image. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

  • image (str, optional): The image to tag
  • repository (str, optional): The repository to set for the tag
  • tag (str, optional): The tag name for the image
  • force (bool, optional): Force tagging of the image
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method tag). See https://docker-py.readthedocs.io/en/stable/api.html for more details
  • **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.docker.images.TagImage.run

(image=None, repository=None, tag=None, force=False, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None)[source]

Task run method.

Args:

  • image (str, optional): The image to tag
  • repository (str, optional): The repository to set for the tag
  • tag (str, optional): The tag name for the image
  • force (bool, optional): Force tagging of the image
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method tag). See https://docker-py.readthedocs.io/en/stable/api.html for more details
Returns:
  • bool: Whether or not the tagging was successful
Raises:
  • ValueError: if either image or repository are None



# CreateContainer

class

prefect.tasks.docker.containers.CreateContainer

(image_name=None, container_name=None, command=None, detach=False, entrypoint=None, environment=None, volumes=None, docker_server_url="unix:///var/run/docker.sock", host_config=None, extra_docker_kwargs=None, **kwargs)[source]

Task for creating a Docker container and optionally running a command. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

  • image_name (str, optional): Name of the image to run
  • container_name (str, optional): A name for the container
  • command (Union[list, str], optional): A single command or a list of commands to run
  • detach (bool, optional): Run container in the background
  • entrypoint (Union[str, list], optional): The entrypoint for the container
  • environment (Union[dict, list], optional): Environment variables to set inside the container, as a dictionary or a list of strings in the format ["SOMEVARIABLE=xxx"]
  • volumes (Union[str, list], optional): List of paths inside the container to use as volumes
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • host_config (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method create_host_config). See https://docker-py.readthedocs.io/en/stable/api.html for more details
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method create_container). See https://docker-py.readthedocs.io/en/stable/api.html for more details
  • **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.docker.containers.CreateContainer.run

(image_name=None, container_name=None, command=None, detach=False, entrypoint=None, environment=None, volumes=None, docker_server_url="unix:///var/run/docker.sock", host_config=None, extra_docker_kwargs=None)[source]

Task run method.

Args:

  • image_name (str, optional): Name of the image to run
  • container_name (str, optional): A name for the container
  • command (Union[list, str], optional): A single command or a list of commands to run
  • detach (bool, optional): Run container in the background
  • entrypoint (Union[str, list]): The entrypoint for the container
  • environment (Union[dict, list]): Environment variables to set inside the container, as a dictionary or a list of strings in the format ["SOMEVARIABLE=xxx"]
  • volumes (Union[str, list], optional): List of paths inside the container to use as volumes
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • host_config (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method create_host_config). See https://docker-py.readthedocs.io/en/stable/api.html for more details
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Dockercall (cf. method create_container). See https://docker-py.readthedocs.io/en/stable/api.html for more details
Returns:
  • str: A string representing the container id
Raises:
  • ValueError: if image_name is None



# GetContainerLogs

class

prefect.tasks.docker.containers.GetContainerLogs

(container_id=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, **kwargs)[source]

Task for getting the logs of a Docker container. Note: This does not stream logs. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

  • container_id (str, optional): The id of a container to retrieve logs from
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method logs). See https://docker-py.readthedocs.io/en/stable/api.html for more details
  • **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.docker.containers.GetContainerLogs.run

(container_id=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None)[source]

Task run method.

Args:

  • container_id (str, optional): The id of a container to retrieve logs from
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method logs). See https://docker-py.readthedocs.io/en/stable/api.html for more details
Returns:
  • str: A string representation of the logs from the container
Raises:
  • ValueError: if container_id is None



# ListContainers

class

prefect.tasks.docker.containers.ListContainers

(all_containers=False, filters=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, **kwargs)[source]

Task for listing Docker containers. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

  • all_containers (bool, optional): Show all containers. Only running containers are shown by default
  • filters (dict, optional): Filter the results. See https://docker-py.readthedocs.io/en/stable/containers.html for more details
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method containers). See https://docker-py.readthedocs.io/en/stable/api.html for more details
  • **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.docker.containers.ListContainers.run

(all_containers=False, filters=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None)[source]

Task run method.

Args:

  • all_containers (bool, optional): Show all containers. Only running containers are shown by default
  • filters (dict, optional): Filter the results. See https://docker-py.readthedocs.io/en/stable/containers.html for more details
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method container). See https://docker-py.readthedocs.io/en/stable/api.html for more details
Returns:
  • list: A list of dicts, one per container



# StartContainer

class

prefect.tasks.docker.containers.StartContainer

(container_id=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, **kwargs)[source]

Task for starting a Docker container that runs the (optional) command it was created with. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

  • container_id (str, optional): The id of a container to start
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method start). See https://docker-py.readthedocs.io/en/stable/api.html for more details
  • **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.docker.containers.StartContainer.run

(container_id=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None)[source]

Task run method.

Args:

  • container_id (str, optional): The id of a container to start
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method start). See https://docker-py.readthedocs.io/en/stable/api.html for more details
Raises:
  • ValueError: if container_id is None



# StopContainer

class

prefect.tasks.docker.containers.StopContainer

(container_id=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, **kwargs)[source]

Task for stopping a Docker container. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

  • container_id (str, optional): The id of a container to stop
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method stop). See https://docker-py.readthedocs.io/en/stable/api.html for more details
  • **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.docker.containers.StopContainer.run

(container_id=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None)[source]

Task run method.

Args:

  • container_id (str, optional): The id of a container to stop
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method stop). See https://docker-py.readthedocs.io/en/stable/api.html for more details
Raises:
  • ValueError: if container_id is None



# RemoveContainer

class

prefect.tasks.docker.containers.RemoveContainer

(container_id=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, **kwargs)[source]

Task for removing a Docker container. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

  • container_id (str, optional): The id of a container to remove
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method remove_container). See https://docker-py.readthedocs.io/en/stable/api.html for more details
  • **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.docker.containers.RemoveContainer.run

(container_id=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None)[source]

Task run method.

Args:

  • container_id (str, optional): The id of a container to remove
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method remove_container). See https://docker-py.readthedocs.io/en/stable/api.html for more details
Raises:
  • ValueError: if container_id is None



# WaitOnContainer

class

prefect.tasks.docker.containers.WaitOnContainer

(container_id=None, docker_server_url="unix:///var/run/docker.sock", raise_on_exit_code=True, extra_docker_kwargs=None, **kwargs)[source]

Task for waiting on an already started Docker container. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

  • container_id (str, optional): The id of a container to start
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • raise_on_exit_code (bool, optional): whether to raise a FAIL signal for a nonzero exit code; defaults to True
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method wait). See https://docker-py.readthedocs.io/en/stable/api.html for more details
  • **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.docker.containers.WaitOnContainer.run

(container_id=None, docker_server_url="unix:///var/run/docker.sock", raise_on_exit_code=True, extra_docker_kwargs=None)[source]

Task run method.

Args:

  • container_id (str, optional): The id of a container to wait on
  • docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
  • raise_on_exit_code (bool, optional): whether to raise a FAIL signal for a nonzero exit code; defaults to True
  • extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method wait). See https://docker-py.readthedocs.io/en/stable/api.html for more details
Returns:
  • dict: a dictionary with StatusCode and Error keys
Raises:
  • ValueError: if container_id is None
  • FAIL: if raise_on_exit_code is True and the container exits with a nonzero exit code



This documentation was auto-generated from commit ffa9a6c
on February 1, 2023 at 18:44 UTC