Docker Tasks

---

Collection of tasks for orchestrating Docker images and containers.

Note: If running these tasks from inside of a docker container itself there are some extra requirements needed for that to work. The container needs to be able to talk to a Docker server. There are a few ways to accomplish this:

1. Use a base image that has Docker installed and running (e.g. https://hub.docker.com/_/docker)
2. Installing the Docker CLI in the base image
3. Talking to an outside (but accessible) Docker API and providing it to the tasks' docker_server_url parameter

It may also help to run your container (which will run the Prefect Docker tasks) with extra privileges. (e.g. --privileged=true) and then installing Docker in the container.

BuildImage

class

prefect.tasks.docker.images.BuildImage

(path=None, tag=None, nocache=False, rm=True, forcerm=False, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, stream_logs=False, **kwargs)[source]

Task for building a Docker image. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

• path (str, optional): The path to the directory containing the Dockerfile
• tag (str, optional): The tag to give the final image
• nocache (bool, optional): Don't use cache when set to True
• rm (bool, optional): Remove intermediate containers; defaults to True
• forcerm (bool, optional): Always remove intermediate containers, even after unsuccessful builds; defaults to False
• docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
• extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method build). See https://docker-py.readthedocs.io/en/stable/api.html for more details
• stream_logs (bool, optional): Adds debug logs with build image status info from Docker
• **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:
prefect.tasks.docker.images.BuildImage.run (path=None, tag=None, nocache=False, rm=True, forcerm=False, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, stream_logs=False)[source]
Task run method. Args: path (str, optional): The path to the directory containing the Dockerfile tag (str, optional): The tag to give the final image nocache (bool, optional): Don't use cache when set to True rm (bool, optional): Remove intermediate containers; defaults to True forcerm (bool, optional): Always remove intermediate containers, even after unsuccessful builds; defaults to False docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method build). See https://docker-py.readthedocs.io/en/stable/api.html for more details stream_logs (bool, optional): Adds debug logs with build image status info from Docker Returns: List[dict]: a cleaned dictionary of the output of client.build Raises: ValueError: if either path is None

---

ListImages

class

prefect.tasks.docker.images.ListImages

(repository_name=None, all_layers=False, filters=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, **kwargs)[source]

Task for listing Docker images. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

• repository_name (str, optional): Only show images belonging to this repository; if not provided then it will list all images from the local Docker server
• all_layers (bool, optional): Show intermediate image layers
• filters (dict, optional): Filter the results. See https://docker-py.readthedocs.io/en/stable/images.html for more details
• docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
• extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method images). See https://docker-py.readthedocs.io/en/stable/api.html for more details
• **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:
prefect.tasks.docker.images.ListImages.run (repository_name=None, all_layers=False, filters=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None)[source]
Task run method. Args: repository_name (str, optional): Only show images belonging to this repository; if not provided then it will list all images from the local Docker server all_layers (bool, optional): Show intermediate image layers filters (dict, optional): Filter the results. See https://docker-py.readthedocs.io/en/stable/images.html for more details docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method images). See https://docker-py.readthedocs.io/en/stable/api.html for more details Returns: list: A list of dictionaries containing information about the images found

---

PullImage

class

prefect.tasks.docker.images.PullImage

(repository=None, tag=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, stream_logs=False, **kwargs)[source]

Task for pulling a Docker image. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

• repository (str, optional): The repository to pull the image from
• tag (str, optional): The tag of the image to pull; if not specified then the latest tag will be pulled
• docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
• extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method pull). See https://docker-py.readthedocs.io/en/stable/api.html for more details
• stream_logs (bool, optional): Adds debug logs with pull status info from Docker
• **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:
prefect.tasks.docker.images.PullImage.run (repository=None, tag=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, stream_logs=False)[source]
Task run method. Args: repository (str, optional): The repository to pull the image from tag (str, optional): The tag of the image to pull; if not specified then the latest tag will be pulled docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method pull). See https://docker-py.readthedocs.io/en/stable/api.html for more details stream_logs (bool, optional): Adds debug logs with pull status info from Docker Returns: str: The output from Docker for pulling the image Raises: ValueError: if repository is None HTTPError: if image doesn't exist or provided repository/tag is not correct

---

PushImage

class

prefect.tasks.docker.images.PushImage

(repository=None, tag=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, **kwargs)[source]

Task for pushing a Docker image. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

• repository (str, optional): The repository to push the image to
• tag (str, optional): The tag for the image to push; if not specified then the latest tag will be pushed
• docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
• extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method push). See https://docker-py.readthedocs.io/en/stable/api.html for more details
• **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:
prefect.tasks.docker.images.PushImage.run (repository=None, tag=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None)[source]
Task run method. Args: repository (str, optional): The repository to push the image to tag (str, optional): The tag for the image to push; if not specified then the latest tag will be pushed docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method push). See https://docker-py.readthedocs.io/en/stable/api.html for more details Returns: str: The output from Docker for pushing the image Raises: ValueError: if repository is None

---

RemoveImage

class

prefect.tasks.docker.images.RemoveImage

(image=None, force=False, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, **kwargs)[source]

Task for removing a Docker image. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

• image (str, optional): The image to remove
• force (bool, optional): Force removal of the image
• docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
• extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method remove_image). See https://docker-py.readthedocs.io/en/stable/api.html for more details
• **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:
prefect.tasks.docker.images.RemoveImage.run (image=None, force=False, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None)[source]
Task run method. Args: image (str, optional): The image to remove force (bool, optional): Force removal of the image docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method remove_image). See https://docker-py.readthedocs.io/en/stable/api.html for more details Raises: ValueError: if image is None

---

TagImage

class

prefect.tasks.docker.images.TagImage

(image=None, repository=None, tag=None, force=False, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, **kwargs)[source]

Task for tagging a Docker image. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

• image (str, optional): The image to tag
• repository (str, optional): The repository to set for the tag
• tag (str, optional): The tag name for the image
• force (bool, optional): Force tagging of the image
• docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
• extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method tag). See https://docker-py.readthedocs.io/en/stable/api.html for more details
• **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:
prefect.tasks.docker.images.TagImage.run (image=None, repository=None, tag=None, force=False, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None)[source]
Task run method. Args: image (str, optional): The image to tag repository (str, optional): The repository to set for the tag tag (str, optional): The tag name for the image force (bool, optional): Force tagging of the image docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method tag). See https://docker-py.readthedocs.io/en/stable/api.html for more details Returns: bool: Whether or not the tagging was successful Raises: ValueError: if either image or repository are None

---

CreateContainer

class

prefect.tasks.docker.containers.CreateContainer

(image_name=None, container_name=None, command=None, detach=False, entrypoint=None, environment=None, volumes=None, docker_server_url="unix:///var/run/docker.sock", host_config=None, extra_docker_kwargs=None, **kwargs)[source]

Task for creating a Docker container and optionally running a command. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

• image_name (str, optional): Name of the image to run
• container_name (str, optional): A name for the container
• command (Union[list, str], optional): A single command or a list of commands to run
• detach (bool, optional): Run container in the background
• entrypoint (Union[str, list], optional): The entrypoint for the container
• environment (Union[dict, list], optional): Environment variables to set inside the container, as a dictionary or a list of strings in the format ["SOMEVARIABLE=xxx"]
• volumes (Union[str, list], optional): List of paths inside the container to use as volumes
• docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
• host_config (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method create_host_config). See https://docker-py.readthedocs.io/en/stable/api.html for more details
• extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method create_container). See https://docker-py.readthedocs.io/en/stable/api.html for more details
• **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:
prefect.tasks.docker.containers.CreateContainer.run (image_name=None, container_name=None, command=None, detach=False, entrypoint=None, environment=None, volumes=None, docker_server_url="unix:///var/run/docker.sock", host_config=None, extra_docker_kwargs=None)[source]
Task run method. Args: image_name (str, optional): Name of the image to run container_name (str, optional): A name for the container command (Union[list, str], optional): A single command or a list of commands to run detach (bool, optional): Run container in the background entrypoint (Union[str, list]): The entrypoint for the container environment (Union[dict, list]): Environment variables to set inside the container, as a dictionary or a list of strings in the format ["SOMEVARIABLE=xxx"] volumes (Union[str, list], optional): List of paths inside the container to use as volumes docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided host_config (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method create_host_config). See https://docker-py.readthedocs.io/en/stable/api.html for more details extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Dockercall (cf. method create_container). See https://docker-py.readthedocs.io/en/stable/api.html for more details Returns: str: A string representing the container id Raises: ValueError: if image_name is None

---

GetContainerLogs

class

prefect.tasks.docker.containers.GetContainerLogs

(container_id=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, **kwargs)[source]

Task for getting the logs of a Docker container. Note: This does not stream logs. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

• container_id (str, optional): The id of a container to retrieve logs from
• docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
• extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method logs). See https://docker-py.readthedocs.io/en/stable/api.html for more details
• **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:
prefect.tasks.docker.containers.GetContainerLogs.run (container_id=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None)[source]
Task run method. Args: container_id (str, optional): The id of a container to retrieve logs from docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method logs). See https://docker-py.readthedocs.io/en/stable/api.html for more details Returns: str: A string representation of the logs from the container Raises: ValueError: if container_id is None

---

ListContainers

class

prefect.tasks.docker.containers.ListContainers

(all_containers=False, filters=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, **kwargs)[source]

Task for listing Docker containers. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

• all_containers (bool, optional): Show all containers. Only running containers are shown by default
• filters (dict, optional): Filter the results. See https://docker-py.readthedocs.io/en/stable/containers.html for more details
• docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
• extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method containers). See https://docker-py.readthedocs.io/en/stable/api.html for more details
• **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:
prefect.tasks.docker.containers.ListContainers.run (all_containers=False, filters=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None)[source]
Task run method. Args: all_containers (bool, optional): Show all containers. Only running containers are shown by default filters (dict, optional): Filter the results. See https://docker-py.readthedocs.io/en/stable/containers.html for more details docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method container). See https://docker-py.readthedocs.io/en/stable/api.html for more details Returns: list: A list of dicts, one per container

---

StartContainer

class

prefect.tasks.docker.containers.StartContainer

(container_id=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, **kwargs)[source]

Task for starting a Docker container that runs the (optional) command it was created with. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

• container_id (str, optional): The id of a container to start
• docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
• extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method start). See https://docker-py.readthedocs.io/en/stable/api.html for more details
• **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:
prefect.tasks.docker.containers.StartContainer.run (container_id=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None)[source]
Task run method. Args: container_id (str, optional): The id of a container to start docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method start). See https://docker-py.readthedocs.io/en/stable/api.html for more details Raises: ValueError: if container_id is None

---

StopContainer

class

prefect.tasks.docker.containers.StopContainer

(container_id=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, **kwargs)[source]

Task for stopping a Docker container. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

• container_id (str, optional): The id of a container to stop
• docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
• extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method stop). See https://docker-py.readthedocs.io/en/stable/api.html for more details
• **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:
prefect.tasks.docker.containers.StopContainer.run (container_id=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None)[source]
Task run method. Args: container_id (str, optional): The id of a container to stop docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method stop). See https://docker-py.readthedocs.io/en/stable/api.html for more details Raises: ValueError: if container_id is None

---

RemoveContainer

class

prefect.tasks.docker.containers.RemoveContainer

(container_id=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None, **kwargs)[source]

Task for removing a Docker container. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

• container_id (str, optional): The id of a container to remove
• docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
• extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method remove_container). See https://docker-py.readthedocs.io/en/stable/api.html for more details
• **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:
prefect.tasks.docker.containers.RemoveContainer.run (container_id=None, docker_server_url="unix:///var/run/docker.sock", extra_docker_kwargs=None)[source]
Task run method. Args: container_id (str, optional): The id of a container to remove docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method remove_container). See https://docker-py.readthedocs.io/en/stable/api.html for more details Raises: ValueError: if container_id is None

---

WaitOnContainer

class

prefect.tasks.docker.containers.WaitOnContainer

(container_id=None, docker_server_url="unix:///var/run/docker.sock", raise_on_exit_code=True, extra_docker_kwargs=None, **kwargs)[source]

Task for waiting on an already started Docker container. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

• container_id (str, optional): The id of a container to start
• docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided
• raise_on_exit_code (bool, optional): whether to raise a FAIL signal for a nonzero exit code; defaults to True
• extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method wait). See https://docker-py.readthedocs.io/en/stable/api.html for more details
• **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor

methods:
prefect.tasks.docker.containers.WaitOnContainer.run (container_id=None, docker_server_url="unix:///var/run/docker.sock", raise_on_exit_code=True, extra_docker_kwargs=None)[source]
Task run method. Args: container_id (str, optional): The id of a container to wait on docker_server_url (str, optional): URL for the Docker server. Defaults to unix:///var/run/docker.sock however other hosts such as tcp://0.0.0.0:2375 can be provided raise_on_exit_code (bool, optional): whether to raise a FAIL signal for a nonzero exit code; defaults to True extra_docker_kwargs (dict, optional): Extra keyword arguments to pass through to the Docker call (cf. method wait). See https://docker-py.readthedocs.io/en/stable/api.html for more details Returns: dict: a dictionary with StatusCode and Error keys Raises: ValueError: if container_id is None FAIL: if raise_on_exit_code is True and the container exits with a nonzero exit code

---

This documentation was auto-generated from commit bd9182e
on July 31, 2024 at 18:02 UTC