Prefect Docs

# Run Configuration

# RunConfig

class

prefect.run_configs.base.RunConfig

(env=None, labels=None)[source]

Base class for RunConfigs.

A "run config" is an object for configuring a flow run, which maps to a specific agent backend.

Args:

  • env (dict, optional): Additional environment variables to set
  • labels (Iterable[str], optional): an iterable of labels to apply to this run config. Labels are string identifiers used by Prefect Agents for selecting valid flow runs when polling for work

methods:                                                                                                                                                       

prefect.run_configs.base.RunConfig.serialize

()[source]

Returns a serialized version of the RunConfig.

Returns:

  • dict: the serialized RunConfig


# UniversalRun

class

prefect.run_configs.base.UniversalRun

(env=None, labels=None)[source]

Configure a flow-run to run universally on any Agent.

Unlike the other agent-specific RunConfig classes (e.g. LocalRun for the Local Agent), the UniversalRun run config is compatible with any agent. This can be useful for flows that don't require any custom configuration other than flow labels, allowing for transitioning a flow between agent types without any config changes.

Args:

  • env (dict, optional): Additional environment variables to set
  • labels (Iterable[str], optional): an iterable of labels to apply to this run config. Labels are string identifiers used by Prefect Agents for selecting valid flow runs when polling for work
Examples:

Use the defaults set on the agent:

flow.run_config = UniversalRun()

Configure additional labels:

flow.run_config = UniversalRun(env={"SOME_VAR": "value"}, labels=["label-1", "label-2"])

# LocalRun

class

prefect.run_configs.local.LocalRun

(env=None, working_dir=None, labels=None)[source]

Configure a flow-run to run as a Local Process.

Args:

  • env (dict, optional): Additional environment variables to set for the process
  • working_dir (str, optional): Working directory in which to start the process, must already exist. If not provided, will be run in the same directory as the agent.
  • labels (Iterable[str], optional): an iterable of labels to apply to this run config. Labels are string identifiers used by Prefect Agents for selecting valid flow runs when polling for work
Examples:

Use the defaults set on the agent:

flow.run_config = LocalRun()

Run in a specified working directory:

flow.run_config = LocalRun(working_dir="/path/to/working-directory")

Set an environment variable in the flow run process:

flow.run_config = LocalRun(env={"SOME_VAR": "value"})

# DockerRun

class

prefect.run_configs.docker.DockerRun

(image=None, env=None, labels=None)[source]

Configure a flow-run to run as a Docker container.

Args:

  • image (str, optional): The image to use
  • env (dict, optional): Additional environment variables to set in the container
  • labels (Iterable[str], optional): an iterable of labels to apply to this run config. Labels are string identifiers used by Prefect Agents for selecting valid flow runs when polling for work
Examples:

Use the defaults set on the agent:

flow.run_config = DockerRun()

Set an environment variable in the flow run process:

flow.run_config = DockerRun(env={"SOME_VAR": "value"})

# KubernetesRun

class

prefect.run_configs.kubernetes.KubernetesRun

(job_template_path=None, job_template=None, image=None, env=None, cpu_limit=None, cpu_request=None, memory_limit=None, memory_request=None, service_account_name=None, image_pull_secrets=None, labels=None, image_pull_policy=None)[source]

Configure a flow-run to run as a Kubernetes Job.

Kubernetes jobs are configured by filling in a job template at runtime. A job template can be specified either as a path (to be read in at runtime) or an in-memory object (which will be stored along with the flow in Prefect Cloud/Server). By default the job template configured on the Agent is used.

Args:

  • job_template_path (str, optional): Path to a job template to use. If a local path (no file scheme, or a file/local scheme), the job template will be loaded on initialization and stored on the KubernetesRun object as the job_template field. Otherwise the job template will be loaded at runtime on the agent. Supported runtime file schemes include (s3, gcs, and agent (for paths local to the runtime agent)).
  • job_template (str or dict, optional): An in-memory job template to use. Can be either a dict, or a YAML string.
  • image (str, optional): The image to use
  • env (dict, optional): Additional environment variables to set on the job
  • cpu_limit (float or str, optional): The CPU limit to use for the job
  • cpu_request (float or str, optional): The CPU request to use for the job
  • memory_limit (str, optional): The memory limit to use for the job
  • memory_request (str, optional): The memory request to use for the job
  • service_account_name (str, optional): A service account name to use for this job. If present, overrides any service account configured on the agent or in the job template.
  • image_pull_secrets (list, optional): A list of image pull secrets to use for this job. If present, overrides any image pull secrets configured on the agent or in the job template.
  • labels (Iterable[str], optional): an iterable of labels to apply to this run config. Labels are string identifiers used by Prefect Agents for selecting valid flow runs when polling for work
  • image_pull_policy (str, optional): The imagePullPolicy to use for the job. https://kubernetes.io/docs/concepts/configuration/overview/#container-images
Examples:

Use the defaults set on the agent:

flow.run_config = KubernetesRun()

Use a local job template, which is stored along with the Flow in Prefect Cloud/Server:

flow.run_config = KubernetesRun(
    job_template_path="my_custom_template.yaml"
)

Use a job template stored in S3, but override the image and CPU limit:

flow.run_config = KubernetesRun(
    job_template_path="s3://example-bucket/my_custom_template.yaml",
    image="example/my-custom-image:latest",
    cpu_limit=2,
)

Use an image not tagged with :latest, and set the image pull policy to Always:

flow.run_config = KubernetesRun(
    image="example/my-custom-image:my-tag,
    image_pull_policy="Always"
)

# ECSRun

class

prefect.run_configs.ecs.ECSRun

(task_definition=None, task_definition_path=None, task_definition_arn=None, image=None, env=None, cpu=None, memory=None, task_role_arn=None, execution_role_arn=None, run_task_kwargs=None, labels=None)[source]

Configure a flow-run to run as an ECS Task.

ECS Tasks are composed of task definitions and runtime parameters.

Task definitions can be configured using either the task_definition, task_definition_path, or task_definition_arn parameters. If neither is specified, the default configured on the agent will be used. At runtime this task definition will be registered once per flow version - subsequent runs of the same flow version will reuse the existing definition.

Runtime parameters can be specified via run_task_kwargs. These will be merged with any runtime parameters configured on the agent when starting the task.

Args:

  • task_definition (dict, optional): An in-memory task definition spec to use. The flow will be executed in a container named flow - if a container named flow isn't part of the task definition, Prefect will add a new container with that name. See the ECS.Client.register_task_definition docs for more information on task definitions. Note that this definition will be stored directly in Prefect Cloud/Server - use task_definition_path instead if you wish to avoid this.
  • task_definition_path (str, optional): Path to a task definition spec to use. If a local path (no file scheme, or a file/local scheme), the task definition will be loaded on initialization and stored on the ECSRun object as the task_definition field. Otherwise the task definition will be loaded at runtime on the agent. Supported runtime file schemes include (s3, gcs, and agent (for paths local to the runtime agent)).
  • task_definition_arn (str, optional): A pre-registered task definition ARN to use (either family, family:version, or a full task definition ARN). This task definition must include a container named flow (which will be used to run the flow).
  • image (str, optional): The image to use for this task. If not provided, will be either inferred from the flow's storage (if using Docker storage), or use the default configured on the agent.
  • env (dict, optional): Additional environment variables to set on the task.
  • cpu (int or str, optional): The amount of CPU available to the task. Can be ether an integer in CPU units (e.g. 1024), or a string using vCPUs (e.g. "1 vcpu"). Note that ECS imposes strict limits on what values this can take, see the ECS documentation for more information.
  • memory (int or str, optional): The amount of memory available to the task. Can be ether an integer in MiB (e.g. 1024), or a string with units (e.g. "1 GB"). Note that ECS imposes strict limits on what values this can take, see the ECS documentation for more information.
  • task_role_arn (str, optional): The name or full ARN for the IAM role to use for this task. If not provided, the default on the agent will be used (if configured).
  • execution_role_arn (str, optional): The execution role ARN to use when registering a task definition for this task. If not provided, the default on the agent will be used (if configured).
  • run_task_kwargs (dict, optional): Additional keyword arguments to pass to run_task when starting this task. See the ECS.Client.run_task docs for more information.
  • labels (Iterable[str], optional): An iterable of labels to apply to this run config. Labels are string identifiers used by Prefect Agents for selecting valid flow runs when polling for work
Examples:

Use the defaults set on the agent:

flow.run_config = ECSRun()

Use the default task definition, but override the image and CPU:

flow.run_config = ECSRun(
    image="example/my-custom-image:latest",
    cpu="2 vcpu",
)

Use an explicit task definition stored in s3, but override the image and CPU:

flow.run_config = ECSRun(
    task_definition="s3://bucket/path/to/task.yaml",
    image="example/my-custom-image:latest",
    cpu="2 vcpu",
)

This documentation was auto-generated from commit n/a
on July 1, 2021 at 18:35 UTC

← Storage →