Prefect Docs

# AWS Tasks

This module contains a collection of tasks for interacting with AWS resources.

All AWS related tasks can be authenticated using the AWS_CREDENTIALS Prefect Secret that should be a dictionary with two keys: "ACCESS_KEY" and "SECRET_ACCESS_KEY". See Third Party Authentication for more information.

# S3Download

class

prefect.tasks.aws.s3.S3Download

(bucket=None, boto_kwargs=None, **kwargs)[source]

Task for downloading data from an S3 bucket and returning it as a string. Note that all initialization arguments can optionally be provided or overwritten at runtime.

For authentication, there are two options: you can set the AWS_CREDENTIALS Prefect Secret containing your AWS access keys which will be passed directly to the boto3 client, or you can configure your flow's runtime environment for boto3.

Args:

  • bucket (str, optional): the name of the S3 Bucket to download from
  • boto_kwargs (dict, optional): additional keyword arguments to forward to the boto client.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.aws.s3.S3Download.run

(key, credentials=None, bucket=None, compression=None, as_bytes=False)[source]

Task run method.

Args:

  • key (str): the name of the Key within this bucket to retrieve
  • credentials (dict, optional): your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.
  • bucket (str, optional): the name of the S3 Bucket to download from
  • compression (str, optional): specifies a file format for decompression, decompressing data upon download. Currently supports 'gzip'.
  • as_bytes (bool, optional): If true, result will be returned as bytes instead of str. Defaults to False.
Returns:
  • str: the contents of this Key / Bucket, as a string or bytes


# S3Upload

class

prefect.tasks.aws.s3.S3Upload

(bucket=None, boto_kwargs=None, **kwargs)[source]

Task for uploading string data (e.g., a JSON string) to an S3 bucket. Note that all initialization arguments can optionally be provided or overwritten at runtime.

For authentication, there are two options: you can set a Prefect Secret containing your AWS access keys which will be passed directly to the boto3 client, or you can configure your flow's runtime environment for boto3.

Args:

  • bucket (str, optional): the name of the S3 Bucket to upload to
  • boto_kwargs (dict, optional): additional keyword arguments to forward to the boto client.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.aws.s3.S3Upload.run

(data, key=None, credentials=None, bucket=None, compression=None)[source]

Task run method.

Args:

  • data (str): the data payload to upload
  • key (str, optional): the Key to upload the data under; if not provided, a random uuid will be created
  • credentials (dict, optional): your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.
  • bucket (str, optional): the name of the S3 Bucket to upload to
  • compression (str, optional): specifies a file format for compression, compressing data before upload. Currently supports 'gzip'.
Returns:
  • str: the name of the Key the data payload was uploaded to


# S3List

class

prefect.tasks.aws.s3.S3List

(bucket=None, **kwargs)[source]

Task for listing files from an S3 bucket. Note that all initialization arguments can optionally be provided or overwritten at runtime.

For authentication, there are two options: you can set the AWS_CREDENTIALS Prefect Secret containing your AWS access keys which will be passed directly to the boto3 client, or you can configure your flow's runtime environment for boto3.

Args:

  • bucket (str, optional): the name of the S3 Bucket to list the files of.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.aws.s3.S3List.run

(prefix, delimiter="", page_size=None, max_items=None, credentials=None, bucket=None, last_modified_begin=None, last_modified_end=None)[source]

Task run method.

Args:

  • prefix (str): the name of the prefix within this bucket to retrieve objects from
  • delimiter (str): indicates the key hierarchy
  • page_size (int): controls the number of items returned per page of each result
  • max_items (int): limits the maximum number of total items returned during pagination
  • credentials (dict, optional): your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.
  • bucket (str, optional): the name of the S3 Bucket to list the files of
  • last_modified_begin (str, optional): keep items with LastModified greater than or equal to given value. timestamp should be in RFC 3339, ISO 8601 or any pendulum supported format.
  • last_modified_end (str, optional): keep items with LastModified less than or equal to given value. timestamp should be in RFC 3339, ISO 8601 or any pendulum supported format.
Returns:
  • list[str]: A list of keys that match the given prefix.


# LambdaCreate

class

prefect.tasks.aws.lambda_function.LambdaCreate

(function_name, runtime, role, handler, zip_file=None, bucket="", bucket_key="", object_version=None, description="", function_timeout=3, memorysize=128, publish=True, subnet_ids=None, security_group_ids=None, dead_letter_config=None, environment_variables=None, kms_key_arn="", function_tags=None, tracing_config="PassThrough", layers=None, boto_kwargs=None, **kwargs)[source]

Task for creating a Lambda function.

Args:

  • function_name (str): name of the Lambda function to create
  • runtime (str): the identifier of the function's runtime
  • role (str): the Amazon Resource Name of the function's execution role
  • handler (str): the name of the method within your code that Lambda calls to execute your function
  • zip_file (str): path to zip file containing code for Lambda function, either zip_file or (bucket and bucket_key) must be passed
  • bucket (str): an S3 bucket in the same AWS region as your function
  • bucket_key (str): the Amazon S3 key of the deployment package
  • object_version (str, optional): for versioned S3 objects, the version of the deployment package to use
  • description (str, optional): description of Lambda function
  • function_timeout (int, optional): Lambda function timeout in seconds, default is 3 seconds
  • memorysize (int, optional): amount of memory that Lambda function has access to in MB, must be a multiple of 64 MB, default is 128
  • publish (bool, optional): set to True to publish the first version of the function during creation, defaults to True
  • subnet_ids (List[str], optional): list of subnet ids for vpc configuration
  • security_group_ids (List[str], optional): list of security group ideas for vpc configuration
  • dead_letter_config (dict, optional): a dead letter queue configuration that specifies the queue or topic where Lambda sends asynchronous events when they fail processing
  • environment_variables (dict, optional): key-value pairs of environment variables to pass to the Lambda function
  • kms_key_arn (str, optional): the ARN of the AWS key management service used to encrypt your function's environment variables, if not provided, AWS Lambda uses a default service key
  • function_tags (dict, optional): a list of tags to apply to the function, string to string map
  • tracing_config (str, optional): set to Active to samle and trace a subset of incoming requests with Amazon X-Ray
  • layers (List[str], optional): a list of function layers to add to the function's execution environment, specify each layer by its ARN
  • boto_kwargs (dict, optional): additional keyword arguments to forward to the boto client.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.aws.lambda_function.LambdaCreate.run

(credentials=None)[source]

Task run method. Creates Lambda function.

Args:

  • credentials (dict, optional): your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.
Returns:
  • json: response from AWS CreateFunction endpoint


# LambdaDelete

class

prefect.tasks.aws.lambda_function.LambdaDelete

(function_name, qualifier="", boto_kwargs=None, **kwargs)[source]

Task for deleting a Lambda function.

Args:

  • function_name (str): name of the Lambda function to delete
  • qualifier (str, optional): specify a version to delete, if not provided, the function will be deleted entirely
  • boto_kwargs (dict, optional): additional keyword arguments to forward to the boto client.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.aws.lambda_function.LambdaDelete.run

(credentials=None)[source]

Task run method. Deletes Lambda function.

Args:

  • credentials (dict, optional): your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.
Returns:
  • dict: response from AWS DeleteFunction endpoint


# LambdaInvoke

class

prefect.tasks.aws.lambda_function.LambdaInvoke

(function_name, invocation_type="RequestResponse", log_type="None", client_context=None, payload="null", qualifier="$LATEST", boto_kwargs=None, **kwargs)[source]

Task to invoke a Lambda function.

Args:

  • function_name (str): the name of the Lambda funciton to invoke
  • invocation_type (str, optional): the invocation type of Lambda function, default is RequestResponse other options include Event and DryRun
  • log_type (str, optional): set to 'Tail' to include the execution log in the response
  • client_context (dict, optional): data to pass to the function in the context object, dict object will be transformed into base64 encoded json automatically
  • payload (bytes or seekable file-like object): the JSON provided to Lambda function as input
  • qualifier (str, optional): specify a version or alias to invoke a published version of the function, defaults to $LATEST
  • boto_kwargs (dict, optional): additional keyword arguments to forward to the boto client.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.aws.lambda_function.LambdaInvoke.run

(function_name=None, payload=None, credentials=None)[source]

Task run method. Invokes Lambda function.

Args:

  • function_name (str): the name of the Lambda funciton to invoke
  • payload (bytes or seekable file-like object): the JSON provided to Lambda function as input
  • credentials (dict, optional): your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.
Returns:
  • dict: response from AWS Invoke endpoint


# LambdaList

class

prefect.tasks.aws.lambda_function.LambdaList

(master_region="ALL", function_version="ALL", marker=None, max_items=50, boto_kwargs=None, **kwargs)[source]

Task to list Lambda functions.

Args:

  • master_region (str, optional): for Lambda@Edge functions, the AWS region of the master function
  • function_version (str, optional): the version of a function, default is 'ALL'
  • marker (str, optional): specify the pagination token that's returned by a previous request to retreive the next page of results
  • max_items (int, optional): specify a value between 1 and 50 to limit the number of functions in the response
  • boto_kwargs (dict, optional): additional keyword arguments to forward to the boto client.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.aws.lambda_function.LambdaList.run

(credentials=None)[source]

Task fun method. Lists all Lambda functions.

Args:

  • credentials (dict, optional): your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.
Returns:
  • dict: a list of Lambda functions from AWS ListFunctions endpoint


# StepActivate

class

prefect.tasks.aws.step_function.StepActivate

(state_machine_arn, execution_name, execution_input="{}", boto_kwargs=None, **kwargs)[source]

Task for activating an AWS Step Function.

Args:

  • state_machine_arn (str): the Amazon Resource Name (ARN) of the state machine to execute
  • execution_name (str): the name of the execution, this name must be unique for your AWS account, region, and state machine for 90 days
  • execution_input (str, optional): string that contains the JSON input data for the execution
  • boto_kwargs (dict, optional): additional keyword arguments to forward to the boto client.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.aws.step_function.StepActivate.run

(credentials=None)[source]

Task run method. Activates AWS Step function.

Args:

  • credentials (dict, optional): your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.
Returns:
  • dict: response from AWS StartExecution endpoint


# AWSSecretsManager

class

prefect.tasks.aws.secrets_manager.AWSSecretsManager

(secret=None, boto_kwargs=None, **kwargs)[source]

Task for retrieving secrets from an AWS Secrets Manager and returning it as a dictionary. Note that all initialization arguments can optionally be provided or overwritten at runtime.

For authentication, there are two options: you can set the AWS_CREDENTIALS Prefect Secret containing your AWS access keys which will be passed directly to the boto3 client, or you can configure your flow's runtime environment for boto3.

Args:

  • secret (str, optional): the name of the secret to retrieve
  • boto_kwargs (dict, optional): additional keyword arguments to forward to the boto client.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.aws.secrets_manager.AWSSecretsManager.run

(secret=None, credentials=None)[source]

Task run method.

Args:

  • secret (str): the name of the secret to retrieve
  • credentials (dict, optional): your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.
Returns:
  • dict: the contents of this secret, as a dictionary


# BatchSubmit

class

prefect.tasks.aws.batch.BatchSubmit

(job_name=None, job_definition=None, job_queue=None, boto_kwargs=None, **kwargs)[source]

Task for submitting a job to AWS batch.

For authentication, there are two options: you can set the AWS_CREDENTIALS Prefect Secret containing your AWS access keys which will be passed directly to the boto3 client, or you can configure your flow's runtime environment for boto3.

Args:

  • job_name (str, optional): The AWS batch job name.
  • job_definition (str, optional): The AWS batch job definition.
  • job_queue (str, optional): Name of the AWS batch job queue.
  • boto_kwargs (dict, optional): additional kekyword arguments to forward to the boto client.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.aws.batch.BatchSubmit.run

(job_name=None, job_definition=None, job_queue=None, batch_kwargs=None, credentials=None)[source]

Submit a job to the AWS Batch job service.

Args:

  • job_name (str, optional): The AWS batch job name.
  • job_definition (str, optional): The AWS batch job definition.
  • job_queue (str, optional): Name of the AWS batch job queue.
  • batch_kwargs (dict, optional): Additional keyword arguments to pass to the boto3 submit_job function. See the submit_job # noqa documentation for more details.
  • credentials (dict, optional): your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.


# AWSClientWait

class

prefect.tasks.aws.client_waiter.AWSClientWait

(client=None, waiter_name=None, waiter_definition=None, boto_kwargs=None, **kwargs)[source]

Task for waiting on a long-running AWS job. Uses the underlying boto3 waiter functionality.

For authentication, there are two options: you can set the AWS_CREDENTIALS Prefect Secret containing your AWS access keys which will be passed directly to the boto3 client, or you can configure your flow's runtime environment for boto3.

Args:

  • client (str, optional): The AWS client on which to wait (e.g., 'batch', 'ec2', etc)
  • waiter_name (str, optional): The name of the waiter to instantiate. Can be a boto-supported waiter or one of prefect's custom waiters. Currently, prefect offers three additional waiters for AWS Batch: "JobExists" waits for a job to be instantiated, "JobRunning" waits for a job to start running, and "JobComplete" waits for a job to finish. You can find the definitions for all prefect-defined waiters here. # noqa You may also use a custom waiter name, if you supply an accompanying waiter definition dict.
  • waiter_definition (dict, optional): A valid custom waiter model, as a dict. Note that if you supply a custom definition, it is assumed that the provided 'waiter_name' is contained within the waiter definition dict.
  • boto_kwargs (dict, optional): additional kekyword arguments to forward to the boto client.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.aws.client_waiter.AWSClientWait.run

(client=None, waiter_name=None, waiter_definition=None, waiter_kwargs=None, credentials=None)[source]

Task for waiting on a long-running AWS job. Uses the underlying boto3 waiter functionality.

Args:

  • client (str): The AWS client on which to wait (e.g., 'batch', 'ec2', etc)
  • waiter_name (str, optional): The name of the waiter to instantiate. Can be a boto-supported waiter or one of prefect's custom waiters. Currently, prefect offers three additional waiters for AWS Batch: "JobExists" waits for a job to be instantiated, "JobRunning" waits for a job to start running, and "JobComplete" waits for a job to finish. You can find the definitions for all prefect-defined waiters here. # noqa You may also use a custom waiter name, if you supply an accompanying waiter definition dict.
  • waiter_definition (dict, optional): A valid custom waiter model, as a dict. Note that if you supply a custom definition, it is assumed that the provided 'waiter_name' is contained within the waiter definition dict.
  • waiter_kwargs (dict, optional): Arguments to pass to the waiter.wait(...) method. Will depend upon the specific waiter being called.
  • credentials (dict, optional): your AWS credentials passed from an upstream Secret task; this Secret must be a JSON string with two keys: ACCESS_KEY and SECRET_ACCESS_KEY which will be passed directly to boto3. If not provided here or in context, boto3 will fall back on standard AWS rules for authentication.


This documentation was auto-generated from commit n/a
on July 1, 2021 at 18:35 UTC

← Azure Tasks →