Prefect Docs

# Azure Tasks

This module contains a collection of tasks for interacting with Azure resources.

# BlobStorageDownload

class

prefect.tasks.azure.blobstorage.BlobStorageDownload

(azure_credentials_secret="AZ_CONNECTION_STRING", container=None, **kwargs)[source]

Task for downloading data from an Blob Storage container and returning it as a string. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

  • azure_credentials_secret (str, optional): the name of the Prefect Secret that stores your Azure credentials; this Secret must be an Azure connection string
  • container (str, optional): the name of the Azure Blob Storage to download from
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.azure.blobstorage.BlobStorageDownload.run

(blob_name, azure_credentials_secret="AZ_CONNECTION_STRING", container=None)[source]

Task run method.

Args:

  • blob_name (str): the name of the blob within this container to retrieve
  • azure_credentials_secret (str, optional): the name of the Prefect Secret that stores your Azure credentials; this Secret must be an Azure connection string
  • container (str, optional): the name of the Blob Storage container to download from
Returns:
  • str: the contents of this blob_name / container, as a string


# BlobStorageUpload

class

prefect.tasks.azure.blobstorage.BlobStorageUpload

(azure_credentials_secret="AZ_CONNECTION_STRING", container=None, overwrite=False, **kwargs)[source]

Task for uploading string data (e.g., a JSON string) to an Azure Blob Storage container. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

  • azure_credentials_secret (str, optional): the name of the Prefect Secret that stores your Azure credentials; this Secret must be an Azure connection string
  • container (str, optional): the name of the Azure Blob Storage to upload to
  • overwrite (bool, optional): if True, an existing blob with the same name will be overwritten. Defaults to False and an error will be thrown if the blob already exists.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.azure.blobstorage.BlobStorageUpload.run

(data, blob_name=None, azure_credentials_secret="AZ_CONNECTION_STRING", container=None, overwrite=False)[source]

Task run method.

Args:

  • data (str): the data payload to upload
  • blob_name (str, optional): the name to upload the data under; if not provided, a random uuid will be created
  • azure_credentials_secret (str, optional): the name of the Prefect Secret that stores your Azure credentials; this Secret must be an Azure connection string
  • container (str, optional): the name of the Blob Storage container to upload to
  • overwrite (bool, optional): if True, an existing blob with the same name will be overwritten. Defaults to False and an error will be thrown if the blob already exists.
Returns:
  • str: the name of the blob the data payload was uploaded to


# CosmosDBCreateItem

class

prefect.tasks.azure.cosmosdb.CosmosDBCreateItem

(url=None, database_or_container_link=None, item=None, azure_credentials_secret="AZ_CREDENTIALS", options=None, **kwargs)[source]

Task for creating an item in a Azure Cosmos database. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

  • url (str, optional): The url to the database.
  • database_or_container_link (str, optional): link to the database or container.
  • item (dict, optional): the item to create
  • azure_credentials_secret (str, optional): the name of the Prefect Secret that stores your Azure credentials; this Secret must be JSON string with the key AZ_COSMOS_AUTH. The value should be dictionary containing masterKey or resourceTokens, where the masterKey value is the default authorization key to use to create the client, and resourceTokens value is the alternative authorization key.
  • options (dict, optional): options to be passed to the azure.cosmos.cosmos_client.CosmosClient.CreateItem method.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.azure.cosmosdb.CosmosDBCreateItem.run

(url=None, database_or_container_link=None, item=None, azure_credentials_secret="AZ_CREDENTIALS", options=None)[source]

Task run method.

Args:

  • url (str, optional): The url to the database.
  • database_or_container_link (str, optional): link to the database or container.
  • item (dict, optional): the item to create
  • azure_credentials_secret (str, optional): the name of the Prefect Secret that stores your Azure credentials; this Secret must be JSON string with the key AZ_COSMOS_AUTH. The value should be dictionary containing masterKey or resourceTokens, where the masterKey value is the default authorization key to use to create the client, and resourceTokens value is the alternative authorization key.
  • options (dict, optional): options to be passed to the azure.cosmos.cosmos_client.CosmosClient.CreateItem method.
Returns:
  • (dict): the created item.


# CosmosDBReadItems

class

prefect.tasks.azure.cosmosdb.CosmosDBReadItems

(url=None, document_or_container_link=None, azure_credentials_secret="AZ_CREDENTIALS", options=None, **kwargs)[source]

Task for reading items from a Azure Cosmos database. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

  • url (str, optional): The url to the database.
  • document_or_container_link (str, optional): link to a document or container. If a document link is provided, the document in question is returned, otherwise all docuements are returned.
  • azure_credentials_secret (str, optional): the name of the Prefect Secret that stores your Azure credentials; this Secret must be JSON string with the key AZ_COSMOS_AUTH. The value should be dictionary containing masterKey or resourceTokens, where the masterKey value is the default authorization key to use to create the client, and resourceTokens value is the alternative authorization key.
  • options (dict, optional): options to be passed to the azure.cosmos.cosmos_client.CosmosClient.ReadItem or ReadItems method.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.azure.cosmosdb.CosmosDBReadItems.run

(url=None, document_or_container_link=None, azure_credentials_secret="AZ_CREDENTIALS", options=None)[source]

Task run method.

Args:

  • url (str, optional): The url to the database.
  • document_or_container_link (str, optional): link to a document or container. If a document link is provided, the document in question is returned, otherwise all docuements are returned.
  • azure_credentials_secret (str, optional): the name of the Prefect Secret that stores your Azure credentials; this Secret must be JSON string with the key AZ_COSMOS_AUTH. The value should be dictionary containing masterKey or resourceTokens, where the masterKey value is the default authorization key to use to create the client, and resourceTokens value is the alternative authorization key.
  • options (dict, optional): options to be passed to the azure.cosmos.cosmos_client.CosmosClient.ReadItem or ReadItems method.
Returns:
  • (dict or list)): a single document or all documents.


# CosmosDBQueryItems

class

prefect.tasks.azure.cosmosdb.CosmosDBQueryItems

(url=None, database_or_container_link=None, query=None, azure_credentials_secret="AZ_CREDENTIALS", options=None, partition_key=None, **kwargs)[source]

Task for creating an item in a Azure Cosmos database. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

  • url (str, optional): The url to the database.
  • database_or_container_link (str, optional): link to the database or container.
  • query (dict, optional): the query to run
  • azure_credentials_secret (str, optional): the name of the Prefect Secret that stores your Azure credentials; this Secret must be JSON string with the key AZ_COSMOS_AUTH. The value should be dictionary containing masterKey or resourceTokens, where the masterKey value is the default authorization key to use to create the client, and resourceTokens value is the alternative authorization key.
  • options (dict, optional): options to be passed to the azure.cosmos.cosmos_client.CosmosClient.QueryItems method.
  • partition_key (str, None): Partition key for the query.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor

methods:                                                                                                                                                       

prefect.tasks.azure.cosmosdb.CosmosDBQueryItems.run

(url=None, database_or_container_link=None, query=None, azure_credentials_secret="AZ_CREDENTIALS", options=None, partition_key=None)[source]

Task run method.

Args:

  • url (str, optional): The url to the database.
  • database_or_container_link (str, optional): link to the database or container.
  • query (dict, optional): the query to run
  • azure_credentials_secret (str, optional): the name of the Prefect Secret that stores your Azure credentials; this Secret must be JSON string with the key AZ_COSMOS_AUTH. The value should be dictionary containing masterKey or resourceTokens, where the masterKey value is the default authorization key to use to create the client, and resourceTokens value is the alternative authorization key.
  • options (dict, optional): options to be passed to the azure.cosmos.cosmos_client.CosmosClient.QueryItems method.
  • partition_key (str, None): Partition key for the query.
Returns:
  • (list): a list containing the query results, one item per row.


# DatafactoryCreate

class

prefect.tasks.azure.datafactory.DatafactoryCreate

(datafactory_name=None, resource_group_name=None, azure_credentials_secret="AZ_CREDENTIALS", location="eastus", polling_interval=10, options=None, **kwargs)[source]

Task for creating an Azure datafactory. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

  • datafactory_name (str): Name of the datafactory to create.
  • resource_group_name (str): Name of the resource group.
  • azure_credentials_secret (str): Name of the Prefect Secret that stores your Azure credentials; This Secret must be JSON string with the keys subscription_id, client_id, secret, and tenant. Defaults to "AZ_CREDENTIALS".
  • location (str, optional): The location of the datafactory.
  • polling_interval (int, optional): The interval, in seconds, to check the provisioning state of the datafactory Defaults to 10.
  • options (dict, optional): The options to be passed to the create_or_update method.
  • **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor.

methods:                                                                                                                                                       

prefect.tasks.azure.datafactory.DatafactoryCreate.run

(datafactory_name=None, resource_group_name=None, azure_credentials_secret=None, location=None, polling_interval=None, options=None)[source]

Create an Azure datafactory.

Args:

  • datafactory_name (str): Name of the datafactory to create.
  • resource_group_name (str): Name of the resource group.
  • azure_credentials_secret (str): Name of the Prefect Secret that stores your Azure credentials; This Secret must be JSON string with the keys subscription_id, client_id, secret, and tenant. Defaults to "AZ_CREDENTIALS".
  • location (str, optional): The location of the datafactory.
  • polling_interval (int, optional): The interval, in seconds, to check the provisioning state of the datafactory Defaults to 10.
  • options (dict, optional): The options to be passed to the create_or_update method.
Returns: The datafactory name.


# PipelineCreate

class

prefect.tasks.azure.datafactory.PipelineCreate

(datafactory_name=None, resource_group_name=None, pipeline_name=None, activities=None, azure_credentials_secret="AZ_CREDENTIALS", parameters=None, options=None, **kwargs)[source]

Task for creating an Azure datafactory pipeline. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

  • datafactory_name (str): Name of the datafactory to create.
  • resource_group_name (str): Name of the resource group.
  • pipeline_name (str): Name of the pipeline.
  • activities (list): The list of activities to run in the pipeline.
  • azure_credentials_secret (str, optional): Name of the Prefect Secret that stores your Azure credentials; This Secret must be JSON string with the keys subscription_id, client_id, secret, and tenant. Defaults to "AZ_CREDENTIALS".
  • parameters (dict): The parameters to be used in pipeline.
  • options (dict, optional): The options to be passed to the create_or_update method.
  • **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor.

methods:                                                                                                                                                       

prefect.tasks.azure.datafactory.PipelineCreate.run

(datafactory_name=None, resource_group_name=None, pipeline_name=None, activities=None, azure_credentials_secret=None, parameters=None, options=None)[source]

Create an Azure datafactory pipeline.

Args:

  • datafactory_name (str): Name of the datafactory to create.
  • resource_group_name (str): Name of the resource group.
  • pipeline_name (str): Name of the pipeline.
  • activities (list): The list of activities to run in the pipeline.
  • parameters (dict): The parameters to be used in pipeline.
  • azure_credentials_secret (str, optional): Name of the Prefect Secret that stores your Azure credentials; This Secret must be JSON string with the keys subscription_id, client_id, secret, and tenant. Defaults to "AZ_CREDENTIALS".
  • options (dict, optional): The options to be passed to the create_or_update method.
Returns: The pipeline name.


# PipelineRun

class

prefect.tasks.azure.datafactory.PipelineRun

(datafactory_name=None, resource_group_name=None, pipeline_name=None, azure_credentials_secret="AZ_CREDENTIALS", parameters=None, polling_interval=10, last_updated_after=None, last_updated_before=None, **kwargs)[source]

Task for creating an Azure datafactory pipeline run. Note that all initialization arguments can optionally be provided or overwritten at runtime.

Args:

  • datafactory_name (str): Name of the datafactory to create.
  • resource_group_name (str): Name of the resource group.
  • pipeline_name (str): Name of the pipeline.
  • azure_credentials_secret (str, optional): Name of the Prefect Secret that stores your Azure credentials; This Secret must be JSON string with the keys subscription_id, client_id, secret, and tenant. Defaults to "AZ_CREDENTIALS".
  • parameters (dict, optional): The parameters to be used in pipeline.
  • polling_interval (int, optional): The interval, in seconds, to check the status of the run. Defaults to 10.
  • last_updated_after (datetime, optional): The time at or after which the run event was updated; used to filter and query the pipeline run, and defaults to yesterday.
  • last_updated_before (datetime, optional): The time at or before which the run event was updated; used to filter and query the pipeline run and defaults to tomorrow.
  • **kwargs (dict, optional): Additional keyword arguments to pass to the Task constructor.

methods:                                                                                                                                                       

prefect.tasks.azure.datafactory.PipelineRun.run

(datafactory_name=None, resource_group_name=None, pipeline_name=None, azure_credentials_secret=None, parameters=None, polling_interval=None, last_updated_after=None, last_updated_before=None)[source]

Create an Azure datafactory pipeline run.

Args:

  • datafactory_name (str): Name of the datafactory to create.
  • resource_group_name (str): Name of the resource group.
  • pipeline_name (str): Name of the pipeline.
  • azure_credentials_secret (str, optional): Name of the Prefect Secret that stores your Azure credentials; This Secret must be JSON string with the keys subscription_id, client_id, secret, and tenant. Defaults to "AZ_CREDENTIALS".
  • parameters (dict, optional): The parameters to be used in pipeline.
  • polling_interval (int, optional): The interval, in seconds, to check the status of the run. Defaults to 10.
  • last_updated_after (datetime, optional): The time at or after which the run event was updated; used to filter and query the pipeline run, and defaults to yesterday.
  • last_updated_before (datetime, optional): The time at or before which the run event was updated; used to filter and query the pipeline run and defaults to tomorrow.
Returns: The pipeline run response.


This documentation was auto-generated from commit bd9182e
on July 31, 2024 at 18:02 UTC

← Azure ML Service Tasks →