Prefect Docs

# Monte Carlo Tasks

Verified by Prefect

These tasks have been tested and verified by Prefect.

# MonteCarloCreateOrUpdateLineage

class

prefect.tasks.monte_carlo.monte_carlo_lineage.MonteCarloCreateOrUpdateLineage

(source=None, destination=None, api_key_id=None, api_token=None, prefect_context_tags=True, expire_at=None, **kwargs)[source]

Task for creating or updating a lineage node for the given source and destination nodes, as well as for creating a lineage edge between those nodes.

Args:

  • source (dict, optional): a source node configuration. Expected to include the following keys: node_name, object_id, object_type, resource_name, and optionally also a list of tags.
  • destination (dict, optional): a destination node configuration. Expected to include the following keys: node_name, object_id, object_type, resource_name, and optionally also a list of tags.
  • api_key_id (string, optional): to use this task, you need to pass the Monte Carlo API credentials. Those can be created using https://getmontecarlo.com/settings/api. To avoid passing the credentials in plain text, you can leverage the PrefectSecret task in your flow.
  • api_token (string, optional): the API token. To avoid passing the credentials in plain text, you can leverage PrefectSecret task in your flow.
  • prefect_context_tags (bool, optional): whether to automatically add tags with Prefect context.
  • expire_at (string, optional): date and time indicating when to expire a source-destination edge. You can expire specific lineage nodes. If this value is set, the edge between a source and destination nodes will expire on a given date. Expected format: "YYYY-MM-DDTHH:mm:ss.SSS". For example, "2042-01-01T00:00:00.000".
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor.
Example source:

    source = dict(
        # this may be a shorter version of object_id
        node_name="example_table_name",
        object_id="example_table_name",
        # "table" is recommended, but you can use any string, e.g. "csv_file"
        object_type="table",
        resource_name="name_your_source_system",
        tags=[{"propertyName": "prefect_test_key", "propertyValue": "prefect_test_value"}]
    )

Example destination:

    destination = dict(
        # the full name of a data warehouse table
        node_name="db_name:schema_name.table_name",
        object_id="db_name:schema_name.table_name",
        # "table" is recommended, but you can use any string, e.g. "csv_file"
        object_type="table",
        resource_name="your_dwh_resource_name",
        tags=[{"propertyName": "prefect_test_key", "propertyValue": "prefect_test_value"}]
    )
methods:                                                                                                                                                       

prefect.tasks.monte_carlo.monte_carlo_lineage.MonteCarloCreateOrUpdateLineage.run

(source=None, destination=None, api_key_id=None, api_token=None, prefect_context_tags=True, expire_at=None)[source]

Creates or updates a lineage node for the given source and destination, and creates a lineage edge between them. If a list of tags is set for either source or destination (or both), the task adds those to the source and/or destination nodes. If no tags are set, the task creates a node without custom tags. Additionally, if the prefect_context_tags flag is set to True (default), the task creates tags with information from Prefect context for debugging data downtime issues.

Args:

  • source (dict, optional): a source node configuration. Expected to include the following keys: node_name, object_id, object_type, resource_name, and optionally also a list of tags.
  • destination (dict, optional): a destination node configuration. Expected to include the following keys: node_name, object_id, object_type, resource_name, and optionally also a list of tags.
  • api_key_id (string, optional): to use this task, you need to pass the Monte Carlo API credentials. Those can be created using https://getmontecarlo.com/settings/api. To avoid passing the credentials in plain text, you can leverage the PrefectSecret task in your flow.
  • api_token (string, optional): the API token. To avoid passing the credentials in plain text, you can leverage PrefectSecret task in your flow.
  • prefect_context_tags (bool, optional): whether to automatically add tags with Prefect context.
  • expire_at (string, optional): date and time indicating when to expire a source-destination edge. You can expire specific lineage nodes. If this value is set, the edge between a source and destination nodes will expire on a given date. Expected format: "YYYY-MM-DDTHH:mm:ss.SSS". For example, "2042-01-01T00:00:00.000".
Returns:
  • dict: a GraphQL response dictionary with the edge ID.


# MonteCarloCreateOrUpdateNodeWithTags

class

prefect.tasks.monte_carlo.monte_carlo_lineage.MonteCarloCreateOrUpdateNodeWithTags

(node_name=None, object_id=None, object_type=None, resource_name=None, lineage_tags=None, api_key_id=None, api_token=None, prefect_context_tags=True, **kwargs)[source]

Task for creating or updating a lineage node in the Monte Carlo Catalog and enriching it with multiple metadata tags. If the prefect_context_tags flag is set to True (default), the task additionally creates tags with Prefect context for debugging data downtime issues.

Args:

  • node_name (string, optional): the display name of a lineage node.
  • object_id (string, optional): the object ID of a lineage node.
  • object_type (string, optional): the object type of a lineage node - usually, either "table" or "view".
  • resource_name (string, optional): name of the data warehouse or custom resource. All resources can be retrieved via a separate task.
  • lineage_tags (list, optional): a list of dictionaries where each dictionary is a single tag.
  • api_key_id (string, optional): to use this task, you need to pass the Monte Carlo API credentials. Those can be created using https://getmontecarlo.com/settings/api. To avoid passing the credentials in plain text, you can leverage the PrefectSecret task in your flow.
  • api_token (string, optional): the API token. To avoid passing the credentials in plain text, you can leverage the PrefectSecret task in your flow.
  • prefect_context_tags (bool, optional): whether to automatically add tags with Prefect context.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor.
The expected format for lineage_tags:

    [{"propertyName": "tag_name", "propertyValue": "your_tag_value"},
    {"propertyName": "another_tag_name", "propertyValue": "your_tag_value"}]
methods:                                                                                                                                                       

prefect.tasks.monte_carlo.monte_carlo_lineage.MonteCarloCreateOrUpdateNodeWithTags.run

(node_name=None, object_id=None, object_type=None, resource_name=None, lineage_tags=None, api_key_id=None, api_token=None, prefect_context_tags=True)[source]

Creates or updates multiple metadata tags for a given lineage node in the Monte Carlo Catalog. If the prefect_context_tags flag is set to True (default), the task additionally creates a tag with the Prefect context dictionary for debugging data downtime issues.

Args:

  • node_name (string, optional): the display name of a lineage node.
  • object_id (string, optional): the object ID of a lineage node.
  • object_type (string, optional): the object type of a lineage node - usually, either "table" or "view".
  • resource_name (string, optional): name of the data warehouse or custom resource. All resources can be retrieved via a separate task.
  • lineage_tags (list, optional): a list of dictionaries where each dictionary is a single tag.
  • api_key_id (string, optional): to use this task, you need to pass the Monte Carlo API credentials. Those can be created using https://getmontecarlo.com/settings/api. To avoid passing the credentials in plain text, you can leverage the PrefectSecret task in your flow.
  • api_token (string, optional): the API token. To avoid passing the credentials in plain text, you can leverage the PrefectSecret task in your flow.
  • prefect_context_tags (bool, optional): whether to automatically add a tag with Prefect context.
Returns:
  • string: MCON - a Monte Carlo internal ID of the node.


# MonteCarloGetResources

class

prefect.tasks.monte_carlo.monte_carlo_lineage.MonteCarloGetResources

(api_key_id=None, api_token=None, **kwargs)[source]

Task for querying resources using the Monte Carlo API. You can use this task to find the name of your data warehouse or other resource, and use it as resource_name in other Monte Carlo tasks.

Args:

  • api_key_id (string, optional): to use this task, you need to pass the Monte Carlo API credentials. Those can be created using https://getmontecarlo.com/settings/api. To avoid passing the credentials in plain text, you can leverage the PrefectSecret task in your flow.
  • api_token (string, optional): the API token. To avoid passing the credentials in plain text, you can leverage the PrefectSecret task in your flow.
  • **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor.

methods:                                                                                                                                                       

prefect.tasks.monte_carlo.monte_carlo_lineage.MonteCarloGetResources.run

(api_key_id=None, api_token=None)[source]

Retrieve all Monte Carlo resources.

Args:

  • api_key_id (string, optional): to use this task, you need to pass the Monte Carlo API credentials. Those can be created using https://getmontecarlo.com/settings/api. To avoid passing the credentials in plain text, you can leverage the PrefectSecret task in your flow.
  • api_token (string, optional): the API token. To avoid passing the credentials in plain text, you can leverage the PrefectSecret task in your flow.
Returns:
  • list: Monte Carlo resources, including all existing data warehouses and custom resources.


This documentation was auto-generated from commit bd9182e
on July 31, 2024 at 18:02 UTC

← MySQL Tasks →