Prefect Docs

# Airbyte Tasks

Verified by Prefect

These tasks have been tested and verified by Prefect.

This module contains a task for triggering Airbyte connection sync jobs as well as a configuration export

# AirbyteConnectionTask

class

prefect.tasks.airbyte.airbyte.AirbyteConnectionTask

(airbyte_server_host="localhost", airbyte_server_port=8000, airbyte_api_version="v1", connection_id=None, stream_output=False, **kwargs)[source]

Task for triggering Airbyte Connections, where "A connection is a configuration for syncing data between a source and a destination." For more information refer to the Airbyte docs

This task assumes that the Airbyte Open-Source, since "For Airbyte Open-Source you don't need the API Token for Authentication! All endpoints are possible to access using the API without it." For more information refer to the Airbyte docs

Args:

  • airbyte_server_host (str, optional): Hostname of Airbyte server where connection is configured. Defaults to localhost.
  • airbyte_server_port (str, optional): Port that the Airbyte server is listening on. Defaults to 8000.
  • airbyte_api_version (str, optional): Version of Airbyte API to use to trigger connection sync. Defaults to v1.
  • connection_id (str, optional): Default connection id to use for sync jobs, if none is specified to run.
  • stream_output (Union[bool, int, str], optional): specifies whether this task should log the output as it occurs, and at what logging level. If True is passed, the logging level defaults to INFO; otherwise, any integer or string value that's passed will be treated as the log level, provided the logging library can successfully interpret it.
  • **kwargs (Any, optional): additional kwargs to pass to the base Task constructor

methods:                                                                                                                                                       

prefect.tasks.airbyte.airbyte.AirbyteConnectionTask.run

(airbyte_server_host=None, airbyte_server_port=None, airbyte_api_version=None, connection_id=None, poll_interval_s=15)[source]

Task run method for triggering an Airbyte Connection.

It is assumed that the user will have previously configured a Source & Destination into a Connection. e.g. MySql -> CSV

An invocation of run will attempt to start a sync job for the specified connection_id representing the Connection in Airbyte.

run will poll Airbyte Server for the Connection status and will only complete when the sync has completed or when it receives an error status code from an API call.

Args:

  • airbyte_server_host (str, optional): Hostname of Airbyte server where connection is configured. Will overwrite the value provided at init if provided.
  • airbyte_server_port (str, optional): Port that the Airbyte server is listening on. Will overwrite the value provided at init if provided.
  • airbyte_api_version (str, optional): Version of Airbyte API to use to trigger connection sync. Will overwrite the value provided at init if provided.
  • connection_id (str, optional): if provided, will overwrite the value provided at init.
  • poll_interval_s (int, optional): this task polls the Airbyte API for status, if provided this value will override the default polling time of 15 seconds.
Returns:
  • dict: connection_id (str) and succeeded_at (timestamp str)


# AirbyteConfigurationExport

class

prefect.tasks.airbyte.airbyte.AirbyteConfigurationExport

(airbyte_server_host="localhost", airbyte_server_port=8000, airbyte_api_version="v1", **kwargs)[source]

Task for triggering an export of the Airbyte configuration.

This task assumes that you are using Airbyte Open-Source, since "For Airbyte Open-Source you don't need the API Token for Authentication! All endpoints are possible to access using the API without it." For more information refer to the Airbyte docs.

Args:

  • airbyte_server_host (str, optional): Hostname of Airbyte server where connection is configured. Defaults to localhost.
  • airbyte_server_port (str, optional): Port that the Airbyte server is listening on. Defaults to 8000.
  • airbyte_api_version (str, optional): Version of Airbyte API to use to trigger connection sync. Defaults to v1.
  • **kwargs (Any, optional): Additional kwargs to pass to the base Task constructor.
Returns:
  • compressed gz file byte array of Airbyte configuration data. Airbyte requires this file type (or .tar) for configuration imports.

Example: from prefect import Flow from prefect.tasks.airbyte import AirbyteConfigurationExport import gzip

@task def unzip(export): with gzip.open('airbyte.gz', 'wb') as f: f.write(export)

airbyte = AirbyteConfigurationExport()

with Flow("airbyte_export") as flow: export = airbyte() unzip(export)

methods:                                                                                                                                                       

prefect.tasks.airbyte.airbyte.AirbyteConfigurationExport.run

(airbyte_server_host=None, airbyte_server_port=None, airbyte_api_version=None)[source]

Task run method for triggering an export of an Airbyte configuration

Args:

  • airbyte_server_host (str, optional): Hostname of Airbyte server where connection is configured. Will overwrite the value provided at init if provided.
  • airbyte_server_port (str, optional): Port that the Airbyte server is listening on. Will overwrite the value provided at init if provided.
  • airbyte_api_version (str, optional): Version of Airbyte API to use to trigger connection sync. Will overwrite the value provided at init if provided.
Returns:
  • byte array of Airbyte configuration data


This documentation was auto-generated from commit bd9182e
on July 31, 2024 at 18:02 UTC

← Airtable Tasks →