Prefect Docs

# Airbyte Tasks

Verified by Prefect

These tasks have been tested and verified by Prefect.

This module contains a task for triggering Airbyte connection sync jobs

# AirbyteConnectionTask

class

prefect.tasks.airbyte.airbyte.AirbyteConnectionTask

(airbyte_server_host="localhost", airbyte_server_port=8000, airbyte_api_version="v1", connection_id=None, **kwargs)[source]

Task for triggering Airbyte Connections, where "A connection is a configuration for syncing data between a source and a destination." For more information refer to the Airbyte docs

This task assumes that the Airbyte Open-Source, since "For Airbyte Open-Source you don't need the API Token for Authentication! All endpoints are possible to access using the API without it." For more information refer to the Airbyte docs

Args:

  • airbyte_server_host (str, optional): Hostname of Airbyte server where connection is configured. Defaults to localhost.
  • airbyte_server_port (str, optional): Port that the Airbyte server is listening on. Defaults to 8000.
  • airbyte_api_version (str, optional): Version of Airbyte API to use to trigger connection sync. Defaults to v1.
  • connection_id (str, optional): Default connection id to use for sync jobs, if none is specified to run.
  • **kwargs (Any, optional): additional kwargs to pass to the base Task constructor

methods:                                                                                                                                                       

prefect.tasks.airbyte.airbyte.AirbyteConnectionTask.run

(airbyte_server_host=None, airbyte_server_port=None, airbyte_api_version=None, connection_id=None, poll_interval_s=15)[source]

Task run method for triggering an Airbyte Connection.

It is assumed that the user will have previously configured a Source & Destination into a Connection. e.g. MySql -> CSV

An invocation of run will attempt to start a sync job for the specified connection_id representing the Connection in Airbyte.

run will poll Airbyte Server for the Connection status and will only complete when the sync has completed or when it receives an error status code from an API call.

Args:

  • airbyte_server_host (str, optional): Hostname of Airbyte server where connection is configured. Will overwrite the value provided at init if provided.
  • airbyte_server_port (str, optional): Port that the Airbyte server is listening on. Will overwrite the value provided at init if provided.
  • airbyte_api_version (str, optional): Version of Airbyte API to use to trigger connection sync. Will overwrite the value provided at init if provided.
  • connection_id (str, optional): if provided, will overwrite the value provided at init.
  • poll_interval_s (int, optional): this task polls the Airbyte API for status, if provided this value will override the default polling time of 15 seconds.
Returns:
  • dict: connection_id (str) and succeeded_at (timestamp str)


This documentation was auto-generated from commit n/a
on February 23, 2022 at 19:26 UTC

← Airtable Tasks →