# Changelog
# 0.14.22 beta
Released on June 15, 2021.
# Enhancements
- Use
functools.update_wrapper
forFunctionTask
- #4608 - Add ability to merge reference tasks when combining two flows - #4644
- Add client side check for key value size - #4655
- Ensure stack traces are included in logs during task run exceptions - #4657
- Add
poll_interval
parameter toStartFlowRun
to define the polling interval when waiting for the flow run to finish - #4641 - Add ability to set task timeouts with
timedelta
objects - #4619
# Fixes
- Add ssh documentation - #4539
# Contributors
# 0.14.21 beta
Released on June 2, 2021.
# Features
- Add interface for backend key-value metadata store - #4499
# Enhancements
- Keep intermediate docker layers when using Docker Storage to improve caching - #4584
# Fixes
- Fix possible race condition in
LocalResult
directory creation - #4587 - Use absolute paths when registering flows in
prefect register
- #4593 - Propagate storage labels (e.g. hostname label on
Local
storage) when registering flows withprefect register
- #4593 - Fix small-flow parallelism issues with multiprocess
LocalDaskExecutor
- #4602 - Cleanly handle unpicklable exceptions in tasks - #4605
# Contributors
# 0.14.20 beta
Released on May 25, 2021.
# Enhancements
- Refactor
Agent
base class for readability - #4341 - Display the agent config id on agent startup if set - #4524
- Add debug logs during agent auth verification - #4547
- Sending logs to Cloud can be globally disabled via config in addition to the agent flag - #4487
# Task Library
- Enable sending attachments with emails in the
EmailTask
- #4457 - Add Google Cloud Platform
GCPSecret
task - #4561
# Fixes
- Fix
import_object
handling of submodules that are not attributes - #4513 - Fix
DockerStorage
building with python slim image - #4523 - Gracefully handle events with missing timestamps in K8s agent - #4544
- Fix bug where agent uses originally scheduled start time instead of latest state time - #4568
# Deprecations
logging.log_to_cloud
has been deprecated in favor ofcloud.send_flow_run_logs
- #4487
# Contributors
# 0.14.19 beta
Released on May 11, 2021 as a hotfix for 0.14.18
# Fixes
- Fix docker container name error while using docker agents - #4511
# 0.14.18 beta
Released on May 11, 2021.
# Enhancements
- Add an
image_pull_policy
kwarg toKubernetesRun
- #4462 - Allow
prefect build/register --module
to accept full import path to flow - #4468 - Add
hello world
flow to prefect module - #4470 - Set docker container names to flow run names when running with the docker agent - #4485
# Task Library
- Add basic implementation of
SendGrid
to Task Library - #4450 - Log link to flow run in
StartFlowRun
- #4458 - Allow passing in
io.BytesIO
toGCSUpload
- #4482
# Fixes
- Fix logging errors within
BigQuery
task - #4419 - Remove unnecessary docker interface detection in Docker agent - #4446
- Upgrade
kubernetes
package requirement upper limit - #4452 - Fix Prefect server startup check for custom server port - #4501
# Contributors
# 0.14.17 beta
Released on April 27, 2021.
# Features
- Add git storage - #4418
# Enhancements
- Add test coverage for threaded
LocalDaskExecutor
timeouts - #4217 - Add environment variable support to UniversalRunConfig - #4383
- Adds column name fetching to PostgresFetch task - #4414
- Allow external Postgres with
prefect server start
command - #4424
# Fixes
- Pass reference tasks states instead of task states to terminal_state_handler - #4409
- Check for AWS_RETRY_MODE variable before setting default in
ECSAgent
- #4417 - Fixed bug from Flow.copy() not copying the slugs dictionary - #4435
- Fix compatibility with PyGithub >= 1.55 - #4440
# Contributors
# 0.14.16 beta
Released on April 14, 2021.
# Enhancements
- Added support for Bitbucket cloud into Bitbucket storage class #4318
- Update docs for API keys - #4313
- Add option to disable the deletion of finished Prefect jobs in the Kubernetes agent - #4351
- Improve messaging when a flow is skipped during registration - #4373
- Display a more helpful error when calling a task outside a flow context - #4374
- Lower required docker-compose version and add note to docs - #4396
- Increase healthcheck intervals for Prefect Server - #4396
- Allow the
ShellTask
to be used on win32 - #4397
# Fixes
- Fix ShellTask docstring - #4360
- Fix incorrect unused task tracking - #4368
- Add error handling to timeouts that fail during result pickling or passing - #4384
# Contributors
# 0.14.15 beta
Released on April 5, 2021.
# Enhancements
- Add terminal flow state handler override - #4198
- When manually set,
prefect.context.date
will be used to determine dependent values - #4295 prefect.context.date
will be cast to aDateTime
object if given a parsable string - #4295- Expand logging for
DaskExecutor
, including the cluster dashboard address (if available) - #4321 - Add ability to stream ShellTask logs with level INFO - #4322
- Add architecture diagram to docs - #4187
- Speed up flow validation logic - #4347
- Pin dask upper package versions - #4350
# Task Library
# Fixes
- Fix
DatabricksRunNow
task attribute override behavior - #4309 - Use default flow labels when triggering flow runs from CLI - #4316
- Improve ECS agent error messages, and fix bug that prevented using ECS agent on Fargate with ECR - #4325
- Support imports from local directory when registering/building flows via CLI - #4332
- Speedup flushing of logs to cloud/server on flow run shutdown, avoiding lost logs on platforms that SIGKILL the process after a short period - #4334
# Contributors
# 0.14.14 beta
Released on March 25, 2021.
# Task Library
- Ensures Snowflake Query Task output is serializable #3744
# Fixes
- Always load task run info prior to running a task - #4296
# 0.14.13 beta
Released on March 24, 2021.
# Features
- Add new improved
prefect register
CLI command. This command supports registering multiple flows at once (through multiple file paths or directories), and also includes a new--watch
flag for watching files and automatically re-registering upon changes. - #4256 - New
prefect build
CLI command for building flows. Artifacts produced by this command can then be used byprefect register
to register flows without requiring the source. - #4282
# Enhancements
- Use explicit exception chaining #3306
- Add
as_bytes
option toS3Download
task - #4238 - Improve error message when a loaded flow doesn't match the version stored in Prefect Cloud/Server - #4259
- Support setting flow run labels from cli - #4266
- Support setting default
image
in--job-template
/--task-definition
in Kubernetes/ECS agents - #4270
# Task Library
- Adds logging of cell outputs to Jupyter task - [#4265] (https://github.com/PrefectHQ/prefect/issues/4265)
- Add
user
andpassword
as runtime parameters to Exasol tasks - #4268
# Fixes
- Fix bug where sometimes the global
prefect.context
wouldn't be respected during a flow run - #4287
# Deprecations
- Deprecate the old
prefect register flow
CLI command in favor ofprefect register
- #4256 - Deprecate
user
andpassword
arguments to Exasol task constructors in favor of runtime parameters - #4268
# Contributors
# 0.14.12 beta
Released on March 10, 2021.
# Enhancements
- Upgrade hasura to 1.3.3 in Prefect Server - #4126
- Add
--docker-client-timeout
flag to docker agent, for configuring the timeout for all docker API requests - #4232 - Make
--slug
flag optional inprefect server create-tenant
- #4240
# Task Library
- Adds new filesystem
Copy
andRemove
tasks - #4202
# Fixes
- Don't forward
nout
to mapped tasks - #4206 - Move
command
,environment
,cpu
,memory
,execution_role_arn
, andtask_role_arn
configuration for ECS tasks from definition time to run time in the ECS agent - #4211 - Register (and deregister) a new task definition for every flow run in ECS agent - #4211
- Fix
Task
signature generation in the presence of with variadic kwargs - #4235 - Ensure
Flow
is serializable usingpickle
- #4209
# Contributors
# 0.14.11 beta
Released on March 3, 2021.
# Features
- Add command
prefect server config
to output configured docker-compose yaml - #4176
# Enhancements
- Add
project_name
andproject_id
to context during Cloud/Server flow runs - #4083 - Better error message when flow not found in file - #4182
- Improve generated names for
GetItem
tasks - #4183 - Add
base_url
option toGitHub
storage - #4194
# Task Library
- Add filehandling tasks
Move
,Unzip
,Zip
- #4131 - Add
msg_plain
,email_to_cc
, andemail_to_bcc
options toEmailTask
- #4157 - Add
jar_params
option toDatabricksRunNow
task - #4157
# Fixes
- Make task slug generation robust to modifying existing task names - #4189
- Forward
client_options
toS3Result
fromS3
storage - #4195
# Contributors
# 0.14.10 beta
Released on February 23, 2021.
# Fixes
- Dynamically import compression libraries for
CompressedSerializer
- #4150 - Support passing environment variables containing
=
through agent CLI--env
flag - #4160
# 0.14.9 beta
Released on February 16, 2021.
# Enhancements
- Add
CompressedSerializer
class - #4063 - Allow
Client
timeout seconds to be configurable through configuration - #4118
# Task Library
- Update
FilterTask
to allow logging the filtered output via a function - #4121 - Add
FivetranSyncTask
, to manage your Fivetran connector sync process - #4116
# Fixes
- Reduce the number of boto3 clients created across Prefect when interacting with AWS services - #4115
# Deprecations
- Deprecate the
use_session
argument in all AWS-related components - #4115
# Contributors
# 0.14.8 beta
Released on February 11, 2021.
# Enhancements
- Add option to provide version group ID to
prefect run flow
CLI command - #4100
# Fixes
- Fix bug in agent healthcheck route that was introduced in 0.14.7 - #4109
# 0.14.7 beta
Released on February 10, 2021.
# Enhancements
- Support multiple docker networks with Docker Agent - #3986
- Add healthchecks to prefect server - #4041
- Raise custom
TimeoutError
for task timeouts to allow for more granular user control - #4091 - Add
access_token_secret
toGitHub
,GitLab
, andBitbucket
storage, making the Prefect secret containing an access token to these services configurable - #4059 - Add
--skip-if-flow-metadata-unchanged
toprefect register flow
CLI command that avoids bumping flow version if flow metadata has not changed - #4061 - Add
--skip-if-exists
toprefect create project
CLI command that safely skips if the project has already been created - #4061 - Add new
Module
storage class, for referencing flows importable from a Python module - #4073 - Drop resource limits from manifest generated using
prefect agent kubernetes install
- #4077
# Task Library
- Add new tasks for communication with an Exasol database - #4044
# Fixes
- Fix task decorator chaining by using
inspect.unwrap
instead of__wrap__
- #4053 - Forward Prefect backend type to deployed flow runs, ensuring backend-specific logic functions properly - #4076
- Patch around bug in dask's multiprocessing scheduler introduced in Dask 2021.02.0 - #4089
# Deprecations
- Docker agent
network
kwarg deprecated in favor ofnetworks
- #3986
# Breaking Changes
- Remove unused
Storage.get_env_runner
method - #4059 - Remove private utilities in
prefect.utilities.git
- #4059
# Contributors
# 0.14.6 beta
Released on February 2, 2021.
# Enhancements
- Add option to provide flow ID to
run flow
CLI command - #4021 - Flow name and project are no longer required options when calling
run flow
CLI command - #4021
# Task Library
- Add GCSBlobExists which checks for the existence of an object in a given GCS bucket - #4025
- Use boto3 session in
S3Upload
andS3Download
tasks, to ensure thread-safe execution - #3981
# Fixes
- Fix issue with fixed duration Paused states not resuming properly - #4031
# Contributors
# 0.14.5 beta
Released on January 26, 2021.
# Enhancements
- S3 storage now logs
ETag
,LastModified
timestamp, andVersionId
(if present) when loading a flow - #3995 GitHub
storage now logs the commit sha used when loading a flow - #3998GitHub
storage now loads from a repo's default branch, allowing default branch names other than 'master' - #3998- Improve error message when Secrets are missing with Server - #4003
- Better error message when passing parameters to
StartFlowRun
constructor - #4008 - Add warning if user-defined class shadows an attribute used by the base class - #4011
- Add support for
EXTRA_PIP_PACKAGES
environment variable inprefecthq/prefect
images, simplifying installation of dependencies during development - #4013 - Add execution role arn parameter to ecs run config and agent - #4015
# Task Library
- Add
ConnectGetNamespacedPodExec
task which runs an exec command in provided pod container - #3991 - Ensure connection secrets can be passed to Databricks tasks at runtime - #4001
# Fixes
- Fix Agent registration possibly skipping on server connection issues - #3972
GCSUpload
task now explicitely fails when ran on non-supported types - #3978- Make logging to Prefect cloud more robust in the presence of errors or process shutdown - #3989
- Handle setting state for missing flow runs in Kubernetes agent resource management - #4006
# Contributors
# 0.14.4 beta
Released on January 19, 2021.
# Enhancements
# Task Library
- Stream log output from Kubernetes RunNamespacedJob - #3715
- Add ReadNamespacedPodLogs which reads or streams logs from Kubernetes pod - #3715
- Add SQL Server task to query SQL Server databases - #3958
- Add chunking to GCP storage tasks - #3968
# Fixes
- Properly handle
NotImplementedError
exceptions raised by a result's serializer - #3964 - Fix support for storing multiple flows in a single script in storage - #3969
- Fix regression in
apply_map
which prevented use incase
/resource_manager
blocks - #3975
# Contributors
# 0.14.3 beta
Released on January 13, 2021.
# Enhancements
- Better errors/warnings when flow fails to load in execution environment - #3940
# Task Library
- Add an Asana task to add tasks to an asana project - #3935
# Fixes
- Fix
prefect server start
failure when given a custom graphql host port - #3933 - Fix Kubernetes Agent attempting to report container info for failed pods when no container statuses are found - #3941
- Avoid race condition when creating task run artifacts for mapped tasks - #3953
- Propogate agent labels info to k8s flow runs, to match other agent behavior - #3954
# 0.14.2 beta
Released on January 6, 2021.
# Features
- Support for specifying
run_config
for an individual flow run - #3903 - Allow the usage of a
profile_name
onget_boto_client
- #3916
# Enhancements
- Support executing Prefect agents/flows without having the
prefect
CLI on path - #3918
# Task Library
- Add support for specifying a
run_config
inStartFlowRun
- #3903 - Task to add Trello card for task library - #3910
# Fixes
- Remove unused
description
field onTask
serializer - #3917 - Fix edge case in
apply_map
that resulted in cycles in theFlow
graph - #3920 - Support storing multiple local flows with the same name when using
Local
storage - #3923 - Fix bug in
prefect.context
contextmanager that resulted in context fields reverting to their initially configured values - #3924
# Contributors
# 0.14.1 beta
Released on December 29, 2020.
# Enhancements
- Make
setup
method optional forresource_manager
tasks - #3869 - Add labels to all containers managed by the docker agent - #3893
- Add
prefect server stop
command for stopping the server - #3899 - Add
--detach
toprefect server start
for running the server in the background - #3899
# Fixes
- Add support for
google-cloud-storage
< 1.31.0 - #3875 - Fix use of
imagePullSecrets
/serviceAccountName
in k8s agent - #3884 - Fix
read_bytes_from_path
to work properly with S3 - #3885 - Change default
idempotency_key
inStartFlowRun
to usetask_run_id
instead offlow_run_id
- #3892
# 0.14.0 beta
Released on December 16, 2020.
# Features
- New flow run configuration system based on
RunConfig
types, see here for more info
# Enhancements
- Kubernetes Agent now reports events for pending pods created by prefect jobs - #3783
- Using
--rbac
for Kubernetes Agent install command now includes theevents
resource - #3783 - Add orchestration-based dependencies to the
prefecthq/prefect
Docker image - #3804 - Add a slimmed down
prefecthq/prefect:core
Docker image that only contains base dependencies - #3804 - Docker storage now installs all orchestration-based dependencies when using default image - #3804
- Add warning on flow registration if
flow.executor
is set but the flow is using the legacyflow.environment
configuration system - #3808 - Echoing prefect config file as JSON to be able to parse it with jq in the terminal - #3818
- Produce artifact for RunGreatExpectationsValidation even if validation fails - #3829
execute flow-run
command now sends flow run log in the case of an error - #3832- Changed name of logs raised by the Kubernetes Agent if they stem from infrastructure events - #3832
- Add
tini
to the official Prefect docker images - #3839 - Remove task run level heartbeats for performance - #3842
# Task Library
- Fix mising
job_id
inDatabricksRunNow
task initialization - #3793
# Fixes
- Fix Azure result byte decoding of blob data - #3846
- Prefect kubernetes agent no longer relies on existence of any fields in configured Kubernetes Job Template - #3805
- Accept old envvar style configuration for Kubernetes agent for
--service-account-name
/--image-pull-secrets
options - #3814 - Pass
as_user=False
when usingclient.get_cloud_url
inStartFlowRun
- #3850 - Fix AWS boto3 utility passing duplicate kwargs to client initialization - #3857
# Deprecations
- Storage classes have been moved from
prefect.environments.storage
toprefect.storage
, the old import paths have been deprecated accordingly - #3796 - Executor classes have been moved from
prefect.engine.executors
toprefect.executors
, the old import paths have been deprecated accordingly - #3798 - Deprecated use of
storage_labels
boolean kwarg on local agent - #3800 - Deprecated use of
--storage-labels
option from agentstart
CLI command - #3800 - Deprecates all
Environment
classes - users should transition to settingflow.run_config
instead offlow.environment
- #3811 - Deprecate the Fargate Agent in favor of the ECS Agent - #3812
# Breaking Changes
- Using in-cluster installs of the Kubernetes Agent now requires RBAC for the
events
resource - #3783 - Removed setting of default labels on storage objects and the local agent - #3800
- Remove deprecated
RemoteEnvironment
/RemoteDaskEnvironment
- #3802 - Remove deprecated
executor_kwargs
argument toKubernetesJobEnvironment
/FargateTaskEnvironment
- #3802 - Remove deprecated
prefect run cloud
/prefect run server
CLI commands - #3803 - Remove deprecated
prefect execute cloud-flow
CLI command - #3803 - Stop building the
prefecthq/prefect:all_extras
image and switch flow deployment default to usingprefecthq/prefect:{core_version}
- #3804 - Flows now use
RunConfig
based deployments by default - legacyEnvironment
based deployments are now opt-in only - #3806 - Remove deprecated
prefect.contrib
module - #3813 - Remove all references to result handlers and safe results - #3838
- Remove option to enable deprecated Kubernetes resource manager in agent install CLI command - #3840
# Contributors
# 0.13.19 beta
Released on December 8, 2020.
# Enhancements
- Use explicit exception chaining - #3306
- Support Bitbucket as storage option - #3711
- Surface pod failures and container errors in jobs deployed with the Kubernetes Agent - 3747
- Support timeout option in GCS tasks - #3732
- Added storage option for AWS CodeCommit - #3733
- Add the image used for a flow-run to the flow run environment as
prefect.context.image
- #3746 - Add
UniversalRun
run-config that works with all agents - #3750 - Support flows that have no run-config or environment - #3750
- Allow Docker storage environment vars to be used in commands - #3755
- Add
service_account_name
andimage_pull_secrets
options toKubernetesRun
andKubernetesAgent
- #3778 - Add a new Client function
delete_project
- #3728
# Task Library
- Add task to fetch data from Dremio - #3734
- Add
RunGreatExpectationsValidation
task - #3753 - Add the option to post markdown artifacts from the
RunGreatExpectationsValidation
task - #3753
# Fixes
- Fix state attempting to read result from absent upstream result - #3618
- Replace deprecated download_as_string method with download_as_bytes method - #3741
- Fix default image whenever working on a non-tagged commit - #3748
- Fix type-casting for task timeout defaults loaded from config - #3761
- Fix the
ref
default on GitHub storage - #3764 - Fix rare cancellation bug when running with external Dask cluster - #3770
# Deprecations
- Deprecated the
RunGreatExpectationsCheckpoint
task in favor ofRunGreatExpectationsValidation
- #3766
# Contributors
# 0.13.18 beta
Released on November 30, 2020.
# Enhancements
- Display formatted graphql errors on client request failure - #3632
- Refactor Core Client API calls for performance - #3730
# Task Library
- Refactor execute query code for
PostgresExecute
,PostgresExecuteMany
, andPostgresFetch
tasks - #3714 - Fix
PicklingError
inBigQueryLoadFile
andBigQueryLoadGoogleCloudStorage
- #3724 - Allow custom exporter for
ExecuteNotebook
task - #3725 - Properly forward
location
parameter in bigquery tasks - #3726 - Support passing
helper_script
toShellTask
/DBTShellTask
at runtime - #3729
# Fixes
- Fix bug with docker storage throwing exception while trying to display output - #3717
# Contributors
# 0.13.17 beta
Released on November 24, 2020.
# Features
- Improved support for Tasks returning multiple results - #3697
# Enhancements
- Allow chaining of
Task
imperative dependency calls - #3696 - Add
task_definition_arn
toECSRun
run-config - #3681 - Rerun
resource_manager
tasks when restarting flows from failed - #3689 - Raise nice warning if user passes
Task
instance toTask
constructor, rather than when calling theTask
(or usingTask.map
/Task.set_dependencies
) - #3691 - Always use tenant slug in output of Client
get_cloud_url
function - #3692
# Task Library
- Add enhancement to
StartFlowRun
task to create link artifact for started flow run - #3692 - Add a new postgres task
PostgresExecuteMany
- #3703 - Add debug logging for Docker tasks
PullImage
andBuildImage
- #3672 ShellTask
returns output on failure - #3649ShellTask
allows streaming of output independently of the number of lines returned - #3649
# Fixes
- Make
serialized_hash
handle unordered task sets correctly - #3682 - Docker storage build error logs were not always displayed - #3693
- Fix automatic quoting of Docker storage environment variable values - #3694
- Use
exist_ok
flag inos.makedirs
to avoid race condition in local storage class - #3679
# Contributors
# 0.13.16 beta
Released on November 17, 2020.
# Enhancements
- Experimental support for Python 3.9 - #3411
# Fixes
- Fixes Flow.replace freezing reference tasks - #3655
- Fixed bug where
flow.serialized_hash()
could return inconsistent values across new python instances - #3654
# Contributors
# 0.13.15 beta
Released on November 11, 2020.
# Features
- Add API for storing task run artifacts in the backend - #3581
# Enhancements
- Allow for setting
Client
headers before loading tenant when running with Prefect Server - #3515 - Checkpoint all iterations of Looped tasks - #3619
- Add
ref
option to GitHub storage for specifying branches other than master - #3638 - Added
ExecuteNotebook
task for running Jupyter notebooks - #3599 - Pass
day_or
croniter argument to CronClock and CronSchedule - #3612 Client.create_project
andprefect create project
will skip creating the project if the project already exists - #3630- Update deployments extension to AppsV1Api - #3637
PrefectSecret
andEnvVarSecret
tasks no longer require secret names be provided at flow creation time - #3641
# Fixes
- Fix issue with retrying mapped pipelines on dask - #3519
- Task arguments take precedence when generating
task_run_name
- #3605 - Fix breaking change in flow registration with old server versions - #3642
- Task arguments take precedence when generating templated targets and locations - #3627
# Breaking Changes
- Environment variable config values now parse without requiring escaping backslashes - #3603
# Contributors
# 0.13.14 beta
Released on November 5, 2020.
# Features
flow.register
accepts an idempotency key to prevent excessive flow versions from being created - #3590- Added
flow.serialized_hash()
for easy generation of hash keys from the serialized flow - #3590
# Enhancements
- Add option to select
cursor_type
for MySQLFetch task - #3574 - Add new
ECSAgent
andECSRun
run config - #3585 - Display exception information on
prefect create project
failure - #3589 prefect diagnostics
no longer displays keys that have values matching the default config - #3593- Allow use of multiple image pull secrets in
KubernetesAgent
,DaskKubernetesEnvironment
- #3596 - Added FROM to explicitly chain exceptions in src/prefect/tasks/twitter - #3602
- Add UTC offset to default logging.datefmt; logging timestamp converter now follows Python default behavior - #3607
- Improve error message when API responds with 400 status code - #3615
# Deprecations
- Deprecate
prefect agent start <kind>
in favor ofprefect agent <kind> start
- #3610 - Deprecate
prefect agent install <kind>
in favor ofprefect agent <kind> install
- #3610
# Contributors
# 0.13.13 beta
Released on October 27, 2020.
# Enhancements
- Don't stop execution if the task runner fails to load a cached result - #3378
- Add option to specify
networkMode
for tasks created by the Fargate Agent - #3546 - Allows to schedule flow runs at an arbitrary time with StartFlowRun - #3573
# Fixes
- Use
BlobServiceClient
instead ofBlockBlobService
to connect to azure blob in azure tasks - #3562 - Tasks with
log_stdout=True
work with non-utf8 output - #3563
# Contributors
# 0.13.12 beta
Released on October 20, 2020.
# Enhancements
- Agents now submit flow runs in order of scheduled start times - #3165
- Updating k8s tutorial docs to include instructions on how to provide access to S3 from kubernetes deployments on AWS - #3200
- Adds option to specify default values for GetItem and GetAttr tasks - #3489
- Allow disabling default storage labels for the
LocalAgent
- #3503 - Improve overall functionality of docs search, full list of changes here - #3504
- Add
LocalRun
implementation forrun_config
based flows - #3527 - Add
DockerRun
implementation forrun_config
based flows - #3537 - Raise a better error message when trying to register a flow with a schedule using custom filter functions - #3450
RenameFlowRunTask
: use defaultflow_run_id
value from context - #3548- Raise a better error message when trying to register a flow with parameters with JSON-incompatible defaults - #3549
# Task Library
- Extended
GCSUpload
task to allow uploading of bytes/gzip data - #3507 - Allow setting runtime
webook_secret
onSlackTask
and kwarg forwebhook_secret
retrieved fromPrefectSecret
task - #3522
# Fixes
- Fix
get flow-runs
anddescribe flow-runs
CLI commands querying of removedduration
field - #3517 - Fix multiprocess based timeout handler on linux - #3526
- Fix API doc generation incorrectly compiling mocked imports - #3504
- Fix multiprocessing scheduler failure while running tasks with timeouts - #3511
- Update Fargate task definition validation - #3514
- Fix bug in k8s where the resource-manager would sometimes silently crash on errors - #3521
- Add labels from
flow.storage
forrun_config
based flows - #3527 - Fix LocalAgent PYTHONPATH construction on Windows - #3551
# Deprecations
FlowRunTask
,RenameFlowRunTask
, andCancelFlowRunTask
have been renamed toStartFlowRun
,RenameFlowRun
, andCancelFlowRun
respectively - #3539
# Contributors
# 0.13.11 beta
Released on October 14, 2020.
# Features
- Allow for schedules that emit custom Flow Run labels - #3483
# Enhancements
- Use explicit exception chaining - #3306
- S3List filtering using the LastModified value - #3460
- Add Gitlab storage - #3461
- Extend module storage capabilities - #3463
- Support adding additional flow labels in
prefect register flow
- #3465 - Strict Type for default value of a Parameter - #3466
- Enable automatic script upload for file-based storage when using S3 and GCS - #3482
- Allow for passing labels to
client.create_flow_run
- #3483 - Display flow group ID in registration output URL instead of flow ID to avoid redirect in UI - #3500
- Add informative error log when local storage fails to load flow - #3475
# Task Library
- Add cancel flow run task - #3484
- Add new
BatchSubmit
task for submitting jobs to AWS batch - #3366 - Add new
AWSClientWait
task for waiting on long-running AWS jobs - #3366 - Add GetAttr task - #3481
# Fixes
- Fix default profile directory creation behavior - #3037
- Fix
DaskKubernetesEnvironment
overwriting log attributes for custom specs - #3231 - Fix default behavior for
dbt_kwargs
in the dbt task to provide an empty string - #3280 - Fix containerDefinitions environment validation - #3452
- Raise a better error when calling
flow.register()
from within aFlow
context - #3467 - Fix task cancellation on Python 3.8 to properly interrupt long blocking calls - #3474
# Contributors
- Aaron Richter
- Alessandro Lollo
- Bruno Casarotti
- Mariia Kerimova
- Max Del Giudice
- Michal Baumgartner
- Panagiotis Simakis
- Raphael Riel
- Shalika Singhal
- Zach McQuiston
- heyitskevin
# 0.13.10 beta
Released on October 6, 2020.
# Enhancements
- Add option to template task run name at runtime when using backend API - #2100
- Add
set_task_run_name
Client function - #2100 - Use 'from' to explicitly chain exceptions - #3306
- Update error message when registering flow to non-existant project - #3418
- Add
flow.run_config
, an experimental design for configuring deployed flows - #3333 - Allow python path in Local storage - #3351
- Enable agent registration for server users - #3385
- Added FROM to explicitly chain exceptions in src/prefect/utilities - #3429
# Task Library
- Add keypair auth for snowflake - #3404
- Add new
RenameFlowRunTask
for renaming a currently running flow - #3285.
# Fixes
- Fix mypy typing for
target
kwarg on base Task class - #2100 - Fix Fargate Agent not parsing cpu and memory provided as integers - #3423
- Fix MySQL Tasks breaking on opening a context - #3426
# Contributors
# 0.13.9 beta
Released on September 29, 2020.
# Features
- Allow for scheduling the same flow at the same time with multiple parameter values - #2510
# Enhancements
# Task Library
- New task for the task library to create an item in Monday - #3387
- Add option to specify
run_name
forFlowRunTask
- #3393
# Contributors
# 0.13.8 beta
Released on September 22, 2020.
# Enhancements
- Allow passing context values as JSON string from CLI - #3347
- Allow copying of directories into Docker image - #3299
- Adds schedule filters for month end or month start and specific day - #3330
- Support configuring executor on flow, not on environment - #3338
- Support configuring additional docker build commands on
Docker
storage - #3342 - Support submission retries within the k8s agent - #3344
- Expose
flow_run_name
toflow.run()
for local runs - #3364
# Task Library
- Add contributing documentation for task library - #3360
- Remove duplicate task library documentation in favor of API reference docs - #3360
# Fixes
- Fix issue with constants when copying Flows - #3319
- Fix
DockerAgent
with--show-flow-logs
to work on windows/osx (with python >= 3.8) - #3339 - Fix mypy type checking for tasks created with
prefect.task
- #3346 - Fix bug in
flow.visualize()
where no output would be generated when running withPYTHONOPTIMIZE=1
- #3352 - Fix typo in
DaskCloudProviderEnvironment
logs - #3354
# Deprecations
- Deprecate the use of the
/contrib
directory - #3360 - Deprecate importing
Databricks
andMySQL
tasks fromprefect.contrib.tasks
, should useprefect.tasks
instead - #3360
# Contributors
# 0.13.7 beta
Released on September 16, 2020.
# Enhancements
# Fixes
- Fix issue with result configuration not being respected by autogenerated tasks - #2989
- Fix issue with result templating that failed on task arguments named 'value' - #3034
- Fix issue restarting Mapped pipelines with no result- #3246
- Fix handling of Prefect Signals when Task state handlers are called - #3258
- Allow using
apply_map
under acase
orresource_manager
block - #3293 - Fix bug with interaction between
case
blocks andConstant
tasks which resulted in some tasks never skipping - #3293 - Fix bug in
DaskExecutor
where not all client timeouts could be configured via settingdistributed.comm.timeouts.connect
- #3317
# Task Library
- Adds a compression argument to both S3Upload and S3Download, allowing for compression of data upon upload and decompression of data upon download - #3259
# Contributors
# 0.13.6 beta
Released on September 9, 2020.
# Enhancements
- Adds logger to global context to remove friction on running task unit tests - #3256
- Expand FunctionTask AttributeError Message - #3248
- Add backend info to diagnostics - #3265
- Ellipsis Support for GraphQL DSL - #3268
# Task Library
- Add
DatabricksRunNow
task for running Spark jobs on Databricks - #3247 - Add GitHub
CreateIssueComment
task - #3269 - Add
S3List
task for listing keys in an S3 bucket - #3282 - Add
boto_kwargs
to AWS tasks - #3275
# Fixes
- Make identifier optional in
KubernetesAgent.replace_job_spec_yaml()
- #3251 - Change
https://localhost
tohttp://localhost
in the welcome message - #3271
# Contributors
- Ashmeet Lamba
- Ashton Sidhu
- Bas Nijholt
- Harutaka Kawamura
- James Lamb
- Max Del Giudice
- Robin Beer
- Rowan Molony
- Shunwen
# 0.13.5 beta
Released on September 1, 2020.
# Enhancements
- Begin storing the width of mapped pipelines on the parent Mapped state - #3233
- Kubernetes agent now manages lifecycle of prefect jobs in its namespace - #3158
- Move agent heartbeat to background thread - #3158
- Handles
ModuleNotFound
errors in the storage healthcheck - #3225 - Raises the
warnings.warn
stack level to 2 to reduce duplicate warning messages - #3225 - Add some extra output to the
client.register
print output for visibility - #3225 - CLI help text docstrings are now auto documented using the API documentation parser - #3225
DaskExecutor
now logs dask worker add/removal events - #3227
# Fixes
- Fix issue with passing --env-vars flag to K8s Agent Install manifest - #3239
- Fix edge case with
add_edge
method - #3230
# Deprecations
- Kubernetes resource manager is now deprecated and the functionality is moved into the Kubernetes agent - #3158
# Contributors
# 0.13.4 beta
Released on August 25, 2020.
# Enhancements
- Allow for setting path to a custom job YAML spec on the Kubernetes Agent - #3046
- Use better coupled versioning scheme for Core / Server / UI images - #3204
- Added option to mount volumes with KubernetesAgent - #1234
- Add more kwargs to State.children and State.parents for common access patterns - #3212
- Reduce size of
prefecthq/prefect
Docker image - #3215
# Task Library
- Add
DatabricksSubmitRun
task for submitting Spark jobs on Databricks - #3166
# Fixes
- Fix Apollo service error output while waiting for GraphQL service with
prefect server start
- #3150 - Fix
--api
CLI option not being respected by agent Client - #3186 - Fix state message when using targets - #3216
# Contributors
# 0.13.3 beta
Released on August 18, 2020.
# Enhancements
- Make use of
kubernetes
extra logger in theDaskKubernetesEnvironment
optional - #2988 - Make Client robust to simplejson - #3151
- Raise Warning instead of Exception during storage healthcheck when Result type is not provided - #3146
- Add
server create-tenant
for creating a tenant on the server - #3147 - Cloud logger now responds to logging level - #3179
# Task Library
- Add support for
host_config
and arbitrary keyword arguments inDocker
tasks - #3173
# Fixes
- Fix empty string
imagePullSecrets
issue on AKS by removing if not set - #3142 - Fix querying for cached states with no
cache_key
- #3168 - Fix access to
core_version
in Agent'sget_flow_run_command()
- #3177
# Breaking Changes
- DaskKubernetesEnvironment no longer logs Kubernetes errors by default - #2988
- Logging level in Cloud now defaults to INFO - #3179
# Contributors
# 0.13.2 beta
Released on August 11, 2020.
# Features
# Enhancements
- Agents set flow run execution command based on flow's core version - #3113
- Clean up extra labels on jobs created by Kubernetes agent - #3129
# Task Library
- Return
LoadJob
object inBigQueryLoad
tasks - #3086
# Fixes
- Fix bug with
LocalDaskExecutor('processes')
that allowed tasks to be run multiple times in certain cases - #3127 - Add toggle to bypass bug in
slack_notifier
that attempted to connect to backend even if the backend didn't exist - #3136
# Contributors
# 0.13.1 beta
Released on August 6, 2020.
# Fixes
- Fix issue with 0.13.0 agents not able to run Flows registered with older Core versions - #3111
# 0.13.0 beta
Released on August 6, 2020.
# Features
# Enhancements
- Only supply versions when setting
SUBMITTED
andRUNNING
states - #2730 - Gracefully recover from version lock errors - #2731
- Add
--ui-version
server start CLI option to run a specific UI image - #3087 - Agent querying of flow runs now passes active tenant ID - #3087
- Ignore calls to flow.register when parsing a flow using file based storage - #3051
# Task Library
- Allow idempotency keys in
FlowRunTask
when using server backend - #3006 - Require project name in
FlowRunTask
when using server backend - #3006
# Fixes
- Fix use of absolute path in Docker storage on Windows - #3044
- Determine if checkpointing is enabled from config set in the flow-runner process - #3085
- Fix
--no-ui
server start CLI option still attempting to pull UI image - #3087
# Deprecations
- Deprecate
execute cloud-flow
CLI command in favor ofexecute flow-run
- #3087 - Deprecate
run server/cloud
CLI commands in favor ofrun flow
- #3087
# Breaking Changes
- Move server and UI code out into separate repositories - #3087
- Project names are now required when managing flows with the core server - #3087
# Contributors
# 0.12.6 beta
Released on July 28, 2020.
# Features
- Add
flatten
operator for unnesting and flat-maps - #2898
# Enhancements
- Add retry_on_api_error flag to client methods - #3012
- Add
reg_allow_list
option for Docker Agent - #3026 - Update FargateTaskEnvironment to throw if task definition is inconsistent with existing task definition - #3031
# Fixes
- Cleanup to ShellTask to close open stdout file which was observable in some cases - #3002
- Fix check of flow existence in storage object
get_flow
to only occur when provided - #3027 - Use fullname and tag when Docker Storage determines if build was successful - #3029
- Prevent duplicated agent labels - #3029
# Deprecations
prefect.utilities.tasks.unmapped
moved toprefect.utilities.edges.unmapped
- #2898
# Breaking Changes
- Remove
dbt
extra from dependencies - #3018
# Contributors
# version=0.12.5 beta
Released on July 21, 2020.
# Features
- Add
resource_manager
api for cleaner setup/cleanup of temporary resources - #2913
# Enhancements
- Add
new_flow_context
to FlowRunTask for configurable context - #2941 - All storage types now support file-based storage - #2944
- Turn work stealing ON by default on Dask K8s environment - #2973
- Send regular heartbeats while waiting to retry / dequeue - #2977
- Cached states now validate based on
hashed_inputs
for more efficient storage - #2984 - Simplify creation of optional parameters with default of
None
- #2995
# Task Library
# Fixes
- Don't send idempotency key when running against a local backend - #3001
- Fix bug in
DaskExecutor
when running with external cluster where dask clients could potentially be leaked - #3009
# Deprecations
- All states have deprecated the usage of
cached_inputs
- #2984
# Breaking Changes
- Remove password from Postgres tasks' initialization methods for security - #1345
# Contributors
# 0.12.4 beta
Released on July 14, 2020.
# Enhancements
- Improve output formatting of
prefect describe
CLI - #2934 - Add new
wait
kwarg to Flow Run Task for reflecting the flow run state in the task - #2935 - Separate build-time and run-time job spec details in KubernetsJobEnvironment - #2950
# Task Library
- Implement RunNamespacedJob task for Kubernetes - #2916
- Add
log_stderr
option toShellTask
andDbtShellTask
for logging the full output from stderr - #2961
# Fixes
- Ensure
is_serializable
always uses same executable for subprocess. - #1262 - Fix issue with Mapped tasks not always reloading child state results on reruns - #2656
- Fix
FargateTaskEnvironment
attempting to retrieve authorization token when not present - #2940 - Fix issue with Metastates compounding - #2965
# Contributors
# 0.12.3 beta
Released on July 8, 2020.
# Enhancements
- Update
flow.slugs
duringflow.replace
- #2919 flow.update
accepts the optional kwargmerge_parameters
that allows flows to be updated with commonParameters
- #2501- Added poke handler to notify agent process of available flow runs - #2914
- Add
Cancelling
state for indicating a flow-run that is being cancelled, but may still have tasks running - #2923
# Task Library
- Add
ReadAirtableRow
task - #2843 - Add
container_name
kwarg toCreateContainer
Docker task - #2904 - Adds an
extra_docker_kwargs
argument toCreateContainer
Docker task - #2915
# Fixes
- Fix issue with short-interval IntervalClocks that had a start_date far in the past - #2906
- When terminating early, executors ensure all pending work is cancelled/completed before returning, ensuring no lingering background processing - #2920
# Contributors
# 0.12.2 beta
Released on June 30, 2020.
# Features
- Add
apply_map
, a function to simplify creating complex mapped pipelines - #2846
# Enhancements
- Make storage location inside Docker storage configurable - #2865
- Send heartbeats on each iteration of the Cloud task runner's retry loop - #2893
# Task Library
- Add option to BigQueryTask to return query as dataframe - #2862
# Server
- None
# Fixes
- Add more context keys when running locally so that templating is consistent between local and Cloud runs - #2662
- Fix Fargate agent not parsing string provided containerDefinitions - #2875
- Fix Fargate agent providing empty parameters if not set - #2878
- Fix issue with Queued task runs flooding agents with work - #2884
- Add missing
prefect register flow
to CLI help text - #2895
# Deprecations
- None
# Breaking Changes
- None
# Contributors
# 0.12.1 beta
Released on June 25, 2020.
# Features
- Task slugs are now stable across rebuilds of the same Flow - #2531
- Support configuring executors for
LocalEnvironment
,KubernetesJobEnvironment
, andFargateTaskEnvironment
- #2805 - Flows can now be stored and executed using file-based storage - #2840
# Enhancements
- Add option to set
repositoryCredentials
on Fargate AgentcontainerDefinitions
- #2822 - Update GraphQL endpoint to
/graphql
- #2669 - Allow Cloud Flow Runners to interact properly with Queued runs - #2741
- Add
Result
serializers - #2755 - Simplify
DaskExecutor
internals - #2817 - Set task names in
LocalDaskExecutor
- #2819 - Flows registered without an image set will default to
all_extras
- #2828 - Improve error message when sending unauthorized requests to Cloud - #2810
- Forward state change status back to core - #2839
- Add GitHub storage for storing flows as files in a GitHub repo - #2840
- Add
prefect register flow
CLI command for registering flows from files - #2840 - Add default
GITHUB_ACCESS_TOKEN
secret - #2840 - Create utility function for getting Kubernetes client - #2845
# Task Library
- Adds a MySQL task using pymysql driver - #2124
- Add some tasks for working with Google Sheets - #2614
- Add support for HTML content in the EmailTask - #2811
# Server
- Failing to set a state raises errors more aggressively - #2708
# Fixes
- Fix
all_extras
tag not being set during CI job to build image - #2801 - Quiet no candidate Cached states were valid debug logging - #2815
- Fix
LocalEnvironment
execute function's use of the flow object - #2804 - Properly set task names when using
DaskExecutor
- #2814 - Fix the
LocalDaskExecutor
to only compute tasks once, not multiple times - #2819 - Generate key names for mapped tasks that work better with Dask's dashboard - #2831
- Fix FlowRunTask when running against locally deployed Server - #2832
- Make sure image from Docker storage is always used with KubernetesJobEnvironment - #2838
- Change Environment.run_flow() to prefer executor from flow's environment - #2849
# Deprecations
- Deprecate
RemoteEnvironment
in favor ofLocalEnvironment
- #2805 - Deprecate
RemoteDaskEnvironment
in favor ofLocalEnvironment
with aDaskExecutor
- #2805 - Deprecate
executor_kwargs
inKubernetesJobEnvironment
andFargateTaskEnvironment
in favor ofexecutor
- #2805
# Breaking Changes
- Remove previously deprecated
SynchronousExecutor
- #2826
# Contributors
# 0.12.0 beta
Released on June 17, 2020.
# Features
- Depth First Execution with Mapping on Dask - #2646
- Support use of cloud storage with containerized environments - #2517,#2796
# Enhancements
- Add flag to include hostname on local storage - #2653
- Add option to set
image_pull_secret
directly onDaskKubernetesEnvironment
- #2657 - Allow for custom callables for Result locations - #2577
- Ensure all Parameter values, included non-required defaults, are present in context - #2698
- Use absolute path for
LocalResult
location for disambiguation - #2698 - Retry client requests when receiving an
API_ERROR
code in the response - #2705 - Reduce size of serialized tasks when running on Dask - #2707
- Extend run state signatures for future development - #2718
- Update set_flow_run_state for future meta state use - #2725
- Add an optional
flow
argument tomerge
to support using it when not inside a flow context - #2727 - Add option to set service account name on Prefect jobs created by Kubernetes agent - #2547
- Add option to set imagePullPolicy on Prefect jobs created by Kubernetes agent - #2721
- Add option to set API url on agent start CLI command - #2633
- Add CI step to build
prefecthq/prefect:all_extras
Docker image for bundling all Prefect dependencies - #2745 - Move
Parameter
to a standalone module - #2758 - Validate Cached states based on hashed inputs - #2763
- Add
validate_configuration
utility to Fargate Agent for verifying it can manage tasks properly - #2768 - Add option to specify task targets as callables - #2769
- Improve
State.__repr__
when there is no message - #2773 - Add support for db argument at run time in the SQLiteQuery and SQLiteScript - #2782
- Add support for mapped argument in control flows - #2784
- Use pagination in kubernetes resource manager to reduce memory usage - #2794
# Task Library
- Adds a task to expose Great Expectations checkpoints as a node in a Prefect pipeline - #2489
# Server
- None
# Fixes
- Fix flow.visualize cleanup of source files when using
filename
- #2726 - Fix
S3Result
handling of AWS credentials provided through kwargs - #2747 - Fix
DaskKubernetesEnvironment
requiring that anenv
block is set when using custom specs - #2657 - Fix
PostgresExecute
task auto commit when commit is set toFalse
- #2658 - Remove need for
{filename}
in mapped templates - #2640 - Fix issue with Results erroring out on multi-level mapped pipelines - #2716
- Fix issue with dask resource tags not being respected - #2735
- Ensure state deserialization works even when another StateSchema exists - #2738
- Remove implicit payload size restriction from Apollo - #2764
- Fix issue with declared storage secrets in K8s job environment and Dask K8s environment - #2780
- Fix context handling for Cloud when working with in-process retries - #2783
# Deprecations
- Accessing
prefect.core.task.Parameter
is deprecated in favor ofprefect.core.parameter.Parameter
- #2758
# Breaking Changes
- Environment
setup
andexecute
function signatures now accept Flow objects - #2796 create_flow_run_job
logic has been moved intoexecute
forDaskKubernetesEnvironment
andKubernetesJobEnvironment
- #2796
# Contributors
# 0.11.5 beta
Released on June 2, 2020.
# Features
- None
# Enhancements
- Allow for manual approval of locally Paused tasks - #2693
- Task instances define a
__signature__
attribute, for improved introspection and tab-completion - #2602 - Tasks created with
@task
forward the wrapped function's docstring - #2602 - Support creating temporary dask clusters from within a
DaskExecutor
- #2667 - Add option for setting any build kwargs on Docker storage - #2668
- Add flow run ID option to
get logs
CLI command - #2671 - Add ID to output of
get
command forflows
andflow-runs
- #2671
# Task Library
- None
# Server
- None
# Fixes
- Fix issue with Google imports being tied together - #2661
- Don't warn about unused tasks defined inline and copied - #2677
- Remove unnecessary volume mount from dev infrastructure Docker compose - #2676
- Fix issue with instantiating LocalResult on Windows with dir from other drive - #2683
- Fix invalid IP address error when running
server start
on Ubuntu using rootless Docker - #2691
# Deprecations
- Deprecate
local_processes
and**kwargs
arguments forDaskExecutor
- #2667 - Deprecate
address='local'
forDaskExecutor
- #2667
# Breaking Changes
- None
# Contributors
# 0.11.4 beta
Released on May 27, 2020.
# Fixes
- Revert GraphQL endpoint change - #2660
# 0.11.3 beta
Released on May 27, 2020.
# Features
- None
# Enhancements
- Add option to set volumes on
server start
CLI command - #2560 - Add
case
to top-level namespace - #2609 - Use host IP for
hostname
label in cases whereLocalAgent
is in container using host network - #2618 - Add option to set TLS configuration on client created by Docker storage - #2626
- The
start_time
of aPaused
state defaults toNone
- #2617 - Raise more informative error when Cloud Secret doesn't exist - #2620
- Update GraphQL endpoint to
/graphql
- #2651
# Task Library
- None
# Fixes
- Kubernetes agent resource manager is more strict about what resources it manages - #2641
- Fix error when adding
Parameter
to flow undercase
statement - #2608 - Fix
S3Result
attempting to load data when checking existence - #2623
# Deprecations
- Deprecate
private_registry
anddocker_secret
options onDaskKubernetesEnvironment
- #2630
# Breaking Changes
- Kubernetes labels associated with Prefect flow runs now have a
prefect.io/
prefix (e.g.prefect.io/identifier
) - #2641
# Contributors
# 0.11.2 beta
Released on May 19, 2020.
# Features
- None
# Enhancements
- Allow log configuration in Fargate Agent - #2589
- Reuse
prefect.context
for openingFlow
contexts - #2581 - Show a warning when tasks are created in a flow context but not added to a flow - #2584
# Server
- Add API healthcheck tile to the UI - #2395
# Task Library
- None
# Fixes
- Fix type for Dask Security in RemoteDaskEnvironment - #2571
- Fix issue with
log_stdout
not correctly storing returned data on the task run state - #2585 - Ensure result locations are updated from targets when copying tasks with
task_args
- #2590 - Fix
S3Result
exists function handling ofNoSuchKey
error - #2585 - Fix confusing language in Telemetry documentation - #2593
- Fix
LocalAgent
not registering with Cloud using default labels - #2587 - Fix flow's
run_agent
function passing aset
of labels to Agent instead of alist
- #2600
# Deprecations
- None
# Breaking Changes
- None
# Contributors
# 0.11.1 beta
Released on May 15, 2020.
# Fixes
- Fix duplicate agent label literal eval parsing - #2569
# 0.11.0 beta
Released on May 14, 2020.
# Features
# Enhancements
- Allow slack_task to accept a dictionary for the message parameter to build a specially-structured JSON Block - #2541
- Support using
case
for control flow with the imperative api - #2546 flow.visualize
is now able to accept aformat
argument to specify the output file type - #2447- Docker storage now writes flows to
/opt
dir to remove need for root permissions - #2025 - Add option to set secrets on Storage objects - #2507
- Add reserved default Secret names and formats for working with cloud platforms - #2507
- Add unique naming option to the jobs created by the
KubernetesJobEnvironment
- #2553 - Use
ast.literal_eval
for configuration values - #2536 - Prevent local cycles even if flow validation is deferred - #2565
# Server
- Add "cancellation-lite" semantic by preventing task runs from running if the flow run isn't running - #2535
- Add minimal telemetry to Prefect Server - #2467
# Task Library
- Add tasks to create issues for Jira and Jira Service Desk #2431
- Add
DbtShellTask
, an extension of ShellTask for working with data build tool (dbt) - #2526 - Add
prefect.tasks.gcp.bigquery.BigQueryLoadFile
- #2423
# Fixes
- Fix bug in Kubernetes agent
deployment.yaml
with a misconfigured liveness probe - #2519 - Fix checkpointing feature not being able to be disabled when using server backend - #2438
# Deprecations
- Result Handlers are now deprecated in favor of the new Result interface - #2507
# Breaking Changes
- Allow for setting docker daemon at build time using DOCKER_HOST env var to override base_url in docker storage - #2482
- Ensure all calls to
flow.run()
use the same execution logic - #1994 - Moved
prefect.tasks.cloud
toprefect.tasks.prefect
- #2404 - Trigger signature now accepts a dictionary of
[Edge, State]
to allow for more customizable trigger behavior - #2298 - Remove all uses of
credentials_secret
from task library in favor ofPrefectSecret
tasks - #2507 - Remove
Bytes
andMemory
storage objects - #2507
# Contributors
# 0.10.7 beta
Released on May 6, 2020.
# Features
- None
# Enhancements
- Agents now support an optional HTTP health check, for use by their backing orchestration layer (e.g. k8s, docker, supervisord, ...) - #2406
- Sets dask scheduler default to "threads" on LocalDaskExecutor to provide parallelism - #2494
- Enhance agent verbose logs to include provided kwargs at start - #2486
- Add
no_cloud_logs
option to all Agent classes for an easier way to disable sending logs to backend - #2484 - Add option to set flow run environment variables on Kubernetes agent install - #2424
# Task Library
- Add new
case
control-flow construct, for nicer management of conditional tasks - #2443
# Fixes
- Give a better error for non-serializable callables when registering with cloud/server - #2491
- Fix runners retrieving invalid
context.caches
on runs started directly from a flow runner - #2403
# Deprecations
- None
# Breaking Changes
- Remove the Nomad agent - #2492
# Contributors
- None
# 0.10.6 beta
Released on May 5, 2020.
# Features
- Add DaskCloudProviderEnvironment to dynamically launch Dask clusters, e.g. on AWS Fargate - #2360
# Enhancements
- Add
botocore_config
option to Fargate agent for setting botocore configuration when interacting with boto3 client - #2170 - Don't create a
None
task for a null condition when usingifelse
- #2449 - Add support for EC2 launch type in Fargate Agent and
FargateTaskEnvironment
- #2421 - Add
flow_id
to context for Flow runs - #2461 - Allow users to inject custom context variables into their logger formats - #2462
- Add option to set backend on
agent install
CLI command - #2478
# Task Library
- None
# Fixes
- Fix
start_server.sh
script when an env var is undefined - #2450 - Fix
server start
CLI command not respectingversion
kwarg on tagged releases - #2435 - Fix issue with non-JSON serializable args being used to format log messages preventing them from shipping to Cloud - #2407
- Fix issue where ordered Prefect collections use lexical sorting, not numerical sorting, which can result in unexpected ordering - #2452
- Fix issue where Resource Manager was failing due to non-JSON timestamp in log writing - #2474
- Fix periodic error in local agent process management loop - #2419
# Deprecations
- None
# Breaking Changes
- None
# Contributors
# 0.10.5 beta
Released on Apr 28, 2020.
# Features
- None
# Enhancements
- Added serializer for
RemoteDaskEnvironment
- #2369 server start
CLI command now defaults to image build based on current Prefect installation version - #2375- Add option to set
executor_kwargs
onKubernetesJobEnvironment
andFargateTaskEnvironment
- #2258 - Add map index to task logs for mapped task runs - #2402
- Agents can now register themselves with Cloud for better management - #2312
- Adding support for
environment
,secrets
, andmountPoints
via configurablecontainerDefinitions
to the Fargate Agent - #2397 - Add flag for disabling Docker agent interface check on Linux - #2361
# Task Library
- Add Pushbullet notification task to send notifications to mobile - #2366
- Add support for Docker volumes and filtering in
prefect.tasks.docker
- #2384
# Fixes
- Fix Docker storage path issue when registering flows on Windows machines - #2332
- Fix issue with refreshing Prefect Cloud tokens - #2409
- Resolve invalid escape sequence deprecation warnings - #2414
# Deprecations
- None
# Breaking Changes
- None
# Contributors
# 0.10.4 beta
Released on Apr 21, 2020.
# Enhancements
- Agent connection step shows which endpoint it is connected to and checks API connectivity - #2372
# Breaking Changes
- Revert changes to
ifelse
&switch
(added in #2310), removing implicit creation ofmerge
tasks - #2379
# 0.10.3 beta
Released on Apr 21, 2020.
# Features
- None
# Enhancements
- Allow GraphQL endpoint configuration via
config.toml
for remote deployments of the UI - #2338 - Add option to connect containers created by Docker agent to an existing Docker network - #2334
- Expose
datefmt
as a configurable logging option in Prefect configuration - #2340 - The Docker agent configures containers to auto-remove on completion - #2347
- Use YAML's safe load and dump commands for the
server start
CLI command - #2352 - New
RemoteDaskEnvironment
specifically for running Flows on an existing Dask cluster - #2367
# Task Library
- None
# Fixes
- Fix
auth create-token
CLI command specifying deprecatedrole
instead ofscope
- #2336 - Fix local schedules not continuing to schedule on errors outside of runner's control - #2133
- Fix
get_latest_cached_states
pulling incorrect upstream cached states when using Core server as the backend - #2343
# Deprecations
- None
# Breaking Changes
- None
# Contributors
# 0.10.2 beta
Released on Apr 14, 2020.
# Features
- None
# Enhancements
- Task logical operators (e.g.
And
,Or
, ...) no longer implicitly cast tobool
- #2303 - Allow for dynamically changing secret names at runtime - #2302
- Update
ifelse
andswitch
to return tasks representing the output of the run branch - #2310
# Task Library
- Rename the base secret tasks for clarity - #2302
# Fixes
- Fix possible subprocess deadlocks when sending stdout to
subprocess.PIPE
- #2293, #2295 - Fix issue with Flow registration to non-standard Cloud backends - #2292
- Fix issue with registering Flows with Server that have required scheduled Parameters - #2296
- Fix interpolation of config for dev services CLI for Apollo - #2299
- Fix pytest Cloud and Core server backend fixtures - #2319
- Fix
AzureResultHandler
choosing an empty Secret over provided connection string - #2316 - Fix containers created by Docker agent not being able to reach out to host API - #2324
# Deprecations
- None
# Breaking Changes
- Remove
env_var
initialization fromEnvVarSecret
in favor ofname
- #2302
# Contributors
# 0.10.1 beta
Released on Apr 7, 2020.
# Features
# Enhancements
- Add flags to
prefect server start
for disabling service port mapping - #2228 - Add options to
prefect server start
for mapping to host ports - #2228 - Return
flow_run_id
from CLIrun
methods for programmatic use - #2242 - Add JSON output option to
describe
CLI commands - #1813 - Add ConstantResult for eventually replacing ConstantResultHandler - #2145
- Add new
diagnostics
mode for timing requests made to Cloud - #2283
# Task Library
- Make
project_name
optional forFlowRunTask
to allow for use with Prefect Core's server - #2266 - Adds
prefect.tasks.docker.container.RemoveContainer
# Fixes
- Fix
S3ResultHandler
safe retrieval of_client
attribute - #2232 - Change default log
timestamp
value in database to be identical to other tables instead of a hard coded value - #2230
# Deprecations
- None
# Breaking Changes
- None
# Contributors
# 0.10.0 beta
Released on Mar 29, 2020.
# Features
- Open source database backend, GraphQL API and UI - #2218
- Add
prefect server start
CLI command for spinning up database and UI - #2214
# Enhancements
- Add ValidationFailed state and signal in anticipation of validating task outputs - #2143
- Add max polling option to all agents - #2037
- Add GCSResult type #2141
- Add Result.validate method that runs validator functions initialized on Result #2144
- Convert all GraphQL calls to have consistent casing - #2185 #2198
- Add
prefect backend
CLI command for switching between Prefect Core server and Prefect Cloud - #2203 - Add
prefect run server
CLI command for starting flow runs without use of project name - #2203 - Make
project_name
optional during flow registration to support Prefect Core's server - #2203 - Send flow run and task run heartbeat at beginning of run time - #2203
# Task Library
- None
# Fixes
- Fix issue with heartbeat failing if any Cloud config var is not present - #2190
- Fix issue where
run cloud
CLI command would pull final state before last batch of logs - #2192 - Fix issue where the
S3ResultHandler
would attempt to access uninitialized attribute - #2204
# Deprecations
- None
# Breaking Changes
- Drop support for Python 3.5 - #2191
- Remove
Client.write_run_log
- #2184 - Remove
Client.deploy
andflow.deploy
- #2183
# Contributors
- None
# 0.9.8
Released on Mar 18, 2020.
# Features
- None
# Enhancements
- Update Cloud config name for heartbeat settings - #2081
- Add examples to Interactive API Docs - #2122
- Allow users to skip Docker healthchecks - #2150
- Add exists, read, and write interfaces to Result #2139
- Add Cloud UI links to Slack Notifications - #2112
# Task Library
- None
# Fixes
- Fix S3ResultHandler use of a new boto3 session per thread - #2108
- Fix issue with stateful function reference deserialization logic mutating state - #2159
- Fix issue with
DateClock
serializer - #2166 - Fix issue with scheduling required parameters - #2166
# Deprecations
- Deprecate cache_* and result_handler options on Task and Flow objects #2140
# Breaking Changes
- None
# Contributors
# 0.9.7 beta
Released on Mar 4, 2020.
# Fixes
- Change
task.log_stdout
retrieval from task runner togetattr
in order to preserve running flows of older 0.9.x versions - #2120
# 0.9.6 beta
Released on Mar 4, 2020.
# Features
- Add new diagnostics utility to assist in troubleshooting issues - #2062
- Add a jira_notification state handler to create jira tickets for failed tasks or flows - #1861
- Add support for Python 3.8 - #2080
# Enhancements
- Add PIN 15 (skip refactor) - #2070
- Update docs and docstrings related to Result Handlers - #1792
- Add volume option to Docker Agent - #2013
DaskKubernetesEnvironment
now elevates important autoscaling logs as well as possible Kubernetes issues - #2089- Add optional
scheduler_logs
kwarg to theDaskKubernetesEnvironment
- #2089 - Add ERROR log if heartbeat process dies - #2097
- Enable stdout logging from inside a task with the kwarg
log_stdout=True
- #2092 - Direct links to Cloud flows and flow runs now shown on creation time - #2109
- Update docs related to using Context - #2077
# Task Library
- Fix expanding of
V1DeleteOptions
kwargs for Kubernetes tasks - #2083
# Fixes
- Fix
extra_loggers
config variable not being able to be set via environment variable - #2089 - Fix environments not passing down their
extra_loggers
to any created infrastructure - #2089 - Don't mutate data when serializing or deserializing - #2098
# Deprecations
- None
# Breaking Changes
- None
# Contributors
# 0.9.5 beta
Released on Feb 21, 2020.
# Features
- None
# Enhancements
- Better exception for unsubscriptable mapping arguments - #1821
- Upload package to PyPI on tag push to master - #2030
- Add DaskGateway tip to docs - #1959
- Improve package import time - #2046
# Task Library
- Fix
V1DeleteOptions
call for Kubernetes tasks - #2050 - Add kwargs to
V1DeleteOptions
for Kubernetes tasks - #2051
# Fixes
- Ensure microseconds are respected on
start_date
provided to CronClock - #2031 - Fix duplicate Client connections when using
--logs
flag fromrun cloud
CLI command - #2056
# Deprecations
- None
# Breaking Changes
- None
# Contributors
# 0.9.4 beta
Released on Feb 14, 2020.
# Features
- None
# Enhancements
- Add incremental tutorial - #1953
- Improve error handling for unsupported callables - #1993
- Accept additional
boto3
client parameters in S3 storage - #2000 - Add optional
version_group_id
kwarg tocreate_flow_run
for a stable API for flow runs - #1987 - Add
extra_loggers
logging configuration for non-Prefect logs in stdout and cloud - #2010
# Task Library
- None
# Fixes
- Ensure
ifelse
casts its condition tobool
prior to evaluation - #1991 - Do not perform
ast.literal_eval
on cpu and memory task_definition kwargs for Fargate Agent - #2010 - Fix new agent processing with Threadpool causing problem for Fargate Agent with task revisions enabled - #2022
# Deprecations
- None
# Breaking Changes
- Remove Airflow Tasks - #1992
# Contributors
# 0.9.3 beta
Released on Feb 05, 2020.
# Features
- None
# Enhancements
- Improve heartbeat functionality to be robust across platforms - #1973
- Run storage healthchecks on other options besides Docker - 1963
- Cloud logger now attempts to elevate logger errors to flow run logs - #1961
- Attach Flow and Task attributes to LogRecords - #1938
# Task Library
- None
# Fixes
- Fix uncaught Fargate Agent kwarg parse SyntaxError from
literal_eval
- #1968 - Fix FargateTaskEnvironment passing empty auth token to run task - #1976
- Fix imagePullSecrets not being automatically passed to jobs created by Kubernetes Agent - #1982
# Deprecations
- None
# Breaking Changes
- Remove cancellation hooks - #1973
# Contributors
- None
# 0.9.2 beta
Released on Jan 30, 2020.
# Features
- Allow for parameter defaults to vary based on clock - #1946
# Enhancements
- More graceful handling of Agents competing for work - #1956
# Task Library
- None
# Fixes
- Eliminated possible duplicate flow run issue in all agents - #1956
# Deprecations
- None
# Breaking Changes
- None
# Contributors
- None
# 0.9.1 beta
Released on Jan 24, 2020.
# Features
- None
# Enhancements
- Docker daemon reconnect attempts + exit on heartbeat failure -#1918
- More responsive agent shutdown - #1921
- Background all agent flow deployment attempts - #1928
- Add show_flow_logs to Docker agent #1929
- Add per-task checkpointing opt-out - #1933
- The Task 'checkpoint' kwarg will no longer be deprecated to allow opt-out - #1933
# Task Library
- None
# Fixes
- Fix the Fargate Agent not parsing kwargs as literals - #1926
- Fix issue with result handler default persisting from initialization - #1936
# Deprecations
- None
# Breaking Changes
- None
# Contributors
- None
# 0.9.0 beta
Released on Jan 15, 2020.
# Features
- Added the ability to leverage native ECS task definition revisions for flow versions in Fargate agent - #1870
- Added the ability to pull in kwargs per flow version from S3 on flow submission in Fargate agent - #1870
- Add sensible default result handlers to non-Docker storage options - #1888
# Enhancements
- Allow for task looping to beyond Python's maximum recursion depth - #1862
- Prevent duplication of stdout logs from multiple instantiated agents - #1866
- Allow intervals less than 60 seconds in
IntervalClock
s - #1880 - Introduce new
Secret.exists
method for checking whether a Secret is available - #1882 - Introduce new
-e
CLI options on agent start commands to allow passing environment variables to flow runs - #1878 - Stop persisting
None
when calling result handlers - #1894 - Change Cancelled state to indicate Finished instead of Failed - #1903
- All States now store
cached_inputs
for easier recovery from failure - #1898 - Always checkpoint tasks which have result handlers - #1898
# Task Library
- Remove implicit requirement that Google Tasks use Prefect Cloud Secrets - #1882
# Fixes
- Enforce provision of
max_retries
if specifyingretry_delay
for aTask
- #1875 - Fix issue with reduce tasks in
flow.visualize()
- #1793
# Deprecations
- The checkpointing kwarg will be removed from Tasks as it is now a default behavior - #1898
# Breaking Changes
- Remove default value for
aws_credentials_secret
on all S3 hooks - #1886 - Remove
config.engine.result_handler
section of Prefect config - #1888 - Remove default value for
credentials_secret
onGCSResultHandler
- #1888 - Remove default value for
azure_credentials_secret
onAzureResultHandler
- #1888
# Contributors
# 0.8.1 beta
Released on Dec 17, 2019.
# Features
- None
# Enhancements
- Enhanced treatment of nested and ordered constant values - #1829
- Add
on_datetime
,on_date
, andat_time
filters - #1837 - Add
--latest
flag for Kubernetes Agent install CLI command - #1842 - Add
--no-cloud-logs
flag for all agents to optionally opt-out of logs being sent to Prefect Cloud - #1843 - Agents mark Flow Runs as
Failed
if a deployment error occurs - #1848 Submitted
states from Agents include deployment identifier information - #1848- Update heartbeats to respect Cloud flow settings - #1851
- Add flow run name to
prefect.context
- #1855 - Add
--namespace
option for Kubernetes Agent start CLI command - #1859 - Add Prefect job resource configuration for Kubernetes Agent - #1859
# Task Library
- Add task for scheduling a flow run - #1871
# Fixes
- Fix Agent deployment errors interrupting full list of found Flow Runs - #1848
- Fix issue with a single bad log preventing all logs from being sent to Cloud - #1845
- Fix Kubernetes Agent passing empty default namespace - #1839
# Deprecations
- None
# Breaking Changes
- None
# Contributors
- None
# 0.8.0 beta
Released on Dec 11, 2019.
# Features
- Added new Local Agent to run Flows from Local Storage, S3 Storage, and GCS Storage - #1819
- Added Azure Blob Storage for Flows - #1831
- Added GCS Storage for Flows - #1809
- Added S3 Storage for Flows - #1753
# Enhancements
- Add
--rbac
flag toprefect agent install
for Kubernetes Agent - #1822 - Add
prefect agent install
option to outputsupervisord.conf
file for Local Agent - #1819 - Add convenience
parents()
andchildren()
classmethods to all State objects for navigating the hierarchy - #1784 - Add new
not_all_skipped
trigger and set it as the default for merge tasks - #1768
# Task Library
- Azure Blob tasks now use newer
BlockBlobService
with connection string authentication - #1831
# Fixes
- Fix issue with
flow.visualize()
for mapped tasks which are skipped - #1765 - Fix issue with timeouts only being softly enforced - #1145, #1686
- Log agent errors using
write_run_logs
instead of the deprecatedwrite_run_log
- #1791 - Fix issue with
flow.update()
not transferring constants - #1785
# Deprecations
flow.deploy
is deprecated in favor offlow.register
- #1819
# Breaking Changes
- Default Flow storage is now
Local
instead ofDocker
- #1819 - Docker based
LocalAgent
is renamedDockerAgent
- #1819 prefect agent start
now defaults to newLocalAgent
- #1819
# Contributors
- None
# 0.7.3 beta
Released on Nov 26, 2019.
# Features
- Add graceful cancellation hooks to Flow and Task runners - #1758
# Enhancements
- Add option to specify a run name for
cloud run
CLI command - #1756 - Add
work_stealing
option toDaskKubernetesEnvironment
- #1760 - Improve heartbeat thread management - #1770
- Add unique scheduler Job name to
DaskKubernetesEnvironment
- #1772 - Add informative error when trying to map with the
LocalDaskExecutor
using processes - #1777
# Task Library
- None
# Fixes
- Fix issue with heartbeat thread deadlocking dask execution when using a
worker_client
- #1750 - Fix issue with Environments not calling
run_flow
on Environment stored on Flow object - #1752 - Fix issue with Docker build context when providing custom docker files - #1762
# Deprecations
- None
# Breaking Changes
- None
# Contributors
- None
# 0.7.2 beta
Released on Nov 15, 2019.
# Features
- Allow users to provide a custom version group ID for controlling Cloud versioning - #1665
- Stop autogenerating constant tasks - #1730
# Enhancements
- Raise an informative error when context objects are pickled - #1710
- Add an option to pass in
run_name
to a flow run to override the auto-generated names when callingcreate_flow_run
#1661 - Add informative logs in the event that a heartbeat thread dies - #1721
- Loosen Job spec requirements for
KubernetesJobEnvironment
- #1713 - Loosen
containerDefinitions
requirements forFargateTaskEnvironment
- #1713 - Local Docker agent proactively fails flow runs if image cannot be pulled - #1395
- Add graceful keyboard interrupt shutdown for all agents - #1731
agent start
CLI command now allows for Agent kwargs - #1737- Add users to specify a custom Dockerfile for Docker storage - #1738
- Expose
labels
kwarg inflow.deploy
for convenient labeling of Flows - #1742
# Task Library
- None
# Fixes
FargateTaskEnvironment
now uses providedfamily
for task definition naming - #1713- Fix executor initialization missing
self
inKubernetesJobEnvironment
- #1713 - FixÂ
identifier_label
 not being generated on each run for Kubernetes based environments - #1718 - Fix issue where users could not override their user config path when deploying Docker to Cloud - #1719
- Respect order of inputs in merge - #1736
# Deprecations
- None
# Breaking Changes
- None
# Contributors
# 0.7.1 beta
Released on Nov 5, 2019
# Features
- None
# Enhancements
- Add a
save
/load
interface to Flows - #1685, #1695 - Add option to specify
aws_session_token
for theFargateTaskEnvironment
- #1688 - Add
EnvVarSecrets
for loading sensitive information from environment variables - #1683 - Add an informative version header to all Cloud client requests - #1690
- Auto-label Flow environments when using Local storage - #1696
- Batch upload logs to Cloud in a background thread for improved performance - #1691
- Include agent labels within each flow's configuration environment - #1671
# Task Library
- None
# Fixes
- Fix Fargate Agent access defaults and environment variable support - #1687
- Removed default python version for docker builds - #1705
- Attempt to install prefect in any docker image (if it is not already installed) - #1704
- Kubernetes Agent deployment yaml now respects new
prefecthq/prefect
image tagging convention - #1707
# Deprecations
- None
# Breaking Changes
- None
# Contributors
- None
# 0.7.0 To Affinity and Beyond beta
Released October 29, 2019
# Features
- Flow Affinity: Environments and Agents now support labeling for execution specification - #1651
- Add new Secret Tasks for a pluggable and reusable Secrets API - #1346, #1587
# Enhancements
- Add the ability to delete task tag limits using the client - #1622
- Adds an "Ask for help" button with a link to the prefect.io support page - #1637
- Reduces the size of the
prefecthq/prefect
Docker image by ~400MB, which is now the base Docker image used in Flows - #1648 - Add a new healthcheck for environment dependencies - #1653
- Add default 30 second timeout to Client requests - #1672
# Task Library
- Add new Secret Tasks for a pluggable and reusable Secrets API - #1346, #1587
- Add support for directly passing credentials to task library tasks, instead of passing secret names - #1667
# Fixes
- Fix defaults for unspecified ARNs in the Fargate Agent - #1634
- Fix ShellTask return value on empty stdout - #1632
- Fix issue with some Cloud Secrets not being converted from strings - #1655
- Fix issue with Agent logging config setting not working - #1657
- Fix issue with SnowflakeQuery tasks not working - #1663
# Deprecations
- Tasks that accepted the name of a secret (often
credentials_secret
) will raise a deprecation warning - #1667
# Breaking Changes
- Fargate Agent now takes in all boto3 camel case arguments instead of specific snake case options - #1649
kubernetes
is no longer installed by default in deployed flow images - #1653- Tasks that accepted the name of a secret (often
credentials_secret
) no longer have a default value for that argument, as it has been deprecated - #1667
# Contributors
# 0.6.7 Oh Six Seven beta
Released October 16, 2019
# Features
- Environments now allow for optional
on_start
andon_exit
callbacks - #1610
# Enhancements
- Raise more informative error when calling
flow.visualize()
if Graphviz executable not installed - #1602 - Allow authentication to Azure Blob Storage with SAS token - #1600
- Additional debug logs to
Docker Container
andDocker Image
tasks - #920 - Changes to Fargate agent to support temporary credentials and IAM role based credentials within AWS compute such as a container or ec2 instance. #1607
- Local Secrets set through environment variable now retain their casing - #1601
- Agents can accept an optional
name
for logging and debugging - #1612 - Added AWS configuration options for Fargate Agent (task_role_arn, execution_role_arn) - #1614
- Change EmailTask to accept SMTP server settings as well as an email_from kwarg - #1619
- Add the ability to delete task tag limits using the client - #1622
# Task Library
- Add
return_all
kwarg toShellTask
for optionally returning all lines of stdout - #1598 - Add
CosmosDBCreateItem
,CosmosDBReadItems
,CosmosDBQueryItems
and for interacting with data stored on Azure Cosmos DB - #1617
# Fixes
- Fix issue with running local Flow without a schedule containing cached tasks - #1599
- Remove blank string for
task_run_id
in k8s resource manager - #1604 - Fix issue with merge task not working for pandas dataframes and numpy arrays - #1609
# Deprecations
- None
# Breaking Changes
- Local Secrets set through environment variable now retain their casing - #1601
# Contributors
# 0.6.6 Wait For It beta
Released October 3, 2019
# Features
- Added
KubernetesJobEnvironment
- #1548 - Add ability to enforce Task concurrency limits by tag in Prefect Cloud - #1570
- Added
FargateTaskEnvironment
- #1592
# Enhancements
- Allow the
Client
to more gracefully handle failed login attempts on initialization - #1535 - Replace
DotDict
withbox.Box
- #1518 - Store
cached_inputs
on Failed states and call their result handlers if they were provided - #1557 raise_on_exception
no longer raises for Prefect Signals, as these are typically intentional / for control flow - #1562run cloud
CLI command takes in optional--parameters
as a file path pointing to a JSON file - #1582- Always consider
Constant
tasks successful and unpack them immediately instead of submitting them for execution - #1527
# Task Library
- Add
BlobStorageDownload
andBlobStorageUpload
for interacting with data stored on Azure Blob Storage - #1538 - Loosen Kubernetes Tasks' requirement of an API secret key - #1559
- Add tasks for working in Azure Machine Learning Serviec with Datastores and Datasets - #1590
# Fixes
- Fix issue with certain Pause / Resume / Retry pipelines retrying indefinitely - #1177
- Kubernetes Agent deployment YAML generation defaults to local Prefect version - #1573
- Fix issue with custom result handlers not working when called in
cached_inputs
- #1585
# Deprecations
- None
# Breaking Changes
- None
# Contributors
# 0.6.5 Agents of Environmental Change beta
Released September 20, 2019
# Features
- Added Fargate agent - #1521
- Custom user-written environments can be deployed to Prefect Cloud - #1534, #1537
# Enhancements
- Allow for Agents to correctly run in environments with differently calibrated clocks - #1402
- Refactor
RemoteEnvironment
to utilize theget_flow
storage interface - #1476 - Ensure Task logger is available in context throughout every pipeline step of the run - #1509
- Skip Docker registry pushing and pulling on empty
registry_url
attribute - #1525 - Agents now log platform errors to flow runs which cannot deploy - #1528
- Updating
ShellTask
to work more like Airflow Bash Operator for streaming logs and returning values - #1451 - Agents now have a verbose/debug logging option for granular output - #1532
DaskKubernetesEnvironment
now allows for custom scheduler and worker specs - #1543, #1537
# Task Library
- None
# Fixes
- Fix map error by removing
imagePullSecrets
from Kubernetes Agent install if not provided - #1524 - Fix issue with two INFO logs not being associated with the Task Run in Cloud - #1526
execute
CLI command can now load custom environments off of the flow object - #1534
# Deprecations
- None
# Breaking Changes
- Update
ShellTask
to return only the last line of stdout, as a string - #1451
# Contributors
# 0.6.4 I installed Docker on a Windows machine and all I got was this release beta
Released September 10, 2019
# Features
- Improve Windows compatibility for local development and deploying to Prefect Cloud - #1441, #1456, #1465, #1466
# Enhancements
- Add OS platform check to Local Agent for running on Windows machines - #1441
- Add
--base-url
argument for Docker daemons toagent start
CLI command - #1441 - Add environment labels for organizing / tagging different Flow execution environments - #1438
- Use
-U
option when installingprefect
in Docker containers to override base image version - #1461 - Remove restriction that prevented
DotDict
classes from having keys that shadowed dict methods - #1462 - Added livenessProbe to Kubernetes Agent - #1474
- Ensure external Dask Clusters do not require Prefect Cloud environment variables to run Cloud flows - #1481
# Task Library
- None
# Fixes
- Fix incorrect import in
DaskKubernetesEnvironment
job template - #1458 - Raise error on Agents started without an appropriate API token - #1459
- Fix bug when calling
as_nested_dict
onDotDicts
with anitems
key - #1462 - Fix
--resource-manager
flag on agent install invalidatingimagePullSecrets
- #1469 - Fix issue with user-written result handlers in Prefect Cloud preventing some states from being set - #1480
# Deprecations
- None
# Breaking Changes
- None
# Contributors
# 0.6.3 Retry Release beta
Released August 30, 2019
Maintenance release.
# Fixes
- Fix issue with reduced mapped tasks not respecting retries - #1436
# 0.6.2 Onboards and Upwards beta
Released August 30, 2019
# Features
- Added Local, Kubernetes, and Nomad agents - #1341
- Add the ability for Tasks to sequentially loop - #1356
# Enhancements
- Adds a copy to clipboard button for codeblocks - #213
- Updates Vuepress to v1.0.3 - #770
- Introduce configurable default for storage class on Flows - #1044
- Allow for min and max workers to be specified in
DaskKubernetesEnvironment
- #1338 - Use task and flow names for corresponding logger names for better organization - #1355
Paused
states subclassScheduled
and can have predefined expirations - #1375- Introduce new Flow health checks prior to Cloud deployment - #1372
- Improve logging functionality to include tracebacks - #1374
- Improve CLI user experience while working with Cloud - #1384
- Users can now create projects from the CLI - #1388
- Add a health check to confirm that serialized flows are valid prior to Cloud deploy - #1397
- Add
task_slug
,flow_id
, andflow_run_id
to context - #1405 - Support persistent
scheduled_start_time
for scheduled flow runs when run locally withflow.run()
- #1418, #1429 - Add
task_args
toTask.map
- #1390 - Add auth flows for
USER
-scoped Cloud API tokens - #1423 - Add
AzureResultHandler
for handling results to / from Azure Blob storage containers - #1421 - Add new configurable
LocalDaskExecutor
- #1336 - Add CLI commands for working with Prefect Cloud auth - #1431
# Task Library
- Add new
SnowflakeQuery
task for using snowflake data warehouse - #1113
# Fixes
- Fix issue with Docker storage not respecting user-provided image names - #1335
- Fix issue with local retries in Cloud not always running in-process - #1348
# Deprecations
- Rename
SynchronousExecutor
asLocalDaskExecutor
- #1434
# Breaking Changes
- Rename
CloudEnvironment
toDaskKubernetesEnvironment
- #1250 - Remove unused
queue
method from all executors - #1434
# Contributors
# 0.6.1 Prefect Timing beta
Released August 8, 2019
# Features
- Introduce new
flows.checkpointing
configuration setting for checkpointing Tasks in local execution - #1283 - Introduce new, flexible
Schedule
objects - #1320
# Enhancements
- Allow passing of custom headers in
Client
calls - #1255 - Autogenerate informative names and tags for Docker images built for Flow storage - #1237
- Allow mixed-case configuration keys (environment variables are interpolated as lowercase) - #1288
- Ensure state handler errors are logged informatively - #1326
# Task Library
- Add
BigQueryLoadGoogleCloudStorage
task for loading data into BigQuery from Google Cloud Storage #1317
# Fixes
- Fix issue with logs not always arriving in long-standing Dask clusters - #1244
- Fix issue with
BuildImage
docker task not actually running to completion - #1243 - Fix
run --logs
CLI command not exiting on flow run finished state - #1319
# Deprecations
OneTimeSchedule
andUnionSchedule
are deprecated, but remain callable as convenience functions - #1320- Old-style schedules can be deserialized as new-style schedules, but support will eventually be dropped - #1320
# Breaking Changes
prefect.Client.graphql()
andprefect.Client.post()
now use an explicit keyword, not**kwargs
, for variables or parameters - #1259auth add
CLI command replaced withauth login
- #1319
# Contributors
- None
# 0.6.0 Cloud Ready beta
Released July 16, 2019
# Features
- Add the Prefect CLI for working with core objects both locally and in cloud - #1059
- Add RemoteEnvironment for simple executor based executions - #1215
- Add the ability to share caches across Tasks and Flows - #1222
- Add the ability to submit tasks to specific dask workers for task / worker affinity - #1229
# Enhancements
- Refactor mapped caching to be independent of order - #1082
- Refactor caching to allow for caching across multiple runs - #1082
- Allow for custom secret names in Result Handlers - #1098
- Have
execute cloud-flow
CLI immediately set the flow run state toFailed
if environment fails - #1122 - Validate configuration objects on initial load - #1136
- Add
auto_generated
property to Tasks for convenient filtering - #1135 - Disable dask work-stealing in Kubernetes via scheduler config - #1166
- Implement backoff retry settings on Client calls - #1187
- Explicitly set Dask keys for a better Dask visualization experience - #1218
- Implement a local cache which persists for the duration of a Python session - #1221
- Implement in-process retries for Cloud Tasks which request retry in less than one minute - #1228
- Support
Client.login()
with API tokens - #1240 - Add live log streaming for
prefect run cloud
command - #1241
# Task Library
- Add task to trigger AWS Step function workflow #1012
- Add task to copy files within Google Cloud Storage - #1206
- Add task for downloading files from Dropbox - #1205
# Fixes
- Fix issue with mapped caching in Prefect Cloud - #1096
- Fix issue with Result Handlers deserializing incorrectly in Cloud - #1112
- Fix issue caused by breaking change in
marshmallow==3.0.0rc7
- #1151 - Fix issue with passing results to Prefect signals - #1163
- Fix issue with
flow.update
not preserving mapped edges - #1164 - Fix issue with Parameters and Context not being raw dictionaries - #1186
- Fix issue with asynchronous, long-running mapped retries in Prefect Cloud - #1208
- Fix issue with automatically applied collections to task call arguments when using the imperative API - #1211
# Breaking Changes
- The CLI command
prefect execute-flow
andprefect execute-cloud-flow
no longer exist - #1059 - The
slack_notifier
state handler now uses awebhook_secret
kwarg to pull the URL from a Secret - #1075 - Use GraphQL for Cloud logging - #1193
- Remove the
CloudResultHandler
default result handler - #1198 - Rename
LocalStorage
toLocal
- #1236
# Contributors
# 0.5.5 Season 8 beta
Released May 31, 2019
Bugfix to address an unpinned dependency
# 0.5.4 A Release Has No Name beta
Released May 28, 2019
# Features
- Add new
UnionSchedule
for combining multiple schedules, allowing for complex schedule specifications - #428 - Allow for Cloud users to securely pull Docker images from private registries - #1028
# Enhancements
- Add
prefect_version
kwarg toDocker
storage for controlling the version of prefect installed into your containers - #1010, #533 - Warn users if their Docker storage base image uses a different python version than their local machine - #999
- Add flow run id to k8s labels on Cloud Environment jobs / pods for easier filtering in deployment - #1016
- Allow for
SlackTask
to pull the Slack webhook URL from a custom named Secret - #1023 - Raise informative errors when Docker storage push / pull fails - #1029
- Standardized
__repr__
s for various classes, to remove inconsistencies - #617 - Allow for use of local images in Docker storage - #1052
- Allow for doc tests and doc generation to run without installing
all_extras
- #1057
# Task Library
- Add task for creating new branches in a GitHub repository - #1011
- Add tasks to create, delete, invoke, and list AWS Lambda functions #1009
- Add tasks for integration with spaCy pipelines #1018
- Add tasks for querying Postgres database #1022
- Add task for waiting on a Docker container to run and optionally raising for nonzero exit code - #1061
- Add tasks for communicating with Redis #1021
# Fixes
- Ensure that state change handlers are called even when unexpected initialization errors occur - #1015
- Fix an issue where a mypy assert relied on an unavailable import - #1034
- Fix an issue where user configurations were loaded after config interpolation had already taken place - #1037
- Fix an issue with saving a flow visualization to a file from a notebook - #1056
- Fix an issue in which mapped tasks incorrectly tried to run when their upstream was skipped - #1068
- Fix an issue in which mapped tasks were not using their caches locally - #1067
# Breaking Changes
- Changed the signature of
configuration.load_configuration()
- #1037 - Local Secrets now raise
ValueError
s when not found in context - #1047
# Contributors
# 0.5.3 The Release is Bright and Full of Features beta
Released May 7, 2019
# Features
# Enhancements
- Flow now has optional
storage
keyword - #936 - Flow
environment
argument now defaults to aCloudEnvironment
- #936 Queued
states acceptstart_time
arguments - #955- Add new
Bytes
andMemory
storage classes for local testing - #956, #961 - Add new
LocalEnvironment
execution environment for local testing - #957 - Add new
Aborted
state for Flow runs which are cancelled by users - #959 - Added an
execute-cloud-flow
CLI command for working with cloud deployed flows - #971 - Add new
flows.run_on_schedule
configuration option for affecting the behavior offlow.run
- #972 - Allow for Tasks with
manual_only
triggers to be root tasks - #667 - Allow compression of serialized flows #993
- Allow for serialization of user written result handlers - #623
- Allow for state to be serialized in certain triggers and cache validators - #949
- Add new
filename
keyword toflow.visualize
for automatically saving visualizations - #1001 - Add new
LocalStorage
option for storing Flows locally - #1006
# Task Library
- None
# Fixes
- Fix Docker storage not pulling correct flow path - #968
- Fix
run_flow
loading to decode properly by use cloudpickle - #978 - Fix Docker storage for handling flow names with spaces and weird characters - #969
- Fix non-deterministic issue with mapping in the DaskExecutor - #943
# Breaking Changes
- Remove
flow.id
andtask.id
attributes - #940 - Removed old WIP environments - #936 (Note: Changes from #936 regarding environments don't break any Prefect code because environments weren't used yet outside of Cloud.)
- Update
flow.deploy
andclient.deploy
to useset_schedule_active
kwarg to match Cloud - #991 - Removed
Flow.generate_local_task_ids()
- #992
# Contributors
- None
# 0.5.2 Unredacted beta
Released April 19, 2019
# Features
- Implement two new triggers that allow for specifying bounds on the number of failures or successes - #933
# Enhancements
DaskExecutor(local_processes=True)
supports timeouts - #886- Calling
Secret.get()
from within a Flow context raises an informative error - #927 - Add new keywords to
Task.set_upstream
andTask.set_downstream
for handling keyed and mapped dependencies - #823 - Downgrade default logging level to "INFO" from "DEBUG" - #935
- Add start times to queued states - #937
- Add
is_submitted
to states - #944 - Introduce new
ClientFailed
state - #938
# Task Library
- Add task for sending Slack notifications via Prefect Slack App - #932
# Fixes
- Fix issue with timeouts behaving incorrectly with unpickleable objects - #886
- Fix issue with Flow validation being performed even when eager validation was turned off - #919
- Fix issue with downstream tasks with
all_failed
triggers running if an upstream Client call fails in Cloud - #938
# Breaking Changes
- Remove
prefect make user config
from cli commands - #904 - Change
set_schedule_active
keyword in Flow deployments toset_schedule_inactive
to match Cloud - #941
# Contributors
- None
# 0.5.1 It Takes a Village beta
Released April 4, 2019
# Features
- API reference documentation is now versioned - #270
- Add
S3ResultHandler
for handling results to / from S3 buckets - #879 - Add ability to use
Cached
states across flow runs in Cloud - #885
# Enhancements
- Bump to latest version of
pytest
(4.3) - #814 Client.deploy
accepts optionalbuild
kwarg for avoiding building Flow environment - #876- Bump
distributed
to 1.26.1 for enhanced security features - #878 - Local secrets automatically attempt to load secrets as JSON - #883
- Add task logger to context for easily creating custom logs during task runs - #884
# Task Library
- Add
ParseRSSFeed
for parsing a remote RSS feed - #856 - Add tasks for working with Docker containers and imaged - #864
- Add task for creating a BigQuery table - #895
# Fixes
- Only checkpoint tasks if running in cloud - #839, #854
- Adjusted small flake8 issues for names, imports, and comparisons - #849
- Fix bug preventing
flow.run
from properly using cached tasks - #861 - Fix tempfile usage in
flow.visualize
so that it runs on Windows machines - #858 - Fix issue caused by Python 3.5.2 bug for Python 3.5.2 compatibility - #857
- Fix issue in which
GCSResultHandler
was not pickleable - #879 - Fix issue with automatically converting callables and dicts to tasks - #894
# Breaking Changes
- Change the call signature of
Dict
task fromrun(**task_results)
torun(keys, values)
- #894
# Contributors
# 0.5.0 Open Source Launch! beta
Released March 24, 2019
# Features
- Add
checkpoint
option for individualTask
s, as well as a globalcheckpoint
config setting for storing the results of Tasks using their result handlers - #649 - Add
defaults_from_attrs
decorator to easily constructTask
s whose attributes serve as defaults forTask.run
- #293 - Environments follow new hierarchy (PIN-3) - #670
- Add
OneTimeSchedule
for one-time execution at a specified time - #680 flow.run
is now a blocking call which will run the Flow, on its schedule, and execute full state-based execution (including retries) - #690- Pre-populate
prefect.context
with various formatted date strings during execution - #704 - Add ability to overwrite task attributes such as "name" when calling tasks in the functional API - #717
- Release Prefect Core under the Apache 2.0 license - #762
# Enhancements
- Refactor all
State
objects to store fully hydratedResult
objects which track information about how results should be handled - #612, #616 - Add
google.cloud.storage
as an optional extra requirement so that theGCSResultHandler
can be exposed better - #626 - Add a
start_time
check for Scheduled flow runs, similar to the one for Task runs - #605 - Project names can now be specified for deployments instead of IDs - #633
- Add a
createProject
mutation function to the client - #633 - Add timestamp to auto-generated API docs footer - #639
- Refactor
Result
interface intoResult
andSafeResult
- #649 - The
manual_only
trigger will pass ifresume=True
is found in context, which indicates that aResume
state was passed - #664 - Added DockerOnKubernetes environment (PIN-3) - #670
- Added Prefect docker image (PIN-3) - #670
defaults_from_attrs
now accepts a splatted list of arguments - #676- Add retry functionality to
flow.run(on_schedule=True)
for local execution - #680 - Add
helper_fns
keyword toShellTask
for pre-populating helper functions to commands - #681 - Convert a few DEBUG level logs to INFO level logs - #682
- Added DaskOnKubernetes environment (PIN-3) - #695
- Load
context
from Cloud when running flows - #699 - Add
Queued
state - #705 flow.serialize()
will always serialize its environment, regardless ofbuild
- #696flow.deploy()
now raises an informative error if your container cannot deserialize the Flow - #711- Add
_MetaState
as a parent class for states that modify other states - #726 - Add
flow
keyword argument toTask.set_upstream()
andTask.set_downstream()
- #749 - Add
is_retrying()
helper method to allState
objects - #753 - Allow for state handlers which return
None
- #753 - Add daylight saving time support for
CronSchedule
- #729 - Add
idempotency_key
andcontext
arguments toClient.create_flow_run
- #757 - Make
EmailTask
more secure by pulling credentials from secrets - #706
# Task Library
- Add
GCSUpload
andGCSDownload
for uploading / retrieving string data to / from Google Cloud Storage - #673 - Add
BigQueryTask
andBigQueryInsertTask
for executing queries against BigQuery tables and inserting data - #678, #685 - Add
FilterTask
for filtering out lists of results - #637 - Add
S3Download
andS3Upload
for interacting with data stored on AWS S3 - #692 - Add
AirflowTask
andAirflowTriggerDAG
tasks to the task library for running individual Airflow tasks / DAGs - #735 - Add
OpenGitHubIssue
andCreateGitHubPR
tasks for interacting with GitHub repositories - #771 - Add Kubernetes tasks for deployments, jobs, pods, and services - #779
- Add Airtable tasks - #803
- Add Twitter tasks - #803
- Add
GetRepoInfo
for pulling GitHub repository information - #816
# Fixes
- Fix edge case in doc generation in which some
Exception
s' call signature could not be inspected - #513 - Fix bug in which exceptions raised within flow runner state handlers could not be sent to Cloud - #628
- Fix issue wherein heartbeats were not being called on a fixed interval - #669
- Fix issue wherein code blocks inside of method docs couldn't use
**kwargs
- #658 - Fix bug in which Prefect-generated Keys for S3 buckets were not properly converted to strings - #698
- Fix next line after Docker Environment push/pull from overwriting progress bar - #702
- Fix issue with
JinjaTemplate
not being pickleable - #710 - Fix issue with creating secrets from JSON documents using the Core Client - #715
- Fix issue with deserialization of JSON secrets unnecessarily calling
json.loads
- #716 - Fix issue where
IntervalSchedules
didn't respect daylight saving time after serialization - #729
# Breaking Changes
- Remove the
BokehRunner
and associated webapp - #609 - Rename
ResultHandler
methods fromserialize
/deserialize
towrite
/read
- #612 - Refactor all
State
objects to store fully hydratedResult
objects which track information about how results should be handled - #612, #616 Client.create_flow_run
now returns a string instead of aGraphQLResult
object to match the API ofdeploy
- #630flow.deploy
andclient.deploy
require aproject_name
instead of an ID - #633- Upstream state results now take precedence for task inputs over
cached_inputs
- #591 - Rename
Match
task (used inside control flow) toCompareValue
- #638 Client.graphql()
now returns a response with up to two keys (data
anderrors
). Previously thedata
key was automatically selected - #642ContainerEnvironment
was changed toDockerEnvironment
- #670- The environment
from_file
was moved toutilities.environments
- #670 - Removed
start_tasks
argument fromFlowRunner.run()
andcheck_upstream
argument fromTaskRunner.run()
- #672 - Remove support for Python 3.4 - #671
flow.run
is now a blocking call which will run the Flow, on its schedule, and execute full state-based execution (including retries) - #690- Remove
make_return_failed_handler
asflow.run
now returns all task states - #693 - Refactor Airflow migration tools into a single
AirflowTask
in the task library for running individual Airflow tasks - #735 name
is now required on all Flow objects - #732- Separate installation "extras" packages into multiple, smaller extras - #739
Flow.parameters()
always returns a set of parameters - #756
# 0.4.1 beta
Released January 31, 2019
# Features
- Add ability to run scheduled flows locally via
on_schedule
kwarg inflow.run()
- #519 - Allow tasks to specify their own result handlers, ensure inputs and outputs are stored only when necessary, and ensure no raw data is sent to the database - #587
# Enhancements
- Allow for building
ContainerEnvironment
s locally without pushing to registry - #514 - Make mapping more robust when running children tasks multiple times - #541
- Always prefer
cached_inputs
over upstream states, if available - #546 - Add hooks to
FlowRunner.initialize_run()
for manipulating task states and contexts - #548 - Improve state-loading strategy for Prefect Cloud - #555
- Introduce
on_failure
kwarg to Tasks and Flows for user-friendly failure callbacks - #551 - Include
scheduled_start_time
in context for Flow runs - #524 - Add GitHub PR template - #542
- Allow flows to be deployed to Prefect Cloud without a project id - #571
- Introduce serialization schemas for ResultHandlers - #572
- Add new
metadata
attribute to States for managing user-generated results - #573 - Add new 'JSONResultHandler' for serializing small bits of data without external storage - #576
- Use
JSONResultHandler
for all Parameter caching - #590
# Fixes
- Fixed
flow.deploy()
attempting to access a nonexistent string attribute - #503 - Ensure all logs make it to the logger service in deployment - #508, #552
- Fix a situation where
Paused
tasks would be treated asPending
and run - #535 - Ensure errors raised in state handlers are trapped appropriately in Cloud Runners - #554
- Ensure unexpected errors raised in FlowRunners are robustly handled - #568
- Fixed non-deterministic errors in mapping caused by clients resolving futures of other clients - #569
- Older versions of Prefect will now ignore fields added by newer versions when deserializing objects - #583
- Result handler failures now result in clear task run failures - #575
- Fix issue deserializing old states with empty metadata - #590
- Fix issue serializing
cached_inputs
- #594
# Breaking Changes
- Move
prefect.client.result_handlers
toprefect.engine.result_handlers
- #512 - Removed
inputs
kwarg fromTaskRunner.run()
- #546 - Moves the
start_task_ids
argument fromFlowRunner.run()
toEnvironment.run()
- #544, #545 - Convert
timeout
kwarg fromtimedelta
tointeger
- #540 - Remove
timeout
kwarg fromexecutor.wait
- #569 - Serialization of States will ignore any result data that hasn't been processed - #581
- Removes
VersionedSchema
in favor of implicit versioning: serializers will ignore unknown fields and thecreate_object
method is responsible for recreating missing ones - #583 - Convert and rename
CachedState
to a successful state namedCached
, and also remove the superfluouscached_result
attribute - #586
# 0.4.0 beta
Released January 8, 2019
# Features
- Add support for Prefect Cloud - #374, #406, #473, #491
- Add versioned serialization schemas for
Flow
,Task
,Parameter
,Edge
,State
,Schedule
, andEnvironment
objects - #310, #318, #319, #340 - Add ability to provide
ResultHandler
s for storing private result data - #391, #394, #430 - Support depth-first execution of mapped tasks and tracking of both the static "parent" and dynamic "children" via
Mapped
states - #485
# Enhancements
- Add new
TimedOut
state for task execution timeouts - #255 - Use timezone-aware dates throughout Prefect - #325
- Add
description
andtags
arguments toParameters
- #318 - Allow edge
key
checks to be skipped in order to create "dummy" flows from metadata - #319 - Add new
names_only
keyword toflow.parameters
- #337 - Add utility for building GraphQL queries and simple schemas from Python objects - #342
- Add links to downloadable Jupyter notebooks for all tutorials - #212
- Add
to_dict
convenience method forDotDict
class - #341 - Refactor requirements to a custom
ini
file specification - #347 - Refactor API documentation specification to
toml
file - #361 - Add new SQLite tasks for basic SQL scripting and querying - #291
- Executors now pass
map_index
into theTaskRunner
s - #373 - All schedules support
start_date
andend_date
parameters - #375 - Add
DateTime
marshmallow field for timezone-aware serialization - #378 - Adds ability to put variables into context via the config - #381
- Adds new
client.deploy
method for adding new flows to the Prefect Cloud - #388 - Add
id
attribute toTask
class - #416 - Add new
Resume
state for resuming fromPaused
tasks - #435 - Add support for heartbeats - #436
- Add new
Submitted
state for signaling thatScheduled
tasks have been handled - #445 - Add ability to add custom environment variables and copy local files into
ContainerEnvironment
s - #453 - Add
set_secret
method to Client for creating and setting the values of user secrets - #452 - Refactor runners into
CloudTaskRunner
andCloudFlowRunner
classes - #431 - Added functions for loading default
engine
classes from config - #477
# Fixes
- Fixed issue with
GraphQLResult
reprs - #374 CronSchedule
produces expected results across daylight savings time transitions - #375utilities.serialization.Nested
properly respectsmarshmallow.missing
values - #398- Fixed issue in capturing unexpected mapping errors during task runs - #409
- Fixed issue in
flow.visualize()
so that mapped flow states can be passed and colored - #387 - Fixed issue where
IntervalSchedule
was serialized at "second" resolution, not lower - #427 - Fixed issue where
SKIP
signals were preventing multiple layers of mapping - #455 - Fixed issue with multi-layer mapping in
flow.visualize()
- #454 - Fixed issue where Prefect Cloud
cached_inputs
weren't being used locally - #434 - Fixed issue where
Config.set_nested
would have an error if the provided key was nested deeper than an existing terminal key - #479 - Fixed issue where
state_handlers
were not called for certain signals - #494
# Breaking Changes
- Remove
NoSchedule
andDateSchedule
schedule classes - #324 - Change
serialize()
method to use schemas rather than custom dict -