# Upgrading to Prefect 1.0
Prefect 1.0 includes some important changes that may require updates to your flow and task definitions.
Importantly, many features previously marked as deprecated have been removed. This means flows using deprecated features will encounter errors rather than warnings.
The following sections describe changes you should be aware of to ensure that your flows work as expected with Prefect 1.0.
# API keys replace authentication tokens
API keys replace authentication tokens to authenticate users and service accounts with the Prefect Cloud API. Existing authentication tokens will be ignored by the client.
See Removing API tokens to learn how to remove old tokens from your environment; this is not required but we recommend you revoke and delete old tokens. See Using API keys with older versions of Prefect for details about how you can use API keys in place of tokens in certain situations (such as a VM or container built with an older version of Prefect).
The Prefect CLI commands
list-tokens have been removed.
prefect auth login and
prefect auth logout commands now use API keys. Previously, logging out with an authentication token would just reset your tenant and access token, but leave the token on disk. Now, we retain that behavior the first time the command is called, but if you call it a second time, we delete the token. This allows users to easily remove their token in favor of an API key.
Agents will use the same API key as the client unless a different key is provided with
--key. Previously, there was a dedicated environment variable for providing authentication tokens to agents, but this will now be ignored. If you have logged in to Prefect Cloud on your machine, you may start any kind of agent and the agent will use your API key to query for flow runs and pass it to the flow run for execution.
Unlike tokens, API keys can be associated with multiple tenants. When using an API key with its non-default tenant, you must use the CLI to switch tenants, provide the tenant ID with the
--tenant-id option, or set the
PREFECT__CLOUD__TENANT_ID environment variable.
# install agent, specifying API key and tenant $ prefect agent [agent-type] install --key [api-key] --tenant-id [tenant-id] # start agent, specifying API key and tenant $ prefect agent [agent-type] start --key [api-key] --tenant-id [tenant-id]
See "API keys for simple authentication" in the blog post Prefect 0.15.0: A New Flow Run Experience for additional details.
# Support for environments
Flow environments, deprecated since Prefect 0.14.0, have been removed completely. Use
RunConfig objects to define where and how a flow run should be executed as described in Run Configuration.
If you do not have
flow.environment configured explicitly on your flow, there is no impact on your flows and no migration steps are needed.
If you still have environment configurations, the topic Upgrading Environments to RunConfig provides detailed instructions for migrating to
# Registering and running flows with the CLI
The Prefect CLI commands to register and run flows have been revised, changing the syntax and adding new functionality.
# Registering flows
prefect register command replaces the
prefect register flow command to Register a flow with the CLI.
- Allows registering multiple flows in a single call. Flows can be specified by
--path(path to a file or directory containing flows) or by
--module(an importable Python module containing flows). Both options can be specified multiple times in a single call for more flexibility.
--nameflag can be used to only register flows with a specific name (or names). If unspecified, all flows found are registered.
- By default, for script-based storage such as GitHub, a flow will only be re-registered if it is structurally different than the existing version. This means that small edits to the source of tasks won't require re-registration, and the CLI will automatically detect this to avoid needlessly bumping the version. This can be disabled by passing in
--force. If you pickle (default behavior) like Local storage, you do need to register for changes to take effect because you need the file in storage to change.
--watchflag enables watching for changes in a directory, path, or module and re-registering flows if a change is detected. This can be used during development to automatically re-register your flows on save (as needed), or as part of a deployment for users who want to watch and auto-update flows from a specific directory.
These features simplify auto-registering flows from within CI. For example, with GitHub actions you might add the following step to your CI workflow to auto-register all flows in a
flows/ directory on merge.
- name: Register Flows if: github.ref == 'refs/heads/master' run: prefect register flows --project testing --path flows
The changes in flow registration require Prefect Server 2021.09.02 or later. Prefect Server will need to be upgraded before flows can be registered from this version.
# Running flows
prefect run command replaces the
prefect run flow command to run a flow from the CLI.
prefect run can run flows locally without the backend (Prefect Server or Prefect Cloud), with the backend by submitting to an agent, or with the backend but without an agent. It takes many options for lookup including a Python import name, a file path, the flow ID, the flow group ID, flow name, or project name. The flow run state change and log display has been entirely rewritten to be nice looking.
prefect run supports the following options:
|-i, --id||The UUID of a flow or flow group to run. If a flow group id is given, the latest flow id will be used for the run.|
|--project||The name of the Prefect project containing the flow to run.|
|-p, --path||The path to a file containing the flow to run.|
|-m, --module||The python module name containing the flow to run.|
|-n, --name||The name of a flow to run from the specified file/module/project. If the source contains multiple flows, this must be provided.|
|--label||A label to add to the flow run. May be passed multiple times to specify multiple labels. If not passed, the labels from the flow group will be used.|
|--run-name||A name to assign to the flow run.|
|--context||A key, value pair (key=value) specifying a flow context variable. The value will be interpreted as JSON. May be passed multiple times to specify multiple context values. Nested values may be set by passing a dict.|
|--param||A key, value pair (key=value) specifying a flow parameter. The value will be interpreted as JSON. May be passed multiple times to specify multiple parameter values.|
|--log-level||The log level to set for the flow run. If passed, the level must be a valid Python logging level name. If this option is not passed, the default level for the flow will be used. Valid values include DEBUG, INFO, WARNING, ERROR, or CRITICAL.|
|--param-file||The path to a JSON file containing parameter keys and values. Any parameters passed with |
|--idempotency-key||A key to prevent duplicate flow runs. If a flow run has already been started with the provided value, the command will display information for the existing run. If using |
|--execute||Execute the flow run in-process without an agent. If this process exits, the flow run will be marked as 'Failed'.|
|-s, --schedule||Execute the flow run according to the schedule attached to the flow. If this flag is set, this command will wait between scheduled flow runs. If the flow has no schedule, this flag will be ignored. If used with a non-local run, an exception will be thrown.|
|-q, --quiet||Disable verbose messaging about the flow run and just print the flow run ID.|
|--no-logs||Disable streaming logs from the flow run to this terminal. Only state changes will be displayed. Only applicable when |
|-w, --watch||Wait for the flow run to finish executing and display status information.|
For example, run a flow in a script locally:
$ prefect run -p hello-world.py
Run a flow with a non-default parameter locally:
$ prefect run -m prefect.hello_world --param name=Marvin
Run a registered flow with the backend with custom labels:
$ prefect run -n "hello-world" --label example --label hello
Run a registered flow and execute locally without an agent (for more information, see "Agentless execution" in the blog post Prefect 0.15.0: A New Flow Run Experience):
$ prefect run -n "hello-world" --execute
# Tasks for sub-flows
Introduced in 0.15.0, Prefect includes new tasks that give you more flexibility around working with sub-flow execution — also known as "flow-of-flows" — and result passing:
create_flow_runlets you programmatically create a flow run in the backend for a registered flow.
wait_for_flow_runlets you wait for a flow run to finish executing while receiving state and log information regarding the flow run.
get_task_run_resultwaits for a task run to complete and returns the result.
# Imports have moved
Imports for some Prefect modules have moved:
- Artifacts functions now imported from
Parameternow imported from
- Exceptions now imported from
- Executors now imported from
These imports were available at both paths previously, but will only be available at the new path now.
# iCal recurrence rules schedules
Prefect now supports rich recurrence rule scheduling following the iCal RRules standard and
rrule module. This feature does not impact existing schedules using interval clocks, cron clocks, and so on, but provides convenient, new syntax for creating repetitive schedules. See Recurrence Rule Clocks for details.
This feature was contributed by a Prefect community member. To learn more, see the original pull request Support for RRule (iCal style recurrence rule) clocks/schedules.
# Drop support for Python 3.6
With Prefect 1.0, we no longer support Python 3.6. Some features will not work as expected if you are using Python 3.6. The minimum recommended version is Python 3.7.
# Prefect server services local by default
Services run by the Prefect CLI only accept local connections by default (they listen to localhost instead of 0.0.0.0).
When configuring Prefect server, you can use the
--expose option if you want to connect from a remote location. This exposes the server to external hosts by listening to 0.0.0.0 instead of localhost.
$ prefect server config --expose
# Additional changes
Prefect 1.0 includes a number of additional minor changes:
- The AWS Fargate agent has been removed. Use the ECS agent instead to deploy flow runs as AWS ECS tasks on either EC2 or Fargate.
- For the Docker agent the deprecated
DockerAgent(docker_interface=...)argument has been removed and will now raise an exception if passed.
log_to_cloudsetting is now ignored. Use
send_flow_run_logsinstead. See Logging with a backend for details.