Overview

Looking for the latest Prefect 2 release? Prefect 2 and Prefect Cloud 2 have been released for General Availability. See https://docs.prefect.io/ for details.

Prefect Agents are lightweight processes for orchestrating flow runs. Agents run inside a user's architecture, and are responsible for starting and monitoring flow runs. During operation the agent process queries the Prefect API for any scheduled flow runs, and allocates resources for them on their respective deployment platforms.

Note that both Prefect Agents (and flow runs) only send requests out to the Prefect API, and never receive requests themselves. This is part of our Hybrid Execution Model, and helps keep your code and data safe.

A single agent can manage many concurrent flow runs - the only reason to have multiple active agents is if you need to support flow runs on different deployment platforms.

Agent Types

Prefect supports several different agent types for deploying on different platforms.

• Local: The Local Agent executes flow runs as local processes.

• Docker: The Docker Agent executes flow runs in docker containers.

• Kubernetes: The Kubernetes Agent executes flow runs as Kubernetes Jobs.

• GCP Vertex: The Vertex Agent executes flow runs as Vertex Custom Jobs.

• AWS ECS: The ECS Agent executes flow runs as AWS ECS tasks (on either ECS or Fargate).

See their respective documentation for more information on each type.

Usage

Prefect agents can be started via the CLI, using prefect agent <AGENT TYPE> start. For example, to start a local agent:

prefect agent local start

Alternatively, all Prefect Agents can also be run using the Python API.

from prefect.agent.local import LocalAgent

LocalAgent().start()

Common Configuration Options

The following configuration options are shared for all agents.

Authentication Cloud

Prefect agents rely on the use of an API key from Prefect Cloud. For information on how to create and configure API keys, see the API key documentation.

In addition to the methods outlined there, you may pass an API key via CLI to an agent

$ prefect agent <agent-type> start --key "API_KEY"

Prefect API Address

Prefect agents query the API for any pending flow runs. By default the address used is:

• https://api.prefect.io for Prefect Cloud
• http://localhost:4200 for Prefect Server

If needed, you can manually configure the address through the CLI:

prefect agent <AGENT TYPE> start --api <API ADDRESS>

Labels

Agents have an optional labels argument which allows for separation of execution when using multiple agents. This is especially useful for teams wanting to run specific flows on different clusters. For more information on labels and how to use them see the Run Configuration docs.

By default, agents have no set labels and will only pick up runs from flows with no specified labels. Labels can be provided to an agent through a few methods:

prefect agent <AGENT TYPE> start --label dev --label staging

from prefect.agent.docker import DockerAgent

DockerAgent(labels=["dev", "staging"]).start()

# ~/.prefect/config.toml
[cloud.agent]
labels = ["dev", "staging"]

export PREFECT__CLOUD__AGENT__LABELS='["dev", "staging"]'

Environment Variables

All agents have a --env flag for configuring environment variables to set on all flow runs managed by that agent. This can be useful for things you want applied to all flow runs, whereas the env option in a flow's RunConfig only applies to runs of a specific flow.

prefect agent <AGENT TYPE> start --env KEY=VALUE --env KEY2=VALUE2

from prefect.agent.docker import DockerAgent

DockerAgent(env_vars={"KEY": "VALUE", "KEY2": "VALUE2"})

Agent Automations Cloud

Users on Standard or Enterprise licenses in Cloud can create an agent automation to notify them if all agents from a configuration group (agent config ids can be added to multiple agents) have not queried for work in a certain time frame. To do so go to the automations tab of the dashboard in the UI and set up an agent configuration then copy the agent config id that is provided once your automation is created. You can then provide the agent configuration to your agent using the --agent-config-id flag:

prefect agent <AGENT TYPE> start --agent-config-id <AGENT CONFIG ID>

Note - Agent automations can only be added as a flag when starting an agent at present. They can not be added at install.

Health Checks

Agents can optionally run a private HTTP server for use as a health check. Health checks can be used by common orchestration services (e.g. supervisord, docker, kubernetes, ...) to check that the agent is running properly and take actions (such as restarting the agent) if it's not.

A few ways to configure:

prefect agent <AGENT TYPE> start --agent-address http://localhost:8080

from prefect.agent.docker import DockerAgent

DockerAgent(agent_address="http://localhost:8080").start()

# ~/.prefect/config.toml
[cloud.agent]
agent_address = "http://localhost:8080"

$ export PREFECT__CLOUD__AGENT__AGENT_ADDRESS=http://localhost:8080

If enabled, the HTTP health check will be available via the /api/health route at the configured agent address. This route returns 200 OK if the agent is healthy, and will error otherwise.