# Kubernetes Agent
The Kubernetes Agent deploys flow runs as Kubernetes Jobs. It can be run both in-cluster (recommended for production deployments) as well as external to the cluster.
# Requirements
If running locally, note that the required dependencies for the Kubernetes
Agent aren't installed by default. If
you're a pip user you'll need to add the kubernetes extra. Likewise, with
conda you'll need to install the extra kubernetes package:
If you're deploying the Kubernetes Agent in-cluster, you won't need to worry about this.
Prefect Server
In order to use this agent with Prefect Server the server's GraphQL API endpoint must be accessible. This may require changes to your Prefect Server deployment and/or configuring the Prefect API address on the agent.
# Flow Configuration
The Kubernetes Agent will deploy flows using either a
UniversalRun (the
default) or KubernetesRun run_config. Using a KubernetesRun object lets you customize the deployment
environment for a flow (exposing env, image, cpu_limit, etc...):
from prefect.run_configs import KubernetesRun
# Configure extra environment variables for this flow,
# and set a custom image
flow.run_config = KubernetesRun(
env={"SOME_VAR": "VALUE"},
image="my-custom-image"
)
See the KubernetesRun documentation for more information.
# Agent Configuration
The Kubernetes agent can be started from the Prefect CLI as
prefect agent kubernetes start
API Keys Cloud
When using Prefect Cloud, this will require a service account API key, see here for more information.
Below we cover a few common configuration options, see the CLI docs for a full list of options.
# Authentication
If running external to the cluster, you'll need to ensure your active kubeconfig is valid and for the proper cluster.
When running in-cluster, the Agent pod will need the proper RBAC credentials. This is covered in more detail below.
# Namespace
By default the agent will deploy flow run jobs into the default namespace.
You can use the --namespace option to change this:
prefect agent kubernetes start --namespace my-namespace
# Service Account
You can use the --service-account-name option to configure a service
account
to use for all Jobs deployed by the agent.
prefect agent kubernetes start --service-account-name my-account
Flows can override this agent default by passing the service_account_name option to
their respective
KubernetesRun run_config.
# Image Pull Secrets
Likewise, the --image-pull-secrets option can be used to configure image
pull
secrets
for all Jobs deployed by the agent. Multiple secrets can be provided as a
comma-separated list. Note that the secrets must already exist within the same
namespace as the agent.
prefect agent kubernetes start --image-pull-secrets secret-1,secret-2
Flows can override this agent default by passing the image_pull_secrets option to
their respective
KubernetesRun run_config.
# Custom Job Template
For deeper customization of the Kubernetes Job created by the agent, you may
want to make use of a custom job template. This template can either be
configured per-flow (on the
KubernetesRun run_config), or on the Agent as a default for flows that don't provide their
own template.
As an example, providing a custom job template will allow you to customize the Kubernetes Job name prefix from prefect-job to something more descriptive for each flow and its associated workload. The prefix is specified in the job template's metadata.name field.
For reference, the default template packaged with Prefect can be found here.
To provide your own job template, you can use the --job-template flag. This
takes a path to a job template YAML file. The path can be local to the agent,
or stored in cloud storage on either GCS or S3 (note that in this case, the
agent will need credentials for S3/GCS access).
# Using a local file
prefect agent kubernetes start --job-template /path/to/my_template.yaml
# Stored on S3
prefect agent kubernetes start --job-template s3://bucket/path/to/my_template.yaml
# Running In-Cluster
For production deployments, we recommend running the Kubernetes Agent inside the Kubernetes Cluster as a Deployment. This helps ensure your agent is active and healthy.
To make this easier, the Prefect CLI provides a command for generating an example Kubernetes Manifest you can use for creating such a deployment.
prefect agent kubernetes install
The install command accepts many of the same options as the start command -
see the CLI documentation for
all available options.
The generated manifest can be piped to kubectl apply, or manually edited to
further customize the deployment.
prefect agent kubernetes install -k API_KEY | kubectl apply --namespace=my-namespace -f -
Once created, you should be able to see the agent deployment running in your cluster:
$ kubectl get deploy
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
prefect-agent 1 1 1 0 2s
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
prefect-agent-845798bb59-s7wxg 1/1 Running 0 5s
# RBAC
The Kubernetes Agent requires certain
RBAC
permissions to work with jobs in its namespace. The proper RBAC configuration
can be generated by providing the --rbac flag to prefect agent kubernetes install- for reference we provide it below as well.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: default
name: prefect-agent-rbac
rules:
- apiGroups: ['batch', 'extensions']
resources: ['jobs']
verbs: ['*']
- apiGroups: ['']
resources: ['events', 'pods']
verbs: ['*']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
namespace: default
name: prefect-agent-rbac
subjects:
- kind: ServiceAccount
name: default
roleRef:
kind: Role
name: prefect-agent-rbac
apiGroup: rbac.authorization.k8s.io
# Additional Permissions
If you are using Amazon EKS for your kubernetes deployment and you need S3 access, note that S3 is not accessible by default to Amazon EKS. To enable S3 access by your kubernetes cluster on EKS, add the necessary permissions (AmazonS3FullAccess or AmazonS3ReadOnlyAccess) directly to the NodeInstanceRole used by aws-auth-cm.yaml after launching worker nodes and before applying aws-auth-cm.yaml with kubectl.