Blocks¶
Prefect blocks store configuration and provide an interface for interacting with external systems.
Blocks expose methods that provide functionality specific to the systems they interface with. For example, blocks can be used to download data from or upload data to an S3 bucket, query data from or write data to a database, or send a message to a Slack channel.
Overview¶
Block types are Python classes with a handy UI webform for configuration. Blocks are instantiation of these classes with specific values.
Configure blocks through Python code or via a form in the UI. Access blocks for use in Python code.
Block values are stored in Prefect Cloud or your self-hosted Prefect server instance. Blocks can be shared with other users in your Prefect Cloud workspace.
To see block types available for configuration, use prefect block type ls from the CLI or navigate to the Blocks page in the UI and click +.
Blocks and parameters
Blocks are useful for configuration that needs to be shared across flow runs and between flows.
For configuration that will change between flow runs, we recommend using parameters.
Prefect built-in blocks¶
Commonly used block types come built-in with Prefect. These block types can be created via the UI and used without installing any additional packages.
| Block | Slug | Description |
|---|---|---|
| Azure | azure |
Stores data as a file on Azure Data Lake and Azure Blob Storage. |
| Custom Webhook | custom-webhook |
Calls custom webhooks. |
| Discord Webhook | discord-webhook |
Calls Discord webhooks. |
| Date Time | date-time |
Stores a datetime value. |
| Docker Container | docker-container |
Runs a command in a container. |
| Docker Registry | docker-registry |
Connects to a Docker registry. Requires a Docker Engine to be connectable. |
| GCS | gcs |
Store data as a file on Google Cloud Storage. |
| GitHub | github |
Interacts with files stored on public GitHub repositories. |
| JSON | json |
Stores JSON data. |
| Kubernetes Cluster Config | kubernetes-cluster-config |
Stores configuration for interaction with Kubernetes clusters. |
| Kubernetes Job | kubernetes-job |
Runs a command as a Kubernetes Job. |
| Local File System | local-file-system |
Stores data as a file on a local file system. |
| Mattermost Webhook | mattermost-webhook |
Sends notifications via a provided Mattermost webhook. |
| Microsoft Teams Webhook | ms-teams-webhook |
Sends notifications via a provided Microsoft Teams webhook. |
| Opsgenie Webhook | opsgenie-webhook |
Sends notifications via a provided Opsgenie webhook. |
| Pager Duty Webhook | pager-duty-webhook |
Sends notifications via a provided PagerDuty webhook. |
| Process | process |
Run a command in a new process. |
| Remote File System | remote-file-system |
Stores data as a file on any remote file system that supports fsspec. |
| S3 | s3 |
Stores data as a file on AWS S3. |
| Secret | secret |
Stores a secret value. The value will be obfuscated when this block is logged or shown in the UI. |
| Sendgrid Email | sendgrid-email |
Sends notifications via Sendgrid email. |
| Slack Webhook | slack-webhook |
Sends notifications via a provided Slack webhook. |
| SMB | smb |
Stores data as a file on a SMB share. |
| String | string |
Stores a string value. |
| Twilio SMS | twilio-sms |
Sends notifications via Twilio SMS. |
Warning
The S3, Azure, GCS, and GitHub blocks are deprecated in favor of the the corresponding S3Bucket, AzureBlobStorageCredentials, GCSBucket, and GitHubRepository blocks found in the Prefect integration libraries.
Blocks in Prefect integration libraries¶
Some block types that appear in the UI can be created immediately, and then the corresponding integration library must be installed for use.
For example, an AWS Secret block can be created, but not used until the prefect-aws library is installed.
Block types can be created by anyone and optionally shared with the community. You'll find block types available for consumption in many of the published Prefect integrations libraries. If a block type is not available in the UI, you can register it via the CLI.
Use existing block types¶
Blocks are classes that subclass the Block base class. They can be instantiated and used like normal classes.
Instantiate blocks¶
To instantiate a block that stores a JSON value, use the JSON block:
from prefect.blocks.system import JSON
json_block = JSON(value={"the_answer": 42})
Save blocks¶
To retrieve this saved value use the .save() method:
json_block.save(name="life-the-universe-everything")
To update saved block value stored for a given block, overwrite the existing block by passing overwrite=True:
json_block.save(overwrite=True)
Create a new JSON block by setting the name parameter to a new value:
json_block.save(name="actually-life-the-universe-everything")
Note that blocks can also be created and updated via the Prefect UI.
Load blocks¶
The block name can be used to load the block:
from prefect import flow
from prefect.blocks.system import JSON
@flow
def what_is_the_answer():
json_block = JSON.load("life-the-universe-everything")
print(json_block.value["the_answer"])
if __name__ == "__main__":
what_is_the_answer() # 42
Alternatively, load a block with the unique slug that is a combination of the block type slug and the block name.
To load our JSON block from above, run the following:
from prefect.blocks.core import Block
json_block = Block.load("json/life-the-universe-everything")
print(json_block.value["the-answer"]) #42
Delete blocks¶
Delete a block with the .delete() method:
from prefect.blocks.core import Block
Block.delete("json/life-the-universe-everything")
Alternatively, use the CLI to delete specific blocks with a given slug or id:
prefect block delete json/life-the-universe-everything
prefect block delete --id <my-id>
Creating new block types¶
To create a custom block type, define a class that subclasses Block. The Block base class builds on Pydantic's BaseModel, so custom blocks can be declared in the same manner as a Pydantic model.
Here's a block that represents a cube and holds information about the length of each edge in inches:
from prefect.blocks.core import Block
class Cube(Block):
edge_length_inches: float
You can include methods on a block to provide functionality. Here's the same cube block with methods to calculate the volume and surface area of the cube:
from prefect.blocks.core import Block
class Cube(Block):
edge_length_inches: float
def get_volume(self):
return self.edge_length_inches**3
def get_surface_area(self):
return 6 * self.edge_length_inches**2
Use the new Cube block type in a flow:
from prefect import flow
rubiks_cube = Cube(edge_length_inches=2.25)
rubiks_cube.save("rubiks-cube")
@flow
def calculate_cube_surface_area(cube_name):
cube = Cube.load(cube_name)
print(cube.get_surface_area())
if __name__ == "__main__":
calculate_cube_surface_area("rubiks-cube") # 30.375
Secret fields¶
All block values are encrypted before being stored.
If you have values that you would not like visible in the UI or in logs, use the SecretStr field type provided by Pydantic to automatically obfuscate those values.
This functionality can be useful for fields that are used to store credentials such as passwords and API tokens.
Here's an example of an AWSCredentials block that uses SecretStr:
from typing import Optional
from prefect.blocks.core import Block
from pydantic import SecretStr # if pydantic version >= 2.0, use: from pydantic.v1 import SecretStr
class AWSCredentials(Block):
aws_access_key_id: Optional[str] = None
aws_secret_access_key: Optional[SecretStr] = None
aws_session_token: Optional[str] = None
profile_name: Optional[str] = None
region_name: Optional[str] = None
Because aws_secret_access_key has the SecretStr type hint assigned to it, the value of that field will not be exposed if the object is logged:
aws_credentials_block = AWSCredentials(
aws_access_key_id="AKIAJKLJKLJKLJKLJKLJK",
aws_secret_access_key="secret_access_key"
)
print(aws_credentials_block)
# aws_access_key_id='AKIAJKLJKLJKLJKLJKLJK' aws_secret_access_key=SecretStr('**********') aws_session_token=None profile_name=None region_name=None
Prefect's SecretDict field type allows you to add a dictionary field to your block that will have values at all levels automatically obfuscated in the UI or in logs.
This functionality is useful for blocks where typing or structure of secret fields is not known until configuration time.
Here's an example of a block that uses SecretDict:
from typing import Dict
from prefect.blocks.core import Block
from prefect.blocks.fields import SecretDict
class SystemConfiguration(Block):
system_secrets: SecretDict
system_variables: Dict
system_configuration_block = SystemConfiguration(
system_secrets={
"password": "p@ssw0rd",
"api_token": "token_123456789",
"private_key": "<private key here>",
},
system_variables={
"self_destruct_countdown_seconds": 60,
"self_destruct_countdown_stop_time": 7,
},
)
system_secrets will be obfuscated when system_configuration_block is displayed, but system_variables will be shown in plain-text:
print(system_configuration_block)
# SystemConfiguration(
# system_secrets=SecretDict('{'password': '**********', 'api_token': '**********', 'private_key': '**********'}'),
# system_variables={'self_destruct_countdown_seconds': 60, 'self_destruct_countdown_stop_time': 7}
# )
Block type metadata¶
The way that a block is displayed can be controlled by metadata fields that can be set on a block type's subclass.
Available metadata fields include:
| Property | Description |
|---|---|
| _block_type_name | Display name of the block in the UI. Defaults to the class name. |
| _block_type_slug | Unique slug used to reference the block type in the API. Defaults to a lowercase, dash-delimited version of the block type name. |
| _logo_url | URL pointing to an image that should be displayed for the block type in the UI. Default to None. |
| _description | Short description of block type. Defaults to docstring, if provided. |
| _code_example | Short code snippet shown in UI for how to load/use block type. Default to first example provided in the docstring of the class, if provided. |
Nested blocks¶
Blocks are composable - a block can be used within other blocks. You can create a block type that uses functionality from another block type by declaring it as an attribute.
Nestable blocks are loosely coupled, as configuration can be changed for each block independently. This allows configuration to be shared across multiple use cases.
To illustrate, here's a an expanded AWSCredentials block that includes the ability to get an authenticated session via the boto3 library:
from typing import Optional
import boto3
from prefect.blocks.core import Block
from pydantic import SecretStr
class AWSCredentials(Block):
aws_access_key_id: Optional[str] = None
aws_secret_access_key: Optional[SecretStr] = None
aws_session_token: Optional[str] = None
profile_name: Optional[str] = None
region_name: Optional[str] = None
def get_boto3_session(self):
return boto3.Session(
aws_access_key_id = self.aws_access_key_id
aws_secret_access_key = self.aws_secret_access_key
aws_session_token = self.aws_session_token
profile_name = self.profile_name
region_name = self.region
)
The AWSCredentials block can be used within an S3Bucket block to provide authentication when interacting with an S3 bucket:
import io
class S3Bucket(Block):
bucket_name: str
credentials: AWSCredentials
def read(self, key: str) -> bytes:
s3_client = self.credentials.get_boto3_session().client("s3")
stream = io.BytesIO()
s3_client.download_fileobj(Bucket=self.bucket_name, key=key, Fileobj=stream)
stream.seek(0)
output = stream.read()
return output
def write(self, key: str, data: bytes) -> None:
s3_client = self.credentials.get_boto3_session().client("s3")
stream = io.BytesIO(data)
s3_client.upload_fileobj(stream, Bucket=self.bucket_name, Key=key)
You can use this S3Bucket block with previously saved AWSCredentials block values in order to interact with the configured S3 bucket:
my_s3_bucket = S3Bucket(
bucket_name="my_s3_bucket",
credentials=AWSCredentials.load("my_aws_credentials")
)
my_s3_bucket.save("my_s3_bucket")
Saving block values like this links the values of the two blocks so that any changes to the values stored for the AWSCredentials block with the name my_aws_credentials will be seen the next time that block values for the S3Bucket block named my_s3_bucket is loaded.
Values for nested blocks can also be hard coded by not first saving child blocks:
my_s3_bucket = S3Bucket(
bucket_name="my_s3_bucket",
credentials=AWSCredentials(
aws_access_key_id="AKIAJKLJKLJKLJKLJKLJK",
aws_secret_access_key="secret_access_key"
)
)
my_s3_bucket.save("my_s3_bucket")
In the above example, the values for AWSCredentials are saved with my_s3_bucket and will not be usable with any other blocks.
Update custom Block types¶
Let's add a bucket_folder field to your custom S3Bucket block that represents the default path to read and write objects from (this field exists on our implementation).
Add the new field to the class definition:
class S3Bucket(Block):
bucket_name: str
credentials: AWSCredentials
bucket_folder: str = None
...
Then register the updated block type with either Prefect Cloud or your self-hosted Prefect server instance.
If you have any existing blocks of this type that were created before the update and you'd prefer to not re-create them, migrate them to the new version of your block type by adding the missing values:
# Bypass Pydantic validation to allow your local Block class to load the old block version
my_s3_bucket_block = S3Bucket.load("my-s3-bucket", validate=False)
# Set the new field to an appropriate value
my_s3_bucket_block.bucket_path = "my-default-bucket-path"
# Overwrite the old block values and update the expected fields on the block
my_s3_bucket_block.save("my-s3-bucket", overwrite=True)
Register blocks¶
Prefect comes with many blocks pre-registered and ready to use. If you do not have a block available for use, you can register it.
Blocks can be registered from a Python module available in the current environment with a CLI command like this:
prefect block register --module prefect_aws.credentials
This command is useful for registering all blocks found within a module in a Prefect Integration library.
Alternatively, if a custom block has been created in a .py file, the block can also be registered with the CLI command:
prefect block register --file my_block.py
The registered block will then be available for configuration.