Parsl Ephemeral Provider for AWS

A provider for the Parsl parallel scripting library that runs workflows on ephemeral AWS resources: instances are created when work arrives and destroyed when it finishes.

License Build Status

Warning

This package is alpha. The public interface and configuration schema may change in any release before 1.0.0.

Key features

  • Three operating modes – standard (direct client-to-worker), detached (a bastion orchestrates jobs so the client can disconnect), and serverless (Lambda or ECS/Fargate workers).

  • Spot support – on-demand or spot, single instances or EC2 Fleet, with price-capacity-optimized allocation and two-minute interruption warnings delivered through EventBridge and SQS.

  • State persistence – file, S3, or SSM Parameter Store, so a provider can be reconstructed after a restart.

  • Pre-provisioned networking – you supply the VPC, subnet, and security group; the provider never creates or deletes them. See Network Prerequisites.

Networking is a prerequisite, not an option

Since v0.7.0 vpc_id, subnet_id, and security_group_id are required for every mode except serverless-with-Lambda workers. Read Network Prerequisites before your first run.

Quick example

import parsl
from parsl.config import Config
from parsl.executors import HighThroughputExecutor
from parsl_ephemeral_provider import EphemeralProvider

provider = EphemeralProvider(
    region="us-east-1",
    instance_type="t3.medium",
    # Pre-provisioned network resources. Required -- see network-prerequisites.
    vpc_id="vpc-0123456789abcdef0",
    subnet_id="subnet-0123456789abcdef0",
    security_group_id="sg-0123456789abcdef0",
    # Block parameters
    init_blocks=1,
    min_blocks=0,
    max_blocks=10,
    # Spot instances at up to 80% of the on-demand price
    use_spot=True,
    spot_max_price_percentage=80,
    # State persistence: "file", "s3", or "parameter_store"
    state_store_type="file",
)

config = Config(
    executors=[
        HighThroughputExecutor(
            label="aws_executor",
            provider=provider,
            # See network-prerequisites: CurveZMQ certificates live in the
            # driver's run_dir and workers cannot read them (#62).
            encrypted=False,
        )
    ]
)

with parsl.load(config):

    @parsl.python_app
    def hello():
        return "Hello from AWS"

    print(hello().result())

image_id is optional: an Amazon Linux 2023 AMI is resolved from AWS’s public SSM parameters for the region and architecture in use.

Using the provider

Integrations

Reference

Development