Parsl Ephemeral Provider Architecture

This document provides an overview of the architecture and design of the Parsl Ephemeral Provider.

Overview

The Parsl Ephemeral Provider runs Parsl workflows on AWS compute that is created when work arrives and destroyed when it finishes.

Networking is not ephemeral. Since v0.7.0 the VPC, subnet, and security group are supplied by you and never created or deleted by the provider — see network-prerequisites.md. Ephemerality applies to compute (instances, fleets, Lambda functions, ECS tasks) and to the launch templates, IAM instance profiles, and CloudFormation stacks the provider creates to run it.

Key components

parsl_ephemeral_provider/
├── provider.py                 # EphemeralProvider — the Parsl interface
├── globus_compute.py           # EphemeralComputeProvider subclass
├── constants.py                # AWS constants and defaults
├── exceptions.py               # Exception hierarchy
├── error_handling.py           # Retry/backoff and polling framework
├── modes/
│   ├── base.py                 # OperatingMode interface
│   ├── standard.py             # Direct client-to-worker
│   ├── detached.py             # Bastion orchestrates; client may disconnect
│   └── serverless.py           # Lambda / ECS-Fargate workers
├── compute/
│   ├── spot_fleet.py           # EC2 Fleet (migrated off Spot Fleet in v0.7.0)
│   ├── spot_fleet_cleanup.py
│   ├── spot_interruption.py    # EventBridge → SQS interruption warnings
│   ├── lambda_func.py
│   └── ecs.py
├── state/
│   ├── base.py                 # Keyed store interface
│   ├── file.py, s3.py, parameter_store.py
├── security/                   # Audit logging, credentials, policy helpers
├── config/security_config.py
├── utils/aws.py                # Session, AMI resolution, tagging, waiters
└── templates/cloudformation/   # bastion, ec2_worker, lambda_worker, ecs_worker

network/, compute/ec2.py, utils/logging.py, and templates/terraform/ were removed in v0.9.0 (#90) — 2,746 LOC that no package module imported. The modes call boto3 directly and #69 removed VPC creation outright, so EC2Manager, VPCManager and SecurityGroupManager routed nothing; their only callers were their own tests.

security/ was not touched. Its modules are exported from security/__init__.py and covered, so removing them would be a public-API break — #90’s original body was wrong to list encryption.py as unreferenced.

Operating modes

See operating_modes.md for configuration. In outline:

Standard mode

The client communicates directly with workers over ZMQ. Suitable when the client has a stable, reachable address — workers connect outbound to the interchange, so the client must accept inbound TCP on the HTEX port range. A client behind NAT cannot use this mode without port forwarding or a VPN; use detached mode instead.

  1. initialize() creates a launch template (IMDSv2 required) and, if asked, an IAM instance profile.

  2. Each submit() launches instances or an EC2 Fleet into your subnet.

  3. Workers connect back to the interchange on the client.

  4. cancel() and shutdown terminate instances and delete the launch template.

Detached mode

A bastion instance runs an orchestrator loop and owns the worker lifecycle, so the client can disconnect entirely. This suits long-running workflows and clients behind NAT.

  1. The client launches a bastion from a CloudFormation stack.

  2. The bastion polls for pending jobs, launches workers, tracks status, and handles cancellations.

  3. The client may disconnect; the workflow continues.

  4. The bastion shuts down after idle_timeout minutes with no work.

The bastion is an autonomous orchestrator, not a network tunnel — which is why an EC2 Instance Connect Endpoint cannot replace it (#88).

Serverless mode

Tasks run on Lambda or ECS/Fargate with no EC2 instances. Best for short, sporadic tasks. Lambda workers run in the Lambda-managed VPC and therefore need none of the three network IDs; ECS/Fargate needs a subnet and security group.

Resource management

The provider creates and manages:

  • Compute: EC2 instances, EC2 Fleets, Lambda functions, ECS tasks

  • Launch templates: one per provider, carrying IMDSv2, shutdown behaviour, and the instance profile

  • IAM instance profile and role: only when auto_create_instance_profile=True, and deleted on shutdown since v0.8.0 (#132). A profile you supply through iam_instance_profile_arn is never touched.

  • CloudFormation stacks: bastion (detached), Lambda and ECS workers (serverless)

  • State storage: a local file, an S3 object, or an SSM parameter

It does not create or delete VPCs, subnets, or security groups.

EC2 resources are tagged so anything left behind is findable, in one of two conventions depending on the mode: standard mode writes CreatedBy=ParslEphemeralProvider and ProviderId=<provider_id>, while detached mode and the serverless fleet write ParslResource=true and ParslWorkflowId=<provider_id>. parsl-ephemeral-cleanup --dry-run queries both and reports the union.

State management

State persistence supports resource tracking across sessions, detached-mode handoff, and cleanup after a crash.

Three backends, selected with state_store_type:

  1. file (default) — local JSON, fcntl locked

  2. s3 — an S3 object; requires s3_bucket

  3. parameter_store — an SSM parameter

All three are keyed: the provider writes under "provider" and the operating mode under "mode", so the two no longer overwrite each other’s document (the v0.6.0 defect that leaked baked AMIs and lost job_map). See state_persistence.md.

Infrastructure as code

CloudFormation templates ship inside the wheel and are loaded with get_cf_template(), not by filesystem path — a wheel install has no source tree. CloudFormation is the only IaC surface: the unused Terraform modules under templates/terraform/ were removed in v0.9.0 (#90).

Error handling and recovery

  • A custom exception hierarchy in exceptions.py

  • Cleanup on initialization failure, scoped to resources the provider created

  • Exponential backoff with jitter for transient AWS API errors, via error_handling.py

error_handling.py offers two shapes, and which one applies depends on how the failure presents:

  • retry_with_backoff wraps a call that raises. Used throughout the compute/ managers.

  • poll_until waits on a call that succeeds and returns a not-yet answer — an instance absent from SSM, a fleet block short of running. Used by StandardMode’s three readiness waits. A retry decorator cannot express this: nothing is thrown, so it would fire zero times and return the not-yet answer as the result. Added in v0.10.0 (#91), which replaced the modes’ hand-rolled flat-interval polling loops.

Both take a RetryConfig for the delay schedule, so a poll and a retry back off and jitter identically. poll_until ignores its max_attempts — a poll is bounded by wall clock, since a slow boot legitimately takes dozens of attempts.

utils/aws.py:wait_for_resource occupies adjacent ground but is not interchangeable: it dispatches to named boto3 waiters, so it covers only states AWS itself publishes a waiter for. Derived states — “this fleet block is running”, which SpotFleetManager computes from instance states — need poll_until.

The two time.sleep calls in modes/detached.py are inside the generated bastion script’s string literal, which runs on the bastion with no access to this package.

Security considerations

  • Least-privilege IAM: EphemeralComputeProvider.minimum_iam_policy() returns the actual action set the provider uses

  • IMDSv2 required on every launch path, set in the launch template

  • No long-lived credentials on instances — workers use an instance profile

  • Resource isolation by provider ID in tags and state keys

Instance profiles created with auto_create_instance_profile=True are deleted on shutdown since v0.8.0 (#132). Deletion is gated on ownership, so a profile supplied through iam_instance_profile_arn survives — supply your own ARN to control that lifecycle yourself. Grant the teardown actions in your IAM policy, or cleanup fails silently and the roles accumulate (#195).

Testing with a local AWS emulator

For testing AWS interactions without real AWS resources, the suite runs against substrate, a local AWS emulator. See substrate_testing.md; it replaced LocalStack in #125.

SPDX-License-Identifier: Apache-2.0 SPDX-FileCopyrightText: 2025-2026 Scott Friedman and Project Contributors