# Parsl Ephemeral Provider Architecture

This document provides an overview of the architecture and design of the Parsl Ephemeral Provider.

## Overview

The Parsl Ephemeral Provider runs Parsl workflows on AWS compute that is
created when work arrives and destroyed when it finishes.

**Networking is not ephemeral.** Since v0.7.0 the VPC, subnet, and security group
are supplied by you and never created or deleted by the provider — see
[network-prerequisites.md](network-prerequisites.md). Ephemerality applies to
compute (instances, fleets, Lambda functions, ECS tasks) and to the launch
templates, IAM instance profiles, and CloudFormation stacks the provider creates
to run it.

## Key components

```
parsl_ephemeral_provider/
├── provider.py                 # EphemeralProvider — the Parsl interface
├── globus_compute.py           # EphemeralComputeProvider subclass
├── constants.py                # AWS constants and defaults
├── exceptions.py               # Exception hierarchy
├── error_handling.py           # Retry/backoff and polling framework
├── modes/
│   ├── base.py                 # OperatingMode interface
│   ├── standard.py             # Direct client-to-worker
│   ├── detached.py             # Bastion orchestrates; client may disconnect
│   └── serverless.py           # Lambda / ECS-Fargate workers
├── compute/
│   ├── spot_fleet.py           # EC2 Fleet (migrated off Spot Fleet in v0.7.0)
│   ├── spot_fleet_cleanup.py
│   ├── spot_interruption.py    # EventBridge → SQS interruption warnings
│   ├── lambda_func.py
│   └── ecs.py
├── state/
│   ├── base.py                 # Keyed store interface
│   ├── file.py, s3.py, parameter_store.py
├── security/                   # Audit logging, credentials, policy helpers
├── config/security_config.py
├── utils/aws.py                # Session, AMI resolution, tagging, waiters
└── templates/cloudformation/   # bastion, ec2_worker, lambda_worker, ecs_worker
```

`network/`, `compute/ec2.py`, `utils/logging.py`, and `templates/terraform/` were
removed in v0.9.0
([#90](https://github.com/scttfrdmn/parsl-ephemeral-provider/issues/90)) — 2,746
LOC that no package module imported. The modes call boto3 directly and #69 removed
VPC creation outright, so `EC2Manager`, `VPCManager` and `SecurityGroupManager`
routed nothing; their only callers were their own tests.

`security/` was *not* touched. Its modules are exported from `security/__init__.py`
and covered, so removing them would be a public-API break — #90's original body was
wrong to list `encryption.py` as unreferenced.

## Operating modes

See [operating_modes.md](operating_modes.md) for configuration. In outline:

### Standard mode

The client communicates directly with workers over ZMQ. Suitable when the client
has a stable, reachable address — workers connect *outbound* to the interchange,
so the client must accept inbound TCP on the HTEX port range. A client behind NAT
cannot use this mode without port forwarding or a VPN; use detached mode instead.

1. `initialize()` creates a launch template (IMDSv2 required) and, if asked, an
   IAM instance profile.
2. Each `submit()` launches instances or an EC2 Fleet into your subnet.
3. Workers connect back to the interchange on the client.
4. `cancel()` and shutdown terminate instances and delete the launch template.

### Detached mode

A bastion instance runs an orchestrator loop and owns the worker lifecycle, so the
client can disconnect entirely. This suits long-running workflows and clients
behind NAT.

1. The client launches a bastion from a CloudFormation stack.
2. The bastion polls for pending jobs, launches workers, tracks status, and
   handles cancellations.
3. The client may disconnect; the workflow continues.
4. The bastion shuts down after `idle_timeout` minutes with no work.

The bastion is an autonomous orchestrator, not a network tunnel — which is why an
EC2 Instance Connect Endpoint cannot replace it
([#88](https://github.com/scttfrdmn/parsl-ephemeral-provider/issues/88)).

### Serverless mode

Tasks run on Lambda or ECS/Fargate with no EC2 instances. Best for short,
sporadic tasks. Lambda workers run in the Lambda-managed VPC and therefore need
none of the three network IDs; ECS/Fargate needs a subnet and security group.

## Resource management

The provider creates and manages:

- **Compute**: EC2 instances, EC2 Fleets, Lambda functions, ECS tasks
- **Launch templates**: one per provider, carrying IMDSv2, shutdown behaviour,
  and the instance profile
- **IAM instance profile and role**: only when `auto_create_instance_profile=True`,
  and deleted on shutdown since v0.8.0
  ([#132](https://github.com/scttfrdmn/parsl-ephemeral-provider/issues/132)). A profile
  you supply through `iam_instance_profile_arn` is never touched.
- **CloudFormation stacks**: bastion (detached), Lambda and ECS workers
  (serverless)
- **State storage**: a local file, an S3 object, or an SSM parameter

It does **not** create or delete VPCs, subnets, or security groups.

EC2 resources are tagged so anything left behind is findable, in one of two
conventions depending on the mode: standard mode writes
`CreatedBy=ParslEphemeralProvider` and `ProviderId=<provider_id>`, while
detached mode and the serverless fleet write `ParslResource=true` and
`ParslWorkflowId=<provider_id>`. `parsl-ephemeral-cleanup --dry-run` queries both and
reports the union.

## State management

State persistence supports resource tracking across sessions, detached-mode
handoff, and cleanup after a crash.

Three backends, selected with `state_store_type`:

1. **`file`** (default) — local JSON, `fcntl` locked
2. **`s3`** — an S3 object; requires `s3_bucket`
3. **`parameter_store`** — an SSM parameter

All three are **keyed**: the provider writes under `"provider"` and the operating
mode under `"mode"`, so the two no longer overwrite each other's document (the
v0.6.0 defect that leaked baked AMIs and lost `job_map`). See
[state_persistence.md](state_persistence.md).

## Infrastructure as code

CloudFormation templates ship inside the wheel and are loaded with
`get_cf_template()`, not by filesystem path — a wheel install has no source tree.
CloudFormation is the only IaC surface: the unused Terraform modules under
`templates/terraform/` were removed in v0.9.0
([#90](https://github.com/scttfrdmn/parsl-ephemeral-provider/issues/90)).

## Error handling and recovery

- A custom exception hierarchy in `exceptions.py`
- Cleanup on initialization failure, scoped to resources the provider created
- Exponential backoff with jitter for transient AWS API errors, via
  `error_handling.py`

`error_handling.py` offers two shapes, and which one applies depends on how the
failure presents:

- **`retry_with_backoff`** wraps a call that *raises*. Used throughout the
  `compute/` managers.
- **`poll_until`** waits on a call that *succeeds* and returns a not-yet answer —
  an instance absent from SSM, a fleet block short of `running`. Used by
  `StandardMode`'s three readiness waits. A retry decorator cannot express this:
  nothing is thrown, so it would fire zero times and return the not-yet answer as
  the result. Added in v0.10.0
  ([#91](https://github.com/scttfrdmn/parsl-ephemeral-provider/issues/91)), which
  replaced the modes' hand-rolled flat-interval polling loops.

Both take a `RetryConfig` for the delay schedule, so a poll and a retry back off
and jitter identically. `poll_until` ignores its `max_attempts` — a poll is bounded
by wall clock, since a slow boot legitimately takes dozens of attempts.

`utils/aws.py:wait_for_resource` occupies adjacent ground but is not
interchangeable: it dispatches to *named* boto3 waiters, so it covers only states
AWS itself publishes a waiter for. Derived states — "this fleet block is running",
which `SpotFleetManager` computes from instance states — need `poll_until`.

The two `time.sleep` calls in `modes/detached.py` are inside the generated bastion
script's string literal, which runs on the bastion with no access to this package.

## Security considerations

- Least-privilege IAM: `EphemeralComputeProvider.minimum_iam_policy()` returns the
  actual action set the provider uses
- IMDSv2 required on every launch path, set in the launch template
- No long-lived credentials on instances — workers use an instance profile
- Resource isolation by provider ID in tags and state keys

Instance profiles created with `auto_create_instance_profile=True` are deleted on
shutdown since v0.8.0
([#132](https://github.com/scttfrdmn/parsl-ephemeral-provider/issues/132)). Deletion is
gated on ownership, so a profile supplied through `iam_instance_profile_arn`
survives — supply your own ARN to control that lifecycle yourself. Grant the
teardown actions in your IAM policy, or cleanup fails silently and the roles
accumulate ([#195](https://github.com/scttfrdmn/parsl-ephemeral-provider/issues/195)).

## Testing with a local AWS emulator

For testing AWS interactions without real AWS resources, the suite runs against
[substrate](https://github.com/scttfrdmn/substrate), a local AWS emulator. See
[substrate_testing.md](substrate_testing.md); it replaced LocalStack in #125.

SPDX-License-Identifier: Apache-2.0
SPDX-FileCopyrightText: 2025-2026 Scott Friedman and Project Contributors