State Persistence

The provider persists its state so that resources can be tracked across sessions, handed to a bastion in detached mode, and cleaned up after a crash.

Overview

State persistence matters because:

Every AWS resource the provider creates is recorded, so nothing is orphaned.
A restarted provider can adopt the resources a previous run left behind instead of duplicating them.
Detached mode needs state the bastion and the client can both read.

Selecting a store

You do not construct a store yourself. Choose one with state_store_type and the provider builds it, wiring in its own session, region, credentials, and audit logger:

from parsl_ephemeral_provider import EphemeralProvider

provider = EphemeralProvider(
    region="us-east-1",
    vpc_id="vpc-0123456789abcdef0",
    subnet_id="subnet-0123456789abcdef0",
    security_group_id="sg-0123456789abcdef0",
    state_store_type="file",  # "file" | "s3" | "parameter_store"
    state_file_path="./aws_provider_state.json",  # used by "file"
)

state_store_type accepts a string or a StateStoreType enum member (from parsl_ephemeral_provider.provider import StateStoreType).

File store (default)

State lives in a local JSON file, written under an fcntl lock.

provider = EphemeralProvider(
    # ... network and compute options ...
    state_store_type="file",
    state_file_path="./aws_provider_state.json",
)

Good for local development and standard mode. Not usable from a bastion, and the state is lost with the machine.

Parameter Store

State lives in an AWS Systems Manager parameter, readable from both the client and the bastion — the reason detached mode prefers it.

provider = EphemeralProvider(
    # ... network and compute options ...
    state_store_type="parameter_store",
    parameter_store_path="/parsl/my-workflow-state",
)

Requires ssm:PutParameter, ssm:GetParameter, and ssm:DeleteParameter. Standard parameters cap at 4 KB, which a workflow with very many tracked resources can reach; use S3 if you expect that.

S3

provider = EphemeralProvider(
    # ... network and compute options ...
    state_store_type="s3",
    s3_bucket="my-parsl-state-bucket",
    s3_key="my-workflow/state.json",
)

s3_bucket is mandatory for this backend — omitting it raises ProviderConfigurationError. By default the bucket must already exist; a missing one is an error rather than something the provider quietly provisions.

Pass s3_create_bucket=True to have the provider create it when it is absent:

provider = EphemeralProvider(
    # ... network and compute options ...
    state_store_type="s3",
    s3_bucket="my-parsl-state-bucket",
    s3_create_bucket=True,
)

Creation also applies a full public-access block and tags the bucket ParslManagedBucket=true, so it is identifiable later. An existing bucket is adopted, not recreated, which is what makes the resume path work — every restart constructs the store again. Requires s3:CreateBucket, s3:PutBucketPublicAccessBlock, and s3:PutBucketTagging on top of the object permissions.

Note that the provider never deletes a bucket it created — no cleanup path calls S3State.delete_bucket_if_empty(), which exists for callers who want it explicitly. State outliving a run is the point of a state store, and an empty bucket costs nothing to keep, so the ownership-tracking that _owns_baked_ami does for AMIs would buy little here.

Recommended store by mode

Mode	Store	Reason
Standard	`file`	Simplest; the client is the only reader
Detached	`parameter_store`	Readable by both client and bastion
Serverless	`s3`	Durable and reachable from Lambda or Fargate

The store interface

All three stores implement the same keyed interface from parsl_ephemeral_provider.state.base.StateStore:

save_state(state_key: str, state_data: Dict[str, Any]) -> None
load_state(state_key: str) -> Optional[Dict[str, Any]]
delete_state(state_key: str) -> None

Keys matter. The provider writes under "provider" and the operating mode under "mode". Before v0.7.0 both wrote a full overwrite to the same slot with different key sets, so each destroyed the other’s fields — which lost job_map and, worse, the baked-AMI ID and its ownership flag, leaking an AMI and its EBS snapshots on every shutdown.

State files written by v0.6.0 and earlier hold a single flat document with no _states wrapper. Those still load: the flat document is offered under every key, each reader takes the fields it recognises, and the first write upgrades the file.

State contents

Under the "provider" key:

provider_id, mode, timestamp
resources — tracked resource IDs and their metadata
job_map — job ID to resource mapping
warm_instances — warm-pool members, in standard mode

Under the "mode" key, mode-specific fields: the validated vpc_id, subnet_id, security_group_id, initialized, the launch template ID, and for standard mode baked_ami_id / owns_baked_ami; for detached mode bastion_id.

{
  "_version": 2,
  "_states": {
    "provider": {
      "provider_id": "8f7e3a4d",
      "mode": "standard",
      "job_map": {"1": {"resource_id": "res-1"}},
      "resources": {"res-1": {"instance_id": "i-087654321", "status": "RUNNING"}}
    },
    "mode": {
      "vpc_id": "vpc-0123456789abcdef0",
      "subnet_id": "subnet-0123456789abcdef0",
      "security_group_id": "sg-0123456789abcdef0",
      "initialized": true
    }
  }
}

Recovery

Recovery is automatic. EphemeralProvider.__init__ loads whatever is stored at the configured location, and if the state records a provider_id and you did not pass one, that ID is adopted — so a second provider pointed at the same state location continues the first one’s work rather than starting a parallel run:

# First run
provider = EphemeralProvider(
    mode="detached",
    state_store_type="parameter_store",
    parameter_store_path="/parsl/my-workflow-state",
    vpc_id="vpc-0123456789abcdef0",
    subnet_id="subnet-0123456789abcdef0",
    security_group_id="sg-0123456789abcdef0",
)

# Later, in a new process: same store and path, so the bastion, tracked jobs,
# and any baked AMI are picked back up.
provider = EphemeralProvider(
    mode="detached",
    state_store_type="parameter_store",
    parameter_store_path="/parsl/my-workflow-state",
    vpc_id="vpc-0123456789abcdef0",
    subnet_id="subnet-0123456789abcdef0",
    security_group_id="sg-0123456789abcdef0",
)

Network IDs are re-validated on load. If a recorded resource has gone, ResourceNotFoundError is raised rather than the ID being silently blanked — the pre-v0.7.0 behaviour, which produced an opaque boto3 error later.

Cleanup

provider.shutdown()

shutdown() terminates tracked compute, deletes the launch template, and deletes the state entry. To remove state without touching resources:

provider.state_store.delete_state("provider")
provider.state_store.delete_state("mode")

Custom stores

Subclass StateStore and implement the three keyed methods:

from typing import Any, Dict, Optional

from parsl_ephemeral_provider.state.base import StateStore

class MyCustomStateStore(StateStore):
    def __init__(self, connection_string: str) -> None:
        self.connection_string = connection_string

    def save_state(self, state_key: str, state_data: Dict[str, Any]) -> None: ...

    def load_state(self, state_key: str) -> Optional[Dict[str, Any]]: ...

    def delete_state(self, state_key: str) -> None: ...

load_state must return None for a key that has never been written — the provider treats that as “first run”, not as an error.

There is no configuration hook for injecting a custom store; assign it after construction (provider.state_store = MyCustomStateStore(...)) before the first submit().

Notes

Keep one state location per workflow. Two concurrent providers sharing a location will adopt each other’s provider_id and fight over the same resources.
Parameter Store operations emit STATE_ACCESS audit events when the provider has an audit_logger.
After any crash, run parsl-ephemeral-cleanup --dry-run --region <region> to check for resources the state no longer names.

SPDX-License-Identifier: Apache-2.0 SPDX-FileCopyrightText: 2025-2026 Scott Friedman and Project Contributors