Parsl

Quiet but deadly Python HPC workflow manager

2025-11-17 — 2025-11-21

Wherein Parsl is described as a Python‑native workflow engine whose DAG is built at runtime and whose Slurm provider is shown to enable scaling from laptop to cluster while methods for unwrapping worker errors are given.

computers are awful

computers are awful together

concurrency hell

premature optimization

python

1 The API

2 How Cluster Execution Works

3 Python Configuration

4 Local execution

5 Providers

6 Debugging Parsl workers

6.1 Error Extraction Wrapper

6.2 Force worker logs to a visible directory

6.3 Instrument Entry Points

7 Common Parsl Worker Errors

7.1 Error: “Dependency failure for task N”

7.2 Error: “ModuleNotFoundError” in worker

7.3 Error: “PickleError: Can’t pickle

7.4 Error: Worker crashes silently

7.5 Worker Error Catching Variations

7.5.1 Variation 1: Accumulate Errors and Continue

7.5.2 Variation 2: Retry Failed Tasks

7.5.3 Variation 3: Extract to Structured Log

7.6 How this all works

7.7 Parsl’s Exception Wrapping Mechanism

Parsl