docs: add explanatory markdown to example notebooks

documentation/conclusion.md

@@ -1,3 +1,17 @@
 # Conclusion 
 Based on the Python Workflow Definition three rather different workflows were implemented in rather different workflow 
-engines. This demonstrates the interoperability achieved with the Python Workflow Definition. 
+engines: a simple arithmetic example coupling two Python functions, a Quantum Espresso energy volume curve calculation 
+with fan-out parallelism, and a file based, multi-tool NFDI4Ing benchmark. In each case the same `workflow.json` was 
+written by one workflow engine and successfully loaded and executed by the others, without any change to the underlying 
+Python functions in `workflow.py`. This demonstrates the interoperability achieved with the Python Workflow Definition, 
+and shows that it scales from small, in-memory Python workflows to larger, file based, multi-environment scientific 
+workflows.
+
+## Where to go next
+* Browse the [example_workflows](https://github.com/pythonworkflow/python-workflow-definition/tree/main/example_workflows) 
+  directory for the full set of notebooks, including engines not covered in this book such as `pyiron_workflow`, 
+  `executorlib` and CWL.
+* See the [README](https://github.com/pythonworkflow/python-workflow-definition) for installation instructions via 
+  `pip` or `conda`.
+* Read the accompanying publication: [J. Janssen et al., A python workflow definition for computational materials 
+  design, Digital Discovery, 2025](https://doi.org/10.1039/D5DD00231A).

documentation/evcurve.md

@@ -1,4 +1,67 @@
 # Energy Volume Curve
 Based on [previous work](https://materialdigital.github.io/ADIS2023/README.html) from the [ADIS 2023 workshop](https://www.mpie.de/4902385/adis2023)
 the calculation of an energy volume curve with the [quantum espresso](https://www.quantum-espresso.org) density 
-functional theory (DFT) simulation code is implemented in the Python Workflow Definition.
+functional theory (DFT) simulation code is implemented in the Python Workflow Definition.
+
+## Workflow
+An energy-volume curve is computed by relaxing a bulk crystal structure, straining it to a series of volumes and
+computing the total energy at each volume with a self-consistent-field (SCF) calculation. The pipeline is implemented
+as five Python functions in [workflow.py](example_workflows/quantum_espresso/workflow.py):
+```python
+def get_bulk_structure(element, a, cubic):
+    # build an ASE bulk crystal structure, e.g. Al
+
+def calculate_qe(working_directory, input_dict):
+    # write a pw.x input file, run "pw.x -in input.pwi > output.pwo" and parse the pwscf.xml output;
+    # input_dict["calculation"] selects "vc-relax" (cell + geometry relaxation) or "scf" (single-point energy)
+
+def generate_structures(structure, strain_lst):
+    # apply each volumetric strain in strain_lst to a structure, returning one structure per strain
+
+def plot_energy_volume_curve(volume_lst, energy_lst):
+    # plot energy against volume and save it as evcurve.png
+```
+`write_input`/`collect_output` handle the file-based `pw.x` input/output (`.pwi`/`.pwo`/`.xml` files), and
+`ase_to_json`/`json_to_ase` (de)serialize ASE `Atoms` objects to/from OPTIMADE-JSON strings, so structures can be
+passed between functions as plain JSON data instead of Python objects.
+
+The workflow combines these functions as: `get_bulk_structure` -> `calculate_qe` (`vc-relax`, once) -> 
+`generate_structures` -> `calculate_qe` (`scf`, once per strained structure) -> `plot_energy_volume_curve`. With five
+strains this means one relaxation followed by five independent SCF calculations that fan out from
+`generate_structures` and feed back into a single plot.
+
+## workflow.json
+[workflow.json](example_workflows/quantum_espresso/workflow.json) encodes this fan-out as five separate
+`workflow.calculate_qe` function nodes, each connected to its own strained structure and its own `working_directory`
+input, all reading the same shared inputs (`pseudopotentials`, `kpts`, `calculation`, `smearing`) via
+`python_workflow_definition.shared.get_dict`. An excerpt showing the pattern for two of the five strained
+calculations:
+```
+{
+  "nodes": [
+    {"id": 2, "type": "function", "value": "workflow.generate_structures"},
+    {"id": 3, "type": "function", "value": "workflow.calculate_qe"},
+    {"id": 4, "type": "function", "value": "workflow.calculate_qe"},
+    {"id": 19, "type": "input", "value": "strain_0", "name": "working_directory_1"},
+    {"id": 20, "type": "function", "value": "python_workflow_definition.shared.get_dict"},
+    {"id": 22, "type": "input", "value": "strain_1", "name": "working_directory_2"},
+    {"id": 23, "type": "function", "value": "python_workflow_definition.shared.get_dict"}
+  ],
+  "edges": [
+    {"target": 3, "targetPort": "working_directory", "source": 19, "sourcePort": null},
+    {"target": 20, "targetPort": "structure", "source": 2, "sourcePort": "s_0"},
+    {"target": 3, "targetPort": "input_dict", "source": 20, "sourcePort": null},
+    {"target": 4, "targetPort": "working_directory", "source": 22, "sourcePort": null},
+    {"target": 23, "targetPort": "structure", "source": 2, "sourcePort": "s_1"},
+    {"target": 4, "targetPort": "input_dict", "source": 23, "sourcePort": null}
+  ]
+}
+```
+Node 2 (`generate_structures`) exposes one output port per strain (`s_0`, `s_1`, ...); each port is wired into its
+own `get_dict`/`calculate_qe` pair. The energy and volume outputs of all five `calculate_qe` nodes are collected back
+into lists with `python_workflow_definition.shared.get_list` before being passed to `plot_energy_volume_curve`.
+
+## Requirements
+Running this workflow (rather than just loading/inspecting the JSON) requires the Quantum Espresso `pw.x` binary,
+matching pseudopotential files (see [espresso/pseudo](example_workflows/quantum_espresso/espresso/pseudo)) and the
+Python dependencies listed in [environment.yml](example_workflows/quantum_espresso/environment.yml).

documentation/intro.md

@@ -13,6 +13,9 @@ Currently supported workflow engines:
 * [aiida-workgraph](https://github.com/aiidateam/aiida-workgraph)
 * [jobflow](https://github.com/materialsproject/jobflow)
 * [pyiron_base](https://github.com/pyiron/pyiron_base)
+* [pyiron_workflow](https://github.com/pyiron/pyiron_workflow)
+* [executorlib](https://github.com/pyiron/executorlib)
+* [CWL](https://www.commonwl.org/) via [cwltool](https://github.com/common-workflow-language/cwltool)
 
 ## Example Workflows
 Three workflows are implemented:

documentation/nfdi.md

@@ -1,4 +1,61 @@
 # NFDI4Ing Benchmark
-To demonstrate the compatibility of the Python Workflow Definition to file based workflows, the workflow benchmark 
-developed as part of [NFDI4Ing](https://www.inggrid.org/article/id/3726/) is implemented for all three simulation codes 
-based on the Python Workflow Definition. 
+To demonstrate the compatibility of the Python Workflow Definition to file based workflows, the workflow benchmark
+developed as part of [NFDI4Ing](https://www.inggrid.org/article/id/3726/) is implemented for all workflow engines
+based on the Python Workflow Definition.
+
+## Workflow
+Unlike the [arithmetic](arithmetic.md) and energy-volume-curve examples, which pass Python objects between functions,
+this benchmark chains together external command line tools that each require their own conda environment (defined in
+`source/envs/*.yaml`) and communicate through files rather than in-memory values. The
+pipeline is defined by six Python functions in [workflow.py](example_workflows/nfdi/workflow.py), each of which
+copies its inputs into a stage-specific subdirectory (`preprocessing`, `processing` or `postprocessing`) and calls
+its external tool with `conda_subprocess.check_output`:
+```python
+def generate_mesh(domain_size: float, source_directory: str) -> str: ...
+def convert_to_xdmf(gmsh_output_file: str) -> dict: ...
+def poisson(meshio_output_xdmf: str, meshio_output_h5: str, source_directory: str) -> dict: ...
+def plot_over_line(poisson_output_pvd_file: str, poisson_output_vtu_file: str, source_directory: str) -> str: ...
+def substitute_macros(pvbatch_output_file: str, ndofs: int, domain_size: float, source_directory: str) -> str: ...
+def compile_paper(macros_tex: str, plot_file: str, source_directory: str) -> str: ...
+```
+* `generate_mesh` runs [`gmsh`](https://gmsh.info) on `unit_square.geo` to mesh a 2D square whose side length is set
+  by the `domain_size` parameter, producing a `.msh` file.
+* `convert_to_xdmf` runs `meshio convert` to turn the gmsh mesh into an XDMF/H5 file pair that the FEM solver can read.
+* `poisson` runs a [FEniCS](https://fenicsproject.org)-based `poisson.py` script that solves the Poisson equation on
+  the mesh, returning the number of degrees of freedom (`numdofs`) alongside the `.pvd`/`.vtu` result files.
+* `plot_over_line` runs ParaView's `pvbatch` with `postprocessing.py` to sample the FEM solution along a line and
+  write it out as a CSV file.
+* `substitute_macros` runs `prepare_paper_macros.py` to fill a LaTeX macro template with the plot data path, the
+  domain size and the number of degrees of freedom.
+* `compile_paper` runs [`tectonic`](https://tectonic-typesetting.github.io) to compile `paper.tex`, which references
+  those macros and the plot, into `paper.pdf`.
+
+The connection of these Python functions is stored in the [workflow.json](example_workflows/nfdi/workflow.json)
+JSON file, following the same `nodes`/`edges` structure as the other examples. Each function is a `function` node,
+`domain_size` and `source_directory` are `input` nodes, and edges connect a function's input port either to another
+function's output port or directly to an input node:
+```
+{
+  "version": "0.1.0",
+  "nodes": [
+    {"id": 0, "type": "function", "value": "workflow.generate_mesh"},
+    {"id": 1, "type": "function", "value": "workflow.convert_to_xdmf"},
+    {"id": 6, "type": "input", "value": 2.0, "name": "domain_size"},
+    {"id": 7, "type": "input", "value": "source", "name": "source_directory"},
+    {"id": 8, "type": "output", "name": "result"}
+  ],
+  "edges": [
+    {"target": 0, "targetPort": "domain_size", "source": 6, "sourcePort": null},
+    {"target": 0, "targetPort": "source_directory", "source": 7, "sourcePort": null},
+    {"target": 1, "targetPort": "gmsh_output_file", "source": 0, "sourcePort": null}
+  ]
+}
+```
+Since `convert_to_xdmf` and `poisson` return a `dict` instead of a single value, the edges that read their output
+set `sourcePort` to the specific dictionary key (e.g. `"xdmf_file"` or `"numdofs"`) rather than `null`.
+
+Because every stage shells out to a different external tool in a different conda environment and passes file paths
+along the graph, this benchmark exercises a different part of the Python Workflow Definition than the arithmetic and
+energy-volume-curve examples: it shows that the same `workflow.json` produced by one engine can be handed to another
+engine, an execution manager like [CWL](https://www.commonwl.org)/`cwltool`, or plain Python, and still reproduce the
+same chain of `gmsh`, `meshio`, FEniCS, `pvbatch` and `tectonic` calls end to end.

example_workflows/arithmetic/aiida.ipynb

@@ -23,12 +23,12 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "source": "# Aiida",
+   "source": "# Aiida\n\nThis notebook defines the arithmetic workflow with [`aiida-workgraph`](https://github.com/aiidateam/aiida-workgraph) and then loads the resulting `workflow.json` into `jobflow`, `pyiron_base` and `pyiron_workflow`, to demonstrate that the same workflow definition can be executed by several different engines.",
    "metadata": {}
   },
   {
    "cell_type": "markdown",
-   "source": "## Define workflow with aiida",
+   "source": "## Define workflow with aiida\n\n`aiida-workgraph` represents a workflow as a `WorkGraph` of `Task`s. Tasks are executed and stored by the AiiDA engine, so we first need to connect to an AiiDA profile with `load_profile()` before any task can run.",
    "metadata": {}
   },
   {
@@ -63,6 +63,11 @@
    "outputs": [],
    "execution_count": 2
   },
+  {
+   "cell_type": "markdown",
+   "source": "The arithmetic functions are imported under a leading underscore because they are still plain Python functions at this point; `wg.add_task` below turns each one into an AiiDA `Task` that can be linked to other tasks' inputs and outputs. `get_prod_and_div` returns a dictionary, so it is wrapped with `task(outputs=['prod', 'div'])` to expose `prod` and `div` as separate output sockets.",
+   "metadata": {}
+  },
   {
    "cell_type": "code",
    "source": "wg = WorkGraph(\"arithmetic\")",
@@ -99,6 +104,11 @@
    "outputs": [],
    "execution_count": 6
   },
+  {
+   "cell_type": "markdown",
+   "source": "With all tasks connected, `write_workflow_json` walks the `WorkGraph`'s tasks and links and serializes them into the PWD `workflow.json` format: each task becomes a `function` node and each link becomes an edge between an input and an output port.",
+   "metadata": {}
+  },
   {
    "cell_type": "code",
    "source": "write_workflow_json(wg=wg, file_name=workflow_json_filename)",
@@ -125,7 +135,7 @@
   },
   {
    "cell_type": "markdown",
-   "source": "## Load Workflow with jobflow",
+   "source": "## Load Workflow with jobflow\n\nThe same `workflow.json` can now be loaded by a different engine. `python_workflow_definition.jobflow.load_workflow_json` reconstructs a jobflow `Flow` from the JSON, which is then executed locally with `run_locally`.",
    "metadata": {}
   },
   {
@@ -180,7 +190,7 @@
   },
   {
    "cell_type": "markdown",
-   "source": "## Load Workflow with pyiron_base",
+   "source": "## Load Workflow with pyiron_base\n\nThe same JSON file is loaded into `pyiron_base`, producing a list of delayed jobs. `.draw()` visualizes the dependency graph and `.pull()` triggers delayed execution, running each job in turn and returning the final result.",
    "metadata": {}
   },
   {
@@ -236,7 +246,7 @@
   {
    "metadata": {},
    "cell_type": "markdown",
-   "source": "## Load Workflow with pyiron_workflow"
+   "source": "## Load Workflow with pyiron_workflow\n\nFinally, the workflow is loaded into `pyiron_workflow`, producing a `Workflow` object whose node graph can be visualized with `.draw()` and executed with `.run()`."
   },
   {
    "metadata": {},
@@ -267,4 +277,4 @@
    "source": "wf.run()"
   }
  ]
-}
+}