From db751acc01c91a874164c5e279f60ed3af4839d4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jan=20Jan=C3=9Fen?= Date: Sun, 26 Jul 2026 19:53:07 +0200 Subject: [PATCH] docs: add explanatory markdown to example notebooks The example notebooks under example_workflows/ had almost no prose, only code cells, which made the documentation hard to follow for new users. Add markdown cells explaining each workflow engine, what each step of the notebook does, and how the "Load Workflow with X" sections demonstrate interoperability, without changing any existing code cell. Also expand documentation/evcurve.md and documentation/nfdi.md (which were a few sentences each) to walk through the workflow.py functions and workflow.json structure the same way documentation/arithmetic.md already did, and lightly extend intro.md/conclusion.md. Co-Authored-By: Claude Sonnet 5 --- documentation/conclusion.md | 16 +- documentation/evcurve.md | 65 ++- documentation/intro.md | 3 + documentation/nfdi.md | 63 ++- example_workflows/arithmetic/aiida.ipynb | 22 +- example_workflows/arithmetic/cwl.ipynb | 28 +- .../arithmetic/executorlib.ipynb | 32 +- .../arithmetic/pyiron_base.ipynb | 18 +- .../arithmetic/pyiron_workflow.ipynb | 28 +- .../arithmetic/universal_workflow.ipynb | 18 +- example_workflows/nfdi/aiida.ipynb | 30 +- example_workflows/nfdi/cwl.ipynb | 20 +- example_workflows/nfdi/executorlib.ipynb | 34 +- example_workflows/nfdi/jobflow.ipynb | 24 +- example_workflows/nfdi/pyiron_base.ipynb | 24 +- example_workflows/nfdi/pyiron_workflow.ipynb | 429 +++++++++++++++++- .../nfdi/universal_workflow.ipynb | 18 +- .../quantum_espresso/aiida.ipynb | 45 +- example_workflows/quantum_espresso/cwl.ipynb | 34 +- .../quantum_espresso/executorlib.ipynb | 32 +- .../quantum_espresso/jobflow.ipynb | 16 +- .../quantum_espresso/pyiron_base.ipynb | 40 +- .../quantum_espresso/pyiron_workflow.ipynb | 55 ++- .../quantum_espresso/universal_workflow.ipynb | 18 +- 24 files changed, 902 insertions(+), 210 deletions(-) diff --git a/documentation/conclusion.md b/documentation/conclusion.md index f0e6582c..5afbcb49 100644 --- a/documentation/conclusion.md +++ b/documentation/conclusion.md @@ -1,3 +1,17 @@ # Conclusion Based on the Python Workflow Definition three rather different workflows were implemented in rather different workflow -engines. This demonstrates the interoperability achieved with the Python Workflow Definition. \ No newline at end of file +engines: a simple arithmetic example coupling two Python functions, a Quantum Espresso energy volume curve calculation +with fan-out parallelism, and a file based, multi-tool NFDI4Ing benchmark. In each case the same `workflow.json` was +written by one workflow engine and successfully loaded and executed by the others, without any change to the underlying +Python functions in `workflow.py`. This demonstrates the interoperability achieved with the Python Workflow Definition, +and shows that it scales from small, in-memory Python workflows to larger, file based, multi-environment scientific +workflows. + +## Where to go next +* Browse the [example_workflows](https://github.com/pythonworkflow/python-workflow-definition/tree/main/example_workflows) + directory for the full set of notebooks, including engines not covered in this book such as `pyiron_workflow`, + `executorlib` and CWL. +* See the [README](https://github.com/pythonworkflow/python-workflow-definition) for installation instructions via + `pip` or `conda`. +* Read the accompanying publication: [J. Janssen et al., A python workflow definition for computational materials + design, Digital Discovery, 2025](https://doi.org/10.1039/D5DD00231A). \ No newline at end of file diff --git a/documentation/evcurve.md b/documentation/evcurve.md index 9c7ac43c..536eec6f 100644 --- a/documentation/evcurve.md +++ b/documentation/evcurve.md @@ -1,4 +1,67 @@ # Energy Volume Curve Based on [previous work](https://materialdigital.github.io/ADIS2023/README.html) from the [ADIS 2023 workshop](https://www.mpie.de/4902385/adis2023) the calculation of an energy volume curve with the [quantum espresso](https://www.quantum-espresso.org) density -functional theory (DFT) simulation code is implemented in the Python Workflow Definition. \ No newline at end of file +functional theory (DFT) simulation code is implemented in the Python Workflow Definition. + +## Workflow +An energy-volume curve is computed by relaxing a bulk crystal structure, straining it to a series of volumes and +computing the total energy at each volume with a self-consistent-field (SCF) calculation. The pipeline is implemented +as five Python functions in [workflow.py](example_workflows/quantum_espresso/workflow.py): +```python +def get_bulk_structure(element, a, cubic): + # build an ASE bulk crystal structure, e.g. Al + +def calculate_qe(working_directory, input_dict): + # write a pw.x input file, run "pw.x -in input.pwi > output.pwo" and parse the pwscf.xml output; + # input_dict["calculation"] selects "vc-relax" (cell + geometry relaxation) or "scf" (single-point energy) + +def generate_structures(structure, strain_lst): + # apply each volumetric strain in strain_lst to a structure, returning one structure per strain + +def plot_energy_volume_curve(volume_lst, energy_lst): + # plot energy against volume and save it as evcurve.png +``` +`write_input`/`collect_output` handle the file-based `pw.x` input/output (`.pwi`/`.pwo`/`.xml` files), and +`ase_to_json`/`json_to_ase` (de)serialize ASE `Atoms` objects to/from OPTIMADE-JSON strings, so structures can be +passed between functions as plain JSON data instead of Python objects. + +The workflow combines these functions as: `get_bulk_structure` -> `calculate_qe` (`vc-relax`, once) -> +`generate_structures` -> `calculate_qe` (`scf`, once per strained structure) -> `plot_energy_volume_curve`. With five +strains this means one relaxation followed by five independent SCF calculations that fan out from +`generate_structures` and feed back into a single plot. + +## workflow.json +[workflow.json](example_workflows/quantum_espresso/workflow.json) encodes this fan-out as five separate +`workflow.calculate_qe` function nodes, each connected to its own strained structure and its own `working_directory` +input, all reading the same shared inputs (`pseudopotentials`, `kpts`, `calculation`, `smearing`) via +`python_workflow_definition.shared.get_dict`. An excerpt showing the pattern for two of the five strained +calculations: +``` +{ + "nodes": [ + {"id": 2, "type": "function", "value": "workflow.generate_structures"}, + {"id": 3, "type": "function", "value": "workflow.calculate_qe"}, + {"id": 4, "type": "function", "value": "workflow.calculate_qe"}, + {"id": 19, "type": "input", "value": "strain_0", "name": "working_directory_1"}, + {"id": 20, "type": "function", "value": "python_workflow_definition.shared.get_dict"}, + {"id": 22, "type": "input", "value": "strain_1", "name": "working_directory_2"}, + {"id": 23, "type": "function", "value": "python_workflow_definition.shared.get_dict"} + ], + "edges": [ + {"target": 3, "targetPort": "working_directory", "source": 19, "sourcePort": null}, + {"target": 20, "targetPort": "structure", "source": 2, "sourcePort": "s_0"}, + {"target": 3, "targetPort": "input_dict", "source": 20, "sourcePort": null}, + {"target": 4, "targetPort": "working_directory", "source": 22, "sourcePort": null}, + {"target": 23, "targetPort": "structure", "source": 2, "sourcePort": "s_1"}, + {"target": 4, "targetPort": "input_dict", "source": 23, "sourcePort": null} + ] +} +``` +Node 2 (`generate_structures`) exposes one output port per strain (`s_0`, `s_1`, ...); each port is wired into its +own `get_dict`/`calculate_qe` pair. The energy and volume outputs of all five `calculate_qe` nodes are collected back +into lists with `python_workflow_definition.shared.get_list` before being passed to `plot_energy_volume_curve`. + +## Requirements +Running this workflow (rather than just loading/inspecting the JSON) requires the Quantum Espresso `pw.x` binary, +matching pseudopotential files (see [espresso/pseudo](example_workflows/quantum_espresso/espresso/pseudo)) and the +Python dependencies listed in [environment.yml](example_workflows/quantum_espresso/environment.yml). \ No newline at end of file diff --git a/documentation/intro.md b/documentation/intro.md index 0b28a3b3..4ad92284 100644 --- a/documentation/intro.md +++ b/documentation/intro.md @@ -13,6 +13,9 @@ Currently supported workflow engines: * [aiida-workgraph](https://github.com/aiidateam/aiida-workgraph) * [jobflow](https://github.com/materialsproject/jobflow) * [pyiron_base](https://github.com/pyiron/pyiron_base) +* [pyiron_workflow](https://github.com/pyiron/pyiron_workflow) +* [executorlib](https://github.com/pyiron/executorlib) +* [CWL](https://www.commonwl.org/) via [cwltool](https://github.com/common-workflow-language/cwltool) ## Example Workflows Three workflows are implemented: diff --git a/documentation/nfdi.md b/documentation/nfdi.md index f1a18cbc..1fa6a863 100644 --- a/documentation/nfdi.md +++ b/documentation/nfdi.md @@ -1,4 +1,61 @@ # NFDI4Ing Benchmark -To demonstrate the compatibility of the Python Workflow Definition to file based workflows, the workflow benchmark -developed as part of [NFDI4Ing](https://www.inggrid.org/article/id/3726/) is implemented for all three simulation codes -based on the Python Workflow Definition. \ No newline at end of file +To demonstrate the compatibility of the Python Workflow Definition to file based workflows, the workflow benchmark +developed as part of [NFDI4Ing](https://www.inggrid.org/article/id/3726/) is implemented for all workflow engines +based on the Python Workflow Definition. + +## Workflow +Unlike the [arithmetic](arithmetic.md) and energy-volume-curve examples, which pass Python objects between functions, +this benchmark chains together external command line tools that each require their own conda environment (defined in +`source/envs/*.yaml`) and communicate through files rather than in-memory values. The +pipeline is defined by six Python functions in [workflow.py](example_workflows/nfdi/workflow.py), each of which +copies its inputs into a stage-specific subdirectory (`preprocessing`, `processing` or `postprocessing`) and calls +its external tool with `conda_subprocess.check_output`: +```python +def generate_mesh(domain_size: float, source_directory: str) -> str: ... +def convert_to_xdmf(gmsh_output_file: str) -> dict: ... +def poisson(meshio_output_xdmf: str, meshio_output_h5: str, source_directory: str) -> dict: ... +def plot_over_line(poisson_output_pvd_file: str, poisson_output_vtu_file: str, source_directory: str) -> str: ... +def substitute_macros(pvbatch_output_file: str, ndofs: int, domain_size: float, source_directory: str) -> str: ... +def compile_paper(macros_tex: str, plot_file: str, source_directory: str) -> str: ... +``` +* `generate_mesh` runs [`gmsh`](https://gmsh.info) on `unit_square.geo` to mesh a 2D square whose side length is set + by the `domain_size` parameter, producing a `.msh` file. +* `convert_to_xdmf` runs `meshio convert` to turn the gmsh mesh into an XDMF/H5 file pair that the FEM solver can read. +* `poisson` runs a [FEniCS](https://fenicsproject.org)-based `poisson.py` script that solves the Poisson equation on + the mesh, returning the number of degrees of freedom (`numdofs`) alongside the `.pvd`/`.vtu` result files. +* `plot_over_line` runs ParaView's `pvbatch` with `postprocessing.py` to sample the FEM solution along a line and + write it out as a CSV file. +* `substitute_macros` runs `prepare_paper_macros.py` to fill a LaTeX macro template with the plot data path, the + domain size and the number of degrees of freedom. +* `compile_paper` runs [`tectonic`](https://tectonic-typesetting.github.io) to compile `paper.tex`, which references + those macros and the plot, into `paper.pdf`. + +The connection of these Python functions is stored in the [workflow.json](example_workflows/nfdi/workflow.json) +JSON file, following the same `nodes`/`edges` structure as the other examples. Each function is a `function` node, +`domain_size` and `source_directory` are `input` nodes, and edges connect a function's input port either to another +function's output port or directly to an input node: +``` +{ + "version": "0.1.0", + "nodes": [ + {"id": 0, "type": "function", "value": "workflow.generate_mesh"}, + {"id": 1, "type": "function", "value": "workflow.convert_to_xdmf"}, + {"id": 6, "type": "input", "value": 2.0, "name": "domain_size"}, + {"id": 7, "type": "input", "value": "source", "name": "source_directory"}, + {"id": 8, "type": "output", "name": "result"} + ], + "edges": [ + {"target": 0, "targetPort": "domain_size", "source": 6, "sourcePort": null}, + {"target": 0, "targetPort": "source_directory", "source": 7, "sourcePort": null}, + {"target": 1, "targetPort": "gmsh_output_file", "source": 0, "sourcePort": null} + ] +} +``` +Since `convert_to_xdmf` and `poisson` return a `dict` instead of a single value, the edges that read their output +set `sourcePort` to the specific dictionary key (e.g. `"xdmf_file"` or `"numdofs"`) rather than `null`. + +Because every stage shells out to a different external tool in a different conda environment and passes file paths +along the graph, this benchmark exercises a different part of the Python Workflow Definition than the arithmetic and +energy-volume-curve examples: it shows that the same `workflow.json` produced by one engine can be handed to another +engine, an execution manager like [CWL](https://www.commonwl.org)/`cwltool`, or plain Python, and still reproduce the +same chain of `gmsh`, `meshio`, FEniCS, `pvbatch` and `tectonic` calls end to end. diff --git a/example_workflows/arithmetic/aiida.ipynb b/example_workflows/arithmetic/aiida.ipynb index 743e787f..bd067d85 100644 --- a/example_workflows/arithmetic/aiida.ipynb +++ b/example_workflows/arithmetic/aiida.ipynb @@ -23,12 +23,12 @@ "cells": [ { "cell_type": "markdown", - "source": "# Aiida", + "source": "# Aiida\n\nThis notebook defines the arithmetic workflow with [`aiida-workgraph`](https://github.com/aiidateam/aiida-workgraph) and then loads the resulting `workflow.json` into `jobflow`, `pyiron_base` and `pyiron_workflow`, to demonstrate that the same workflow definition can be executed by several different engines.", "metadata": {} }, { "cell_type": "markdown", - "source": "## Define workflow with aiida", + "source": "## Define workflow with aiida\n\n`aiida-workgraph` represents a workflow as a `WorkGraph` of `Task`s. Tasks are executed and stored by the AiiDA engine, so we first need to connect to an AiiDA profile with `load_profile()` before any task can run.", "metadata": {} }, { @@ -63,6 +63,11 @@ "outputs": [], "execution_count": 2 }, + { + "cell_type": "markdown", + "source": "The arithmetic functions are imported under a leading underscore because they are still plain Python functions at this point; `wg.add_task` below turns each one into an AiiDA `Task` that can be linked to other tasks' inputs and outputs. `get_prod_and_div` returns a dictionary, so it is wrapped with `task(outputs=['prod', 'div'])` to expose `prod` and `div` as separate output sockets.", + "metadata": {} + }, { "cell_type": "code", "source": "wg = WorkGraph(\"arithmetic\")", @@ -99,6 +104,11 @@ "outputs": [], "execution_count": 6 }, + { + "cell_type": "markdown", + "source": "With all tasks connected, `write_workflow_json` walks the `WorkGraph`'s tasks and links and serializes them into the PWD `workflow.json` format: each task becomes a `function` node and each link becomes an edge between an input and an output port.", + "metadata": {} + }, { "cell_type": "code", "source": "write_workflow_json(wg=wg, file_name=workflow_json_filename)", @@ -125,7 +135,7 @@ }, { "cell_type": "markdown", - "source": "## Load Workflow with jobflow", + "source": "## Load Workflow with jobflow\n\nThe same `workflow.json` can now be loaded by a different engine. `python_workflow_definition.jobflow.load_workflow_json` reconstructs a jobflow `Flow` from the JSON, which is then executed locally with `run_locally`.", "metadata": {} }, { @@ -180,7 +190,7 @@ }, { "cell_type": "markdown", - "source": "## Load Workflow with pyiron_base", + "source": "## Load Workflow with pyiron_base\n\nThe same JSON file is loaded into `pyiron_base`, producing a list of delayed jobs. `.draw()` visualizes the dependency graph and `.pull()` triggers delayed execution, running each job in turn and returning the final result.", "metadata": {} }, { @@ -236,7 +246,7 @@ { "metadata": {}, "cell_type": "markdown", - "source": "## Load Workflow with pyiron_workflow" + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the workflow is loaded into `pyiron_workflow`, producing a `Workflow` object whose node graph can be visualized with `.draw()` and executed with `.run()`." }, { "metadata": {}, @@ -267,4 +277,4 @@ "source": "wf.run()" } ] -} +} \ No newline at end of file diff --git a/example_workflows/arithmetic/cwl.ipynb b/example_workflows/arithmetic/cwl.ipynb index 9912fa92..b0c0f40e 100644 --- a/example_workflows/arithmetic/cwl.ipynb +++ b/example_workflows/arithmetic/cwl.ipynb @@ -21,6 +21,12 @@ "nbformat_minor": 5, "nbformat": 4, "cells": [ + { + "cell_type": "markdown", + "id": "7a93c81a", + "source": "# CWL\n\n[`cwltool`](https://github.com/common-workflow-language/cwltool) is the reference implementation of the [Common Workflow Language (CWL)](https://www.commonwl.org/), a standard for describing command-line based workflows. Unlike the other engines in this repository, CWL has no Python API: workflows are described in YAML/JSON documents and executed by the external `cwltool` command. This notebook converts the arithmetic `workflow.json` into an equivalent set of CWL files and executes them with `cwltool`.", + "metadata": {} + }, { "id": "377fef56-484d-491c-b19e-1be6931e44eb", "cell_type": "code", @@ -41,6 +47,12 @@ "outputs": [], "execution_count": 2 }, + { + "cell_type": "markdown", + "id": "55b1a180", + "source": "`write_workflow` reads the existing `workflow.json` and generates the corresponding CWL description: one `CommandLineTool` `.cwl` file per function node, a `workflow.cwl` file that wires the steps together, and a `workflow.yml` file listing the input values (each serialized to its own pickle file). Every generated tool invokes `python -m python_workflow_definition.cwl`, which loads the target function from `workflow.py` and calls it with the pickled inputs.", + "metadata": {} + }, { "id": "5303c059-8ae4-4557-858e-b4bd64eac711", "cell_type": "code", @@ -51,6 +63,12 @@ "outputs": [], "execution_count": 3 }, + { + "cell_type": "markdown", + "id": "0581c24f", + "source": "Running `cwltool workflow.cwl workflow.yml` executes the steps in dependency order, passing intermediate results between steps as pickle files, and prints the location of the final output file.", + "metadata": {} + }, { "id": "df302bd2-e9b6-4595-979c-67c46414d986", "cell_type": "code", @@ -62,11 +80,17 @@ { "name": "stdout", "output_type": "stream", - "text": "/srv/conda/envs/notebook/bin/cwltool:11: DeprecationWarning: Nesting argument groups is deprecated.\n sys.exit(run())\n\u001B[1;30mINFO\u001B[0m /srv/conda/envs/notebook/bin/cwltool 3.1.20250110105449\n\u001B[1;30mINFO\u001B[0m Resolved 'workflow.cwl' to 'file:///home/jovyan/example_workflows/arithmetic/workflow.cwl'\n\u001B[1;30mINFO\u001B[0m [workflow ] start\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_prod_and_div_0\n\u001B[1;30mINFO\u001B[0m [step get_prod_and_div_0] start\n\u001B[1;30mINFO\u001B[0m [job get_prod_and_div_0] /tmp/_1apt559$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/dyuxg6ka/stgf337bac5-2f8e-4489-bd51-83491b1d5f0e/workflow.py \\\n --function=workflow.get_prod_and_div \\\n --arg_x=/tmp/dyuxg6ka/stgd87c2fe4-ca8e-4031-9af0-751d6944cbe7/x.pickle \\\n --arg_y=/tmp/dyuxg6ka/stg7bcb8ced-0e5b-442e-a0e0-2b3117afe2d8/y.pickle\n\u001B[1;30mINFO\u001B[0m [job get_prod_and_div_0] completed success\n\u001B[1;30mINFO\u001B[0m [step get_prod_and_div_0] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_sum_1\n\u001B[1;30mINFO\u001B[0m [step get_sum_1] start\n\u001B[1;30mINFO\u001B[0m [job get_sum_1] /tmp/ysau_yra$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/rm6c9qek/stgfef68b90-5d69-4da9-a952-b9c2f2a1eb76/workflow.py \\\n --function=workflow.get_sum \\\n --arg_x=/tmp/rm6c9qek/stg35728be7-5a6d-487b-bd02-345581007663/prod.pickle \\\n --arg_y=/tmp/rm6c9qek/stgc0992786-237e-4308-9d9e-814ebbfb0319/div.pickle\n\u001B[1;30mINFO\u001B[0m [job get_sum_1] completed success\n\u001B[1;30mINFO\u001B[0m [step get_sum_1] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_square_2\n\u001B[1;30mINFO\u001B[0m [step get_square_2] start\n\u001B[1;30mINFO\u001B[0m [job get_square_2] /tmp/o3gzya8t$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/6a3sltnz/stg58ccf048-a5c3-4ba3-a24f-c64adfff84cd/workflow.py \\\n --function=workflow.get_square \\\n --arg_x=/tmp/6a3sltnz/stg70203cec-cbd4-4fea-a3de-70e9373d9c9d/result.pickle\n\u001B[1;30mINFO\u001B[0m [job get_square_2] completed success\n\u001B[1;30mINFO\u001B[0m [step get_square_2] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] completed success\n{\n \"result_file\": {\n \"location\": \"file:///home/jovyan/example_workflows/arithmetic/result.pickle\",\n \"basename\": \"result.pickle\",\n \"class\": \"File\",\n \"checksum\": \"sha1$1fd8f217331336b1226b84e351c56fb9a8d93685\",\n \"size\": 21,\n \"path\": \"/home/jovyan/example_workflows/arithmetic/result.pickle\"\n }\n}\u001B[1;30mINFO\u001B[0m Final process status is success\n" + "text": "/srv/conda/envs/notebook/bin/cwltool:11: DeprecationWarning: Nesting argument groups is deprecated.\n sys.exit(run())\n\u001b[1;30mINFO\u001b[0m /srv/conda/envs/notebook/bin/cwltool 3.1.20250110105449\n\u001b[1;30mINFO\u001b[0m Resolved 'workflow.cwl' to 'file:///home/jovyan/example_workflows/arithmetic/workflow.cwl'\n\u001b[1;30mINFO\u001b[0m [workflow ] start\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_prod_and_div_0\n\u001b[1;30mINFO\u001b[0m [step get_prod_and_div_0] start\n\u001b[1;30mINFO\u001b[0m [job get_prod_and_div_0] /tmp/_1apt559$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/dyuxg6ka/stgf337bac5-2f8e-4489-bd51-83491b1d5f0e/workflow.py \\\n --function=workflow.get_prod_and_div \\\n --arg_x=/tmp/dyuxg6ka/stgd87c2fe4-ca8e-4031-9af0-751d6944cbe7/x.pickle \\\n --arg_y=/tmp/dyuxg6ka/stg7bcb8ced-0e5b-442e-a0e0-2b3117afe2d8/y.pickle\n\u001b[1;30mINFO\u001b[0m [job get_prod_and_div_0] completed success\n\u001b[1;30mINFO\u001b[0m [step get_prod_and_div_0] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_sum_1\n\u001b[1;30mINFO\u001b[0m [step get_sum_1] start\n\u001b[1;30mINFO\u001b[0m [job get_sum_1] /tmp/ysau_yra$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/rm6c9qek/stgfef68b90-5d69-4da9-a952-b9c2f2a1eb76/workflow.py \\\n --function=workflow.get_sum \\\n --arg_x=/tmp/rm6c9qek/stg35728be7-5a6d-487b-bd02-345581007663/prod.pickle \\\n --arg_y=/tmp/rm6c9qek/stgc0992786-237e-4308-9d9e-814ebbfb0319/div.pickle\n\u001b[1;30mINFO\u001b[0m [job get_sum_1] completed success\n\u001b[1;30mINFO\u001b[0m [step get_sum_1] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_square_2\n\u001b[1;30mINFO\u001b[0m [step get_square_2] start\n\u001b[1;30mINFO\u001b[0m [job get_square_2] /tmp/o3gzya8t$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/6a3sltnz/stg58ccf048-a5c3-4ba3-a24f-c64adfff84cd/workflow.py \\\n --function=workflow.get_square \\\n --arg_x=/tmp/6a3sltnz/stg70203cec-cbd4-4fea-a3de-70e9373d9c9d/result.pickle\n\u001b[1;30mINFO\u001b[0m [job get_square_2] completed success\n\u001b[1;30mINFO\u001b[0m [step get_square_2] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] completed success\n{\n \"result_file\": {\n \"location\": \"file:///home/jovyan/example_workflows/arithmetic/result.pickle\",\n \"basename\": \"result.pickle\",\n \"class\": \"File\",\n \"checksum\": \"sha1$1fd8f217331336b1226b84e351c56fb9a8d93685\",\n \"size\": 21,\n \"path\": \"/home/jovyan/example_workflows/arithmetic/result.pickle\"\n }\n}\u001b[1;30mINFO\u001b[0m Final process status is success\n" } ], "execution_count": 4 }, + { + "cell_type": "markdown", + "id": "c483a640", + "source": "The workflow's final output is written to `result.pickle`; unpickling it gives the same numeric result produced by the other engines.", + "metadata": {} + }, { "id": "2942dbba-ea0a-4d20-be5c-ed9992d09ff8", "cell_type": "code", @@ -94,4 +118,4 @@ "execution_count": null } ] -} +} \ No newline at end of file diff --git a/example_workflows/arithmetic/executorlib.ipynb b/example_workflows/arithmetic/executorlib.ipynb index 509b7316..612c6152 100644 --- a/example_workflows/arithmetic/executorlib.ipynb +++ b/example_workflows/arithmetic/executorlib.ipynb @@ -4,17 +4,13 @@ "cell_type": "markdown", "id": "c39b76fb-259f-4e16-a44d-02a295c82386", "metadata": {}, - "source": [ - "# executorlib" - ] + "source": "# executorlib\n\nThis notebook defines the arithmetic workflow with [`executorlib`](https://github.com/pyiron/executorlib), a lightweight `concurrent.futures`-style executor for up-scaling Python functions to HPC resources, and then loads the resulting `workflow.json` into `aiida-workgraph`, `jobflow`, `pyiron_base` and `pyiron_workflow`." }, { "cell_type": "markdown", "id": "3638419b-a0cb-49e2-b157-7fbb1acde90f", "metadata": {}, - "source": [ - "## Define workflow with executorlib" - ] + "source": "## Define workflow with executorlib\n\n`executorlib`'s `SingleNodeExecutor` implements the standard `Executor` interface: `.submit()` schedules a function call and immediately returns a `Future`. Passing `export_workflow_filename` to the executor records every submitted call and its dependencies as a side effect and writes them out as a PWD `workflow.json` once the `with` block exits." }, { "cell_type": "code", @@ -46,6 +42,12 @@ "workflow_json_filename = \"executorlib_arithmetic.json\"" ] }, + { + "cell_type": "markdown", + "id": "a6c2ee50", + "source": "`get_prod_and_div` returns a dictionary, so `get_item_from_future` is used to extract the `prod` and `div` keys from its future's result before they can be passed on as separate arguments to `get_sum`.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 4, @@ -157,9 +159,7 @@ "cell_type": "markdown", "id": "a4c0faaf-e30d-4ded-8e9f-57f97f51b14c", "metadata": {}, - "source": [ - "## Load Workflow with aiida" - ] + "source": "## Load Workflow with aiida\n\nThe exported `workflow.json` is loaded into `aiida-workgraph`. As with the other notebooks, an AiiDA profile must be connected via `load_profile()` before the resulting `WorkGraph` can be run." }, { "cell_type": "code", @@ -260,9 +260,7 @@ "cell_type": "markdown", "id": "0c3503e1-0a32-40e1-845d-3fd9ec3c4c19", "metadata": {}, - "source": [ - "## Load Workflow with jobflow" - ] + "source": "## Load Workflow with jobflow\n\nThe same JSON is loaded into a jobflow `Flow` and executed locally with `run_locally`." }, { "cell_type": "code", @@ -336,9 +334,7 @@ "cell_type": "markdown", "id": "406fd07dd4bd8006", "metadata": {}, - "source": [ - "## Load Workflow with pyiron_base" - ] + "source": "## Load Workflow with pyiron_base\n\nLoading the workflow into `pyiron_base` produces a list of delayed jobs; `.draw()` visualizes the dependency graph and `.pull()` triggers delayed execution of the whole chain." }, { "cell_type": "code", @@ -495,9 +491,7 @@ "cell_type": "markdown", "id": "406d62d0-76eb-411f-a7fb-f866d5cec9c1", "metadata": {}, - "source": [ - "## Load Workflow with pyiron_workflow" - ] + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the workflow is loaded into a `pyiron_workflow` `Workflow` object, drawn with `.draw()` and executed with `.run()`." }, { "cell_type": "code", @@ -1128,4 +1122,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file diff --git a/example_workflows/arithmetic/pyiron_base.ipynb b/example_workflows/arithmetic/pyiron_base.ipynb index 18742f20..eaba0a52 100644 --- a/example_workflows/arithmetic/pyiron_base.ipynb +++ b/example_workflows/arithmetic/pyiron_base.ipynb @@ -24,13 +24,13 @@ { "id": "c39b76fb-259f-4e16-a44d-02a295c82386", "cell_type": "markdown", - "source": "# pyiron", + "source": "# pyiron\n\nThis notebook defines the arithmetic workflow with [`pyiron_base`](https://github.com/pyiron/pyiron_base) and then loads the resulting `workflow.json` into `aiida-workgraph`, `jobflow` and `pyiron_workflow`.", "metadata": {} }, { "id": "3638419b-a0cb-49e2-b157-7fbb1acde90f", "cell_type": "markdown", - "source": "## Define workflow with pyiron_base", + "source": "## Define workflow with pyiron_base\n\nThe `job` decorator from `pyiron_base` turns a plain Python function into a pyiron job: calling the decorated function no longer executes it immediately, but returns a delayed object that can be chained with other jobs. For functions such as `get_prod_and_div` that return a dictionary, `output_key_lst` declares the dictionary keys so that individual entries (`prod`, `div`) can be wired as separate outputs to downstream jobs.", "metadata": {} }, { @@ -113,6 +113,12 @@ "outputs": [], "execution_count": 8 }, + { + "cell_type": "markdown", + "id": "9cecf662", + "source": "None of these calls run yet, they only build up a graph of delayed jobs. `write_workflow_json` walks that graph starting from the final `result` job and serializes it into the PWD `workflow.json` format.", + "metadata": {} + }, { "id": "e464da97-16a1-4772-9a07-0a47f152781d", "cell_type": "code", @@ -142,7 +148,7 @@ { "id": "a4c0faaf-e30d-4ded-8e9f-57f97f51b14c", "cell_type": "markdown", - "source": "## Load Workflow with aiida", + "source": "## Load Workflow with aiida\n\nThe exported `workflow.json` is loaded into `aiida-workgraph`. An AiiDA profile must be connected via `load_profile()` before the resulting `WorkGraph` can be run.", "metadata": {} }, { @@ -217,7 +223,7 @@ { "id": "0c3503e1-0a32-40e1-845d-3fd9ec3c4c19", "cell_type": "markdown", - "source": "## Load Workflow with jobflow", + "source": "## Load Workflow with jobflow\n\nThe same JSON is loaded into a jobflow `Flow` and executed locally with `run_locally`.", "metadata": {} }, { @@ -277,7 +283,7 @@ { "metadata": {}, "cell_type": "markdown", - "source": "## Load Workflow with pyiron_workflow", + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the workflow is loaded into a `pyiron_workflow` `Workflow` object, drawn with `.draw()` and executed with `.run()`.", "id": "406fd07dd4bd8006" }, { @@ -319,4 +325,4 @@ "id": "d36522a1c315b7f5" } ] -} +} \ No newline at end of file diff --git a/example_workflows/arithmetic/pyiron_workflow.ipynb b/example_workflows/arithmetic/pyiron_workflow.ipynb index 2dd96a50..7946d492 100644 --- a/example_workflows/arithmetic/pyiron_workflow.ipynb +++ b/example_workflows/arithmetic/pyiron_workflow.ipynb @@ -4,17 +4,13 @@ "cell_type": "markdown", "id": "9db99131-e5ef-49a9-93f6-13c44782c119", "metadata": {}, - "source": [ - "# pyiron_workflow" - ] + "source": "# pyiron_workflow\n\nThis notebook defines the arithmetic workflow with [`pyiron_workflow`](https://github.com/pyiron/pyiron_workflow) and then loads the resulting `workflow.json` into `aiida-workgraph`, `jobflow` and `pyiron_base`." }, { "cell_type": "markdown", "id": "d16df60e-1359-4d4c-bb95-2b994860d04a", "metadata": {}, - "source": [ - "## Define workflow in pyiron_workflow" - ] + "source": "## Define workflow in pyiron_workflow\n\n`to_function_node` wraps a plain Python function into a `pyiron_workflow` node that can be added to a `Workflow` graph. Nodes are assembled into a graph with attribute access: assigning `wf. = node(...)` both registers the node under that name and connects its inputs to the outputs of other nodes already on the graph." }, { "cell_type": "code", @@ -96,6 +92,12 @@ "wf.square_result = get_square(x=wf.tmp_sum)" ] }, + { + "cell_type": "markdown", + "id": "7d2460f1", + "source": "`wf.draw()` renders the resulting node graph, and `write_workflow_json` serializes `wf.graph_as_dict` into the PWD `workflow.json` format.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 7, @@ -760,9 +762,7 @@ "cell_type": "markdown", "id": "1681d449-0266-4da4-8d87-40a055cf3149", "metadata": {}, - "source": [ - "## Load Workflow with aiida" - ] + "source": "## Load Workflow with aiida\n\nThe exported `workflow.json` is loaded into `aiida-workgraph`. An AiiDA profile must be connected via `load_profile()` before the resulting `WorkGraph` can be run." }, { "cell_type": "code", @@ -811,9 +811,7 @@ "cell_type": "markdown", "id": "bac61097-e666-443b-9446-4ebb9a3e6b5f", "metadata": {}, - "source": [ - "# Load Workflow with jobflow" - ] + "source": "## Load Workflow with jobflow\n\nThe same JSON is loaded into a jobflow `Flow` and executed locally with `run_locally`." }, { "cell_type": "code", @@ -860,9 +858,7 @@ "cell_type": "markdown", "id": "f26c9650-649f-436d-8d19-b2cd75cf6488", "metadata": {}, - "source": [ - "# Load Workflow with pyiron_base" - ] + "source": "## Load Workflow with pyiron_base\n\nFinally, the workflow is loaded into `pyiron_base`, producing a list of delayed jobs. `.draw()` visualizes the dependency graph and `.pull()` triggers delayed execution of the whole chain." }, { "cell_type": "code", @@ -917,4 +913,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file diff --git a/example_workflows/arithmetic/universal_workflow.ipynb b/example_workflows/arithmetic/universal_workflow.ipynb index a8cfc73b..a100ed2d 100644 --- a/example_workflows/arithmetic/universal_workflow.ipynb +++ b/example_workflows/arithmetic/universal_workflow.ipynb @@ -23,12 +23,12 @@ "cells": [ { "cell_type": "markdown", - "source": "# Load Simple Workflow", + "source": "# Load Simple Workflow\n\nThis notebook loads a single, pre-existing `workflow.json` with several different engines in turn. Since the JSON only describes function nodes, inputs, outputs and the edges between them, it does not matter which engine originally wrote it — any of the engines below can load and execute it.", "metadata": {} }, { "cell_type": "markdown", - "source": "## Plot", + "source": "## Plot\n\nBefore running the workflow with any engine, `python_workflow_definition.plot.plot` renders `workflow.json` directly as a graph of function, input and output nodes — independent of any particular workflow engine.", "metadata": {} }, { @@ -60,7 +60,7 @@ }, { "cell_type": "markdown", - "source": "## Aiida ", + "source": "## Aiida\n\nLoading the JSON into `aiida-workgraph` requires connecting to an AiiDA profile first via `load_profile()`; the workflow is then reconstructed as a `WorkGraph` and executed with `.run()`.", "metadata": {} }, { @@ -130,7 +130,7 @@ }, { "cell_type": "markdown", - "source": "## executorlib", + "source": "## executorlib\n\n`python_workflow_definition.executorlib.load_workflow_json` submits each function node to the given `Executor` and returns the `Future` of the final node; calling `.result()` blocks until the whole chain has completed.", "metadata": {} }, { @@ -180,7 +180,7 @@ }, { "cell_type": "markdown", - "source": "## Jobflow", + "source": "## Jobflow\n\nThe JSON is reconstructed as a jobflow `Flow` and executed locally with `run_locally`.", "metadata": {} }, { @@ -235,7 +235,7 @@ }, { "cell_type": "markdown", - "source": "## pyiron", + "source": "## pyiron\n\nLoading the JSON into `pyiron_base` produces a list of delayed jobs; `.draw()` visualizes the dependency graph and `.pull()` triggers execution of the whole chain.", "metadata": {} }, { @@ -291,7 +291,7 @@ { "metadata": {}, "cell_type": "markdown", - "source": "## Load Workflow with pyiron_workflow" + "source": "## pyiron_workflow\n\nFinally, the workflow is loaded into a `pyiron_workflow` `Workflow` object, drawn with `.draw()` and executed with `.run()`." }, { "metadata": {}, @@ -323,7 +323,7 @@ }, { "cell_type": "markdown", - "source": "## Python", + "source": "## Python\n\nAs a baseline, `python_workflow_definition.purepython.load_workflow_json` interprets the JSON directly in plain Python, with no workflow engine involved: it evaluates each function node in dependency order and returns the final result.", "metadata": {} }, { @@ -354,4 +354,4 @@ "execution_count": 19 } ] -} +} \ No newline at end of file diff --git a/example_workflows/nfdi/aiida.ipynb b/example_workflows/nfdi/aiida.ipynb index 30378ca8..ef72f1bf 100644 --- a/example_workflows/nfdi/aiida.ipynb +++ b/example_workflows/nfdi/aiida.ipynb @@ -24,13 +24,13 @@ { "id": "106ded66-d202-46ac-82b0-2755ca309bdd", "cell_type": "markdown", - "source": "# Aiida\n\nhttps://github.com/BAMresearch/NFDI4IngScientificWorkflowRequirements", + "source": "# Aiida\n\nhttps://github.com/BAMresearch/NFDI4IngScientificWorkflowRequirements\n\nThis notebook defines the NFDI4Ing file-based workflow benchmark with [`aiida-workgraph`](https://github.com/aiidateam/aiida-workgraph) and then loads the resulting `workflow.json` into `jobflow`, `pyiron_base` and `pyiron_workflow` to show that the same JSON file can be executed by different engines. Every stage of the workflow calls an external command-line tool (`gmsh`, `meshio`, a FEniCS-based Poisson solver, ParaView's `pvbatch`, `tectonic`) in its own conda environment and passes file paths between stages rather than in-memory Python objects — see [`workflow.py`](workflow.py) for the six pipeline functions.", "metadata": {} }, { "id": "11e09b78-cb72-465f-9c8b-5b77f0aa729c", "cell_type": "markdown", - "source": "## Define workflow with aiida", + "source": "## Define workflow with aiida\n\n`aiida-workgraph` represents a workflow as a `WorkGraph` of `Task`s. `load_profile()` connects to the local AiiDA profile/database that stores provenance and results for every task. `generate_mesh`, `plot_over_line`, `substitute_macros` and `compile_paper` each return a single value and can be added to the graph unchanged, but `convert_to_xdmf` and `poisson` return a `dict`, so they are wrapped with `task(outputs=[...])` to expose each dictionary key as its own output socket that later tasks can connect to.", "metadata": {} }, { @@ -82,6 +82,12 @@ "outputs": [], "execution_count": 4 }, + { + "cell_type": "markdown", + "id": "e16e7b61", + "source": "AiiDA tracks every input as a typed, storable node, so the shared `domain_size` and `source_directory` values are wrapped as `orm.Float` and `orm.Str` before being connected to any task.", + "metadata": {} + }, { "id": "37c9d988-1755-446c-9f7b-c32f99e280d4", "cell_type": "code", @@ -112,6 +118,12 @@ "outputs": [], "execution_count": 7 }, + { + "cell_type": "markdown", + "id": "2e764162", + "source": "Each `wg.add_task` call below adds one stage of the pipeline: mesh generation, XDMF conversion, the Poisson solve, line-plot postprocessing, LaTeX macro substitution and paper compilation. Outputs are wired to the next task's inputs with `.outputs.`; since the underlying functions communicate through files on disk, these connections are really file paths handed from one task to the next, not shared Python objects.", + "metadata": {} + }, { "id": "71d411b6-cbec-489e-99e3-ba71680bcb5b", "cell_type": "code", @@ -201,6 +213,12 @@ ], "execution_count": 14 }, + { + "cell_type": "markdown", + "id": "a7bd7df7", + "source": "The `wg` widget above renders the assembled task graph. Once satisfied, `write_workflow_json` serializes it into the PWD `workflow.json` format — one `function` node per task, `input` nodes for `domain_size`/`source_directory`, and edges recording which task output feeds which task's input port, as shown by the `cat` output below.", + "metadata": {} + }, { "id": "fb23ad9c-76fd-4c0b-b546-e305d6c49796", "cell_type": "code", @@ -240,7 +258,7 @@ { "id": "11a829e2-face-469f-b343-2c95763b1f13", "cell_type": "markdown", - "source": "## Load Workflow with jobflow", + "source": "## Load Workflow with jobflow\n\n`python_workflow_definition.jobflow.load_workflow_json` reconstructs the same six-stage pipeline as a jobflow `Flow`, purely from `aiida_nfdi.json` — no reference to the `aiida-workgraph` objects defined above. `run_locally` then executes the jobs in dependency order, re-running `gmsh`, `meshio`, the Poisson solver, `pvbatch` and `tectonic` exactly as before.", "metadata": {} }, { @@ -300,7 +318,7 @@ { "id": "397b16a2-e1ec-4eec-8562-1c84f585c347", "cell_type": "markdown", - "source": "## Load Workflow with pyiron_base", + "source": "## Load Workflow with pyiron_base\n\n`python_workflow_definition.pyiron_base.load_workflow_json` turns the JSON graph into a list of delayed pyiron jobs; `.draw()` visualizes their dependencies and `.pull()` triggers execution, running each job in turn and returning the final `paper.pdf` path.", "metadata": {} }, { @@ -359,7 +377,7 @@ { "metadata": {}, "cell_type": "markdown", - "source": "## Load Workflow with pyiron_workflow", + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the same JSON is loaded into a `pyiron_workflow` `Workflow` object. `.draw()` shows the node graph and `.run()` executes it end to end, again shelling out to each external tool in turn.", "id": "afaf2bc0b3a491ca" }, { @@ -395,4 +413,4 @@ "id": "c1427e3ce0716f64" } ] -} +} \ No newline at end of file diff --git a/example_workflows/nfdi/cwl.ipynb b/example_workflows/nfdi/cwl.ipynb index fc40e635..5fc2d266 100644 --- a/example_workflows/nfdi/cwl.ipynb +++ b/example_workflows/nfdi/cwl.ipynb @@ -21,6 +21,12 @@ "nbformat_minor": 5, "nbformat": 4, "cells": [ + { + "cell_type": "markdown", + "id": "e88c3a45", + "source": "# CWL\n\nThis notebook executes the NFDI4Ing file-based workflow benchmark's `workflow.json` with the [Common Workflow Language](https://www.commonwl.org/) via [`cwltool`](https://github.com/common-workflow-language/cwltool), instead of generating the JSON from a Python workflow engine like the other notebooks in this directory. `python_workflow_definition.cwl.write_workflow` translates the PWD graph into a CWL `workflow.cwl` plus one `CommandLineTool` wrapper per pipeline function; CWL then passes data between steps as pickled files on disk rather than in-memory Python objects, which matches this benchmark's file-based nature.", + "metadata": {} + }, { "id": "377fef56-484d-491c-b19e-1be6931e44eb", "cell_type": "code", @@ -64,7 +70,7 @@ { "id": "b0cf73b9-ea21-4437-8d2a-c51b65bbfa86", "cell_type": "markdown", - "source": "# Overwrite source directory with absolute path", + "source": "## Overwrite source directory with absolute path\n\n`workflow.json` stores `source_directory` as the relative path `\"source\"`. Because `cwltool` stages each step's inputs into its own temporary working directory, a relative path would no longer resolve once execution starts, so it is rewritten to an absolute path before export.", "metadata": {} }, { @@ -100,7 +106,7 @@ { "id": "a9540ba7-f15a-4d04-86aa-0cf2ad4ac185", "cell_type": "markdown", - "source": "# Execute workflow", + "source": "## Execute workflow\n\n`write_workflow` reads `workflow.json` and writes out `workflow.cwl` (the CWL `Workflow` definition), one `_.cwl` `CommandLineTool` per pipeline step, and `workflow.yml` with the input values pickled to disk. `cwltool` then runs the six steps in dependency order, invoking `gmsh`, `meshio`, the Poisson solver, `pvbatch` and `tectonic` in turn, and writes the workflow's single output — the path to the compiled `paper.pdf` — to `result.pickle`.", "metadata": {} }, { @@ -124,11 +130,17 @@ { "name": "stdout", "output_type": "stream", - "text": "/srv/conda/envs/notebook/bin/cwltool:11: DeprecationWarning: Nesting argument groups is deprecated.\n sys.exit(run())\n\u001B[1;30mINFO\u001B[0m /srv/conda/envs/notebook/bin/cwltool 3.1.20250110105449\n\u001B[1;30mINFO\u001B[0m Resolved 'workflow.cwl' to 'file:///home/jovyan/example_workflows/nfdi/workflow.cwl'\n\u001B[1;30mINFO\u001B[0m [workflow ] start\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step generate_mesh_0\n\u001B[1;30mINFO\u001B[0m [step generate_mesh_0] start\n\u001B[1;30mINFO\u001B[0m [job generate_mesh_0] /tmp/xjzxjjxg$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/lint8hha/stgaa9fa1cf-dae6-4402-aa25-3da9289e3937/workflow.py \\\n --function=workflow.generate_mesh \\\n --arg_domain_size=/tmp/lint8hha/stg94834a09-0454-4ad4-bc93-b8f3a1dc72f4/domain_size.pickle \\\n --arg_source_directory=/tmp/lint8hha/stg03fa6508-a0ea-4379-97b1-e3815dfe7395/source_directory.pickle\n\u001B[1;30mINFO\u001B[0m [job generate_mesh_0] Max memory used: 60MiB\n\u001B[1;30mINFO\u001B[0m [job generate_mesh_0] completed success\n\u001B[1;30mINFO\u001B[0m [step generate_mesh_0] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step convert_to_xdmf_1\n\u001B[1;30mINFO\u001B[0m [step convert_to_xdmf_1] start\n\u001B[1;30mINFO\u001B[0m [job convert_to_xdmf_1] /tmp/q47niux0$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/pd0inpeh/stg928009b1-efc5-45d7-9746-e1af8d96f0c5/workflow.py \\\n --function=workflow.convert_to_xdmf \\\n --arg_gmsh_output_file=/tmp/pd0inpeh/stg03750262-f7c4-4bb9-b558-0b7ea0a54cf3/result.pickle\n\u001B[1;30mINFO\u001B[0m [job convert_to_xdmf_1] Max memory used: 69MiB\n\u001B[1;30mINFO\u001B[0m [job convert_to_xdmf_1] completed success\n\u001B[1;30mINFO\u001B[0m [step convert_to_xdmf_1] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step poisson_2\n\u001B[1;30mINFO\u001B[0m [step poisson_2] start\n\u001B[1;30mINFO\u001B[0m [job poisson_2] /tmp/e045_wvq$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/rsa0i3o4/stgfd3d067f-0d0e-46b6-8b1c-e63a936cca99/workflow.py \\\n --function=workflow.poisson \\\n --arg_meshio_output_xdmf=/tmp/rsa0i3o4/stgebd53ba4-dbbc-4769-b359-f1f95611d02d/xdmf_file.pickle \\\n --arg_meshio_output_h5=/tmp/rsa0i3o4/stga4bcfd98-5bf0-4da3-b1e6-a4d55a00cc30/h5_file.pickle \\\n --arg_source_directory=/tmp/rsa0i3o4/stg003e0d81-81e0-4138-9e97-103a910c925e/source_directory.pickle\n\u001B[1;30mINFO\u001B[0m [job poisson_2] Max memory used: 81MiB\n\u001B[1;30mINFO\u001B[0m [job poisson_2] completed success\n\u001B[1;30mINFO\u001B[0m [step poisson_2] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step plot_over_line_3\n\u001B[1;30mINFO\u001B[0m [step plot_over_line_3] start\n\u001B[1;30mINFO\u001B[0m [job plot_over_line_3] /tmp/0tov09ih$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/z9a5oiuc/stgcde66eb7-274f-4ae3-833b-c8d99fe3bbb1/workflow.py \\\n --function=workflow.plot_over_line \\\n --arg_poisson_output_vtu_file=/tmp/z9a5oiuc/stg144f313e-e286-4800-a3d2-86a6cae36a98/vtu_file.pickle \\\n --arg_source_directory=/tmp/z9a5oiuc/stg224c0c40-7cb8-4f1c-921c-013809fb5cbf/source_directory.pickle \\\n --arg_poisson_output_pvd_file=/tmp/z9a5oiuc/stgc4c75ca6-aab0-40be-bfbd-f432d3523d2b/pvd_file.pickle\n\u001B[1;30mINFO\u001B[0m [job plot_over_line_3] Max memory used: 68MiB\n\u001B[1;30mINFO\u001B[0m [job plot_over_line_3] completed success\n\u001B[1;30mINFO\u001B[0m [step plot_over_line_3] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step substitute_macros_4\n\u001B[1;30mINFO\u001B[0m [step substitute_macros_4] start\n\u001B[1;30mINFO\u001B[0m [job substitute_macros_4] /tmp/08mfe8f0$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/5valbs4a/stg15353150-ebc3-49de-93a8-564a25f33270/workflow.py \\\n --function=workflow.substitute_macros \\\n --arg_domain_size=/tmp/5valbs4a/stg0734ab21-1efd-480b-ba2b-cea9ed24012e/domain_size.pickle \\\n --arg_source_directory=/tmp/5valbs4a/stg0c79cde6-159f-4b82-bdb9-aabef8678f9f/source_directory.pickle \\\n --arg_ndofs=/tmp/5valbs4a/stg770f9cb7-d2c3-4ce5-bc00-7a9cdb11d64d/numdofs.pickle \\\n --arg_pvbatch_output_file=/tmp/5valbs4a/stgdd978452-6c84-4308-abf6-7ae2609fc91d/result.pickle\n\u001B[1;30mINFO\u001B[0m [job substitute_macros_4] Max memory used: 60MiB\n\u001B[1;30mINFO\u001B[0m [job substitute_macros_4] completed success\n\u001B[1;30mINFO\u001B[0m [step substitute_macros_4] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step compile_paper_5\n\u001B[1;30mINFO\u001B[0m [step compile_paper_5] start\n\u001B[1;30mINFO\u001B[0m [job compile_paper_5] /tmp/hjjeng16$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/3bq2mxdl/stg34a5d902-97c3-43f1-95eb-5e14d0c68386/workflow.py \\\n --function=workflow.compile_paper \\\n --arg_macros_tex=/tmp/3bq2mxdl/stg184e0acf-5138-4b37-97f0-ba406ed71433/result.pickle \\\n --arg_plot_file=/tmp/3bq2mxdl/stg2360c6e9-26ec-43f5-99a2-26c05668b2fa/result.pickle \\\n --arg_source_directory=/tmp/3bq2mxdl/stg7194b36e-38db-4ef7-af5d-368e9575f15f/source_directory.pickle\n\u001B[1;30mINFO\u001B[0m [job compile_paper_5] Max memory used: 265MiB\n\u001B[1;30mINFO\u001B[0m [job compile_paper_5] completed success\n\u001B[1;30mINFO\u001B[0m [step compile_paper_5] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] completed success\n{\n \"result_file\": {\n \"location\": \"file:///home/jovyan/example_workflows/nfdi/result.pickle\",\n \"basename\": \"result.pickle\",\n \"class\": \"File\",\n \"checksum\": \"sha1$a23617d3f4b4e6970b7f4fb9eb4d2148f9888e58\",\n \"size\": 53,\n \"path\": \"/home/jovyan/example_workflows/nfdi/result.pickle\"\n }\n}\u001B[1;30mINFO\u001B[0m Final process status is success\n" + "text": "/srv/conda/envs/notebook/bin/cwltool:11: DeprecationWarning: Nesting argument groups is deprecated.\n sys.exit(run())\n\u001b[1;30mINFO\u001b[0m /srv/conda/envs/notebook/bin/cwltool 3.1.20250110105449\n\u001b[1;30mINFO\u001b[0m Resolved 'workflow.cwl' to 'file:///home/jovyan/example_workflows/nfdi/workflow.cwl'\n\u001b[1;30mINFO\u001b[0m [workflow ] start\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step generate_mesh_0\n\u001b[1;30mINFO\u001b[0m [step generate_mesh_0] start\n\u001b[1;30mINFO\u001b[0m [job generate_mesh_0] /tmp/xjzxjjxg$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/lint8hha/stgaa9fa1cf-dae6-4402-aa25-3da9289e3937/workflow.py \\\n --function=workflow.generate_mesh \\\n --arg_domain_size=/tmp/lint8hha/stg94834a09-0454-4ad4-bc93-b8f3a1dc72f4/domain_size.pickle \\\n --arg_source_directory=/tmp/lint8hha/stg03fa6508-a0ea-4379-97b1-e3815dfe7395/source_directory.pickle\n\u001b[1;30mINFO\u001b[0m [job generate_mesh_0] Max memory used: 60MiB\n\u001b[1;30mINFO\u001b[0m [job generate_mesh_0] completed success\n\u001b[1;30mINFO\u001b[0m [step generate_mesh_0] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step convert_to_xdmf_1\n\u001b[1;30mINFO\u001b[0m [step convert_to_xdmf_1] start\n\u001b[1;30mINFO\u001b[0m [job convert_to_xdmf_1] /tmp/q47niux0$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/pd0inpeh/stg928009b1-efc5-45d7-9746-e1af8d96f0c5/workflow.py \\\n --function=workflow.convert_to_xdmf \\\n --arg_gmsh_output_file=/tmp/pd0inpeh/stg03750262-f7c4-4bb9-b558-0b7ea0a54cf3/result.pickle\n\u001b[1;30mINFO\u001b[0m [job convert_to_xdmf_1] Max memory used: 69MiB\n\u001b[1;30mINFO\u001b[0m [job convert_to_xdmf_1] completed success\n\u001b[1;30mINFO\u001b[0m [step convert_to_xdmf_1] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step poisson_2\n\u001b[1;30mINFO\u001b[0m [step poisson_2] start\n\u001b[1;30mINFO\u001b[0m [job poisson_2] /tmp/e045_wvq$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/rsa0i3o4/stgfd3d067f-0d0e-46b6-8b1c-e63a936cca99/workflow.py \\\n --function=workflow.poisson \\\n --arg_meshio_output_xdmf=/tmp/rsa0i3o4/stgebd53ba4-dbbc-4769-b359-f1f95611d02d/xdmf_file.pickle \\\n --arg_meshio_output_h5=/tmp/rsa0i3o4/stga4bcfd98-5bf0-4da3-b1e6-a4d55a00cc30/h5_file.pickle \\\n --arg_source_directory=/tmp/rsa0i3o4/stg003e0d81-81e0-4138-9e97-103a910c925e/source_directory.pickle\n\u001b[1;30mINFO\u001b[0m [job poisson_2] Max memory used: 81MiB\n\u001b[1;30mINFO\u001b[0m [job poisson_2] completed success\n\u001b[1;30mINFO\u001b[0m [step poisson_2] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step plot_over_line_3\n\u001b[1;30mINFO\u001b[0m [step plot_over_line_3] start\n\u001b[1;30mINFO\u001b[0m [job plot_over_line_3] /tmp/0tov09ih$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/z9a5oiuc/stgcde66eb7-274f-4ae3-833b-c8d99fe3bbb1/workflow.py \\\n --function=workflow.plot_over_line \\\n --arg_poisson_output_vtu_file=/tmp/z9a5oiuc/stg144f313e-e286-4800-a3d2-86a6cae36a98/vtu_file.pickle \\\n --arg_source_directory=/tmp/z9a5oiuc/stg224c0c40-7cb8-4f1c-921c-013809fb5cbf/source_directory.pickle \\\n --arg_poisson_output_pvd_file=/tmp/z9a5oiuc/stgc4c75ca6-aab0-40be-bfbd-f432d3523d2b/pvd_file.pickle\n\u001b[1;30mINFO\u001b[0m [job plot_over_line_3] Max memory used: 68MiB\n\u001b[1;30mINFO\u001b[0m [job plot_over_line_3] completed success\n\u001b[1;30mINFO\u001b[0m [step plot_over_line_3] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step substitute_macros_4\n\u001b[1;30mINFO\u001b[0m [step substitute_macros_4] start\n\u001b[1;30mINFO\u001b[0m [job substitute_macros_4] /tmp/08mfe8f0$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/5valbs4a/stg15353150-ebc3-49de-93a8-564a25f33270/workflow.py \\\n --function=workflow.substitute_macros \\\n --arg_domain_size=/tmp/5valbs4a/stg0734ab21-1efd-480b-ba2b-cea9ed24012e/domain_size.pickle \\\n --arg_source_directory=/tmp/5valbs4a/stg0c79cde6-159f-4b82-bdb9-aabef8678f9f/source_directory.pickle \\\n --arg_ndofs=/tmp/5valbs4a/stg770f9cb7-d2c3-4ce5-bc00-7a9cdb11d64d/numdofs.pickle \\\n --arg_pvbatch_output_file=/tmp/5valbs4a/stgdd978452-6c84-4308-abf6-7ae2609fc91d/result.pickle\n\u001b[1;30mINFO\u001b[0m [job substitute_macros_4] Max memory used: 60MiB\n\u001b[1;30mINFO\u001b[0m [job substitute_macros_4] completed success\n\u001b[1;30mINFO\u001b[0m [step substitute_macros_4] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step compile_paper_5\n\u001b[1;30mINFO\u001b[0m [step compile_paper_5] start\n\u001b[1;30mINFO\u001b[0m [job compile_paper_5] /tmp/hjjeng16$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/3bq2mxdl/stg34a5d902-97c3-43f1-95eb-5e14d0c68386/workflow.py \\\n --function=workflow.compile_paper \\\n --arg_macros_tex=/tmp/3bq2mxdl/stg184e0acf-5138-4b37-97f0-ba406ed71433/result.pickle \\\n --arg_plot_file=/tmp/3bq2mxdl/stg2360c6e9-26ec-43f5-99a2-26c05668b2fa/result.pickle \\\n --arg_source_directory=/tmp/3bq2mxdl/stg7194b36e-38db-4ef7-af5d-368e9575f15f/source_directory.pickle\n\u001b[1;30mINFO\u001b[0m [job compile_paper_5] Max memory used: 265MiB\n\u001b[1;30mINFO\u001b[0m [job compile_paper_5] completed success\n\u001b[1;30mINFO\u001b[0m [step compile_paper_5] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] completed success\n{\n \"result_file\": {\n \"location\": \"file:///home/jovyan/example_workflows/nfdi/result.pickle\",\n \"basename\": \"result.pickle\",\n \"class\": \"File\",\n \"checksum\": \"sha1$a23617d3f4b4e6970b7f4fb9eb4d2148f9888e58\",\n \"size\": 53,\n \"path\": \"/home/jovyan/example_workflows/nfdi/result.pickle\"\n }\n}\u001b[1;30mINFO\u001b[0m Final process status is success\n" } ], "execution_count": 9 }, + { + "cell_type": "markdown", + "id": "c55f089b", + "source": "The result is unpickled below to confirm it is the expected `paper.pdf` path.", + "metadata": {} + }, { "id": "2942dbba-ea0a-4d20-be5c-ed9992d09ff8", "cell_type": "code", @@ -156,4 +168,4 @@ "execution_count": null } ] -} +} \ No newline at end of file diff --git a/example_workflows/nfdi/executorlib.ipynb b/example_workflows/nfdi/executorlib.ipynb index b1a0f41d..8a2e0bac 100644 --- a/example_workflows/nfdi/executorlib.ipynb +++ b/example_workflows/nfdi/executorlib.ipynb @@ -4,19 +4,13 @@ "cell_type": "markdown", "id": "106ded66-d202-46ac-82b0-2755ca309bdd", "metadata": {}, - "source": [ - "# executorlib\n", - "\n", - "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/BAMresearch/NFDI4IngScientificWorkflowRequirements" - ] + "source": "# executorlib\n\nhttps://github.com/BAMresearch/NFDI4IngScientificWorkflowRequirements\n\nThis notebook defines the NFDI4Ing file-based workflow benchmark with [`executorlib`](https://github.com/pyiron/executorlib) and then loads the resulting `workflow.json` into `aiida`, `jobflow`, `pyiron_base` and `pyiron_workflow`. Every stage of the pipeline shells out to an external command-line tool (`gmsh`, `meshio`, a FEniCS-based Poisson solver, ParaView's `pvbatch`, `tectonic`) in its own conda environment and passes file paths between stages rather than in-memory Python objects — see [`workflow.py`](workflow.py)." }, { "cell_type": "markdown", "id": "91dd48ea-aa7e-4937-a68e-59fc5017eb1e", "metadata": {}, - "source": [ - "## Define workflow with executorlib" - ] + "source": "## Define workflow with executorlib\n\n`executorlib` provides a `concurrent.futures`-compatible `SingleNodeExecutor`: each `exe.submit(func, ...)` call schedules a function call and immediately returns a `Future`. Since `convert_to_xdmf` and `poisson` return a `dict`, `get_item_from_future` is used to pick a single key (e.g. `\"xdmf_file\"`) out of a future's eventual result before passing it on as an input to the next stage." }, { "cell_type": "code", @@ -78,6 +72,12 @@ "workflow_json_filename = \"executorlib_nfdi.json\"" ] }, + { + "cell_type": "markdown", + "id": "57d670cb", + "source": "Passing `export_workflow_filename` to `SingleNodeExecutor` makes it record every `submit` call as a side effect and write them out as a PWD `workflow.json` once the `with` block exits — no separate `write_workflow_json` call is needed. The six `submit` calls below mirror the six pipeline stages: mesh generation, XDMF conversion, the Poisson solve, line-plot postprocessing, LaTeX macro substitution and paper compilation.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 7, @@ -301,9 +301,7 @@ "cell_type": "markdown", "id": "d789971e-8f41-45fa-832a-11fd72dea96e", "metadata": {}, - "source": [ - "## Load Workflow with aiida" - ] + "source": "## Load Workflow with aiida\n\nThe JSON produced by `executorlib` is loaded into an `aiida-workgraph` `WorkGraph` via `load_workflow_json`, after connecting to a local AiiDA profile with `load_profile()`. `wg.run()` then re-executes the whole pipeline under `aiida`." }, { "cell_type": "code", @@ -412,9 +410,7 @@ "cell_type": "markdown", "id": "55dc8d12-dfe6-4465-a368-b7e590ae6800", "metadata": {}, - "source": [ - "## Load Workflow with jobflow" - ] + "source": "## Load Workflow with jobflow\n\nThe same `workflow.json` is reconstructed as a jobflow `Flow` and executed locally with `run_locally`, independently of the `executorlib` objects used to define it." }, { "cell_type": "code", @@ -511,9 +507,7 @@ "cell_type": "markdown", "id": "7ecc0861-6fc8-4788-b05b-fa5ae06b96e3", "metadata": {}, - "source": [ - "# Load Workflow with pyiron_base" - ] + "source": "## Load Workflow with pyiron_base\n\n`load_workflow_json` turns the graph into a list of delayed pyiron jobs; `.draw()` visualizes the dependency chain and `.pull()` triggers execution, returning the path to the final `paper.pdf`." }, { "cell_type": "code", @@ -832,9 +826,7 @@ "cell_type": "markdown", "id": "385acbf585763632", "metadata": {}, - "source": [ - "## Load Workflow with pyiron_workflow" - ] + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the graph is loaded into a `pyiron_workflow` `Workflow`. `.draw()` shows the node graph and `.run()` executes it end to end." }, { "cell_type": "code", @@ -2126,4 +2118,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file diff --git a/example_workflows/nfdi/jobflow.ipynb b/example_workflows/nfdi/jobflow.ipynb index 2cd73f26..77c50414 100644 --- a/example_workflows/nfdi/jobflow.ipynb +++ b/example_workflows/nfdi/jobflow.ipynb @@ -30,13 +30,13 @@ { "id": "106ded66-d202-46ac-82b0-2755ca309bdd", "cell_type": "markdown", - "source": "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/BAMresearch/NFDI4IngScientificWorkflowRequirements", + "source": "/BAMresearch/NFDI4IngScientificWorkflowRequirements\n\nThis notebook defines the NFDI4Ing file-based workflow benchmark with [`jobflow`](https://github.com/materialsproject/jobflow) and then loads the resulting `workflow.json` into `aiida`, `pyiron_base` and `pyiron_workflow`. Every stage of the pipeline shells out to an external command-line tool (`gmsh`, `meshio`, a FEniCS-based Poisson solver, ParaView's `pvbatch`, `tectonic`) in its own conda environment and passes file paths between stages rather than in-memory Python objects — see [`workflow.py`](workflow.py).", "metadata": {} }, { "id": "856b2ba2-93d5-4516-93e1-a1eac49c48f2", "cell_type": "markdown", - "source": "## Define workflow with jobflow", + "source": "## Define workflow with jobflow\n\nThe `job` decorator turns each of the six pipeline functions into a jobflow `Job` that delays execution until the `Flow` is run. Because `convert_to_xdmf` and `poisson` return a `dict`, their outputs are accessed downstream as `.output.` (e.g. `meshio_output_dict.output.xdmf_file`) to wire a single dictionary entry into the next job's input.", "metadata": {} }, { @@ -89,6 +89,12 @@ "outputs": [], "execution_count": 5 }, + { + "cell_type": "markdown", + "id": "1cd3d59e", + "source": "With `source_directory` and `domain_size` fixed, the calls below chain the six jobs together: mesh generation, XDMF conversion, the Poisson solve, line-plot postprocessing, LaTeX macro substitution and paper compilation. Each job runs in its own subdirectory and conda environment (see `source/envs/`), so what actually flows from one job to the next are file paths, not in-memory results.", + "metadata": {} + }, { "id": "8d911f98-3b80-457f-a0f4-3cb37ebf1691", "cell_type": "code", @@ -174,6 +180,12 @@ "outputs": [], "execution_count": 13 }, + { + "cell_type": "markdown", + "id": "71c4ff70", + "source": "The six jobs are collected into a `Flow`, which `write_workflow_json` then serializes to the PWD `workflow.json` format for the other engines to load.", + "metadata": {} + }, { "id": "fb23ad9c-76fd-4c0b-b546-e305d6c49796", "cell_type": "code", @@ -197,7 +209,7 @@ { "id": "11a829e2-face-469f-b343-2c95763b1f13", "cell_type": "markdown", - "source": "## Load Workflow with aiida", + "source": "## Load Workflow with aiida\n\n`load_workflow_json` rebuilds the pipeline as an `aiida-workgraph` `WorkGraph` from `jobflow_nfdi.json`, after connecting to a local AiiDA profile with `load_profile()`. `wg.run()` re-executes the whole pipeline under `aiida`.", "metadata": {} }, { @@ -272,7 +284,7 @@ { "id": "397b16a2-e1ec-4eec-8562-1c84f585c347", "cell_type": "markdown", - "source": "## Load Workflow with pyiron_base", + "source": "## Load Workflow with pyiron_base\n\nThe same JSON is loaded into a list of delayed pyiron jobs; `.draw()` visualizes their dependencies and `.pull()` triggers execution, returning the final `paper.pdf` path.", "metadata": {} }, { @@ -331,7 +343,7 @@ { "metadata": {}, "cell_type": "markdown", - "source": "## Load Workflow with pyiron_workflow", + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the graph is loaded into a `pyiron_workflow` `Workflow`. `.draw()` shows the node graph and `.run()` executes it end to end.", "id": "8e73fa07cebd7b72" }, { @@ -367,4 +379,4 @@ "id": "447506cdca5080e6" } ] -} +} \ No newline at end of file diff --git a/example_workflows/nfdi/pyiron_base.ipynb b/example_workflows/nfdi/pyiron_base.ipynb index f4feda62..164b5af8 100644 --- a/example_workflows/nfdi/pyiron_base.ipynb +++ b/example_workflows/nfdi/pyiron_base.ipynb @@ -24,13 +24,13 @@ { "id": "106ded66-d202-46ac-82b0-2755ca309bdd", "cell_type": "markdown", - "source": "# pyiron\n\nhttps://github.com/BAMresearch/NFDI4IngScientificWorkflowRequirements", + "source": "# pyiron\n\nhttps://github.com/BAMresearch/NFDI4IngScientificWorkflowRequirements\n\nThis notebook defines the NFDI4Ing file-based workflow benchmark with [`pyiron_base`](https://github.com/pyiron/pyiron_base) and then loads the resulting `workflow.json` into `aiida`, `jobflow` and `pyiron_workflow`. Every stage of the pipeline shells out to an external command-line tool (`gmsh`, `meshio`, a FEniCS-based Poisson solver, ParaView's `pvbatch`, `tectonic`) in its own conda environment and passes file paths between stages rather than in-memory Python objects — see [`workflow.py`](workflow.py).", "metadata": {} }, { "id": "91dd48ea-aa7e-4937-a68e-59fc5017eb1e", "cell_type": "markdown", - "source": "## Define workflow with pyiron_base", + "source": "## Define workflow with pyiron_base\n\nThe `job` decorator turns a plain Python function into a delayed pyiron job. Since `convert_to_xdmf` and `poisson` return a `dict`, they are decorated with `output_key_lst=[...]` so each dictionary key becomes a separately addressable output (e.g. `meshio_output_dict.output.xdmf_file`) that can be wired into the next job's input.", "metadata": {} }, { @@ -93,6 +93,12 @@ "outputs": [], "execution_count": 6 }, + { + "cell_type": "markdown", + "id": "b80c6588", + "source": "The calls below chain the six jobs together: mesh generation, XDMF conversion, the Poisson solve, line-plot postprocessing, LaTeX macro substitution and paper compilation. Since each stage runs an external tool in its own conda environment (see `source/envs/`), what is actually passed between jobs are file paths, not in-memory Python values.", + "metadata": {} + }, { "id": "71d411b6-cbec-489e-99e3-ba71680bcb5b", "cell_type": "code", @@ -158,6 +164,12 @@ "outputs": [], "execution_count": 12 }, + { + "cell_type": "markdown", + "id": "4d1ce825", + "source": "Because pyiron builds a delayed call graph rather than an explicit `Flow`/`WorkGraph` object, `write_workflow_json` only needs the final delayed object (`paper_output`) — it walks the graph backwards to discover every upstream job.", + "metadata": {} + }, { "id": "63f29646-3846-4a97-a033-20e9df0ac214", "cell_type": "code", @@ -181,7 +193,7 @@ { "id": "d789971e-8f41-45fa-832a-11fd72dea96e", "cell_type": "markdown", - "source": "## Load Workflow with aiida", + "source": "## Load Workflow with aiida\n\n`load_workflow_json` rebuilds the pipeline as an `aiida-workgraph` `WorkGraph` from `pyiron_base_nfdi.json`, after connecting to a local AiiDA profile with `load_profile()`. `wg.run()` re-executes the whole pipeline under `aiida`.", "metadata": {} }, { @@ -256,7 +268,7 @@ { "id": "55dc8d12-dfe6-4465-a368-b7e590ae6800", "cell_type": "markdown", - "source": "## Load Workflow with jobflow", + "source": "## Load Workflow with jobflow\n\nThe same JSON is reconstructed as a jobflow `Flow` and executed locally with `run_locally`, independently of the `pyiron_base` objects used to define it.", "metadata": {} }, { @@ -316,7 +328,7 @@ { "metadata": {}, "cell_type": "markdown", - "source": "## Load Workflow with pyiron_workflow", + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the graph is loaded into a `pyiron_workflow` `Workflow`. `.draw()` shows the node graph and `.run()` executes it end to end.", "id": "385acbf585763632" }, { @@ -352,4 +364,4 @@ "id": "fff9513c4a127a96" } ] -} +} \ No newline at end of file diff --git a/example_workflows/nfdi/pyiron_workflow.ipynb b/example_workflows/nfdi/pyiron_workflow.ipynb index db1df099..887894b8 100644 --- a/example_workflows/nfdi/pyiron_workflow.ipynb +++ b/example_workflows/nfdi/pyiron_workflow.ipynb @@ -1 +1,428 @@ -{"metadata":{"kernelspec":{"display_name":"Python 3 (ipykernel)","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.12.8","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":5,"nbformat":4,"cells":[{"id":"106ded66-d202-46ac-82b0-2755ca309bdd","cell_type":"markdown","source":"# pyiron\n\nhttps://github.com/BAMresearch/NFDI4IngScientificWorkflowRequirements","metadata":{}},{"id":"91dd48ea-aa7e-4937-a68e-59fc5017eb1e","cell_type":"markdown","source":"## Define workflow with pyiron_workflow","metadata":{}},{"id":"2c9622f5-ab7e-460e-b8e4-8d21413eda77","cell_type":"code","source":"import os","metadata":{"trusted":true},"outputs":[],"execution_count":1},{"id":"d265bb5aa6af79d6","cell_type":"code","source":"from workflow import (\n generate_mesh as _generate_mesh, \n convert_to_xdmf as _convert_to_xdmf,\n poisson as _poisson,\n plot_over_line as _plot_over_line,\n substitute_macros as _substitute_macros,\n compile_paper as _compile_paper,\n)","metadata":{"trusted":true},"outputs":[],"execution_count":2},{"id":"2dced28725813fc1","cell_type":"code","source":"from pyiron_workflow import Workflow, to_function_node\n\nfrom python_workflow_definition.pyiron_workflow import write_workflow_json","metadata":{"trusted":true},"outputs":[],"execution_count":3},{"id":"549ecf27-88ef-4e77-8bd4-b616cfdda2e4","cell_type":"code","source":"generate_mesh = to_function_node(\"generate_mesh\", _generate_mesh, \"generate_mesh\")\nconvert_to_xdmf = to_function_node(\"convert_to_xdmf\", _convert_to_xdmf, \"convert_to_xdmf\")\npoisson = to_function_node(\"poisson\", _poisson, \"poisson\")\nplot_over_line = to_function_node(\"plot_over_line\", _plot_over_line, \"plot_over_line\")\nsubstitute_macros = to_function_node(\"substitute_macros\", _substitute_macros, \"substitute_macros\")\ncompile_paper = to_function_node(\"compile_paper\", _compile_paper, \"compile_paper\")","metadata":{"trusted":true},"outputs":[],"execution_count":4},{"id":"3a75428e-18c7-49cf-8256-23cff58b9d6e","cell_type":"code","source":"wf = Workflow(\"my_workflow\")","metadata":{"trusted":true},"outputs":[],"execution_count":5},{"id":"8d911f98-3b80-457f-a0f4-3cb37ebf1691","cell_type":"code","source":"wf.domain_size = 2.0","metadata":{"trusted":true},"outputs":[],"execution_count":6},{"id":"c6ea980b-6761-4191-8407-7b1f78a4c3ea","cell_type":"code","source":"wf.source_directory = os.path.abspath(os.path.join(os.curdir, \"source\"))","metadata":{"trusted":true},"outputs":[],"execution_count":7},{"id":"71d411b6-cbec-489e-99e3-ba71680bcb5b","cell_type":"code","source":"wf.gmsh_output_file = generate_mesh(\n domain_size=wf.domain_size,\n source_directory=wf.source_directory,\n)","metadata":{"tags":[],"trusted":true},"outputs":[],"execution_count":8},{"id":"1d0d9804-f250-48b3-a5d0-a546d520f79b","cell_type":"code","source":"wf.meshio_output_dict = convert_to_xdmf(\n gmsh_output_file=wf.gmsh_output_file,\n)","metadata":{"tags":[],"trusted":true},"outputs":[],"execution_count":9},{"id":"7b69bcff-e2b1-4d4a-b62c-6a1c86eeb590","cell_type":"code","source":"wf.poisson_dict = poisson(\n meshio_output_xdmf=wf.meshio_output_dict[\"xdmf_file\"], \n meshio_output_h5=wf.meshio_output_dict[\"h5_file\"],\n source_directory=wf.source_directory,\n)","metadata":{"tags":[],"trusted":true},"outputs":[],"execution_count":10},{"id":"3c4a29b0-eb1e-490a-8be0-e03cfff15e0a","cell_type":"code","source":"wf.pvbatch_output_file = plot_over_line(\n poisson_output_pvd_file=wf.poisson_dict[\"pvd_file\"], \n poisson_output_vtu_file=wf.poisson_dict[\"vtu_file\"],\n source_directory=wf.source_directory,\n)","metadata":{"tags":[],"trusted":true},"outputs":[],"execution_count":11},{"id":"a0a4c233-322d-4723-9627-62ca2487bfa9","cell_type":"code","source":"wf.macros_tex_file = substitute_macros( \n pvbatch_output_file=wf.pvbatch_output_file, \n ndofs=wf.poisson_dict[\"numdofs\"], \n domain_size=wf.domain_size,\n source_directory=wf.source_directory,\n)","metadata":{"tags":[],"trusted":true},"outputs":[],"execution_count":12},{"id":"c281408f-e63d-4380-a7e6-c595d49fbb8f","cell_type":"code","source":"wf.paper_output = compile_paper(\n macros_tex=wf.macros_tex_file, \n plot_file=wf.pvbatch_output_file,\n source_directory=wf.source_directory,\n)","metadata":{"trusted":true},"outputs":[],"execution_count":13},{"id":"db4f1e15-3710-4e24-abc2-0fef417043e8","cell_type":"code","source":"wf.draw(size=(10,10))","metadata":{"trusted":true},"outputs":[{"execution_count":14,"output_type":"execute_result","data":{"image/svg+xml":"\n\n\n\n\n\nclustermy_workflow\n\nmy_workflow: Workflow\n\nclustermy_workflowInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowgmsh_output_file\n\n\n\n\n\n\n\ngmsh_output_file: generate_mesh\n\n\nclustermy_workflowgmsh_output_fileInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowgmsh_output_fileOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowmeshio_output_dict\n\n\n\n\n\n\n\nmeshio_output_dict: convert_to_xdmf\n\n\nclustermy_workflowmeshio_output_dictInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366\n\n\n\n\n\n\n\ninjected_GetItem_m7197572853956972366: GetItem\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366Inputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366OutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092\n\n\n\n\n\n\n\ninjected_GetItem_5401042856615209092: GetItem\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092Inputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092OutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowpoisson_dict\n\n\n\n\n\n\n\npoisson_dict: poisson\n\n\nclustermy_workflowpoisson_dictInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowpoisson_dictOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609\n\n\n\n\n\n\n\ninjected_GetItem_5839915180513303609: GetItem\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609Inputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609OutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100\n\n\n\n\n\n\n\ninjected_GetItem_m3497348724979863100: GetItem\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100Inputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100OutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowpvbatch_output_file\n\n\n\n\n\n\n\npvbatch_output_file: plot_over_line\n\n\nclustermy_workflowpvbatch_output_fileInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458\n\n\n\n\n\n\n\ninjected_GetItem_m6105838930489235458: GetItem\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458Inputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458OutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowmacros_tex_file\n\n\n\n\n\n\n\nmacros_tex_file: substitute_macros\n\n\nclustermy_workflowmacros_tex_fileInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowmacros_tex_fileOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowpaper_output\n\n\n\n\n\n\n\npaper_output: compile_paper\n\n\nclustermy_workflowpaper_outputInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowpaper_outputOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\n\nclustermy_workflowInputsrun\n\nrun\n\n\n\nclustermy_workflowOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowInputsgmsh_output_file__domain_size\n\ngmsh_output_file__domain_size: float\n\n\n\nclustermy_workflowgmsh_output_fileInputsdomain_size\n\ndomain_size: float\n\n\n\nclustermy_workflowInputsgmsh_output_file__domain_size->clustermy_workflowgmsh_output_fileInputsdomain_size\n\n\n\n\n\n\nclustermy_workflowInputsgmsh_output_file__source_directory\n\ngmsh_output_file__source_directory: str\n\n\n\nclustermy_workflowgmsh_output_fileInputssource_directory\n\nsource_directory: str\n\n\n\nclustermy_workflowInputsgmsh_output_file__source_directory->clustermy_workflowgmsh_output_fileInputssource_directory\n\n\n\n\n\n\nclustermy_workflowInputsinjected_GetItem_m7197572853956972366__item\n\ninjected_GetItem_m7197572853956972366__item\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366Inputsitem\n\nitem\n\n\n\nclustermy_workflowInputsinjected_GetItem_m7197572853956972366__item->clustermy_workflowinjected_GetItem_m7197572853956972366Inputsitem\n\n\n\n\n\n\nclustermy_workflowInputsinjected_GetItem_5401042856615209092__item\n\ninjected_GetItem_5401042856615209092__item\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092Inputsitem\n\nitem\n\n\n\nclustermy_workflowInputsinjected_GetItem_5401042856615209092__item->clustermy_workflowinjected_GetItem_5401042856615209092Inputsitem\n\n\n\n\n\n\nclustermy_workflowInputspoisson_dict__source_directory\n\npoisson_dict__source_directory: str\n\n\n\nclustermy_workflowpoisson_dictInputssource_directory\n\nsource_directory: str\n\n\n\nclustermy_workflowInputspoisson_dict__source_directory->clustermy_workflowpoisson_dictInputssource_directory\n\n\n\n\n\n\nclustermy_workflowInputsinjected_GetItem_5839915180513303609__item\n\ninjected_GetItem_5839915180513303609__item\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609Inputsitem\n\nitem\n\n\n\nclustermy_workflowInputsinjected_GetItem_5839915180513303609__item->clustermy_workflowinjected_GetItem_5839915180513303609Inputsitem\n\n\n\n\n\n\nclustermy_workflowInputsinjected_GetItem_m3497348724979863100__item\n\ninjected_GetItem_m3497348724979863100__item\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100Inputsitem\n\nitem\n\n\n\nclustermy_workflowInputsinjected_GetItem_m3497348724979863100__item->clustermy_workflowinjected_GetItem_m3497348724979863100Inputsitem\n\n\n\n\n\n\nclustermy_workflowInputspvbatch_output_file__source_directory\n\npvbatch_output_file__source_directory: str\n\n\n\nclustermy_workflowpvbatch_output_fileInputssource_directory\n\nsource_directory: str\n\n\n\nclustermy_workflowInputspvbatch_output_file__source_directory->clustermy_workflowpvbatch_output_fileInputssource_directory\n\n\n\n\n\n\nclustermy_workflowInputsinjected_GetItem_m6105838930489235458__item\n\ninjected_GetItem_m6105838930489235458__item\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458Inputsitem\n\nitem\n\n\n\nclustermy_workflowInputsinjected_GetItem_m6105838930489235458__item->clustermy_workflowinjected_GetItem_m6105838930489235458Inputsitem\n\n\n\n\n\n\nclustermy_workflowInputsmacros_tex_file__domain_size\n\nmacros_tex_file__domain_size: float\n\n\n\nclustermy_workflowmacros_tex_fileInputsdomain_size\n\ndomain_size: float\n\n\n\nclustermy_workflowInputsmacros_tex_file__domain_size->clustermy_workflowmacros_tex_fileInputsdomain_size\n\n\n\n\n\n\nclustermy_workflowInputsmacros_tex_file__source_directory\n\nmacros_tex_file__source_directory: str\n\n\n\nclustermy_workflowmacros_tex_fileInputssource_directory\n\nsource_directory: str\n\n\n\nclustermy_workflowInputsmacros_tex_file__source_directory->clustermy_workflowmacros_tex_fileInputssource_directory\n\n\n\n\n\n\nclustermy_workflowInputspaper_output__source_directory\n\npaper_output__source_directory: str\n\n\n\nclustermy_workflowpaper_outputInputssource_directory\n\nsource_directory: str\n\n\n\nclustermy_workflowInputspaper_output__source_directory->clustermy_workflowpaper_outputInputssource_directory\n\n\n\n\n\n\nclustermy_workflowOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowOutputsWithInjectionpaper_output__compile_paper\n\npaper_output__compile_paper: str\n\n\n\nclustermy_workflowgmsh_output_fileInputsrun\n\nrun\n\n\n\nclustermy_workflowgmsh_output_fileOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowgmsh_output_fileInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowgmsh_output_fileOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowgmsh_output_fileOutputsWithInjectiongenerate_mesh\n\ngenerate_mesh: str\n\n\n\nclustermy_workflowmeshio_output_dictInputsgmsh_output_file\n\ngmsh_output_file: str\n\n\n\nclustermy_workflowgmsh_output_fileOutputsWithInjectiongenerate_mesh->clustermy_workflowmeshio_output_dictInputsgmsh_output_file\n\n\n\n\n\n\nclustermy_workflowmeshio_output_dictInputsrun\n\nrun\n\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowmeshio_output_dictInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjectionconvert_to_xdmf\n\nconvert_to_xdmf: dict\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366Inputsobj\n\nobj\n\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjectionconvert_to_xdmf->clustermy_workflowinjected_GetItem_m7197572853956972366Inputsobj\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092Inputsobj\n\nobj\n\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjectionconvert_to_xdmf->clustermy_workflowinjected_GetItem_5401042856615209092Inputsobj\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366Inputsrun\n\nrun\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366OutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366Inputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366OutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366OutputsWithInjectiongetitem\n\ngetitem\n\n\n\nclustermy_workflowpoisson_dictInputsmeshio_output_xdmf\n\nmeshio_output_xdmf: str\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366OutputsWithInjectiongetitem->clustermy_workflowpoisson_dictInputsmeshio_output_xdmf\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092Inputsrun\n\nrun\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092OutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092Inputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092OutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092OutputsWithInjectiongetitem\n\ngetitem\n\n\n\nclustermy_workflowpoisson_dictInputsmeshio_output_h5\n\nmeshio_output_h5: str\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092OutputsWithInjectiongetitem->clustermy_workflowpoisson_dictInputsmeshio_output_h5\n\n\n\n\n\n\nclustermy_workflowpoisson_dictInputsrun\n\nrun\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowpoisson_dictInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionpoisson\n\npoisson: dict\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609Inputsobj\n\nobj\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionpoisson->clustermy_workflowinjected_GetItem_5839915180513303609Inputsobj\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100Inputsobj\n\nobj\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionpoisson->clustermy_workflowinjected_GetItem_m3497348724979863100Inputsobj\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458Inputsobj\n\nobj\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionpoisson->clustermy_workflowinjected_GetItem_m6105838930489235458Inputsobj\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609Inputsrun\n\nrun\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609OutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609Inputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609OutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609OutputsWithInjectiongetitem\n\ngetitem\n\n\n\nclustermy_workflowpvbatch_output_fileInputspoisson_output_pvd_file\n\npoisson_output_pvd_file: str\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609OutputsWithInjectiongetitem->clustermy_workflowpvbatch_output_fileInputspoisson_output_pvd_file\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100Inputsrun\n\nrun\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100OutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100Inputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100OutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100OutputsWithInjectiongetitem\n\ngetitem\n\n\n\nclustermy_workflowpvbatch_output_fileInputspoisson_output_vtu_file\n\npoisson_output_vtu_file: str\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100OutputsWithInjectiongetitem->clustermy_workflowpvbatch_output_fileInputspoisson_output_vtu_file\n\n\n\n\n\n\nclustermy_workflowpvbatch_output_fileInputsrun\n\nrun\n\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowpvbatch_output_fileInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjectionplot_over_line\n\nplot_over_line: str\n\n\n\nclustermy_workflowmacros_tex_fileInputspvbatch_output_file\n\npvbatch_output_file: str\n\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjectionplot_over_line->clustermy_workflowmacros_tex_fileInputspvbatch_output_file\n\n\n\n\n\n\nclustermy_workflowpaper_outputInputsplot_file\n\nplot_file: str\n\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjectionplot_over_line->clustermy_workflowpaper_outputInputsplot_file\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458Inputsrun\n\nrun\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458OutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458Inputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458OutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458OutputsWithInjectiongetitem\n\ngetitem\n\n\n\nclustermy_workflowmacros_tex_fileInputsndofs\n\nndofs: int\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458OutputsWithInjectiongetitem->clustermy_workflowmacros_tex_fileInputsndofs\n\n\n\n\n\n\nclustermy_workflowmacros_tex_fileInputsrun\n\nrun\n\n\n\nclustermy_workflowmacros_tex_fileOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowmacros_tex_fileInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowmacros_tex_fileOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowmacros_tex_fileOutputsWithInjectionsubstitute_macros\n\nsubstitute_macros: str\n\n\n\nclustermy_workflowpaper_outputInputsmacros_tex\n\nmacros_tex: str\n\n\n\nclustermy_workflowmacros_tex_fileOutputsWithInjectionsubstitute_macros->clustermy_workflowpaper_outputInputsmacros_tex\n\n\n\n\n\n\nclustermy_workflowpaper_outputInputsrun\n\nrun\n\n\n\nclustermy_workflowpaper_outputOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowpaper_outputInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowpaper_outputOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowpaper_outputOutputsWithInjectioncompile_paper\n\ncompile_paper: str\n\n\n\nclustermy_workflowpaper_outputOutputsWithInjectioncompile_paper->clustermy_workflowOutputsWithInjectionpaper_output__compile_paper\n\n\n\n\n\n\n","text/plain":""},"metadata":{}}],"execution_count":14},{"id":"63f29646-3846-4a97-a033-20e9df0ac214","cell_type":"code","source":"workflow_json_filename = \"pyiron_workflow_nfdi.json\"","metadata":{"trusted":true},"outputs":[],"execution_count":15},{"id":"f62111ba-9271-4987-9c7e-3b1c9f9eae7a","cell_type":"code","source":"write_workflow_json(graph_as_dict=wf.graph_as_dict, file_name=workflow_json_filename)","metadata":{"trusted":true},"outputs":[],"execution_count":16},{"id":"d789971e-8f41-45fa-832a-11fd72dea96e","cell_type":"markdown","source":"## Load Workflow with aiida","metadata":{}},{"id":"a6e85e89-5d7a-40eb-809c-ac44974e3fd7","cell_type":"code","source":"from aiida import load_profile\n\nload_profile()","metadata":{"trusted":true},"outputs":[{"execution_count":17,"output_type":"execute_result","data":{"text/plain":"Profile"},"metadata":{}}],"execution_count":17},{"id":"3de84fb7-b01b-4541-868a-92e881eb6e77","cell_type":"code","source":"from python_workflow_definition.aiida import load_workflow_json","metadata":{"trusted":true},"outputs":[],"execution_count":18},{"id":"b33f5528-10cd-47c8-8723-622902978859","cell_type":"code","source":"wg = load_workflow_json(file_name=workflow_json_filename)\nwg","metadata":{"trusted":true},"outputs":[{"execution_count":19,"output_type":"execute_result","data":{"text/plain":"NodeGraphWidget(settings={'minimap': True}, style={'width': '90%', 'height': '600px'}, value={'name': 'WorkGra…","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":1,"model_id":"6641ced7603742b8b28900587d764b7c"}},"metadata":{}}],"execution_count":19},{"id":"15282ca1-d339-40e7-ad68-8a7613ed08da","cell_type":"code","source":"wg.run()","metadata":{"trusted":true},"outputs":[{"name":"stderr","output_type":"stream","text":"05/24/2025 09:30:18 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: generate_mesh1\n05/24/2025 09:30:20 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: generate_mesh1, type: PyFunction, finished.\n05/24/2025 09:30:20 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: convert_to_xdmf2\n05/24/2025 09:30:22 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: convert_to_xdmf2, type: PyFunction, finished.\n05/24/2025 09:30:22 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: poisson3\n05/24/2025 09:30:30 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: poisson3, type: PyFunction, finished.\n05/24/2025 09:30:30 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: plot_over_line4\n05/24/2025 09:30:33 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: plot_over_line4, type: PyFunction, finished.\n05/24/2025 09:30:33 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: substitute_macros5\n05/24/2025 09:30:34 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: substitute_macros5, type: PyFunction, finished.\n05/24/2025 09:30:34 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: compile_paper6\n05/24/2025 09:30:50 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: compile_paper6, type: PyFunction, finished.\n05/24/2025 09:30:50 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: \n05/24/2025 09:30:50 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|finalize]: Finalize workgraph.\n"}],"execution_count":20},{"id":"55dc8d12-dfe6-4465-a368-b7e590ae6800","cell_type":"markdown","source":"## Load Workflow with jobflow","metadata":{}},{"id":"dff46eb8-e0e7-49bb-8c40-0db2df133124","cell_type":"code","source":"from python_workflow_definition.jobflow import load_workflow_json","metadata":{"trusted":true},"outputs":[],"execution_count":21},{"id":"6a189459-84e4-4738-ada1-37ee8c65b2ab","cell_type":"code","source":"from jobflow.managers.local import run_locally","metadata":{"trusted":true},"outputs":[],"execution_count":22},{"id":"6e7f3614-c971-4e2d-83f0-96f0d0fc04de","cell_type":"code","source":"flow = load_workflow_json(file_name=workflow_json_filename)","metadata":{"trusted":true},"outputs":[],"execution_count":23},{"id":"2d87ed45-f5d9-403f-a03a-26be4a47a3ef","cell_type":"code","source":"result = run_locally(flow)\nresult","metadata":{"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":"2025-05-24 09:31:15,165 INFO Started executing jobs locally\n2025-05-24 09:31:15,364 INFO Starting job - generate_mesh (4cc4e3d9-39aa-4c20-a047-a029b6812d61)\n2025-05-24 09:31:16,480 INFO Finished job - generate_mesh (4cc4e3d9-39aa-4c20-a047-a029b6812d61)\n2025-05-24 09:31:16,481 INFO Starting job - convert_to_xdmf (8527c563-c0b1-4ae5-a6c3-00ae25ad2508)\n2025-05-24 09:31:17,834 INFO Finished job - convert_to_xdmf (8527c563-c0b1-4ae5-a6c3-00ae25ad2508)\n2025-05-24 09:31:17,836 INFO Starting job - poisson (58b31d24-51f9-4b9a-ac3e-88a7f01a09ad)\n2025-05-24 09:31:20,318 INFO Finished job - poisson (58b31d24-51f9-4b9a-ac3e-88a7f01a09ad)\n2025-05-24 09:31:20,318 INFO Starting job - plot_over_line (b095dc38-f858-4c34-876c-df8f8b851a5f)\n2025-05-24 09:31:21,717 INFO Finished job - plot_over_line (b095dc38-f858-4c34-876c-df8f8b851a5f)\n2025-05-24 09:31:21,718 INFO Starting job - substitute_macros (656e820d-20d5-4665-bb8a-43807ed6b2e9)\n2025-05-24 09:31:22,542 INFO Finished job - substitute_macros (656e820d-20d5-4665-bb8a-43807ed6b2e9)\n2025-05-24 09:31:22,543 INFO Starting job - compile_paper (52de4e6d-7574-401b-bcdb-ae6c298cc6b9)\n2025-05-24 09:31:24,556 INFO Finished job - compile_paper (52de4e6d-7574-401b-bcdb-ae6c298cc6b9)\n2025-05-24 09:31:24,557 INFO Finished executing jobs locally\n"},{"execution_count":24,"output_type":"execute_result","data":{"text/plain":"{'4cc4e3d9-39aa-4c20-a047-a029b6812d61': {1: Response(output='/home/jovyan/example_workflows/nfdi/preprocessing/square.msh', detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))},\n '8527c563-c0b1-4ae5-a6c3-00ae25ad2508': {1: Response(output={'xdmf_file': '/home/jovyan/example_workflows/nfdi/preprocessing/square.xdmf', 'h5_file': '/home/jovyan/example_workflows/nfdi/preprocessing/square.h5'}, detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))},\n '58b31d24-51f9-4b9a-ac3e-88a7f01a09ad': {1: Response(output={'numdofs': 357, 'pvd_file': '/home/jovyan/example_workflows/nfdi/processing/poisson.pvd', 'vtu_file': '/home/jovyan/example_workflows/nfdi/processing/poisson000000.vtu'}, detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))},\n 'b095dc38-f858-4c34-876c-df8f8b851a5f': {1: Response(output='/home/jovyan/example_workflows/nfdi/postprocessing/plotoverline.csv', detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))},\n '656e820d-20d5-4665-bb8a-43807ed6b2e9': {1: Response(output='/home/jovyan/example_workflows/nfdi/postprocessing/macros.tex', detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))},\n '52de4e6d-7574-401b-bcdb-ae6c298cc6b9': {1: Response(output='/home/jovyan/example_workflows/nfdi/postprocessing/paper.pdf', detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))}}"},"metadata":{}}],"execution_count":24},{"id":"ebd3248b-ff75-40bf-a488-b5e338342671","cell_type":"markdown","source":"## Load Workflow with pyiron_base","metadata":{}},{"id":"c7e1047f-0d42-4777-911d-ab405396401a","cell_type":"code","source":"from python_workflow_definition.pyiron_base import load_workflow_json","metadata":{"trusted":true},"outputs":[],"execution_count":25},{"id":"6aeba274-10d3-4e79-8ee4-4d870f163939","cell_type":"code","source":"delayed_object_lst = load_workflow_json(file_name=workflow_json_filename)\ndelayed_object_lst[-1].draw()","metadata":{"trusted":true},"outputs":[{"output_type":"display_data","data":{"text/plain":"","image/svg+xml":"\n\n\n\n\ncreate_function_job_ba0bf2668d8f7b42ff717c2cb79b5920\n\ncreate_function_job=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d9280>\n\n\n\nplot_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\nplot_file=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8da0>\n\n\n\nplot_file_5007b1d67345934b2ffcdeaaad6ea2ab->create_function_job_ba0bf2668d8f7b42ff717c2cb79b5920\n\n\n\n\n\npoisson_output_vtu_file_51141ca0ec0a07922807865d8a067793\n\npoisson_output_vtu_file=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8a40>\n\n\n\npoisson_output_vtu_file_51141ca0ec0a07922807865d8a067793->plot_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\npvbatch_output_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\npvbatch_output_file=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8da0>\n\n\n\npoisson_output_vtu_file_51141ca0ec0a07922807865d8a067793->pvbatch_output_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\nmacros_tex_02d82cedf90ad323f7eef57ddf383160\n\nmacros_tex=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d9070>\n\n\n\npvbatch_output_file_5007b1d67345934b2ffcdeaaad6ea2ab->macros_tex_02d82cedf90ad323f7eef57ddf383160\n\n\n\n\n\nmeshio_output_h5_0af06cb14e9ddc7585647c44e4d4ef88\n\nmeshio_output_h5=<pyiron_base.project.delayed.DelayedObject object at 0x76dba3cb2600>\n\n\n\nmeshio_output_h5_0af06cb14e9ddc7585647c44e4d4ef88->poisson_output_vtu_file_51141ca0ec0a07922807865d8a067793\n\n\n\n\n\npoisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513\n\npoisson_output_pvd_file=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8a10>\n\n\n\nmeshio_output_h5_0af06cb14e9ddc7585647c44e4d4ef88->poisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513\n\n\n\n\n\nndofs_ac7427db28c0527371c0e9b27a57024b\n\nndofs=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8e30>\n\n\n\nmeshio_output_h5_0af06cb14e9ddc7585647c44e4d4ef88->ndofs_ac7427db28c0527371c0e9b27a57024b\n\n\n\n\n\npoisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513->plot_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\npoisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513->pvbatch_output_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\nndofs_ac7427db28c0527371c0e9b27a57024b->macros_tex_02d82cedf90ad323f7eef57ddf383160\n\n\n\n\n\ngmsh_output_file_6ef41eb684984daa8337ee65c2aa9f18\n\ngmsh_output_file=<pyiron_base.project.delayed.DelayedObject object at 0x76dba89319d0>\n\n\n\ngmsh_output_file_6ef41eb684984daa8337ee65c2aa9f18->meshio_output_h5_0af06cb14e9ddc7585647c44e4d4ef88\n\n\n\n\n\nmeshio_output_xdmf_c383e7d9f83d7cdc8ae04ebf9c264173\n\nmeshio_output_xdmf=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8710>\n\n\n\ngmsh_output_file_6ef41eb684984daa8337ee65c2aa9f18->meshio_output_xdmf_c383e7d9f83d7cdc8ae04ebf9c264173\n\n\n\n\n\nmeshio_output_xdmf_c383e7d9f83d7cdc8ae04ebf9c264173->poisson_output_vtu_file_51141ca0ec0a07922807865d8a067793\n\n\n\n\n\nmeshio_output_xdmf_c383e7d9f83d7cdc8ae04ebf9c264173->poisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513\n\n\n\n\n\nmeshio_output_xdmf_c383e7d9f83d7cdc8ae04ebf9c264173->ndofs_ac7427db28c0527371c0e9b27a57024b\n\n\n\n\n\ndomain_size_f12a7f1986b9dd058dfc666dbe230b20\n\ndomain_size=2.0\n\n\n\ndomain_size_f12a7f1986b9dd058dfc666dbe230b20->gmsh_output_file_6ef41eb684984daa8337ee65c2aa9f18\n\n\n\n\n\ndomain_size_f12a7f1986b9dd058dfc666dbe230b20->macros_tex_02d82cedf90ad323f7eef57ddf383160\n\n\n\n\n\nmacros_tex_02d82cedf90ad323f7eef57ddf383160->create_function_job_ba0bf2668d8f7b42ff717c2cb79b5920\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696\n\nsource_directory=/home/jovyan/example_workflows/nfdi/source\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->create_function_job_ba0bf2668d8f7b42ff717c2cb79b5920\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->plot_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->poisson_output_vtu_file_51141ca0ec0a07922807865d8a067793\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->pvbatch_output_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->poisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->ndofs_ac7427db28c0527371c0e9b27a57024b\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->gmsh_output_file_6ef41eb684984daa8337ee65c2aa9f18\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->macros_tex_02d82cedf90ad323f7eef57ddf383160\n\n\n\n\n"},"metadata":{}}],"execution_count":26},{"id":"ea92c7ad-afef-423c-b8fa-9d24c1267c73","cell_type":"code","source":"delayed_object_lst[-1].pull()","metadata":{"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":"The job generate_mesh_47725c16637f799ac042e47468005db3 was saved and received the ID: 1\nThe job convert_to_xdmf_d6a46eb9a4ec352aa996e783ec3c785f was saved and received the ID: 2\nThe job poisson_3c147fc86db87cf0c0f94bda333f8cd8 was saved and received the ID: 3\nThe job plot_over_line_ef50933291910dadcc8311924971e127 was saved and received the ID: 4\nThe job substitute_macros_63766eafd6b1980c7832dd8c9a97c96e was saved and received the ID: 5\nThe job compile_paper_128d1d58374953c00e95b8de62cbb10b was saved and received the ID: 6\n"},{"execution_count":27,"output_type":"execute_result","data":{"text/plain":"'/home/jovyan/example_workflows/nfdi/postprocessing/paper.pdf'"},"metadata":{}}],"execution_count":27},{"id":"32287743-e59a-467c-92fe-8586f199e36e","cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]} \ No newline at end of file +{ + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.12.8", + "mimetype": "text/x-python", + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "pygments_lexer": "ipython3", + "nbconvert_exporter": "python", + "file_extension": ".py" + } + }, + "nbformat_minor": 5, + "nbformat": 4, + "cells": [ + { + "id": "106ded66-d202-46ac-82b0-2755ca309bdd", + "cell_type": "markdown", + "source": "# pyiron\n\nhttps://github.com/BAMresearch/NFDI4IngScientificWorkflowRequirements\n\nThis notebook defines the NFDI4Ing file-based workflow benchmark with [`pyiron_workflow`](https://github.com/pyiron/pyiron_workflow) and then loads the resulting `workflow.json` into `aiida`, `jobflow` and `pyiron_base`. Every stage of the pipeline shells out to an external command-line tool (`gmsh`, `meshio`, a FEniCS-based Poisson solver, ParaView's `pvbatch`, `tectonic`) in its own conda environment and passes file paths between stages rather than in-memory Python objects — see [`workflow.py`](workflow.py).", + "metadata": {} + }, + { + "id": "91dd48ea-aa7e-4937-a68e-59fc5017eb1e", + "cell_type": "markdown", + "source": "## Define workflow with pyiron_workflow\n\n`to_function_node` wraps each plain Python function into a `pyiron_workflow` node; the three arguments are the node's registered class name, the function itself, and the name of its (single, unlabeled) output. Nodes are assembled into a graph by assigning them as attributes of a `Workflow` object (`wf. = node(...)`), and dictionary outputs such as `convert_to_xdmf`'s are indexed with `wf.meshio_output_dict[\"xdmf_file\"]` when wiring the next node's input.", + "metadata": {} + }, + { + "id": "2c9622f5-ab7e-460e-b8e4-8d21413eda77", + "cell_type": "code", + "source": "import os", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 1 + }, + { + "id": "d265bb5aa6af79d6", + "cell_type": "code", + "source": "from workflow import (\n generate_mesh as _generate_mesh, \n convert_to_xdmf as _convert_to_xdmf,\n poisson as _poisson,\n plot_over_line as _plot_over_line,\n substitute_macros as _substitute_macros,\n compile_paper as _compile_paper,\n)", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 2 + }, + { + "id": "2dced28725813fc1", + "cell_type": "code", + "source": "from pyiron_workflow import Workflow, to_function_node\n\nfrom python_workflow_definition.pyiron_workflow import write_workflow_json", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 3 + }, + { + "id": "549ecf27-88ef-4e77-8bd4-b616cfdda2e4", + "cell_type": "code", + "source": "generate_mesh = to_function_node(\"generate_mesh\", _generate_mesh, \"generate_mesh\")\nconvert_to_xdmf = to_function_node(\"convert_to_xdmf\", _convert_to_xdmf, \"convert_to_xdmf\")\npoisson = to_function_node(\"poisson\", _poisson, \"poisson\")\nplot_over_line = to_function_node(\"plot_over_line\", _plot_over_line, \"plot_over_line\")\nsubstitute_macros = to_function_node(\"substitute_macros\", _substitute_macros, \"substitute_macros\")\ncompile_paper = to_function_node(\"compile_paper\", _compile_paper, \"compile_paper\")", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 4 + }, + { + "id": "3a75428e-18c7-49cf-8256-23cff58b9d6e", + "cell_type": "code", + "source": "wf = Workflow(\"my_workflow\")", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 5 + }, + { + "id": "8d911f98-3b80-457f-a0f4-3cb37ebf1691", + "cell_type": "code", + "source": "wf.domain_size = 2.0", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 6 + }, + { + "id": "c6ea980b-6761-4191-8407-7b1f78a4c3ea", + "cell_type": "code", + "source": "wf.source_directory = os.path.abspath(os.path.join(os.curdir, \"source\"))", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 7 + }, + { + "cell_type": "markdown", + "id": "7c642a24", + "source": "The assignments below build the six-stage pipeline as attributes of `wf`: mesh generation, XDMF conversion, the Poisson solve, line-plot postprocessing, LaTeX macro substitution and paper compilation. Each stage runs an external tool in its own conda environment (see `source/envs/`), so the values connecting them are file paths on disk rather than shared Python objects.", + "metadata": {} + }, + { + "id": "71d411b6-cbec-489e-99e3-ba71680bcb5b", + "cell_type": "code", + "source": "wf.gmsh_output_file = generate_mesh(\n domain_size=wf.domain_size,\n source_directory=wf.source_directory,\n)", + "metadata": { + "tags": [], + "trusted": true + }, + "outputs": [], + "execution_count": 8 + }, + { + "id": "1d0d9804-f250-48b3-a5d0-a546d520f79b", + "cell_type": "code", + "source": "wf.meshio_output_dict = convert_to_xdmf(\n gmsh_output_file=wf.gmsh_output_file,\n)", + "metadata": { + "tags": [], + "trusted": true + }, + "outputs": [], + "execution_count": 9 + }, + { + "id": "7b69bcff-e2b1-4d4a-b62c-6a1c86eeb590", + "cell_type": "code", + "source": "wf.poisson_dict = poisson(\n meshio_output_xdmf=wf.meshio_output_dict[\"xdmf_file\"], \n meshio_output_h5=wf.meshio_output_dict[\"h5_file\"],\n source_directory=wf.source_directory,\n)", + "metadata": { + "tags": [], + "trusted": true + }, + "outputs": [], + "execution_count": 10 + }, + { + "id": "3c4a29b0-eb1e-490a-8be0-e03cfff15e0a", + "cell_type": "code", + "source": "wf.pvbatch_output_file = plot_over_line(\n poisson_output_pvd_file=wf.poisson_dict[\"pvd_file\"], \n poisson_output_vtu_file=wf.poisson_dict[\"vtu_file\"],\n source_directory=wf.source_directory,\n)", + "metadata": { + "tags": [], + "trusted": true + }, + "outputs": [], + "execution_count": 11 + }, + { + "id": "a0a4c233-322d-4723-9627-62ca2487bfa9", + "cell_type": "code", + "source": "wf.macros_tex_file = substitute_macros( \n pvbatch_output_file=wf.pvbatch_output_file, \n ndofs=wf.poisson_dict[\"numdofs\"], \n domain_size=wf.domain_size,\n source_directory=wf.source_directory,\n)", + "metadata": { + "tags": [], + "trusted": true + }, + "outputs": [], + "execution_count": 12 + }, + { + "id": "c281408f-e63d-4380-a7e6-c595d49fbb8f", + "cell_type": "code", + "source": "wf.paper_output = compile_paper(\n macros_tex=wf.macros_tex_file, \n plot_file=wf.pvbatch_output_file,\n source_directory=wf.source_directory,\n)", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 13 + }, + { + "cell_type": "markdown", + "id": "cb9a923d", + "source": "`wf.draw()` renders the assembled node graph; `wf.graph_as_dict` is then handed to `write_workflow_json`, which serializes it into the PWD `workflow.json` format for the other engines to load.", + "metadata": {} + }, + { + "id": "db4f1e15-3710-4e24-abc2-0fef417043e8", + "cell_type": "code", + "source": "wf.draw(size=(10,10))", + "metadata": { + "trusted": true + }, + "outputs": [ + { + "execution_count": 14, + "output_type": "execute_result", + "data": { + "image/svg+xml": "\n\n\n\n\n\nclustermy_workflow\n\nmy_workflow: Workflow\n\nclustermy_workflowInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowgmsh_output_file\n\n\n\n\n\n\n\ngmsh_output_file: generate_mesh\n\n\nclustermy_workflowgmsh_output_fileInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowgmsh_output_fileOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowmeshio_output_dict\n\n\n\n\n\n\n\nmeshio_output_dict: convert_to_xdmf\n\n\nclustermy_workflowmeshio_output_dictInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366\n\n\n\n\n\n\n\ninjected_GetItem_m7197572853956972366: GetItem\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366Inputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366OutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092\n\n\n\n\n\n\n\ninjected_GetItem_5401042856615209092: GetItem\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092Inputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092OutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowpoisson_dict\n\n\n\n\n\n\n\npoisson_dict: poisson\n\n\nclustermy_workflowpoisson_dictInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowpoisson_dictOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609\n\n\n\n\n\n\n\ninjected_GetItem_5839915180513303609: GetItem\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609Inputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609OutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100\n\n\n\n\n\n\n\ninjected_GetItem_m3497348724979863100: GetItem\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100Inputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100OutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowpvbatch_output_file\n\n\n\n\n\n\n\npvbatch_output_file: plot_over_line\n\n\nclustermy_workflowpvbatch_output_fileInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458\n\n\n\n\n\n\n\ninjected_GetItem_m6105838930489235458: GetItem\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458Inputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458OutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowmacros_tex_file\n\n\n\n\n\n\n\nmacros_tex_file: substitute_macros\n\n\nclustermy_workflowmacros_tex_fileInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowmacros_tex_fileOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowpaper_output\n\n\n\n\n\n\n\npaper_output: compile_paper\n\n\nclustermy_workflowpaper_outputInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowpaper_outputOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\n\nclustermy_workflowInputsrun\n\nrun\n\n\n\nclustermy_workflowOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowInputsgmsh_output_file__domain_size\n\ngmsh_output_file__domain_size: float\n\n\n\nclustermy_workflowgmsh_output_fileInputsdomain_size\n\ndomain_size: float\n\n\n\nclustermy_workflowInputsgmsh_output_file__domain_size->clustermy_workflowgmsh_output_fileInputsdomain_size\n\n\n\n\n\n\nclustermy_workflowInputsgmsh_output_file__source_directory\n\ngmsh_output_file__source_directory: str\n\n\n\nclustermy_workflowgmsh_output_fileInputssource_directory\n\nsource_directory: str\n\n\n\nclustermy_workflowInputsgmsh_output_file__source_directory->clustermy_workflowgmsh_output_fileInputssource_directory\n\n\n\n\n\n\nclustermy_workflowInputsinjected_GetItem_m7197572853956972366__item\n\ninjected_GetItem_m7197572853956972366__item\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366Inputsitem\n\nitem\n\n\n\nclustermy_workflowInputsinjected_GetItem_m7197572853956972366__item->clustermy_workflowinjected_GetItem_m7197572853956972366Inputsitem\n\n\n\n\n\n\nclustermy_workflowInputsinjected_GetItem_5401042856615209092__item\n\ninjected_GetItem_5401042856615209092__item\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092Inputsitem\n\nitem\n\n\n\nclustermy_workflowInputsinjected_GetItem_5401042856615209092__item->clustermy_workflowinjected_GetItem_5401042856615209092Inputsitem\n\n\n\n\n\n\nclustermy_workflowInputspoisson_dict__source_directory\n\npoisson_dict__source_directory: str\n\n\n\nclustermy_workflowpoisson_dictInputssource_directory\n\nsource_directory: str\n\n\n\nclustermy_workflowInputspoisson_dict__source_directory->clustermy_workflowpoisson_dictInputssource_directory\n\n\n\n\n\n\nclustermy_workflowInputsinjected_GetItem_5839915180513303609__item\n\ninjected_GetItem_5839915180513303609__item\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609Inputsitem\n\nitem\n\n\n\nclustermy_workflowInputsinjected_GetItem_5839915180513303609__item->clustermy_workflowinjected_GetItem_5839915180513303609Inputsitem\n\n\n\n\n\n\nclustermy_workflowInputsinjected_GetItem_m3497348724979863100__item\n\ninjected_GetItem_m3497348724979863100__item\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100Inputsitem\n\nitem\n\n\n\nclustermy_workflowInputsinjected_GetItem_m3497348724979863100__item->clustermy_workflowinjected_GetItem_m3497348724979863100Inputsitem\n\n\n\n\n\n\nclustermy_workflowInputspvbatch_output_file__source_directory\n\npvbatch_output_file__source_directory: str\n\n\n\nclustermy_workflowpvbatch_output_fileInputssource_directory\n\nsource_directory: str\n\n\n\nclustermy_workflowInputspvbatch_output_file__source_directory->clustermy_workflowpvbatch_output_fileInputssource_directory\n\n\n\n\n\n\nclustermy_workflowInputsinjected_GetItem_m6105838930489235458__item\n\ninjected_GetItem_m6105838930489235458__item\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458Inputsitem\n\nitem\n\n\n\nclustermy_workflowInputsinjected_GetItem_m6105838930489235458__item->clustermy_workflowinjected_GetItem_m6105838930489235458Inputsitem\n\n\n\n\n\n\nclustermy_workflowInputsmacros_tex_file__domain_size\n\nmacros_tex_file__domain_size: float\n\n\n\nclustermy_workflowmacros_tex_fileInputsdomain_size\n\ndomain_size: float\n\n\n\nclustermy_workflowInputsmacros_tex_file__domain_size->clustermy_workflowmacros_tex_fileInputsdomain_size\n\n\n\n\n\n\nclustermy_workflowInputsmacros_tex_file__source_directory\n\nmacros_tex_file__source_directory: str\n\n\n\nclustermy_workflowmacros_tex_fileInputssource_directory\n\nsource_directory: str\n\n\n\nclustermy_workflowInputsmacros_tex_file__source_directory->clustermy_workflowmacros_tex_fileInputssource_directory\n\n\n\n\n\n\nclustermy_workflowInputspaper_output__source_directory\n\npaper_output__source_directory: str\n\n\n\nclustermy_workflowpaper_outputInputssource_directory\n\nsource_directory: str\n\n\n\nclustermy_workflowInputspaper_output__source_directory->clustermy_workflowpaper_outputInputssource_directory\n\n\n\n\n\n\nclustermy_workflowOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowOutputsWithInjectionpaper_output__compile_paper\n\npaper_output__compile_paper: str\n\n\n\nclustermy_workflowgmsh_output_fileInputsrun\n\nrun\n\n\n\nclustermy_workflowgmsh_output_fileOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowgmsh_output_fileInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowgmsh_output_fileOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowgmsh_output_fileOutputsWithInjectiongenerate_mesh\n\ngenerate_mesh: str\n\n\n\nclustermy_workflowmeshio_output_dictInputsgmsh_output_file\n\ngmsh_output_file: str\n\n\n\nclustermy_workflowgmsh_output_fileOutputsWithInjectiongenerate_mesh->clustermy_workflowmeshio_output_dictInputsgmsh_output_file\n\n\n\n\n\n\nclustermy_workflowmeshio_output_dictInputsrun\n\nrun\n\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowmeshio_output_dictInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjectionconvert_to_xdmf\n\nconvert_to_xdmf: dict\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366Inputsobj\n\nobj\n\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjectionconvert_to_xdmf->clustermy_workflowinjected_GetItem_m7197572853956972366Inputsobj\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092Inputsobj\n\nobj\n\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjectionconvert_to_xdmf->clustermy_workflowinjected_GetItem_5401042856615209092Inputsobj\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366Inputsrun\n\nrun\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366OutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366Inputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366OutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366OutputsWithInjectiongetitem\n\ngetitem\n\n\n\nclustermy_workflowpoisson_dictInputsmeshio_output_xdmf\n\nmeshio_output_xdmf: str\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366OutputsWithInjectiongetitem->clustermy_workflowpoisson_dictInputsmeshio_output_xdmf\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092Inputsrun\n\nrun\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092OutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092Inputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092OutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092OutputsWithInjectiongetitem\n\ngetitem\n\n\n\nclustermy_workflowpoisson_dictInputsmeshio_output_h5\n\nmeshio_output_h5: str\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092OutputsWithInjectiongetitem->clustermy_workflowpoisson_dictInputsmeshio_output_h5\n\n\n\n\n\n\nclustermy_workflowpoisson_dictInputsrun\n\nrun\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowpoisson_dictInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionpoisson\n\npoisson: dict\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609Inputsobj\n\nobj\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionpoisson->clustermy_workflowinjected_GetItem_5839915180513303609Inputsobj\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100Inputsobj\n\nobj\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionpoisson->clustermy_workflowinjected_GetItem_m3497348724979863100Inputsobj\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458Inputsobj\n\nobj\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionpoisson->clustermy_workflowinjected_GetItem_m6105838930489235458Inputsobj\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609Inputsrun\n\nrun\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609OutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609Inputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609OutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609OutputsWithInjectiongetitem\n\ngetitem\n\n\n\nclustermy_workflowpvbatch_output_fileInputspoisson_output_pvd_file\n\npoisson_output_pvd_file: str\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609OutputsWithInjectiongetitem->clustermy_workflowpvbatch_output_fileInputspoisson_output_pvd_file\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100Inputsrun\n\nrun\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100OutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100Inputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100OutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100OutputsWithInjectiongetitem\n\ngetitem\n\n\n\nclustermy_workflowpvbatch_output_fileInputspoisson_output_vtu_file\n\npoisson_output_vtu_file: str\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100OutputsWithInjectiongetitem->clustermy_workflowpvbatch_output_fileInputspoisson_output_vtu_file\n\n\n\n\n\n\nclustermy_workflowpvbatch_output_fileInputsrun\n\nrun\n\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowpvbatch_output_fileInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjectionplot_over_line\n\nplot_over_line: str\n\n\n\nclustermy_workflowmacros_tex_fileInputspvbatch_output_file\n\npvbatch_output_file: str\n\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjectionplot_over_line->clustermy_workflowmacros_tex_fileInputspvbatch_output_file\n\n\n\n\n\n\nclustermy_workflowpaper_outputInputsplot_file\n\nplot_file: str\n\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjectionplot_over_line->clustermy_workflowpaper_outputInputsplot_file\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458Inputsrun\n\nrun\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458OutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458Inputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458OutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458OutputsWithInjectiongetitem\n\ngetitem\n\n\n\nclustermy_workflowmacros_tex_fileInputsndofs\n\nndofs: int\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458OutputsWithInjectiongetitem->clustermy_workflowmacros_tex_fileInputsndofs\n\n\n\n\n\n\nclustermy_workflowmacros_tex_fileInputsrun\n\nrun\n\n\n\nclustermy_workflowmacros_tex_fileOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowmacros_tex_fileInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowmacros_tex_fileOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowmacros_tex_fileOutputsWithInjectionsubstitute_macros\n\nsubstitute_macros: str\n\n\n\nclustermy_workflowpaper_outputInputsmacros_tex\n\nmacros_tex: str\n\n\n\nclustermy_workflowmacros_tex_fileOutputsWithInjectionsubstitute_macros->clustermy_workflowpaper_outputInputsmacros_tex\n\n\n\n\n\n\nclustermy_workflowpaper_outputInputsrun\n\nrun\n\n\n\nclustermy_workflowpaper_outputOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowpaper_outputInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowpaper_outputOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowpaper_outputOutputsWithInjectioncompile_paper\n\ncompile_paper: str\n\n\n\nclustermy_workflowpaper_outputOutputsWithInjectioncompile_paper->clustermy_workflowOutputsWithInjectionpaper_output__compile_paper\n\n\n\n\n\n\n", + "text/plain": "" + }, + "metadata": {} + } + ], + "execution_count": 14 + }, + { + "id": "63f29646-3846-4a97-a033-20e9df0ac214", + "cell_type": "code", + "source": "workflow_json_filename = \"pyiron_workflow_nfdi.json\"", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 15 + }, + { + "id": "f62111ba-9271-4987-9c7e-3b1c9f9eae7a", + "cell_type": "code", + "source": "write_workflow_json(graph_as_dict=wf.graph_as_dict, file_name=workflow_json_filename)", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 16 + }, + { + "id": "d789971e-8f41-45fa-832a-11fd72dea96e", + "cell_type": "markdown", + "source": "## Load Workflow with aiida\n\n`load_workflow_json` rebuilds the pipeline as an `aiida-workgraph` `WorkGraph` from `pyiron_workflow_nfdi.json`, after connecting to a local AiiDA profile with `load_profile()`. `wg.run()` re-executes the whole pipeline under `aiida`.", + "metadata": {} + }, + { + "id": "a6e85e89-5d7a-40eb-809c-ac44974e3fd7", + "cell_type": "code", + "source": "from aiida import load_profile\n\nload_profile()", + "metadata": { + "trusted": true + }, + "outputs": [ + { + "execution_count": 17, + "output_type": "execute_result", + "data": { + "text/plain": "Profile" + }, + "metadata": {} + } + ], + "execution_count": 17 + }, + { + "id": "3de84fb7-b01b-4541-868a-92e881eb6e77", + "cell_type": "code", + "source": "from python_workflow_definition.aiida import load_workflow_json", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 18 + }, + { + "id": "b33f5528-10cd-47c8-8723-622902978859", + "cell_type": "code", + "source": "wg = load_workflow_json(file_name=workflow_json_filename)\nwg", + "metadata": { + "trusted": true + }, + "outputs": [ + { + "execution_count": 19, + "output_type": "execute_result", + "data": { + "text/plain": "NodeGraphWidget(settings={'minimap': True}, style={'width': '90%', 'height': '600px'}, value={'name': 'WorkGra…", + "application/vnd.jupyter.widget-view+json": { + "version_major": 2, + "version_minor": 1, + "model_id": "6641ced7603742b8b28900587d764b7c" + } + }, + "metadata": {} + } + ], + "execution_count": 19 + }, + { + "id": "15282ca1-d339-40e7-ad68-8a7613ed08da", + "cell_type": "code", + "source": "wg.run()", + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": "05/24/2025 09:30:18 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: generate_mesh1\n05/24/2025 09:30:20 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: generate_mesh1, type: PyFunction, finished.\n05/24/2025 09:30:20 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: convert_to_xdmf2\n05/24/2025 09:30:22 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: convert_to_xdmf2, type: PyFunction, finished.\n05/24/2025 09:30:22 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: poisson3\n05/24/2025 09:30:30 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: poisson3, type: PyFunction, finished.\n05/24/2025 09:30:30 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: plot_over_line4\n05/24/2025 09:30:33 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: plot_over_line4, type: PyFunction, finished.\n05/24/2025 09:30:33 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: substitute_macros5\n05/24/2025 09:30:34 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: substitute_macros5, type: PyFunction, finished.\n05/24/2025 09:30:34 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: compile_paper6\n05/24/2025 09:30:50 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: compile_paper6, type: PyFunction, finished.\n05/24/2025 09:30:50 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: \n05/24/2025 09:30:50 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|finalize]: Finalize workgraph.\n" + } + ], + "execution_count": 20 + }, + { + "id": "55dc8d12-dfe6-4465-a368-b7e590ae6800", + "cell_type": "markdown", + "source": "## Load Workflow with jobflow\n\nThe same JSON is reconstructed as a jobflow `Flow` and executed locally with `run_locally`, independently of the `pyiron_workflow` objects used to define it.", + "metadata": {} + }, + { + "id": "dff46eb8-e0e7-49bb-8c40-0db2df133124", + "cell_type": "code", + "source": "from python_workflow_definition.jobflow import load_workflow_json", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 21 + }, + { + "id": "6a189459-84e4-4738-ada1-37ee8c65b2ab", + "cell_type": "code", + "source": "from jobflow.managers.local import run_locally", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 22 + }, + { + "id": "6e7f3614-c971-4e2d-83f0-96f0d0fc04de", + "cell_type": "code", + "source": "flow = load_workflow_json(file_name=workflow_json_filename)", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 23 + }, + { + "id": "2d87ed45-f5d9-403f-a03a-26be4a47a3ef", + "cell_type": "code", + "source": "result = run_locally(flow)\nresult", + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": "2025-05-24 09:31:15,165 INFO Started executing jobs locally\n2025-05-24 09:31:15,364 INFO Starting job - generate_mesh (4cc4e3d9-39aa-4c20-a047-a029b6812d61)\n2025-05-24 09:31:16,480 INFO Finished job - generate_mesh (4cc4e3d9-39aa-4c20-a047-a029b6812d61)\n2025-05-24 09:31:16,481 INFO Starting job - convert_to_xdmf (8527c563-c0b1-4ae5-a6c3-00ae25ad2508)\n2025-05-24 09:31:17,834 INFO Finished job - convert_to_xdmf (8527c563-c0b1-4ae5-a6c3-00ae25ad2508)\n2025-05-24 09:31:17,836 INFO Starting job - poisson (58b31d24-51f9-4b9a-ac3e-88a7f01a09ad)\n2025-05-24 09:31:20,318 INFO Finished job - poisson (58b31d24-51f9-4b9a-ac3e-88a7f01a09ad)\n2025-05-24 09:31:20,318 INFO Starting job - plot_over_line (b095dc38-f858-4c34-876c-df8f8b851a5f)\n2025-05-24 09:31:21,717 INFO Finished job - plot_over_line (b095dc38-f858-4c34-876c-df8f8b851a5f)\n2025-05-24 09:31:21,718 INFO Starting job - substitute_macros (656e820d-20d5-4665-bb8a-43807ed6b2e9)\n2025-05-24 09:31:22,542 INFO Finished job - substitute_macros (656e820d-20d5-4665-bb8a-43807ed6b2e9)\n2025-05-24 09:31:22,543 INFO Starting job - compile_paper (52de4e6d-7574-401b-bcdb-ae6c298cc6b9)\n2025-05-24 09:31:24,556 INFO Finished job - compile_paper (52de4e6d-7574-401b-bcdb-ae6c298cc6b9)\n2025-05-24 09:31:24,557 INFO Finished executing jobs locally\n" + }, + { + "execution_count": 24, + "output_type": "execute_result", + "data": { + "text/plain": "{'4cc4e3d9-39aa-4c20-a047-a029b6812d61': {1: Response(output='/home/jovyan/example_workflows/nfdi/preprocessing/square.msh', detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))},\n '8527c563-c0b1-4ae5-a6c3-00ae25ad2508': {1: Response(output={'xdmf_file': '/home/jovyan/example_workflows/nfdi/preprocessing/square.xdmf', 'h5_file': '/home/jovyan/example_workflows/nfdi/preprocessing/square.h5'}, detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))},\n '58b31d24-51f9-4b9a-ac3e-88a7f01a09ad': {1: Response(output={'numdofs': 357, 'pvd_file': '/home/jovyan/example_workflows/nfdi/processing/poisson.pvd', 'vtu_file': '/home/jovyan/example_workflows/nfdi/processing/poisson000000.vtu'}, detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))},\n 'b095dc38-f858-4c34-876c-df8f8b851a5f': {1: Response(output='/home/jovyan/example_workflows/nfdi/postprocessing/plotoverline.csv', detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))},\n '656e820d-20d5-4665-bb8a-43807ed6b2e9': {1: Response(output='/home/jovyan/example_workflows/nfdi/postprocessing/macros.tex', detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))},\n '52de4e6d-7574-401b-bcdb-ae6c298cc6b9': {1: Response(output='/home/jovyan/example_workflows/nfdi/postprocessing/paper.pdf', detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))}}" + }, + "metadata": {} + } + ], + "execution_count": 24 + }, + { + "id": "ebd3248b-ff75-40bf-a488-b5e338342671", + "cell_type": "markdown", + "source": "## Load Workflow with pyiron_base\n\nFinally, the graph is loaded into a list of delayed `pyiron_base` jobs. `.draw()` visualizes the dependency chain and `.pull()` triggers execution, returning the final `paper.pdf` path.", + "metadata": {} + }, + { + "id": "c7e1047f-0d42-4777-911d-ab405396401a", + "cell_type": "code", + "source": "from python_workflow_definition.pyiron_base import load_workflow_json", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 25 + }, + { + "id": "6aeba274-10d3-4e79-8ee4-4d870f163939", + "cell_type": "code", + "source": "delayed_object_lst = load_workflow_json(file_name=workflow_json_filename)\ndelayed_object_lst[-1].draw()", + "metadata": { + "trusted": true + }, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": "", + "image/svg+xml": "\n\n\n\n\ncreate_function_job_ba0bf2668d8f7b42ff717c2cb79b5920\n\ncreate_function_job=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d9280>\n\n\n\nplot_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\nplot_file=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8da0>\n\n\n\nplot_file_5007b1d67345934b2ffcdeaaad6ea2ab->create_function_job_ba0bf2668d8f7b42ff717c2cb79b5920\n\n\n\n\n\npoisson_output_vtu_file_51141ca0ec0a07922807865d8a067793\n\npoisson_output_vtu_file=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8a40>\n\n\n\npoisson_output_vtu_file_51141ca0ec0a07922807865d8a067793->plot_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\npvbatch_output_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\npvbatch_output_file=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8da0>\n\n\n\npoisson_output_vtu_file_51141ca0ec0a07922807865d8a067793->pvbatch_output_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\nmacros_tex_02d82cedf90ad323f7eef57ddf383160\n\nmacros_tex=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d9070>\n\n\n\npvbatch_output_file_5007b1d67345934b2ffcdeaaad6ea2ab->macros_tex_02d82cedf90ad323f7eef57ddf383160\n\n\n\n\n\nmeshio_output_h5_0af06cb14e9ddc7585647c44e4d4ef88\n\nmeshio_output_h5=<pyiron_base.project.delayed.DelayedObject object at 0x76dba3cb2600>\n\n\n\nmeshio_output_h5_0af06cb14e9ddc7585647c44e4d4ef88->poisson_output_vtu_file_51141ca0ec0a07922807865d8a067793\n\n\n\n\n\npoisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513\n\npoisson_output_pvd_file=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8a10>\n\n\n\nmeshio_output_h5_0af06cb14e9ddc7585647c44e4d4ef88->poisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513\n\n\n\n\n\nndofs_ac7427db28c0527371c0e9b27a57024b\n\nndofs=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8e30>\n\n\n\nmeshio_output_h5_0af06cb14e9ddc7585647c44e4d4ef88->ndofs_ac7427db28c0527371c0e9b27a57024b\n\n\n\n\n\npoisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513->plot_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\npoisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513->pvbatch_output_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\nndofs_ac7427db28c0527371c0e9b27a57024b->macros_tex_02d82cedf90ad323f7eef57ddf383160\n\n\n\n\n\ngmsh_output_file_6ef41eb684984daa8337ee65c2aa9f18\n\ngmsh_output_file=<pyiron_base.project.delayed.DelayedObject object at 0x76dba89319d0>\n\n\n\ngmsh_output_file_6ef41eb684984daa8337ee65c2aa9f18->meshio_output_h5_0af06cb14e9ddc7585647c44e4d4ef88\n\n\n\n\n\nmeshio_output_xdmf_c383e7d9f83d7cdc8ae04ebf9c264173\n\nmeshio_output_xdmf=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8710>\n\n\n\ngmsh_output_file_6ef41eb684984daa8337ee65c2aa9f18->meshio_output_xdmf_c383e7d9f83d7cdc8ae04ebf9c264173\n\n\n\n\n\nmeshio_output_xdmf_c383e7d9f83d7cdc8ae04ebf9c264173->poisson_output_vtu_file_51141ca0ec0a07922807865d8a067793\n\n\n\n\n\nmeshio_output_xdmf_c383e7d9f83d7cdc8ae04ebf9c264173->poisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513\n\n\n\n\n\nmeshio_output_xdmf_c383e7d9f83d7cdc8ae04ebf9c264173->ndofs_ac7427db28c0527371c0e9b27a57024b\n\n\n\n\n\ndomain_size_f12a7f1986b9dd058dfc666dbe230b20\n\ndomain_size=2.0\n\n\n\ndomain_size_f12a7f1986b9dd058dfc666dbe230b20->gmsh_output_file_6ef41eb684984daa8337ee65c2aa9f18\n\n\n\n\n\ndomain_size_f12a7f1986b9dd058dfc666dbe230b20->macros_tex_02d82cedf90ad323f7eef57ddf383160\n\n\n\n\n\nmacros_tex_02d82cedf90ad323f7eef57ddf383160->create_function_job_ba0bf2668d8f7b42ff717c2cb79b5920\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696\n\nsource_directory=/home/jovyan/example_workflows/nfdi/source\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->create_function_job_ba0bf2668d8f7b42ff717c2cb79b5920\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->plot_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->poisson_output_vtu_file_51141ca0ec0a07922807865d8a067793\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->pvbatch_output_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->poisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->ndofs_ac7427db28c0527371c0e9b27a57024b\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->gmsh_output_file_6ef41eb684984daa8337ee65c2aa9f18\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->macros_tex_02d82cedf90ad323f7eef57ddf383160\n\n\n\n\n" + }, + "metadata": {} + } + ], + "execution_count": 26 + }, + { + "id": "ea92c7ad-afef-423c-b8fa-9d24c1267c73", + "cell_type": "code", + "source": "delayed_object_lst[-1].pull()", + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": "The job generate_mesh_47725c16637f799ac042e47468005db3 was saved and received the ID: 1\nThe job convert_to_xdmf_d6a46eb9a4ec352aa996e783ec3c785f was saved and received the ID: 2\nThe job poisson_3c147fc86db87cf0c0f94bda333f8cd8 was saved and received the ID: 3\nThe job plot_over_line_ef50933291910dadcc8311924971e127 was saved and received the ID: 4\nThe job substitute_macros_63766eafd6b1980c7832dd8c9a97c96e was saved and received the ID: 5\nThe job compile_paper_128d1d58374953c00e95b8de62cbb10b was saved and received the ID: 6\n" + }, + { + "execution_count": 27, + "output_type": "execute_result", + "data": { + "text/plain": "'/home/jovyan/example_workflows/nfdi/postprocessing/paper.pdf'" + }, + "metadata": {} + } + ], + "execution_count": 27 + }, + { + "id": "32287743-e59a-467c-92fe-8586f199e36e", + "cell_type": "code", + "source": "", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": null + } + ] +} \ No newline at end of file diff --git a/example_workflows/nfdi/universal_workflow.ipynb b/example_workflows/nfdi/universal_workflow.ipynb index 5c1d8890..8ceeaad3 100644 --- a/example_workflows/nfdi/universal_workflow.ipynb +++ b/example_workflows/nfdi/universal_workflow.ipynb @@ -23,12 +23,12 @@ "cells": [ { "cell_type": "markdown", - "source": "# Load Simple Workflow", + "source": "# Load Simple Workflow\n\nThis notebook loads the single, pre-existing NFDI4Ing `workflow.json` — the file-based benchmark from [`workflow.py`](workflow.py) that runs `gmsh`, `meshio`, a FEniCS-based Poisson solver, ParaView's `pvbatch` and `tectonic` in turn — with several different engines. Since the JSON only describes function nodes, inputs, outputs and the edges between them, it does not matter which engine originally wrote it; any of the engines below can load and execute it, each re-running the same chain of external tools.", "metadata": {} }, { "cell_type": "markdown", - "source": "## Plot", + "source": "## Plot\n\nBefore running the workflow with any engine, `python_workflow_definition.plot.plot` renders `workflow.json` directly as a graph of function, input and output nodes — independent of any particular workflow engine.", "metadata": {} }, { @@ -60,7 +60,7 @@ }, { "cell_type": "markdown", - "source": "## Aiida ", + "source": "## Aiida\n\nLoading the JSON into `aiida-workgraph` requires connecting to an AiiDA profile first via `load_profile()`; the workflow is then reconstructed as a `WorkGraph` and executed with `.run()`, which re-runs `generate_mesh`, `convert_to_xdmf`, `poisson`, `plot_over_line`, `substitute_macros` and `compile_paper` in order.", "metadata": {} }, { @@ -130,7 +130,7 @@ }, { "cell_type": "markdown", - "source": "## executorlib", + "source": "## executorlib\n\n`python_workflow_definition.executorlib.load_workflow_json` submits each function node to the given `Executor` and returns the `Future` of the final node (the compiled `paper.pdf` path); calling `.result()` blocks until the whole chain of external tools has completed.", "metadata": {} }, { @@ -180,7 +180,7 @@ }, { "cell_type": "markdown", - "source": "## Jobflow", + "source": "## Jobflow\n\nThe JSON is reconstructed as a jobflow `Flow` and executed locally with `run_locally`.", "metadata": {} }, { @@ -235,7 +235,7 @@ }, { "cell_type": "markdown", - "source": "## pyiron", + "source": "## pyiron\n\nLoading the JSON into `pyiron_base` produces a list of delayed jobs; `.draw()` visualizes the dependency graph and `.pull()` triggers execution of the whole chain, again writing intermediate files to `preprocessing/`, `processing/` and `postprocessing/`.", "metadata": {} }, { @@ -291,7 +291,7 @@ { "metadata": {}, "cell_type": "markdown", - "source": "## Load Workflow with pyiron_workflow" + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the workflow is loaded into a `pyiron_workflow` `Workflow` object, drawn with `.draw()` and executed with `.run()`." }, { "metadata": {}, @@ -324,7 +324,7 @@ { "metadata": {}, "cell_type": "markdown", - "source": "## Python" + "source": "## Python\n\nAs a baseline, `python_workflow_definition.purepython.load_workflow_json` interprets the JSON directly in plain Python, with no workflow engine involved: it evaluates each function node in dependency order — invoking `gmsh`, `meshio`, the Poisson solver, `pvbatch` and `tectonic` via `conda_subprocess` exactly as `workflow.py` defines — and returns the final result." }, { "cell_type": "code", @@ -354,4 +354,4 @@ "execution_count": 19 } ] -} +} \ No newline at end of file diff --git a/example_workflows/quantum_espresso/aiida.ipynb b/example_workflows/quantum_espresso/aiida.ipynb index bddc5166..8b297f21 100644 --- a/example_workflows/quantum_espresso/aiida.ipynb +++ b/example_workflows/quantum_espresso/aiida.ipynb @@ -3,16 +3,12 @@ { "cell_type": "markdown", "metadata": {}, - "source": [ - "# Aiida" - ] + "source": "# aiida-workgraph\n\nThis notebook builds the Quantum Espresso energy-volume-curve workflow directly with [`aiida-workgraph`](https://github.com/aiidateam/aiida-workgraph) and exports it as a PWD `workflow.json`. The second half of the notebook loads that same JSON file with `jobflow`, `pyiron_base` and `pyiron_workflow` to show that a graph built with one engine can be executed by another." }, { "cell_type": "markdown", "metadata": {}, - "source": [ - "## Define workflow with aiida" - ] + "source": "## Define workflow with aiida\n\n`aiida-workgraph` represents a workflow as a `WorkGraph` built from `task`s, each wrapping a Python function; connecting a task's output to another task's input builds up the graph. `load_profile()` connects to the local AiiDA profile/database, which is required before any AiiDA node can be created, since every task and its result are stored there for provenance." }, { "cell_type": "code", @@ -59,6 +55,11 @@ ")" ] }, + { + "cell_type": "markdown", + "source": "`calculate_qe` (defined in [`workflow.py`](workflow.py)) returns a dict with `energy`, `volume` and `structure` keys. Wrapping it with `task(outputs=[...])` tells `aiida-workgraph` to expose these as separate output sockets (`.outputs.energy`, `.outputs.volume`, `.outputs.structure`) instead of a single opaque result, so downstream tasks can connect to individual keys.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 4, @@ -84,9 +85,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": [ - "### Prepare the inputs" - ] + "source": "### Prepare the inputs\n\nScalar inputs are wrapped as AiiDA data types (`orm.Str`, `orm.Float`, `orm.Bool`, `orm.Dict`, `orm.List`) so AiiDA can track their provenance. `strain_lst` are the five volumetric strains (0.9-1.1) that will later be applied to the relaxed structure to sample the energy-volume curve." }, { "cell_type": "code", @@ -111,9 +110,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": [ - "### Actual tasks to construct the EOS workflow" - ] + "source": "### Build the task graph\n\nThe pipeline is: `get_bulk_structure` builds the initial Al structure, `calculate_qe` relaxes it (`vc-relax`), `generate_structures` applies the strains, and one `calculate_qe` (`scf`) task per strained structure computes its energy and volume. `get_dict` bundles the structure, calculation type, k-points, pseudopotentials and smearing into the `input_dict` argument `calculate_qe` expects." }, { "cell_type": "code", @@ -155,6 +152,11 @@ ")" ] }, + { + "cell_type": "markdown", + "source": "`generate_structures` applies each of the five strains to the relaxed structure, producing five strained structures (`s_0` ... `s_4`). This is the fan-out point of the workflow: the loop below adds one independent `scf` `calculate_qe` task per strained structure, and their `energy`/`volume` outputs are collected into two lists (via `get_list` tasks) for the final plot.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 9, @@ -243,6 +245,11 @@ ")" ] }, + { + "cell_type": "markdown", + "source": "Displaying `wg` renders an interactive `NodeGraphWidget` of the task graph, which is useful to check the wiring before running it. `write_workflow_json` then serializes the `WorkGraph` into the PWD `workflow.json` format (nodes = functions/inputs/outputs, edges = data connections) so it can be loaded by another engine.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 14, @@ -861,9 +868,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": [ - "## Load Workflow with jobflow" - ] + "source": "## Load Workflow with jobflow\n\n`load_workflow_json` reconstructs the same graph as a `jobflow` `Flow` of `Job`s. To show that the loaded graph behaves like a native `jobflow` object, the lattice constant `a` of the first job is overridden (4.04 -> 4.05) before the flow is executed locally with `run_locally`." }, { "cell_type": "code", @@ -1099,9 +1104,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": [ - "## Load Workflow with pyiron_base" - ] + "source": "## Load Workflow with pyiron_base\n\n`pyiron_base`'s `load_workflow_json` turns each node into a delayed pyiron job. `.draw()` renders the dependency graph, and `.pull()` triggers execution of the whole chain, saving every intermediate job (relax + 5 strained SCF calculations) to the pyiron project." }, { "cell_type": "code", @@ -1994,9 +1997,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": [ - "## Load Workflow with pyiron_workflow" - ] + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the same JSON is loaded into a `pyiron_workflow` `Workflow` object. Nodes become graph nodes reachable via attribute access (e.g. `wf.get_bulk_structure`), and `.run()` executes them in dependency order." }, { "cell_type": "code", @@ -5796,4 +5797,4 @@ }, "nbformat": 4, "nbformat_minor": 4 -} +} \ No newline at end of file diff --git a/example_workflows/quantum_espresso/cwl.ipynb b/example_workflows/quantum_espresso/cwl.ipynb index 5dc1979d..4c477564 100644 --- a/example_workflows/quantum_espresso/cwl.ipynb +++ b/example_workflows/quantum_espresso/cwl.ipynb @@ -21,6 +21,12 @@ "nbformat_minor": 5, "nbformat": 4, "cells": [ + { + "cell_type": "markdown", + "id": "f9fecca7", + "source": "# CWL\n\nThis notebook executes the Quantum Espresso energy-volume-curve `workflow.json` (produced by one of the other notebooks in this directory, e.g. `aiida.ipynb`) with the [Common Workflow Language](https://www.commonwl.org) via [`cwltool`](https://github.com/common-workflow-language/cwltool). Unlike the other engines, PWD does not build a CWL graph directly in Python - instead it translates an existing `workflow.json` into native CWL files, which `cwltool` then executes as a separate command line process.", + "metadata": {} + }, { "id": "4eca79ef-1053-4f69-89ad-2bee8411068e", "cell_type": "code", @@ -41,6 +47,12 @@ "outputs": [], "execution_count": 2 }, + { + "cell_type": "markdown", + "id": "8da8470b", + "source": "## Translate workflow.json to CWL\n\n`write_workflow` reads the existing `workflow.json` and generates: one `.cwl` `CommandLineTool` per PWD function node (each one invokes `python -m python_workflow_definition.cwl --function=...` to call the matching function in `workflow.py`), a top-level `workflow.cwl` that chains these steps together following the `workflow.json` edges, and a `workflow.yml` job file listing one input file per PWD input node. Since CWL only passes files between steps, every input and every function output is exchanged as a pickle file.", + "metadata": {} + }, { "id": "92e3921b-2bb8-4333-8cfe-4bd27f785d24", "cell_type": "code", @@ -61,6 +73,12 @@ "outputs": [], "execution_count": 4 }, + { + "cell_type": "markdown", + "id": "21fc2973", + "source": "## Run the workflow with cwltool\n\nASE's Quantum Espresso writer resolves the pseudopotential directory from the `ESPRESSO_PSEUDO` environment variable when no explicit path is given. Because `cwltool` runs every step in its own isolated temporary directory, the environment variable has to be forwarded explicitly with `--preserve-environment=ESPRESSO_PSEUDO`, so that `pw.x` can find the pseudopotential file shipped in [`espresso/pseudo`](espresso/pseudo) from any step.", + "metadata": {} + }, { "id": "0192ca74-3971-464b-9435-c156e0b6e623", "cell_type": "code", @@ -82,11 +100,17 @@ { "name": "stdout", "output_type": "stream", - "text": "/srv/conda/envs/notebook/bin/cwltool:11: DeprecationWarning: Nesting argument groups is deprecated.\n sys.exit(run())\n\u001B[1;30mINFO\u001B[0m /srv/conda/envs/notebook/bin/cwltool 3.1.20250110105449\n\u001B[1;30mINFO\u001B[0m Resolved 'workflow.cwl' to 'file:///home/jovyan/example_workflows/quantum_espresso/workflow.cwl'\n\u001B[1;30mINFO\u001B[0m [workflow ] start\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_bulk_structure_0\n\u001B[1;30mINFO\u001B[0m [step get_bulk_structure_0] start\n\u001B[1;30mINFO\u001B[0m [job get_bulk_structure_0] /tmp/9sant4h9$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/va137af6/stga19aac01-d6f9-48d3-ba6b-d8a6e460f525/workflow.py \\\n --function=workflow.get_bulk_structure \\\n --arg_cubic=/tmp/va137af6/stgd2861bb7-939d-449d-9600-71f380ee058d/cubic.pickle \\\n --arg_element=/tmp/va137af6/stg93f5775e-477a-4b2e-87ac-822ff4097346/element.pickle \\\n --arg_a=/tmp/va137af6/stgcfd0bd7b-a186-4b46-80a6-663aeb5d0fa9/a.pickle\n\u001B[1;30mINFO\u001B[0m [job get_bulk_structure_0] Max memory used: 110MiB\n\u001B[1;30mINFO\u001B[0m [job get_bulk_structure_0] completed success\n\u001B[1;30mINFO\u001B[0m [step get_bulk_structure_0] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_dict_13\n\u001B[1;30mINFO\u001B[0m [step get_dict_13] start\n\u001B[1;30mINFO\u001B[0m [job get_dict_13] /tmp/_o8rxcyc$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/1h2uwpt9/stg17e865b8-f1f4-406f-a9c7-b8deddc5695d/smearing.pickle \\\n --arg_calculation=/tmp/1h2uwpt9/stgf1d25320-f3de-48f7-8087-5cdb5531765c/calculation_0.pickle \\\n --arg_kpts=/tmp/1h2uwpt9/stgc9e37196-83cf-49d9-88e1-b5acb7c53e7f/kpts.pickle \\\n --arg_pseudopotentials=/tmp/1h2uwpt9/stgcfb6bfd1-3ae7-4600-b07d-ca107c1a275c/pseudopotentials.pickle \\\n --arg_structure=/tmp/1h2uwpt9/stg0da627e7-f19e-4948-b23e-4552aa692862/result.pickle\n\u001B[1;30mINFO\u001B[0m [job get_dict_13] completed success\n\u001B[1;30mINFO\u001B[0m [step get_dict_13] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step calculate_qe_1\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_1] start\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_1] /tmp/q56p65x4$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/5hwlsb8q/stga00637b3-d27a-4b67-b4b2-5857ffc8f94a/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/5hwlsb8q/stgea0d773c-cef9-49b7-9895-09c22db6f52d/result.pickle \\\n --arg_working_directory=/tmp/5hwlsb8q/stg541840e1-d4a6-4a2c-a220-ab09e72c869d/working_directory_0.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09273] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_1] Max memory used: 273MiB\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_1] completed success\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_1] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step generate_structures_2\n\u001B[1;30mINFO\u001B[0m [step generate_structures_2] start\n\u001B[1;30mINFO\u001B[0m [job generate_structures_2] /tmp/81vwps2h$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/0d6xih7u/stg8da9bfd1-34a7-4282-8446-97a728d4747b/workflow.py \\\n --function=workflow.generate_structures \\\n --arg_structure=/tmp/0d6xih7u/stg89eb15f9-014d-4c20-8197-87c6c7345ffe/structure.pickle \\\n --arg_strain_lst=/tmp/0d6xih7u/stg80ec088c-9b6a-45bd-8565-e68fd3c76210/strain_lst.pickle\n\u001B[1;30mINFO\u001B[0m [job generate_structures_2] Max memory used: 96MiB\n\u001B[1;30mINFO\u001B[0m [job generate_structures_2] completed success\n\u001B[1;30mINFO\u001B[0m [step generate_structures_2] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_dict_25\n\u001B[1;30mINFO\u001B[0m [step get_dict_25] start\n\u001B[1;30mINFO\u001B[0m [job get_dict_25] /tmp/hj3teyvh$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/ky497mud/stgf2ab029c-fd78-4490-a4aa-2ee352fa40a2/smearing.pickle \\\n --arg_calculation=/tmp/ky497mud/stg12f82711-28cb-42e6-80c5-59e7fa946ea8/calculation_1.pickle \\\n --arg_kpts=/tmp/ky497mud/stgd35fecc6-83e5-493e-85ad-caf6acd301cb/kpts.pickle \\\n --arg_pseudopotentials=/tmp/ky497mud/stg9b98aba6-11a6-4718-bd3a-23f160518077/pseudopotentials.pickle \\\n --arg_structure=/tmp/ky497mud/stg00376f4a-a991-469a-b494-27c160e6c15d/s_2.pickle\n\u001B[1;30mINFO\u001B[0m [job get_dict_25] completed success\n\u001B[1;30mINFO\u001B[0m [step get_dict_25] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_dict_20\n\u001B[1;30mINFO\u001B[0m [step get_dict_20] start\n\u001B[1;30mINFO\u001B[0m [job get_dict_20] /tmp/y9drdoe7$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/dsu142t1/stgcba59359-36e1-4bf9-9e6f-e7cf99a23077/smearing.pickle \\\n --arg_calculation=/tmp/dsu142t1/stg56a2646f-2057-4670-bb4e-f55af62c5858/calculation_1.pickle \\\n --arg_kpts=/tmp/dsu142t1/stgc443e9a8-c9b6-4c5e-8321-6a70d347706b/kpts.pickle \\\n --arg_pseudopotentials=/tmp/dsu142t1/stg47b4b44d-c69c-4629-aaed-0e15bf14426e/pseudopotentials.pickle \\\n --arg_structure=/tmp/dsu142t1/stg0e5b3c6e-a315-4eaf-9a66-dbf5669eaada/s_0.pickle\n\u001B[1;30mINFO\u001B[0m [job get_dict_20] completed success\n\u001B[1;30mINFO\u001B[0m [step get_dict_20] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_dict_23\n\u001B[1;30mINFO\u001B[0m [step get_dict_23] start\n\u001B[1;30mINFO\u001B[0m [job get_dict_23] /tmp/9v59c4vd$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/2mc_byfs/stgace4b178-c0ff-406e-95ac-754318766886/smearing.pickle \\\n --arg_calculation=/tmp/2mc_byfs/stg793df1d8-8f8d-4bb3-9c7c-243a48dc494b/calculation_1.pickle \\\n --arg_kpts=/tmp/2mc_byfs/stga852e76a-885f-4e39-8ab4-ecebf95c656f/kpts.pickle \\\n --arg_pseudopotentials=/tmp/2mc_byfs/stgd06acdff-6e0c-476c-be57-1f21b2d03711/pseudopotentials.pickle \\\n --arg_structure=/tmp/2mc_byfs/stg02210df5-d3e2-42b8-a75e-47e2f2c9f274/s_1.pickle\n\u001B[1;30mINFO\u001B[0m [job get_dict_23] completed success\n\u001B[1;30mINFO\u001B[0m [step get_dict_23] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_dict_29\n\u001B[1;30mINFO\u001B[0m [step get_dict_29] start\n\u001B[1;30mINFO\u001B[0m [job get_dict_29] /tmp/q_svsf9g$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/8u8e7db4/stgbba611dc-3b23-4653-a293-f71d492a42af/smearing.pickle \\\n --arg_calculation=/tmp/8u8e7db4/stge9a32972-6649-4eb5-abd7-9db61d929d2d/calculation_1.pickle \\\n --arg_kpts=/tmp/8u8e7db4/stg02098ea5-7487-4a66-bb77-99a6336d7b71/kpts.pickle \\\n --arg_pseudopotentials=/tmp/8u8e7db4/stg571b6259-4599-4c39-8cb2-79a70feb78b8/pseudopotentials.pickle \\\n --arg_structure=/tmp/8u8e7db4/stg64c55a4d-8af6-40bf-abfd-3ec8a690b90b/s_4.pickle\n\u001B[1;30mINFO\u001B[0m [job get_dict_29] completed success\n\u001B[1;30mINFO\u001B[0m [step get_dict_29] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_dict_27\n\u001B[1;30mINFO\u001B[0m [step get_dict_27] start\n\u001B[1;30mINFO\u001B[0m [job get_dict_27] /tmp/vgcwzb84$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/xf32x1ld/stg3cbb6408-2383-49d0-9476-ef3fbee7c165/smearing.pickle \\\n --arg_calculation=/tmp/xf32x1ld/stgeb36ed3b-b442-43ea-b14d-1397875e8839/calculation_1.pickle \\\n --arg_kpts=/tmp/xf32x1ld/stge8854d32-32f3-4a90-8e66-8ce7b1160d8d/kpts.pickle \\\n --arg_pseudopotentials=/tmp/xf32x1ld/stg0dba3c44-e6af-46a9-9f00-c08969392ede/pseudopotentials.pickle \\\n --arg_structure=/tmp/xf32x1ld/stg4223a500-1f85-4651-9130-bd1629008770/s_3.pickle\n\u001B[1;30mINFO\u001B[0m [job get_dict_27] completed success\n\u001B[1;30mINFO\u001B[0m [step get_dict_27] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step calculate_qe_3\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_3] start\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_3] /tmp/v2e_84tz$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/n81xp6_a/stgf98a9803-3329-400e-a03c-02b51d3f3c97/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/n81xp6_a/stgd93a737b-2598-4fb8-bdcb-bb1d2048116f/result.pickle \\\n --arg_working_directory=/tmp/n81xp6_a/stg5ad2a52b-2844-45e4-b227-3d835efa01f6/working_directory_1.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09459] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_3] Max memory used: 240MiB\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_3] completed success\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_3] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step calculate_qe_4\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_4] start\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_4] /tmp/tgwdkhxx$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/l5mihx97/stgf73a20aa-bd97-46af-a923-b8d776da2b98/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/l5mihx97/stgb8c3fcbe-9277-46c4-b8a2-a1171a292580/result.pickle \\\n --arg_working_directory=/tmp/l5mihx97/stg3ae0b93a-2e0b-4f1b-b079-985ea6bbaa72/working_directory_2.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09533] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_4] Max memory used: 213MiB\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_4] completed success\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_4] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step calculate_qe_5\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_5] start\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_5] /tmp/d36x49f6$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/nvm7fsf5/stg384bab04-833c-43f2-bd00-f4b5a5c93eec/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/nvm7fsf5/stg9aa51e21-01b0-4157-b73f-a3feee8a917a/result.pickle \\\n --arg_working_directory=/tmp/nvm7fsf5/stg02ef6345-c598-4127-95c7-e92d0a977c40/working_directory_3.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09611] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_5] Max memory used: 243MiB\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_5] completed success\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_5] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step calculate_qe_6\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_6] start\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_6] /tmp/eiyg8it6$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/708xmf3f/stg791b1717-cfb1-4569-ba75-d648a84e0a72/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/708xmf3f/stg9ebf7054-080c-4173-a8b9-2247d01a0068/result.pickle \\\n --arg_working_directory=/tmp/708xmf3f/stg1afccee5-d9b6-46ed-bc2c-fa36bdb8013d/working_directory_4.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09688] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_6] Max memory used: 245MiB\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_6] completed success\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_6] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step calculate_qe_7\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_7] start\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_7] /tmp/qjgxwxnd$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/g3sp392q/stg0a51a8da-c7a7-4d90-a49c-1817002a62a6/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/g3sp392q/stg67e73a60-3cc6-46dd-b009-cfc0034aec2c/result.pickle \\\n --arg_working_directory=/tmp/g3sp392q/stgf3d265db-820b-4af3-8975-0a6faa0d7a67/working_directory_5.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09770] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_7] Max memory used: 244MiB\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_7] completed success\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_7] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_list_31\n\u001B[1;30mINFO\u001B[0m [step get_list_31] start\n\u001B[1;30mINFO\u001B[0m [job get_list_31] /tmp/mnovytq_$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_list \\\n --arg_0=/tmp/bz0ntajx/stgdab2f59b-b118-44c4-96ef-b4639edaee4e/energy.pickle \\\n --arg_2=/tmp/bz0ntajx/stg2e9bec4d-1824-4152-b003-aab0eac204cd/energy.pickle \\\n --arg_1=/tmp/bz0ntajx/stg9511563e-9784-48a9-8d81-74ade3a084c1/energy.pickle \\\n --arg_4=/tmp/bz0ntajx/stg17845efd-e335-421e-9891-6e627254edb4/energy.pickle \\\n --arg_3=/tmp/bz0ntajx/stg051fa3bc-832a-4d70-b9fb-b9948f51904d/energy.pickle\n\u001B[1;30mINFO\u001B[0m [job get_list_31] completed success\n\u001B[1;30mINFO\u001B[0m [step get_list_31] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_list_30\n\u001B[1;30mINFO\u001B[0m [step get_list_30] start\n\u001B[1;30mINFO\u001B[0m [job get_list_30] /tmp/vc0dzppy$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_list \\\n --arg_0=/tmp/mejux8b1/stg71e3a6a9-c56b-4be8-8720-0ca90ac40513/volume.pickle \\\n --arg_2=/tmp/mejux8b1/stg429d498d-cf56-4561-a653-8048c1213851/volume.pickle \\\n --arg_1=/tmp/mejux8b1/stgee1a0001-20f2-45de-ac4e-77b0d0c59957/volume.pickle \\\n --arg_4=/tmp/mejux8b1/stg0de2a6dc-57cd-4a0c-9637-ef2e63e96fce/volume.pickle \\\n --arg_3=/tmp/mejux8b1/stga4551f25-d5f6-47c6-92ea-452cf694a286/volume.pickle\n\u001B[1;30mINFO\u001B[0m [job get_list_30] completed success\n\u001B[1;30mINFO\u001B[0m [step get_list_30] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step plot_energy_volume_curve_8\n\u001B[1;30mINFO\u001B[0m [step plot_energy_volume_curve_8] start\n\u001B[1;30mINFO\u001B[0m [job plot_energy_volume_curve_8] /tmp/j4v3j3mt$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/_67k4bu3/stg5be48d76-8a92-4b02-b3d6-7bdb4c35c9ce/workflow.py \\\n --function=workflow.plot_energy_volume_curve \\\n --arg_volume_lst=/tmp/_67k4bu3/stgd43e71b0-d16b-4eb2-8d14-83924dafd7d6/result.pickle \\\n --arg_energy_lst=/tmp/_67k4bu3/stg43eeacdd-5042-4f2d-8efc-d8e0d5fe1c14/result.pickle\n\u001B[1;30mINFO\u001B[0m [job plot_energy_volume_curve_8] Max memory used: 112MiB\n\u001B[1;30mINFO\u001B[0m [job plot_energy_volume_curve_8] completed success\n\u001B[1;30mINFO\u001B[0m [step plot_energy_volume_curve_8] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] completed success\n{\n \"result_file\": {\n \"location\": \"file:///home/jovyan/example_workflows/quantum_espresso/result.pickle\",\n \"basename\": \"result.pickle\",\n \"class\": \"File\",\n \"checksum\": \"sha1$dbc1aaddc8b7343d6d33b34edcf608b8f8801918\",\n \"size\": 4,\n \"path\": \"/home/jovyan/example_workflows/quantum_espresso/result.pickle\"\n }\n}\u001B[1;30mINFO\u001B[0m Final process status is success\n" + "text": "/srv/conda/envs/notebook/bin/cwltool:11: DeprecationWarning: Nesting argument groups is deprecated.\n sys.exit(run())\n\u001b[1;30mINFO\u001b[0m /srv/conda/envs/notebook/bin/cwltool 3.1.20250110105449\n\u001b[1;30mINFO\u001b[0m Resolved 'workflow.cwl' to 'file:///home/jovyan/example_workflows/quantum_espresso/workflow.cwl'\n\u001b[1;30mINFO\u001b[0m [workflow ] start\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_bulk_structure_0\n\u001b[1;30mINFO\u001b[0m [step get_bulk_structure_0] start\n\u001b[1;30mINFO\u001b[0m [job get_bulk_structure_0] /tmp/9sant4h9$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/va137af6/stga19aac01-d6f9-48d3-ba6b-d8a6e460f525/workflow.py \\\n --function=workflow.get_bulk_structure \\\n --arg_cubic=/tmp/va137af6/stgd2861bb7-939d-449d-9600-71f380ee058d/cubic.pickle \\\n --arg_element=/tmp/va137af6/stg93f5775e-477a-4b2e-87ac-822ff4097346/element.pickle \\\n --arg_a=/tmp/va137af6/stgcfd0bd7b-a186-4b46-80a6-663aeb5d0fa9/a.pickle\n\u001b[1;30mINFO\u001b[0m [job get_bulk_structure_0] Max memory used: 110MiB\n\u001b[1;30mINFO\u001b[0m [job get_bulk_structure_0] completed success\n\u001b[1;30mINFO\u001b[0m [step get_bulk_structure_0] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_dict_13\n\u001b[1;30mINFO\u001b[0m [step get_dict_13] start\n\u001b[1;30mINFO\u001b[0m [job get_dict_13] /tmp/_o8rxcyc$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/1h2uwpt9/stg17e865b8-f1f4-406f-a9c7-b8deddc5695d/smearing.pickle \\\n --arg_calculation=/tmp/1h2uwpt9/stgf1d25320-f3de-48f7-8087-5cdb5531765c/calculation_0.pickle \\\n --arg_kpts=/tmp/1h2uwpt9/stgc9e37196-83cf-49d9-88e1-b5acb7c53e7f/kpts.pickle \\\n --arg_pseudopotentials=/tmp/1h2uwpt9/stgcfb6bfd1-3ae7-4600-b07d-ca107c1a275c/pseudopotentials.pickle \\\n --arg_structure=/tmp/1h2uwpt9/stg0da627e7-f19e-4948-b23e-4552aa692862/result.pickle\n\u001b[1;30mINFO\u001b[0m [job get_dict_13] completed success\n\u001b[1;30mINFO\u001b[0m [step get_dict_13] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step calculate_qe_1\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_1] start\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_1] /tmp/q56p65x4$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/5hwlsb8q/stga00637b3-d27a-4b67-b4b2-5857ffc8f94a/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/5hwlsb8q/stgea0d773c-cef9-49b7-9895-09c22db6f52d/result.pickle \\\n --arg_working_directory=/tmp/5hwlsb8q/stg541840e1-d4a6-4a2c-a220-ab09e72c869d/working_directory_0.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09273] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_1] Max memory used: 273MiB\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_1] completed success\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_1] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step generate_structures_2\n\u001b[1;30mINFO\u001b[0m [step generate_structures_2] start\n\u001b[1;30mINFO\u001b[0m [job generate_structures_2] /tmp/81vwps2h$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/0d6xih7u/stg8da9bfd1-34a7-4282-8446-97a728d4747b/workflow.py \\\n --function=workflow.generate_structures \\\n --arg_structure=/tmp/0d6xih7u/stg89eb15f9-014d-4c20-8197-87c6c7345ffe/structure.pickle \\\n --arg_strain_lst=/tmp/0d6xih7u/stg80ec088c-9b6a-45bd-8565-e68fd3c76210/strain_lst.pickle\n\u001b[1;30mINFO\u001b[0m [job generate_structures_2] Max memory used: 96MiB\n\u001b[1;30mINFO\u001b[0m [job generate_structures_2] completed success\n\u001b[1;30mINFO\u001b[0m [step generate_structures_2] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_dict_25\n\u001b[1;30mINFO\u001b[0m [step get_dict_25] start\n\u001b[1;30mINFO\u001b[0m [job get_dict_25] /tmp/hj3teyvh$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/ky497mud/stgf2ab029c-fd78-4490-a4aa-2ee352fa40a2/smearing.pickle \\\n --arg_calculation=/tmp/ky497mud/stg12f82711-28cb-42e6-80c5-59e7fa946ea8/calculation_1.pickle \\\n --arg_kpts=/tmp/ky497mud/stgd35fecc6-83e5-493e-85ad-caf6acd301cb/kpts.pickle \\\n --arg_pseudopotentials=/tmp/ky497mud/stg9b98aba6-11a6-4718-bd3a-23f160518077/pseudopotentials.pickle \\\n --arg_structure=/tmp/ky497mud/stg00376f4a-a991-469a-b494-27c160e6c15d/s_2.pickle\n\u001b[1;30mINFO\u001b[0m [job get_dict_25] completed success\n\u001b[1;30mINFO\u001b[0m [step get_dict_25] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_dict_20\n\u001b[1;30mINFO\u001b[0m [step get_dict_20] start\n\u001b[1;30mINFO\u001b[0m [job get_dict_20] /tmp/y9drdoe7$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/dsu142t1/stgcba59359-36e1-4bf9-9e6f-e7cf99a23077/smearing.pickle \\\n --arg_calculation=/tmp/dsu142t1/stg56a2646f-2057-4670-bb4e-f55af62c5858/calculation_1.pickle \\\n --arg_kpts=/tmp/dsu142t1/stgc443e9a8-c9b6-4c5e-8321-6a70d347706b/kpts.pickle \\\n --arg_pseudopotentials=/tmp/dsu142t1/stg47b4b44d-c69c-4629-aaed-0e15bf14426e/pseudopotentials.pickle \\\n --arg_structure=/tmp/dsu142t1/stg0e5b3c6e-a315-4eaf-9a66-dbf5669eaada/s_0.pickle\n\u001b[1;30mINFO\u001b[0m [job get_dict_20] completed success\n\u001b[1;30mINFO\u001b[0m [step get_dict_20] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_dict_23\n\u001b[1;30mINFO\u001b[0m [step get_dict_23] start\n\u001b[1;30mINFO\u001b[0m [job get_dict_23] /tmp/9v59c4vd$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/2mc_byfs/stgace4b178-c0ff-406e-95ac-754318766886/smearing.pickle \\\n --arg_calculation=/tmp/2mc_byfs/stg793df1d8-8f8d-4bb3-9c7c-243a48dc494b/calculation_1.pickle \\\n --arg_kpts=/tmp/2mc_byfs/stga852e76a-885f-4e39-8ab4-ecebf95c656f/kpts.pickle \\\n --arg_pseudopotentials=/tmp/2mc_byfs/stgd06acdff-6e0c-476c-be57-1f21b2d03711/pseudopotentials.pickle \\\n --arg_structure=/tmp/2mc_byfs/stg02210df5-d3e2-42b8-a75e-47e2f2c9f274/s_1.pickle\n\u001b[1;30mINFO\u001b[0m [job get_dict_23] completed success\n\u001b[1;30mINFO\u001b[0m [step get_dict_23] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_dict_29\n\u001b[1;30mINFO\u001b[0m [step get_dict_29] start\n\u001b[1;30mINFO\u001b[0m [job get_dict_29] /tmp/q_svsf9g$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/8u8e7db4/stgbba611dc-3b23-4653-a293-f71d492a42af/smearing.pickle \\\n --arg_calculation=/tmp/8u8e7db4/stge9a32972-6649-4eb5-abd7-9db61d929d2d/calculation_1.pickle \\\n --arg_kpts=/tmp/8u8e7db4/stg02098ea5-7487-4a66-bb77-99a6336d7b71/kpts.pickle \\\n --arg_pseudopotentials=/tmp/8u8e7db4/stg571b6259-4599-4c39-8cb2-79a70feb78b8/pseudopotentials.pickle \\\n --arg_structure=/tmp/8u8e7db4/stg64c55a4d-8af6-40bf-abfd-3ec8a690b90b/s_4.pickle\n\u001b[1;30mINFO\u001b[0m [job get_dict_29] completed success\n\u001b[1;30mINFO\u001b[0m [step get_dict_29] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_dict_27\n\u001b[1;30mINFO\u001b[0m [step get_dict_27] start\n\u001b[1;30mINFO\u001b[0m [job get_dict_27] /tmp/vgcwzb84$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/xf32x1ld/stg3cbb6408-2383-49d0-9476-ef3fbee7c165/smearing.pickle \\\n --arg_calculation=/tmp/xf32x1ld/stgeb36ed3b-b442-43ea-b14d-1397875e8839/calculation_1.pickle \\\n --arg_kpts=/tmp/xf32x1ld/stge8854d32-32f3-4a90-8e66-8ce7b1160d8d/kpts.pickle \\\n --arg_pseudopotentials=/tmp/xf32x1ld/stg0dba3c44-e6af-46a9-9f00-c08969392ede/pseudopotentials.pickle \\\n --arg_structure=/tmp/xf32x1ld/stg4223a500-1f85-4651-9130-bd1629008770/s_3.pickle\n\u001b[1;30mINFO\u001b[0m [job get_dict_27] completed success\n\u001b[1;30mINFO\u001b[0m [step get_dict_27] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step calculate_qe_3\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_3] start\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_3] /tmp/v2e_84tz$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/n81xp6_a/stgf98a9803-3329-400e-a03c-02b51d3f3c97/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/n81xp6_a/stgd93a737b-2598-4fb8-bdcb-bb1d2048116f/result.pickle \\\n --arg_working_directory=/tmp/n81xp6_a/stg5ad2a52b-2844-45e4-b227-3d835efa01f6/working_directory_1.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09459] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_3] Max memory used: 240MiB\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_3] completed success\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_3] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step calculate_qe_4\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_4] start\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_4] /tmp/tgwdkhxx$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/l5mihx97/stgf73a20aa-bd97-46af-a923-b8d776da2b98/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/l5mihx97/stgb8c3fcbe-9277-46c4-b8a2-a1171a292580/result.pickle \\\n --arg_working_directory=/tmp/l5mihx97/stg3ae0b93a-2e0b-4f1b-b079-985ea6bbaa72/working_directory_2.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09533] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_4] Max memory used: 213MiB\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_4] completed success\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_4] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step calculate_qe_5\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_5] start\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_5] /tmp/d36x49f6$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/nvm7fsf5/stg384bab04-833c-43f2-bd00-f4b5a5c93eec/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/nvm7fsf5/stg9aa51e21-01b0-4157-b73f-a3feee8a917a/result.pickle \\\n --arg_working_directory=/tmp/nvm7fsf5/stg02ef6345-c598-4127-95c7-e92d0a977c40/working_directory_3.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09611] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_5] Max memory used: 243MiB\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_5] completed success\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_5] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step calculate_qe_6\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_6] start\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_6] /tmp/eiyg8it6$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/708xmf3f/stg791b1717-cfb1-4569-ba75-d648a84e0a72/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/708xmf3f/stg9ebf7054-080c-4173-a8b9-2247d01a0068/result.pickle \\\n --arg_working_directory=/tmp/708xmf3f/stg1afccee5-d9b6-46ed-bc2c-fa36bdb8013d/working_directory_4.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09688] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_6] Max memory used: 245MiB\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_6] completed success\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_6] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step calculate_qe_7\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_7] start\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_7] /tmp/qjgxwxnd$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/g3sp392q/stg0a51a8da-c7a7-4d90-a49c-1817002a62a6/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/g3sp392q/stg67e73a60-3cc6-46dd-b009-cfc0034aec2c/result.pickle \\\n --arg_working_directory=/tmp/g3sp392q/stgf3d265db-820b-4af3-8975-0a6faa0d7a67/working_directory_5.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09770] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_7] Max memory used: 244MiB\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_7] completed success\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_7] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_list_31\n\u001b[1;30mINFO\u001b[0m [step get_list_31] start\n\u001b[1;30mINFO\u001b[0m [job get_list_31] /tmp/mnovytq_$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_list \\\n --arg_0=/tmp/bz0ntajx/stgdab2f59b-b118-44c4-96ef-b4639edaee4e/energy.pickle \\\n --arg_2=/tmp/bz0ntajx/stg2e9bec4d-1824-4152-b003-aab0eac204cd/energy.pickle \\\n --arg_1=/tmp/bz0ntajx/stg9511563e-9784-48a9-8d81-74ade3a084c1/energy.pickle \\\n --arg_4=/tmp/bz0ntajx/stg17845efd-e335-421e-9891-6e627254edb4/energy.pickle \\\n --arg_3=/tmp/bz0ntajx/stg051fa3bc-832a-4d70-b9fb-b9948f51904d/energy.pickle\n\u001b[1;30mINFO\u001b[0m [job get_list_31] completed success\n\u001b[1;30mINFO\u001b[0m [step get_list_31] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_list_30\n\u001b[1;30mINFO\u001b[0m [step get_list_30] start\n\u001b[1;30mINFO\u001b[0m [job get_list_30] /tmp/vc0dzppy$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_list \\\n --arg_0=/tmp/mejux8b1/stg71e3a6a9-c56b-4be8-8720-0ca90ac40513/volume.pickle \\\n --arg_2=/tmp/mejux8b1/stg429d498d-cf56-4561-a653-8048c1213851/volume.pickle \\\n --arg_1=/tmp/mejux8b1/stgee1a0001-20f2-45de-ac4e-77b0d0c59957/volume.pickle \\\n --arg_4=/tmp/mejux8b1/stg0de2a6dc-57cd-4a0c-9637-ef2e63e96fce/volume.pickle \\\n --arg_3=/tmp/mejux8b1/stga4551f25-d5f6-47c6-92ea-452cf694a286/volume.pickle\n\u001b[1;30mINFO\u001b[0m [job get_list_30] completed success\n\u001b[1;30mINFO\u001b[0m [step get_list_30] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step plot_energy_volume_curve_8\n\u001b[1;30mINFO\u001b[0m [step plot_energy_volume_curve_8] start\n\u001b[1;30mINFO\u001b[0m [job plot_energy_volume_curve_8] /tmp/j4v3j3mt$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/_67k4bu3/stg5be48d76-8a92-4b02-b3d6-7bdb4c35c9ce/workflow.py \\\n --function=workflow.plot_energy_volume_curve \\\n --arg_volume_lst=/tmp/_67k4bu3/stgd43e71b0-d16b-4eb2-8d14-83924dafd7d6/result.pickle \\\n --arg_energy_lst=/tmp/_67k4bu3/stg43eeacdd-5042-4f2d-8efc-d8e0d5fe1c14/result.pickle\n\u001b[1;30mINFO\u001b[0m [job plot_energy_volume_curve_8] Max memory used: 112MiB\n\u001b[1;30mINFO\u001b[0m [job plot_energy_volume_curve_8] completed success\n\u001b[1;30mINFO\u001b[0m [step plot_energy_volume_curve_8] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] completed success\n{\n \"result_file\": {\n \"location\": \"file:///home/jovyan/example_workflows/quantum_espresso/result.pickle\",\n \"basename\": \"result.pickle\",\n \"class\": \"File\",\n \"checksum\": \"sha1$dbc1aaddc8b7343d6d33b34edcf608b8f8801918\",\n \"size\": 4,\n \"path\": \"/home/jovyan/example_workflows/quantum_espresso/result.pickle\"\n }\n}\u001b[1;30mINFO\u001b[0m Final process status is success\n" } ], "execution_count": 6 }, + { + "cell_type": "markdown", + "id": "7ddd08ec", + "source": "`cwltool` runs one step per PWD function node - `get_bulk_structure`, the `vc-relax` `calculate_qe`, `generate_structures`, the five strained `scf` `calculate_qe` steps and finally `plot_energy_volume_curve` - each reading its input pickles and writing its output pickle(s), exactly mirroring the fan-out of one relax followed by five parallel SCF calculations seen in the other notebooks. The workflow's final output is declared as `result_file`, the pickle produced by the last step.", + "metadata": {} + }, { "id": "2942dbba-ea0a-4d20-be5c-ed9992d09ff8", "cell_type": "code", @@ -103,6 +127,12 @@ ], "execution_count": 7 }, + { + "cell_type": "markdown", + "id": "e9ec76ee", + "source": "`plot_energy_volume_curve` has no return value, so `result_file` unpickles to `None` here - that is expected. The actual output of this workflow is the `evcurve.png` plot, written as a side effect into the working directory of the last CWL step.", + "metadata": {} + }, { "id": "60e909ee-d0d0-4bd1-81c8-dd5274ae5834", "cell_type": "code", @@ -114,4 +144,4 @@ "execution_count": null } ] -} +} \ No newline at end of file diff --git a/example_workflows/quantum_espresso/executorlib.ipynb b/example_workflows/quantum_espresso/executorlib.ipynb index b1f60692..e4ebd6f0 100644 --- a/example_workflows/quantum_espresso/executorlib.ipynb +++ b/example_workflows/quantum_espresso/executorlib.ipynb @@ -4,17 +4,13 @@ "cell_type": "markdown", "id": "be2d61b0-0d47-4349-b4b0-1b767c961644", "metadata": {}, - "source": [ - "# executorlib" - ] + "source": "# executorlib\n\nThis notebook builds the Quantum Espresso energy-volume-curve workflow with [`executorlib`](https://github.com/pyiron/executorlib), a lightweight `concurrent.futures`-style executor. Tasks are submitted individually with `.submit()`, and the resulting graph is exported as a PWD `workflow.json` as a side effect. The second half of the notebook loads that same JSON with `aiida-workgraph`, `jobflow` and `pyiron_base`/`pyiron_workflow`." }, { "cell_type": "markdown", "id": "0bad2a57-1bd2-4837-94fe-f8c60e211fae", "metadata": {}, - "source": [ - "## Define workflow with executorlib" - ] + "source": "## Define workflow with executorlib\n\n`SingleNodeExecutor` runs submitted functions in local worker processes. Passing `export_workflow_filename` makes it record every `.submit()` call and its dependencies and write them out as `workflow_json_filename` once the `with` block exits - no separate `write_workflow_json` call is needed, unlike the other engines." }, { "cell_type": "code", @@ -57,6 +53,12 @@ "pseudopotentials = {\"Al\": \"Al.pbe-n-kjpaw_psl.1.0.0.UPF\"}" ] }, + { + "cell_type": "markdown", + "id": "9ce84a62", + "source": "Each `exe.submit()` call schedules one function call and returns a `Future`. Since `calculate_qe` returns a dict, `get_item_from_future` extracts a single key (e.g. `\"structure\"`, `\"energy\"`, `\"volume\"`) from a future's eventual result so it can be passed as an argument to the next `submit()` call. The relaxed structure is strained into `number_of_strains` (5) structures by `generate_structures`, and each strained structure is submitted as its own independent `scf` `calculate_qe` call - the fan-out that produces the 5 points of the energy-volume curve.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 5, @@ -688,9 +690,7 @@ "cell_type": "markdown", "id": "7d75a2f6-6fad-49c8-bd29-37cca1b84441", "metadata": {}, - "source": [ - "## Load Workflow with aiida" - ] + "source": "## Load Workflow with aiida\n\n`load_workflow_json` reconstructs the graph as an `aiida-workgraph` `WorkGraph`. To demonstrate that the loaded graph can still be edited like one built natively with `aiida-workgraph`, the lattice constant `a` is overridden (4.04 -> 4.05) before running it with `wg.run()`." }, { "cell_type": "code", @@ -828,9 +828,7 @@ "cell_type": "markdown", "id": "c4f5c047-c6da-4b54-9007-415faca7a448", "metadata": {}, - "source": [ - "## Load Workflow with jobflow" - ] + "source": "## Load Workflow with jobflow\n\nThe same JSON is loaded as a `jobflow` `Flow` and executed locally with `run_locally`, again after overriding the lattice constant `a`." }, { "cell_type": "code", @@ -966,9 +964,7 @@ "cell_type": "markdown", "id": "9a956c9e", "metadata": {}, - "source": [ - "## Load Workflow with pyiron_base" - ] + "source": "## Load Workflow with pyiron_base\n\n`pyiron_base`'s loader turns the graph into a list of delayed jobs. `.draw()` visualizes the dependency graph and `.pull()` executes the whole chain (relax + 5 strained SCF calculations), after the lattice constant override." }, { "cell_type": "code", @@ -1774,9 +1770,7 @@ "cell_type": "markdown", "id": "406b0429e65b9760", "metadata": {}, - "source": [ - "## Load Workflow with pyiron_workflow" - ] + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the graph is loaded into a `pyiron_workflow` `Workflow`, its lattice constant overridden, drawn with `.draw()`, and executed with `.run()`." }, { "cell_type": "code", @@ -5561,4 +5555,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file diff --git a/example_workflows/quantum_espresso/jobflow.ipynb b/example_workflows/quantum_espresso/jobflow.ipynb index 4e04b74f..d8b1e1fa 100644 --- a/example_workflows/quantum_espresso/jobflow.ipynb +++ b/example_workflows/quantum_espresso/jobflow.ipynb @@ -181,9 +181,7 @@ "outputs_hidden": false } }, - "source": [ - "Next, for the \"Energy vs. Volume\" curve, we meed to specify the number of strained structures and save them into a list object. For each of the strained structures, we will carry out a QE calculation." - ] + "source": "Next, for the \"Energy vs. Volume\" curve, we need to specify the number of strained structures and save them into a list object. This is the fan-out point of the workflow: for each of the `number_of_strains` (5) strained structures, an independent `scf` QE calculation is carried out below." }, { "cell_type": "code", @@ -1027,11 +1025,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": [ - "## Load Workflow with pyiron_base\n", - "\n", - "And we can repeat the same process using `pyiron`." - ] + "source": "## Load Workflow with pyiron_base\n\nAnd we can repeat the same process using `pyiron_base`: `load_workflow_json` turns the graph into a list of delayed jobs, `.draw()` visualizes the dependency graph, and `.pull()` executes the whole chain (relax + 5 strained SCF calculations) after overriding the lattice constant `a`." }, { "cell_type": "code", @@ -1924,9 +1918,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": [ - "## Load Workflow with pyiron_workflow" - ] + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the same JSON is loaded into a `pyiron_workflow` `Workflow`; nodes become graph nodes reachable via attribute access (e.g. `wf.get_bulk_structure`). The lattice constant `a` is overridden again before drawing the graph with `.draw()` and executing it with `.run()`." }, { "cell_type": "code", @@ -5744,4 +5736,4 @@ }, "nbformat": 4, "nbformat_minor": 4 -} +} \ No newline at end of file diff --git a/example_workflows/quantum_espresso/pyiron_base.ipynb b/example_workflows/quantum_espresso/pyiron_base.ipynb index 38703d80..1369193c 100644 --- a/example_workflows/quantum_espresso/pyiron_base.ipynb +++ b/example_workflows/quantum_espresso/pyiron_base.ipynb @@ -4,17 +4,13 @@ "cell_type": "markdown", "id": "be2d61b0-0d47-4349-b4b0-1b767c961644", "metadata": {}, - "source": [ - "# pyiron" - ] + "source": "# pyiron\n\nThis notebook builds the Quantum Espresso energy-volume-curve workflow with [`pyiron_base`](https://github.com/pyiron/pyiron_base). The `job` decorator turns plain functions into delayed pyiron jobs that are only executed once `.pull()` is called on the final result, and the whole chain of jobs is exported as a PWD `workflow.json`. The second half of the notebook loads that same JSON with `aiida-workgraph`, `jobflow` and `pyiron_workflow`." }, { "cell_type": "markdown", "id": "0bad2a57-1bd2-4837-94fe-f8c60e211fae", "metadata": {}, - "source": [ - "## Define workflow with pyiron_base" - ] + "source": "## Define workflow with pyiron_base\n\nWe import the `job` decorator from `pyiron_base` and the PWD writer for `pyiron_base`, together with the Quantum Espresso functions from `workflow.py`." }, { "cell_type": "code", @@ -59,6 +55,12 @@ "workflow_json_filename = \"pyiron_base_qe.json\"" ] }, + { + "cell_type": "markdown", + "id": "d73d32d1", + "source": "`calculate_qe` returns a dict with `energy`, `volume` and `structure` keys. `job(..., output_key_lst=[...])` tells `pyiron_base` which keys to expose as separate delayed outputs (e.g. `calc_mini.output.structure`), so a single downstream job can depend on just one of them instead of the whole dict.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 4, @@ -123,6 +125,12 @@ ")" ] }, + { + "cell_type": "markdown", + "id": "1e0aace0", + "source": "`generate_structures` fans out the relaxed structure into `number_of_strains` (5) strained structures. Since `pyiron_base` builds the graph lazily, it cannot know how many items a delayed dict result will contain, so `list_length` is passed explicitly - this lets `structure_lst` be iterated with `enumerate()` below to create one independent `scf` `calculate_qe` job per strained structure, before anything has actually run.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 8, @@ -179,6 +187,12 @@ ")" ] }, + { + "cell_type": "markdown", + "id": "e7e789dd", + "source": "`write_workflow_json` only needs the single terminal delayed object (`plot`, the plotting job) - `pyiron_base` walks its dependency chain backwards to discover the full graph, unlike `jobflow`'s `Flow` which needs an explicit list of jobs.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 11, @@ -773,9 +787,7 @@ "cell_type": "markdown", "id": "7d75a2f6-6fad-49c8-bd29-37cca1b84441", "metadata": {}, - "source": [ - "## Load Workflow with aiida" - ] + "source": "## Load Workflow with aiida\n\n`load_workflow_json` reconstructs the graph as an `aiida-workgraph` `WorkGraph`. The lattice constant `a` is overridden (4.04 -> 4.05) to show the loaded graph can still be edited like one built natively with `aiida-workgraph`, before running it with `wg.run()`." }, { "cell_type": "code", @@ -925,9 +937,7 @@ "cell_type": "markdown", "id": "c4f5c047-c6da-4b54-9007-415faca7a448", "metadata": {}, - "source": [ - "## Load Workflow with jobflow" - ] + "source": "## Load Workflow with jobflow\n\nThe same JSON is loaded as a `jobflow` `Flow` and executed locally with `run_locally`, again after overriding the lattice constant `a`." }, { "cell_type": "code", @@ -1163,9 +1173,7 @@ "cell_type": "markdown", "id": "406b0429e65b9760", "metadata": {}, - "source": [ - "## Load Workflow with pyiron_workflow" - ] + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the graph is loaded into a `pyiron_workflow` `Workflow`, its lattice constant overridden, drawn with `.draw()`, and executed with `.run()`." }, { "cell_type": "code", @@ -4990,4 +4998,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file diff --git a/example_workflows/quantum_espresso/pyiron_workflow.ipynb b/example_workflows/quantum_espresso/pyiron_workflow.ipynb index 9f7e084f..687be88f 100644 --- a/example_workflows/quantum_espresso/pyiron_workflow.ipynb +++ b/example_workflows/quantum_espresso/pyiron_workflow.ipynb @@ -4,10 +4,7 @@ "cell_type": "markdown", "id": "760acc89-8c02-4bc9-a8f6-2572506b7085", "metadata": {}, - "source": [ - "# pyiron_workflow\n", - "## Define workflow with pyiron_workflow" - ] + "source": "# pyiron_workflow\n## Define workflow with pyiron_workflow\n\nThis notebook builds the Quantum Espresso energy-volume-curve workflow with [`pyiron_workflow`](https://github.com/pyiron/pyiron_workflow). Plain functions are converted to graph nodes with `to_function_node`, assembled into a `Workflow` via attribute access, and exported as a PWD `workflow.json`. The second half of the notebook loads that same JSON with `aiida-workgraph`, `jobflow` and `pyiron_base`." }, { "cell_type": "code", @@ -42,6 +39,12 @@ ")" ] }, + { + "cell_type": "markdown", + "id": "2d3f3ba6", + "source": "`pyiron_workflow` has no built-in equivalent of the `get_list` helper the other engines use to collect several outputs into a list, so `get_values_from_dict` is defined here as a custom node with `as_function_node` to turn a dict of energies/volumes into a plain list for the final plot.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 3, @@ -83,6 +86,12 @@ "plot_energy_volume_curve = to_function_node(\"plot_energy_volume_curve\", _plot_energy_volume_curve, \"plot_energy_volume_curve\", validate_output_labels=False)" ] }, + { + "cell_type": "markdown", + "id": "be84f115", + "source": "`to_function_node` wraps each plain Python function into a `pyiron_workflow` node type. `plot_energy_volume_curve` passes `validate_output_labels=False` since it has no return value, so its output cannot be matched to an explicit output label.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 6, @@ -107,6 +116,12 @@ "wf.pseudopotentials = {\"Al\": \"Al.pbe-n-kjpaw_psl.1.0.0.UPF\"}" ] }, + { + "cell_type": "markdown", + "id": "c117be40", + "source": "Nodes are added to the `Workflow` by assigning their result to an attribute of `wf` (e.g. `wf.structure = get_bulk_structure(...)`), which both registers the node and gives it a name in the graph. Since `calculate_qe` expects a single `input_dict` argument, `inputs_to_dict` is used to bundle the structure, pseudopotentials, k-points, calculation type and smearing into one dict-valued node before the first (`vc-relax`) QE calculation.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 8, @@ -164,6 +179,12 @@ ")" ] }, + { + "cell_type": "markdown", + "id": "6d21ce04", + "source": "`generate_structures` fans out the relaxed structure into `number_of_strains` (5) strained structures (`wf.structure_lst[\"s_0\"]` ... `[\"s_4\"]`). The loop below builds one `input_dict` and one independent `scf` `calculate_qe` node per strained structure, so five QE calculations run to sample the energy-volume curve.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 11, @@ -207,6 +228,12 @@ " job_strain_lst.append(getattr(wf, \"calc_strain_\" + str(i)))" ] }, + { + "cell_type": "markdown", + "id": "edc600a3", + "source": "The `energy` and `volume` outputs of the five strained calculations are bundled with `inputs_to_dict` and then flattened into two plain lists with the `get_values_from_dict` node defined above, ready to be passed to `plot_energy_volume_curve`.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 13, @@ -246,6 +273,12 @@ "wf.plot = plot_energy_volume_curve(volume_lst=wf.volume_lst, energy_lst=wf.energy_lst)" ] }, + { + "cell_type": "markdown", + "id": "e5408a80", + "source": "`.draw()` renders the assembled node graph, and `write_workflow_json` exports it - using `wf.graph_as_dict` - to the PWD `workflow.json` format so it can be loaded by another engine.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 15, @@ -4688,9 +4721,7 @@ "cell_type": "markdown", "id": "fef7107a-5af3-4434-ae4c-a8d45e1d9b61", "metadata": {}, - "source": [ - "## Load Workflow with aiida" - ] + "source": "## Load Workflow with aiida\n\n`load_workflow_json` reconstructs the graph as an `aiida-workgraph` `WorkGraph`. The lattice constant `a` is overridden (4.04 -> 4.05) to show the loaded graph can still be edited like one built natively with `aiida-workgraph`, before running it with `wg.run()`." }, { "cell_type": "code", @@ -4840,9 +4871,7 @@ "cell_type": "markdown", "id": "9226d966-d90a-4ab2-9776-df1b50bd6a49", "metadata": {}, - "source": [ - "## Load Workflow with jobflow" - ] + "source": "## Load Workflow with jobflow\n\nThe same JSON is loaded as a `jobflow` `Flow` and executed locally with `run_locally`, again after overriding the lattice constant `a`." }, { "cell_type": "code", @@ -5084,9 +5113,7 @@ "cell_type": "markdown", "id": "e0c78db2-2ead-45d3-92ed-d087e19952ba", "metadata": {}, - "source": [ - "## Load Workflow with pyiron_base" - ] + "source": "## Load Workflow with pyiron_base\n\nFinally, `pyiron_base`'s loader turns the graph into a list of delayed jobs. `.draw()` visualizes the dependency graph and `.pull()` executes the whole chain (relax + 5 strained SCF calculations), after overriding the lattice constant `a`." }, { "cell_type": "code", @@ -6012,4 +6039,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file diff --git a/example_workflows/quantum_espresso/universal_workflow.ipynb b/example_workflows/quantum_espresso/universal_workflow.ipynb index 12568d24..6e834b72 100644 --- a/example_workflows/quantum_espresso/universal_workflow.ipynb +++ b/example_workflows/quantum_espresso/universal_workflow.ipynb @@ -23,12 +23,12 @@ "cells": [ { "cell_type": "markdown", - "source": "# Load Quantum Espresso Energy Volume Curve Workflow", + "source": "# Load Quantum Espresso Energy Volume Curve Workflow\n\nThis notebook loads the same, already-exported [`workflow.json`](workflow.json) with six different backends in turn - `aiida-workgraph`, `executorlib`, `jobflow`, `pyiron_base`, `pyiron_workflow` and plain Python - to demonstrate that the PWD format is interoperable regardless of which engine originally produced it.", "metadata": {} }, { "cell_type": "markdown", - "source": "## Plot", + "source": "## Plot\n\n`python_workflow_definition.plot.plot` renders the raw node/edge structure of `workflow.json` as a graph, independent of any workflow engine - useful to inspect the relax -> strain -> 5x scf -> plot pipeline before executing it.", "metadata": {} }, { @@ -60,7 +60,7 @@ }, { "cell_type": "markdown", - "source": "## Aiida ", + "source": "## Aiida\n\n`load_workflow_json` reconstructs the graph as an `aiida-workgraph` `WorkGraph`, after connecting to the local AiiDA profile with `load_profile()`. `wg.run()` executes the relax step followed by the 5 parallel strained SCF calculations and the final plot.", "metadata": {} }, { @@ -138,7 +138,7 @@ }, { "cell_type": "markdown", - "source": "## executorlib", + "source": "## executorlib\n\nHere `load_workflow_json` takes a live `exe` (a `SingleNodeExecutor`) and submits the whole graph to it, returning a `Future` for the final result; `.result()` blocks until every node - including the 5 fanned-out SCF calculations - has finished executing.", "metadata": {} }, { @@ -185,7 +185,7 @@ }, { "cell_type": "markdown", - "source": "## jobflow", + "source": "## jobflow\n\nThe graph is loaded as a `jobflow` `Flow` and executed locally with `run_locally`, which runs the relax job, the 5 strained SCF jobs and the plotting job in dependency order.", "metadata": {} }, { @@ -318,7 +318,7 @@ }, { "cell_type": "markdown", - "source": "## pyiron_base", + "source": "## pyiron_base\n\n`load_workflow_json` turns the graph into a list of delayed pyiron jobs; `.draw()` visualizes the dependency graph (including the fan-out into 5 strained SCF jobs) and `.pull()` executes the whole chain.", "metadata": {} }, { @@ -434,7 +434,7 @@ { "metadata": {}, "cell_type": "markdown", - "source": "## Load Workflow with pyiron_workflow" + "source": "## Load Workflow with pyiron_workflow\n\nThe graph is loaded into a `pyiron_workflow` `Workflow`, drawn with `.draw()`, and executed with `.run()`." }, { "metadata": {}, @@ -466,7 +466,7 @@ }, { "cell_type": "markdown", - "source": "## Python", + "source": "## Python\n\n`python_workflow_definition.purepython.load_workflow_json` is a minimal reference implementation with no external workflow engine: it topologically sorts the nodes by their edges and calls each function directly in plain Python, returning the final result.", "metadata": {} }, { @@ -502,4 +502,4 @@ "execution_count": 19 } ] -} +} \ No newline at end of file