diff --git a/documentation/conclusion.md b/documentation/conclusion.md index f0e6582..5afbcb4 100644 --- a/documentation/conclusion.md +++ b/documentation/conclusion.md @@ -1,3 +1,17 @@ # Conclusion Based on the Python Workflow Definition three rather different workflows were implemented in rather different workflow -engines. This demonstrates the interoperability achieved with the Python Workflow Definition. \ No newline at end of file +engines: a simple arithmetic example coupling two Python functions, a Quantum Espresso energy volume curve calculation +with fan-out parallelism, and a file based, multi-tool NFDI4Ing benchmark. In each case the same `workflow.json` was +written by one workflow engine and successfully loaded and executed by the others, without any change to the underlying +Python functions in `workflow.py`. This demonstrates the interoperability achieved with the Python Workflow Definition, +and shows that it scales from small, in-memory Python workflows to larger, file based, multi-environment scientific +workflows. + +## Where to go next +* Browse the [example_workflows](https://github.com/pythonworkflow/python-workflow-definition/tree/main/example_workflows) + directory for the full set of notebooks, including engines not covered in this book such as `pyiron_workflow`, + `executorlib` and CWL. +* See the [README](https://github.com/pythonworkflow/python-workflow-definition) for installation instructions via + `pip` or `conda`. +* Read the accompanying publication: [J. Janssen et al., A python workflow definition for computational materials + design, Digital Discovery, 2025](https://doi.org/10.1039/D5DD00231A). \ No newline at end of file diff --git a/documentation/evcurve.md b/documentation/evcurve.md index 9c7ac43..536eec6 100644 --- a/documentation/evcurve.md +++ b/documentation/evcurve.md @@ -1,4 +1,67 @@ # Energy Volume Curve Based on [previous work](https://materialdigital.github.io/ADIS2023/README.html) from the [ADIS 2023 workshop](https://www.mpie.de/4902385/adis2023) the calculation of an energy volume curve with the [quantum espresso](https://www.quantum-espresso.org) density -functional theory (DFT) simulation code is implemented in the Python Workflow Definition. \ No newline at end of file +functional theory (DFT) simulation code is implemented in the Python Workflow Definition. + +## Workflow +An energy-volume curve is computed by relaxing a bulk crystal structure, straining it to a series of volumes and +computing the total energy at each volume with a self-consistent-field (SCF) calculation. The pipeline is implemented +as five Python functions in [workflow.py](example_workflows/quantum_espresso/workflow.py): +```python +def get_bulk_structure(element, a, cubic): + # build an ASE bulk crystal structure, e.g. Al + +def calculate_qe(working_directory, input_dict): + # write a pw.x input file, run "pw.x -in input.pwi > output.pwo" and parse the pwscf.xml output; + # input_dict["calculation"] selects "vc-relax" (cell + geometry relaxation) or "scf" (single-point energy) + +def generate_structures(structure, strain_lst): + # apply each volumetric strain in strain_lst to a structure, returning one structure per strain + +def plot_energy_volume_curve(volume_lst, energy_lst): + # plot energy against volume and save it as evcurve.png +``` +`write_input`/`collect_output` handle the file-based `pw.x` input/output (`.pwi`/`.pwo`/`.xml` files), and +`ase_to_json`/`json_to_ase` (de)serialize ASE `Atoms` objects to/from OPTIMADE-JSON strings, so structures can be +passed between functions as plain JSON data instead of Python objects. + +The workflow combines these functions as: `get_bulk_structure` -> `calculate_qe` (`vc-relax`, once) -> +`generate_structures` -> `calculate_qe` (`scf`, once per strained structure) -> `plot_energy_volume_curve`. With five +strains this means one relaxation followed by five independent SCF calculations that fan out from +`generate_structures` and feed back into a single plot. + +## workflow.json +[workflow.json](example_workflows/quantum_espresso/workflow.json) encodes this fan-out as five separate +`workflow.calculate_qe` function nodes, each connected to its own strained structure and its own `working_directory` +input, all reading the same shared inputs (`pseudopotentials`, `kpts`, `calculation`, `smearing`) via +`python_workflow_definition.shared.get_dict`. An excerpt showing the pattern for two of the five strained +calculations: +``` +{ + "nodes": [ + {"id": 2, "type": "function", "value": "workflow.generate_structures"}, + {"id": 3, "type": "function", "value": "workflow.calculate_qe"}, + {"id": 4, "type": "function", "value": "workflow.calculate_qe"}, + {"id": 19, "type": "input", "value": "strain_0", "name": "working_directory_1"}, + {"id": 20, "type": "function", "value": "python_workflow_definition.shared.get_dict"}, + {"id": 22, "type": "input", "value": "strain_1", "name": "working_directory_2"}, + {"id": 23, "type": "function", "value": "python_workflow_definition.shared.get_dict"} + ], + "edges": [ + {"target": 3, "targetPort": "working_directory", "source": 19, "sourcePort": null}, + {"target": 20, "targetPort": "structure", "source": 2, "sourcePort": "s_0"}, + {"target": 3, "targetPort": "input_dict", "source": 20, "sourcePort": null}, + {"target": 4, "targetPort": "working_directory", "source": 22, "sourcePort": null}, + {"target": 23, "targetPort": "structure", "source": 2, "sourcePort": "s_1"}, + {"target": 4, "targetPort": "input_dict", "source": 23, "sourcePort": null} + ] +} +``` +Node 2 (`generate_structures`) exposes one output port per strain (`s_0`, `s_1`, ...); each port is wired into its +own `get_dict`/`calculate_qe` pair. The energy and volume outputs of all five `calculate_qe` nodes are collected back +into lists with `python_workflow_definition.shared.get_list` before being passed to `plot_energy_volume_curve`. + +## Requirements +Running this workflow (rather than just loading/inspecting the JSON) requires the Quantum Espresso `pw.x` binary, +matching pseudopotential files (see [espresso/pseudo](example_workflows/quantum_espresso/espresso/pseudo)) and the +Python dependencies listed in [environment.yml](example_workflows/quantum_espresso/environment.yml). \ No newline at end of file diff --git a/documentation/intro.md b/documentation/intro.md index 0b28a3b..4ad9228 100644 --- a/documentation/intro.md +++ b/documentation/intro.md @@ -13,6 +13,9 @@ Currently supported workflow engines: * [aiida-workgraph](https://github.com/aiidateam/aiida-workgraph) * [jobflow](https://github.com/materialsproject/jobflow) * [pyiron_base](https://github.com/pyiron/pyiron_base) +* [pyiron_workflow](https://github.com/pyiron/pyiron_workflow) +* [executorlib](https://github.com/pyiron/executorlib) +* [CWL](https://www.commonwl.org/) via [cwltool](https://github.com/common-workflow-language/cwltool) ## Example Workflows Three workflows are implemented: diff --git a/documentation/nfdi.md b/documentation/nfdi.md index f1a18cb..1fa6a86 100644 --- a/documentation/nfdi.md +++ b/documentation/nfdi.md @@ -1,4 +1,61 @@ # NFDI4Ing Benchmark -To demonstrate the compatibility of the Python Workflow Definition to file based workflows, the workflow benchmark -developed as part of [NFDI4Ing](https://www.inggrid.org/article/id/3726/) is implemented for all three simulation codes -based on the Python Workflow Definition. \ No newline at end of file +To demonstrate the compatibility of the Python Workflow Definition to file based workflows, the workflow benchmark +developed as part of [NFDI4Ing](https://www.inggrid.org/article/id/3726/) is implemented for all workflow engines +based on the Python Workflow Definition. + +## Workflow +Unlike the [arithmetic](arithmetic.md) and energy-volume-curve examples, which pass Python objects between functions, +this benchmark chains together external command line tools that each require their own conda environment (defined in +`source/envs/*.yaml`) and communicate through files rather than in-memory values. The +pipeline is defined by six Python functions in [workflow.py](example_workflows/nfdi/workflow.py), each of which +copies its inputs into a stage-specific subdirectory (`preprocessing`, `processing` or `postprocessing`) and calls +its external tool with `conda_subprocess.check_output`: +```python +def generate_mesh(domain_size: float, source_directory: str) -> str: ... +def convert_to_xdmf(gmsh_output_file: str) -> dict: ... +def poisson(meshio_output_xdmf: str, meshio_output_h5: str, source_directory: str) -> dict: ... +def plot_over_line(poisson_output_pvd_file: str, poisson_output_vtu_file: str, source_directory: str) -> str: ... +def substitute_macros(pvbatch_output_file: str, ndofs: int, domain_size: float, source_directory: str) -> str: ... +def compile_paper(macros_tex: str, plot_file: str, source_directory: str) -> str: ... +``` +* `generate_mesh` runs [`gmsh`](https://gmsh.info) on `unit_square.geo` to mesh a 2D square whose side length is set + by the `domain_size` parameter, producing a `.msh` file. +* `convert_to_xdmf` runs `meshio convert` to turn the gmsh mesh into an XDMF/H5 file pair that the FEM solver can read. +* `poisson` runs a [FEniCS](https://fenicsproject.org)-based `poisson.py` script that solves the Poisson equation on + the mesh, returning the number of degrees of freedom (`numdofs`) alongside the `.pvd`/`.vtu` result files. +* `plot_over_line` runs ParaView's `pvbatch` with `postprocessing.py` to sample the FEM solution along a line and + write it out as a CSV file. +* `substitute_macros` runs `prepare_paper_macros.py` to fill a LaTeX macro template with the plot data path, the + domain size and the number of degrees of freedom. +* `compile_paper` runs [`tectonic`](https://tectonic-typesetting.github.io) to compile `paper.tex`, which references + those macros and the plot, into `paper.pdf`. + +The connection of these Python functions is stored in the [workflow.json](example_workflows/nfdi/workflow.json) +JSON file, following the same `nodes`/`edges` structure as the other examples. Each function is a `function` node, +`domain_size` and `source_directory` are `input` nodes, and edges connect a function's input port either to another +function's output port or directly to an input node: +``` +{ + "version": "0.1.0", + "nodes": [ + {"id": 0, "type": "function", "value": "workflow.generate_mesh"}, + {"id": 1, "type": "function", "value": "workflow.convert_to_xdmf"}, + {"id": 6, "type": "input", "value": 2.0, "name": "domain_size"}, + {"id": 7, "type": "input", "value": "source", "name": "source_directory"}, + {"id": 8, "type": "output", "name": "result"} + ], + "edges": [ + {"target": 0, "targetPort": "domain_size", "source": 6, "sourcePort": null}, + {"target": 0, "targetPort": "source_directory", "source": 7, "sourcePort": null}, + {"target": 1, "targetPort": "gmsh_output_file", "source": 0, "sourcePort": null} + ] +} +``` +Since `convert_to_xdmf` and `poisson` return a `dict` instead of a single value, the edges that read their output +set `sourcePort` to the specific dictionary key (e.g. `"xdmf_file"` or `"numdofs"`) rather than `null`. + +Because every stage shells out to a different external tool in a different conda environment and passes file paths +along the graph, this benchmark exercises a different part of the Python Workflow Definition than the arithmetic and +energy-volume-curve examples: it shows that the same `workflow.json` produced by one engine can be handed to another +engine, an execution manager like [CWL](https://www.commonwl.org)/`cwltool`, or plain Python, and still reproduce the +same chain of `gmsh`, `meshio`, FEniCS, `pvbatch` and `tectonic` calls end to end. diff --git a/example_workflows/arithmetic/aiida.ipynb b/example_workflows/arithmetic/aiida.ipynb index 743e787..bd067d8 100644 --- a/example_workflows/arithmetic/aiida.ipynb +++ b/example_workflows/arithmetic/aiida.ipynb @@ -23,12 +23,12 @@ "cells": [ { "cell_type": "markdown", - "source": "# Aiida", + "source": "# Aiida\n\nThis notebook defines the arithmetic workflow with [`aiida-workgraph`](https://github.com/aiidateam/aiida-workgraph) and then loads the resulting `workflow.json` into `jobflow`, `pyiron_base` and `pyiron_workflow`, to demonstrate that the same workflow definition can be executed by several different engines.", "metadata": {} }, { "cell_type": "markdown", - "source": "## Define workflow with aiida", + "source": "## Define workflow with aiida\n\n`aiida-workgraph` represents a workflow as a `WorkGraph` of `Task`s. Tasks are executed and stored by the AiiDA engine, so we first need to connect to an AiiDA profile with `load_profile()` before any task can run.", "metadata": {} }, { @@ -63,6 +63,11 @@ "outputs": [], "execution_count": 2 }, + { + "cell_type": "markdown", + "source": "The arithmetic functions are imported under a leading underscore because they are still plain Python functions at this point; `wg.add_task` below turns each one into an AiiDA `Task` that can be linked to other tasks' inputs and outputs. `get_prod_and_div` returns a dictionary, so it is wrapped with `task(outputs=['prod', 'div'])` to expose `prod` and `div` as separate output sockets.", + "metadata": {} + }, { "cell_type": "code", "source": "wg = WorkGraph(\"arithmetic\")", @@ -99,6 +104,11 @@ "outputs": [], "execution_count": 6 }, + { + "cell_type": "markdown", + "source": "With all tasks connected, `write_workflow_json` walks the `WorkGraph`'s tasks and links and serializes them into the PWD `workflow.json` format: each task becomes a `function` node and each link becomes an edge between an input and an output port.", + "metadata": {} + }, { "cell_type": "code", "source": "write_workflow_json(wg=wg, file_name=workflow_json_filename)", @@ -125,7 +135,7 @@ }, { "cell_type": "markdown", - "source": "## Load Workflow with jobflow", + "source": "## Load Workflow with jobflow\n\nThe same `workflow.json` can now be loaded by a different engine. `python_workflow_definition.jobflow.load_workflow_json` reconstructs a jobflow `Flow` from the JSON, which is then executed locally with `run_locally`.", "metadata": {} }, { @@ -180,7 +190,7 @@ }, { "cell_type": "markdown", - "source": "## Load Workflow with pyiron_base", + "source": "## Load Workflow with pyiron_base\n\nThe same JSON file is loaded into `pyiron_base`, producing a list of delayed jobs. `.draw()` visualizes the dependency graph and `.pull()` triggers delayed execution, running each job in turn and returning the final result.", "metadata": {} }, { @@ -236,7 +246,7 @@ { "metadata": {}, "cell_type": "markdown", - "source": "## Load Workflow with pyiron_workflow" + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the workflow is loaded into `pyiron_workflow`, producing a `Workflow` object whose node graph can be visualized with `.draw()` and executed with `.run()`." }, { "metadata": {}, @@ -267,4 +277,4 @@ "source": "wf.run()" } ] -} +} \ No newline at end of file diff --git a/example_workflows/arithmetic/cwl.ipynb b/example_workflows/arithmetic/cwl.ipynb index 9912fa9..b0c0f40 100644 --- a/example_workflows/arithmetic/cwl.ipynb +++ b/example_workflows/arithmetic/cwl.ipynb @@ -21,6 +21,12 @@ "nbformat_minor": 5, "nbformat": 4, "cells": [ + { + "cell_type": "markdown", + "id": "7a93c81a", + "source": "# CWL\n\n[`cwltool`](https://github.com/common-workflow-language/cwltool) is the reference implementation of the [Common Workflow Language (CWL)](https://www.commonwl.org/), a standard for describing command-line based workflows. Unlike the other engines in this repository, CWL has no Python API: workflows are described in YAML/JSON documents and executed by the external `cwltool` command. This notebook converts the arithmetic `workflow.json` into an equivalent set of CWL files and executes them with `cwltool`.", + "metadata": {} + }, { "id": "377fef56-484d-491c-b19e-1be6931e44eb", "cell_type": "code", @@ -41,6 +47,12 @@ "outputs": [], "execution_count": 2 }, + { + "cell_type": "markdown", + "id": "55b1a180", + "source": "`write_workflow` reads the existing `workflow.json` and generates the corresponding CWL description: one `CommandLineTool` `.cwl` file per function node, a `workflow.cwl` file that wires the steps together, and a `workflow.yml` file listing the input values (each serialized to its own pickle file). Every generated tool invokes `python -m python_workflow_definition.cwl`, which loads the target function from `workflow.py` and calls it with the pickled inputs.", + "metadata": {} + }, { "id": "5303c059-8ae4-4557-858e-b4bd64eac711", "cell_type": "code", @@ -51,6 +63,12 @@ "outputs": [], "execution_count": 3 }, + { + "cell_type": "markdown", + "id": "0581c24f", + "source": "Running `cwltool workflow.cwl workflow.yml` executes the steps in dependency order, passing intermediate results between steps as pickle files, and prints the location of the final output file.", + "metadata": {} + }, { "id": "df302bd2-e9b6-4595-979c-67c46414d986", "cell_type": "code", @@ -62,11 +80,17 @@ { "name": "stdout", "output_type": "stream", - "text": "/srv/conda/envs/notebook/bin/cwltool:11: DeprecationWarning: Nesting argument groups is deprecated.\n sys.exit(run())\n\u001B[1;30mINFO\u001B[0m /srv/conda/envs/notebook/bin/cwltool 3.1.20250110105449\n\u001B[1;30mINFO\u001B[0m Resolved 'workflow.cwl' to 'file:///home/jovyan/example_workflows/arithmetic/workflow.cwl'\n\u001B[1;30mINFO\u001B[0m [workflow ] start\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_prod_and_div_0\n\u001B[1;30mINFO\u001B[0m [step get_prod_and_div_0] start\n\u001B[1;30mINFO\u001B[0m [job get_prod_and_div_0] /tmp/_1apt559$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/dyuxg6ka/stgf337bac5-2f8e-4489-bd51-83491b1d5f0e/workflow.py \\\n --function=workflow.get_prod_and_div \\\n --arg_x=/tmp/dyuxg6ka/stgd87c2fe4-ca8e-4031-9af0-751d6944cbe7/x.pickle \\\n --arg_y=/tmp/dyuxg6ka/stg7bcb8ced-0e5b-442e-a0e0-2b3117afe2d8/y.pickle\n\u001B[1;30mINFO\u001B[0m [job get_prod_and_div_0] completed success\n\u001B[1;30mINFO\u001B[0m [step get_prod_and_div_0] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_sum_1\n\u001B[1;30mINFO\u001B[0m [step get_sum_1] start\n\u001B[1;30mINFO\u001B[0m [job get_sum_1] /tmp/ysau_yra$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/rm6c9qek/stgfef68b90-5d69-4da9-a952-b9c2f2a1eb76/workflow.py \\\n --function=workflow.get_sum \\\n --arg_x=/tmp/rm6c9qek/stg35728be7-5a6d-487b-bd02-345581007663/prod.pickle \\\n --arg_y=/tmp/rm6c9qek/stgc0992786-237e-4308-9d9e-814ebbfb0319/div.pickle\n\u001B[1;30mINFO\u001B[0m [job get_sum_1] completed success\n\u001B[1;30mINFO\u001B[0m [step get_sum_1] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_square_2\n\u001B[1;30mINFO\u001B[0m [step get_square_2] start\n\u001B[1;30mINFO\u001B[0m [job get_square_2] /tmp/o3gzya8t$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/6a3sltnz/stg58ccf048-a5c3-4ba3-a24f-c64adfff84cd/workflow.py \\\n --function=workflow.get_square \\\n --arg_x=/tmp/6a3sltnz/stg70203cec-cbd4-4fea-a3de-70e9373d9c9d/result.pickle\n\u001B[1;30mINFO\u001B[0m [job get_square_2] completed success\n\u001B[1;30mINFO\u001B[0m [step get_square_2] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] completed success\n{\n \"result_file\": {\n \"location\": \"file:///home/jovyan/example_workflows/arithmetic/result.pickle\",\n \"basename\": \"result.pickle\",\n \"class\": \"File\",\n \"checksum\": \"sha1$1fd8f217331336b1226b84e351c56fb9a8d93685\",\n \"size\": 21,\n \"path\": \"/home/jovyan/example_workflows/arithmetic/result.pickle\"\n }\n}\u001B[1;30mINFO\u001B[0m Final process status is success\n" + "text": "/srv/conda/envs/notebook/bin/cwltool:11: DeprecationWarning: Nesting argument groups is deprecated.\n sys.exit(run())\n\u001b[1;30mINFO\u001b[0m /srv/conda/envs/notebook/bin/cwltool 3.1.20250110105449\n\u001b[1;30mINFO\u001b[0m Resolved 'workflow.cwl' to 'file:///home/jovyan/example_workflows/arithmetic/workflow.cwl'\n\u001b[1;30mINFO\u001b[0m [workflow ] start\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_prod_and_div_0\n\u001b[1;30mINFO\u001b[0m [step get_prod_and_div_0] start\n\u001b[1;30mINFO\u001b[0m [job get_prod_and_div_0] /tmp/_1apt559$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/dyuxg6ka/stgf337bac5-2f8e-4489-bd51-83491b1d5f0e/workflow.py \\\n --function=workflow.get_prod_and_div \\\n --arg_x=/tmp/dyuxg6ka/stgd87c2fe4-ca8e-4031-9af0-751d6944cbe7/x.pickle \\\n --arg_y=/tmp/dyuxg6ka/stg7bcb8ced-0e5b-442e-a0e0-2b3117afe2d8/y.pickle\n\u001b[1;30mINFO\u001b[0m [job get_prod_and_div_0] completed success\n\u001b[1;30mINFO\u001b[0m [step get_prod_and_div_0] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_sum_1\n\u001b[1;30mINFO\u001b[0m [step get_sum_1] start\n\u001b[1;30mINFO\u001b[0m [job get_sum_1] /tmp/ysau_yra$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/rm6c9qek/stgfef68b90-5d69-4da9-a952-b9c2f2a1eb76/workflow.py \\\n --function=workflow.get_sum \\\n --arg_x=/tmp/rm6c9qek/stg35728be7-5a6d-487b-bd02-345581007663/prod.pickle \\\n --arg_y=/tmp/rm6c9qek/stgc0992786-237e-4308-9d9e-814ebbfb0319/div.pickle\n\u001b[1;30mINFO\u001b[0m [job get_sum_1] completed success\n\u001b[1;30mINFO\u001b[0m [step get_sum_1] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_square_2\n\u001b[1;30mINFO\u001b[0m [step get_square_2] start\n\u001b[1;30mINFO\u001b[0m [job get_square_2] /tmp/o3gzya8t$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/6a3sltnz/stg58ccf048-a5c3-4ba3-a24f-c64adfff84cd/workflow.py \\\n --function=workflow.get_square \\\n --arg_x=/tmp/6a3sltnz/stg70203cec-cbd4-4fea-a3de-70e9373d9c9d/result.pickle\n\u001b[1;30mINFO\u001b[0m [job get_square_2] completed success\n\u001b[1;30mINFO\u001b[0m [step get_square_2] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] completed success\n{\n \"result_file\": {\n \"location\": \"file:///home/jovyan/example_workflows/arithmetic/result.pickle\",\n \"basename\": \"result.pickle\",\n \"class\": \"File\",\n \"checksum\": \"sha1$1fd8f217331336b1226b84e351c56fb9a8d93685\",\n \"size\": 21,\n \"path\": \"/home/jovyan/example_workflows/arithmetic/result.pickle\"\n }\n}\u001b[1;30mINFO\u001b[0m Final process status is success\n" } ], "execution_count": 4 }, + { + "cell_type": "markdown", + "id": "c483a640", + "source": "The workflow's final output is written to `result.pickle`; unpickling it gives the same numeric result produced by the other engines.", + "metadata": {} + }, { "id": "2942dbba-ea0a-4d20-be5c-ed9992d09ff8", "cell_type": "code", @@ -94,4 +118,4 @@ "execution_count": null } ] -} +} \ No newline at end of file diff --git a/example_workflows/arithmetic/executorlib.ipynb b/example_workflows/arithmetic/executorlib.ipynb index 509b731..612c615 100644 --- a/example_workflows/arithmetic/executorlib.ipynb +++ b/example_workflows/arithmetic/executorlib.ipynb @@ -4,17 +4,13 @@ "cell_type": "markdown", "id": "c39b76fb-259f-4e16-a44d-02a295c82386", "metadata": {}, - "source": [ - "# executorlib" - ] + "source": "# executorlib\n\nThis notebook defines the arithmetic workflow with [`executorlib`](https://github.com/pyiron/executorlib), a lightweight `concurrent.futures`-style executor for up-scaling Python functions to HPC resources, and then loads the resulting `workflow.json` into `aiida-workgraph`, `jobflow`, `pyiron_base` and `pyiron_workflow`." }, { "cell_type": "markdown", "id": "3638419b-a0cb-49e2-b157-7fbb1acde90f", "metadata": {}, - "source": [ - "## Define workflow with executorlib" - ] + "source": "## Define workflow with executorlib\n\n`executorlib`'s `SingleNodeExecutor` implements the standard `Executor` interface: `.submit()` schedules a function call and immediately returns a `Future`. Passing `export_workflow_filename` to the executor records every submitted call and its dependencies as a side effect and writes them out as a PWD `workflow.json` once the `with` block exits." }, { "cell_type": "code", @@ -46,6 +42,12 @@ "workflow_json_filename = \"executorlib_arithmetic.json\"" ] }, + { + "cell_type": "markdown", + "id": "a6c2ee50", + "source": "`get_prod_and_div` returns a dictionary, so `get_item_from_future` is used to extract the `prod` and `div` keys from its future's result before they can be passed on as separate arguments to `get_sum`.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 4, @@ -157,9 +159,7 @@ "cell_type": "markdown", "id": "a4c0faaf-e30d-4ded-8e9f-57f97f51b14c", "metadata": {}, - "source": [ - "## Load Workflow with aiida" - ] + "source": "## Load Workflow with aiida\n\nThe exported `workflow.json` is loaded into `aiida-workgraph`. As with the other notebooks, an AiiDA profile must be connected via `load_profile()` before the resulting `WorkGraph` can be run." }, { "cell_type": "code", @@ -260,9 +260,7 @@ "cell_type": "markdown", "id": "0c3503e1-0a32-40e1-845d-3fd9ec3c4c19", "metadata": {}, - "source": [ - "## Load Workflow with jobflow" - ] + "source": "## Load Workflow with jobflow\n\nThe same JSON is loaded into a jobflow `Flow` and executed locally with `run_locally`." }, { "cell_type": "code", @@ -336,9 +334,7 @@ "cell_type": "markdown", "id": "406fd07dd4bd8006", "metadata": {}, - "source": [ - "## Load Workflow with pyiron_base" - ] + "source": "## Load Workflow with pyiron_base\n\nLoading the workflow into `pyiron_base` produces a list of delayed jobs; `.draw()` visualizes the dependency graph and `.pull()` triggers delayed execution of the whole chain." }, { "cell_type": "code", @@ -495,9 +491,7 @@ "cell_type": "markdown", "id": "406d62d0-76eb-411f-a7fb-f866d5cec9c1", "metadata": {}, - "source": [ - "## Load Workflow with pyiron_workflow" - ] + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the workflow is loaded into a `pyiron_workflow` `Workflow` object, drawn with `.draw()` and executed with `.run()`." }, { "cell_type": "code", @@ -1128,4 +1122,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file diff --git a/example_workflows/arithmetic/pyiron_base.ipynb b/example_workflows/arithmetic/pyiron_base.ipynb index 18742f2..eaba0a5 100644 --- a/example_workflows/arithmetic/pyiron_base.ipynb +++ b/example_workflows/arithmetic/pyiron_base.ipynb @@ -24,13 +24,13 @@ { "id": "c39b76fb-259f-4e16-a44d-02a295c82386", "cell_type": "markdown", - "source": "# pyiron", + "source": "# pyiron\n\nThis notebook defines the arithmetic workflow with [`pyiron_base`](https://github.com/pyiron/pyiron_base) and then loads the resulting `workflow.json` into `aiida-workgraph`, `jobflow` and `pyiron_workflow`.", "metadata": {} }, { "id": "3638419b-a0cb-49e2-b157-7fbb1acde90f", "cell_type": "markdown", - "source": "## Define workflow with pyiron_base", + "source": "## Define workflow with pyiron_base\n\nThe `job` decorator from `pyiron_base` turns a plain Python function into a pyiron job: calling the decorated function no longer executes it immediately, but returns a delayed object that can be chained with other jobs. For functions such as `get_prod_and_div` that return a dictionary, `output_key_lst` declares the dictionary keys so that individual entries (`prod`, `div`) can be wired as separate outputs to downstream jobs.", "metadata": {} }, { @@ -113,6 +113,12 @@ "outputs": [], "execution_count": 8 }, + { + "cell_type": "markdown", + "id": "9cecf662", + "source": "None of these calls run yet, they only build up a graph of delayed jobs. `write_workflow_json` walks that graph starting from the final `result` job and serializes it into the PWD `workflow.json` format.", + "metadata": {} + }, { "id": "e464da97-16a1-4772-9a07-0a47f152781d", "cell_type": "code", @@ -142,7 +148,7 @@ { "id": "a4c0faaf-e30d-4ded-8e9f-57f97f51b14c", "cell_type": "markdown", - "source": "## Load Workflow with aiida", + "source": "## Load Workflow with aiida\n\nThe exported `workflow.json` is loaded into `aiida-workgraph`. An AiiDA profile must be connected via `load_profile()` before the resulting `WorkGraph` can be run.", "metadata": {} }, { @@ -217,7 +223,7 @@ { "id": "0c3503e1-0a32-40e1-845d-3fd9ec3c4c19", "cell_type": "markdown", - "source": "## Load Workflow with jobflow", + "source": "## Load Workflow with jobflow\n\nThe same JSON is loaded into a jobflow `Flow` and executed locally with `run_locally`.", "metadata": {} }, { @@ -277,7 +283,7 @@ { "metadata": {}, "cell_type": "markdown", - "source": "## Load Workflow with pyiron_workflow", + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the workflow is loaded into a `pyiron_workflow` `Workflow` object, drawn with `.draw()` and executed with `.run()`.", "id": "406fd07dd4bd8006" }, { @@ -319,4 +325,4 @@ "id": "d36522a1c315b7f5" } ] -} +} \ No newline at end of file diff --git a/example_workflows/arithmetic/pyiron_workflow.ipynb b/example_workflows/arithmetic/pyiron_workflow.ipynb index 2dd96a5..7946d49 100644 --- a/example_workflows/arithmetic/pyiron_workflow.ipynb +++ b/example_workflows/arithmetic/pyiron_workflow.ipynb @@ -4,17 +4,13 @@ "cell_type": "markdown", "id": "9db99131-e5ef-49a9-93f6-13c44782c119", "metadata": {}, - "source": [ - "# pyiron_workflow" - ] + "source": "# pyiron_workflow\n\nThis notebook defines the arithmetic workflow with [`pyiron_workflow`](https://github.com/pyiron/pyiron_workflow) and then loads the resulting `workflow.json` into `aiida-workgraph`, `jobflow` and `pyiron_base`." }, { "cell_type": "markdown", "id": "d16df60e-1359-4d4c-bb95-2b994860d04a", "metadata": {}, - "source": [ - "## Define workflow in pyiron_workflow" - ] + "source": "## Define workflow in pyiron_workflow\n\n`to_function_node` wraps a plain Python function into a `pyiron_workflow` node that can be added to a `Workflow` graph. Nodes are assembled into a graph with attribute access: assigning `wf. = node(...)` both registers the node under that name and connects its inputs to the outputs of other nodes already on the graph." }, { "cell_type": "code", @@ -96,6 +92,12 @@ "wf.square_result = get_square(x=wf.tmp_sum)" ] }, + { + "cell_type": "markdown", + "id": "7d2460f1", + "source": "`wf.draw()` renders the resulting node graph, and `write_workflow_json` serializes `wf.graph_as_dict` into the PWD `workflow.json` format.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 7, @@ -760,9 +762,7 @@ "cell_type": "markdown", "id": "1681d449-0266-4da4-8d87-40a055cf3149", "metadata": {}, - "source": [ - "## Load Workflow with aiida" - ] + "source": "## Load Workflow with aiida\n\nThe exported `workflow.json` is loaded into `aiida-workgraph`. An AiiDA profile must be connected via `load_profile()` before the resulting `WorkGraph` can be run." }, { "cell_type": "code", @@ -811,9 +811,7 @@ "cell_type": "markdown", "id": "bac61097-e666-443b-9446-4ebb9a3e6b5f", "metadata": {}, - "source": [ - "# Load Workflow with jobflow" - ] + "source": "## Load Workflow with jobflow\n\nThe same JSON is loaded into a jobflow `Flow` and executed locally with `run_locally`." }, { "cell_type": "code", @@ -860,9 +858,7 @@ "cell_type": "markdown", "id": "f26c9650-649f-436d-8d19-b2cd75cf6488", "metadata": {}, - "source": [ - "# Load Workflow with pyiron_base" - ] + "source": "## Load Workflow with pyiron_base\n\nFinally, the workflow is loaded into `pyiron_base`, producing a list of delayed jobs. `.draw()` visualizes the dependency graph and `.pull()` triggers delayed execution of the whole chain." }, { "cell_type": "code", @@ -917,4 +913,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file diff --git a/example_workflows/arithmetic/universal_workflow.ipynb b/example_workflows/arithmetic/universal_workflow.ipynb index a8cfc73..a100ed2 100644 --- a/example_workflows/arithmetic/universal_workflow.ipynb +++ b/example_workflows/arithmetic/universal_workflow.ipynb @@ -23,12 +23,12 @@ "cells": [ { "cell_type": "markdown", - "source": "# Load Simple Workflow", + "source": "# Load Simple Workflow\n\nThis notebook loads a single, pre-existing `workflow.json` with several different engines in turn. Since the JSON only describes function nodes, inputs, outputs and the edges between them, it does not matter which engine originally wrote it — any of the engines below can load and execute it.", "metadata": {} }, { "cell_type": "markdown", - "source": "## Plot", + "source": "## Plot\n\nBefore running the workflow with any engine, `python_workflow_definition.plot.plot` renders `workflow.json` directly as a graph of function, input and output nodes — independent of any particular workflow engine.", "metadata": {} }, { @@ -60,7 +60,7 @@ }, { "cell_type": "markdown", - "source": "## Aiida ", + "source": "## Aiida\n\nLoading the JSON into `aiida-workgraph` requires connecting to an AiiDA profile first via `load_profile()`; the workflow is then reconstructed as a `WorkGraph` and executed with `.run()`.", "metadata": {} }, { @@ -130,7 +130,7 @@ }, { "cell_type": "markdown", - "source": "## executorlib", + "source": "## executorlib\n\n`python_workflow_definition.executorlib.load_workflow_json` submits each function node to the given `Executor` and returns the `Future` of the final node; calling `.result()` blocks until the whole chain has completed.", "metadata": {} }, { @@ -180,7 +180,7 @@ }, { "cell_type": "markdown", - "source": "## Jobflow", + "source": "## Jobflow\n\nThe JSON is reconstructed as a jobflow `Flow` and executed locally with `run_locally`.", "metadata": {} }, { @@ -235,7 +235,7 @@ }, { "cell_type": "markdown", - "source": "## pyiron", + "source": "## pyiron\n\nLoading the JSON into `pyiron_base` produces a list of delayed jobs; `.draw()` visualizes the dependency graph and `.pull()` triggers execution of the whole chain.", "metadata": {} }, { @@ -291,7 +291,7 @@ { "metadata": {}, "cell_type": "markdown", - "source": "## Load Workflow with pyiron_workflow" + "source": "## pyiron_workflow\n\nFinally, the workflow is loaded into a `pyiron_workflow` `Workflow` object, drawn with `.draw()` and executed with `.run()`." }, { "metadata": {}, @@ -323,7 +323,7 @@ }, { "cell_type": "markdown", - "source": "## Python", + "source": "## Python\n\nAs a baseline, `python_workflow_definition.purepython.load_workflow_json` interprets the JSON directly in plain Python, with no workflow engine involved: it evaluates each function node in dependency order and returns the final result.", "metadata": {} }, { @@ -354,4 +354,4 @@ "execution_count": 19 } ] -} +} \ No newline at end of file diff --git a/example_workflows/nfdi/aiida.ipynb b/example_workflows/nfdi/aiida.ipynb index 30378ca..ef72f1b 100644 --- a/example_workflows/nfdi/aiida.ipynb +++ b/example_workflows/nfdi/aiida.ipynb @@ -24,13 +24,13 @@ { "id": "106ded66-d202-46ac-82b0-2755ca309bdd", "cell_type": "markdown", - "source": "# Aiida\n\nhttps://github.com/BAMresearch/NFDI4IngScientificWorkflowRequirements", + "source": "# Aiida\n\nhttps://github.com/BAMresearch/NFDI4IngScientificWorkflowRequirements\n\nThis notebook defines the NFDI4Ing file-based workflow benchmark with [`aiida-workgraph`](https://github.com/aiidateam/aiida-workgraph) and then loads the resulting `workflow.json` into `jobflow`, `pyiron_base` and `pyiron_workflow` to show that the same JSON file can be executed by different engines. Every stage of the workflow calls an external command-line tool (`gmsh`, `meshio`, a FEniCS-based Poisson solver, ParaView's `pvbatch`, `tectonic`) in its own conda environment and passes file paths between stages rather than in-memory Python objects — see [`workflow.py`](workflow.py) for the six pipeline functions.", "metadata": {} }, { "id": "11e09b78-cb72-465f-9c8b-5b77f0aa729c", "cell_type": "markdown", - "source": "## Define workflow with aiida", + "source": "## Define workflow with aiida\n\n`aiida-workgraph` represents a workflow as a `WorkGraph` of `Task`s. `load_profile()` connects to the local AiiDA profile/database that stores provenance and results for every task. `generate_mesh`, `plot_over_line`, `substitute_macros` and `compile_paper` each return a single value and can be added to the graph unchanged, but `convert_to_xdmf` and `poisson` return a `dict`, so they are wrapped with `task(outputs=[...])` to expose each dictionary key as its own output socket that later tasks can connect to.", "metadata": {} }, { @@ -82,6 +82,12 @@ "outputs": [], "execution_count": 4 }, + { + "cell_type": "markdown", + "id": "e16e7b61", + "source": "AiiDA tracks every input as a typed, storable node, so the shared `domain_size` and `source_directory` values are wrapped as `orm.Float` and `orm.Str` before being connected to any task.", + "metadata": {} + }, { "id": "37c9d988-1755-446c-9f7b-c32f99e280d4", "cell_type": "code", @@ -112,6 +118,12 @@ "outputs": [], "execution_count": 7 }, + { + "cell_type": "markdown", + "id": "2e764162", + "source": "Each `wg.add_task` call below adds one stage of the pipeline: mesh generation, XDMF conversion, the Poisson solve, line-plot postprocessing, LaTeX macro substitution and paper compilation. Outputs are wired to the next task's inputs with `.outputs.`; since the underlying functions communicate through files on disk, these connections are really file paths handed from one task to the next, not shared Python objects.", + "metadata": {} + }, { "id": "71d411b6-cbec-489e-99e3-ba71680bcb5b", "cell_type": "code", @@ -201,6 +213,12 @@ ], "execution_count": 14 }, + { + "cell_type": "markdown", + "id": "a7bd7df7", + "source": "The `wg` widget above renders the assembled task graph. Once satisfied, `write_workflow_json` serializes it into the PWD `workflow.json` format — one `function` node per task, `input` nodes for `domain_size`/`source_directory`, and edges recording which task output feeds which task's input port, as shown by the `cat` output below.", + "metadata": {} + }, { "id": "fb23ad9c-76fd-4c0b-b546-e305d6c49796", "cell_type": "code", @@ -240,7 +258,7 @@ { "id": "11a829e2-face-469f-b343-2c95763b1f13", "cell_type": "markdown", - "source": "## Load Workflow with jobflow", + "source": "## Load Workflow with jobflow\n\n`python_workflow_definition.jobflow.load_workflow_json` reconstructs the same six-stage pipeline as a jobflow `Flow`, purely from `aiida_nfdi.json` — no reference to the `aiida-workgraph` objects defined above. `run_locally` then executes the jobs in dependency order, re-running `gmsh`, `meshio`, the Poisson solver, `pvbatch` and `tectonic` exactly as before.", "metadata": {} }, { @@ -300,7 +318,7 @@ { "id": "397b16a2-e1ec-4eec-8562-1c84f585c347", "cell_type": "markdown", - "source": "## Load Workflow with pyiron_base", + "source": "## Load Workflow with pyiron_base\n\n`python_workflow_definition.pyiron_base.load_workflow_json` turns the JSON graph into a list of delayed pyiron jobs; `.draw()` visualizes their dependencies and `.pull()` triggers execution, running each job in turn and returning the final `paper.pdf` path.", "metadata": {} }, { @@ -359,7 +377,7 @@ { "metadata": {}, "cell_type": "markdown", - "source": "## Load Workflow with pyiron_workflow", + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the same JSON is loaded into a `pyiron_workflow` `Workflow` object. `.draw()` shows the node graph and `.run()` executes it end to end, again shelling out to each external tool in turn.", "id": "afaf2bc0b3a491ca" }, { @@ -395,4 +413,4 @@ "id": "c1427e3ce0716f64" } ] -} +} \ No newline at end of file diff --git a/example_workflows/nfdi/cwl.ipynb b/example_workflows/nfdi/cwl.ipynb index fc40e63..5fc2d26 100644 --- a/example_workflows/nfdi/cwl.ipynb +++ b/example_workflows/nfdi/cwl.ipynb @@ -21,6 +21,12 @@ "nbformat_minor": 5, "nbformat": 4, "cells": [ + { + "cell_type": "markdown", + "id": "e88c3a45", + "source": "# CWL\n\nThis notebook executes the NFDI4Ing file-based workflow benchmark's `workflow.json` with the [Common Workflow Language](https://www.commonwl.org/) via [`cwltool`](https://github.com/common-workflow-language/cwltool), instead of generating the JSON from a Python workflow engine like the other notebooks in this directory. `python_workflow_definition.cwl.write_workflow` translates the PWD graph into a CWL `workflow.cwl` plus one `CommandLineTool` wrapper per pipeline function; CWL then passes data between steps as pickled files on disk rather than in-memory Python objects, which matches this benchmark's file-based nature.", + "metadata": {} + }, { "id": "377fef56-484d-491c-b19e-1be6931e44eb", "cell_type": "code", @@ -64,7 +70,7 @@ { "id": "b0cf73b9-ea21-4437-8d2a-c51b65bbfa86", "cell_type": "markdown", - "source": "# Overwrite source directory with absolute path", + "source": "## Overwrite source directory with absolute path\n\n`workflow.json` stores `source_directory` as the relative path `\"source\"`. Because `cwltool` stages each step's inputs into its own temporary working directory, a relative path would no longer resolve once execution starts, so it is rewritten to an absolute path before export.", "metadata": {} }, { @@ -100,7 +106,7 @@ { "id": "a9540ba7-f15a-4d04-86aa-0cf2ad4ac185", "cell_type": "markdown", - "source": "# Execute workflow", + "source": "## Execute workflow\n\n`write_workflow` reads `workflow.json` and writes out `workflow.cwl` (the CWL `Workflow` definition), one `_.cwl` `CommandLineTool` per pipeline step, and `workflow.yml` with the input values pickled to disk. `cwltool` then runs the six steps in dependency order, invoking `gmsh`, `meshio`, the Poisson solver, `pvbatch` and `tectonic` in turn, and writes the workflow's single output — the path to the compiled `paper.pdf` — to `result.pickle`.", "metadata": {} }, { @@ -124,11 +130,17 @@ { "name": "stdout", "output_type": "stream", - "text": "/srv/conda/envs/notebook/bin/cwltool:11: DeprecationWarning: Nesting argument groups is deprecated.\n sys.exit(run())\n\u001B[1;30mINFO\u001B[0m /srv/conda/envs/notebook/bin/cwltool 3.1.20250110105449\n\u001B[1;30mINFO\u001B[0m Resolved 'workflow.cwl' to 'file:///home/jovyan/example_workflows/nfdi/workflow.cwl'\n\u001B[1;30mINFO\u001B[0m [workflow ] start\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step generate_mesh_0\n\u001B[1;30mINFO\u001B[0m [step generate_mesh_0] start\n\u001B[1;30mINFO\u001B[0m [job generate_mesh_0] /tmp/xjzxjjxg$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/lint8hha/stgaa9fa1cf-dae6-4402-aa25-3da9289e3937/workflow.py \\\n --function=workflow.generate_mesh \\\n --arg_domain_size=/tmp/lint8hha/stg94834a09-0454-4ad4-bc93-b8f3a1dc72f4/domain_size.pickle \\\n --arg_source_directory=/tmp/lint8hha/stg03fa6508-a0ea-4379-97b1-e3815dfe7395/source_directory.pickle\n\u001B[1;30mINFO\u001B[0m [job generate_mesh_0] Max memory used: 60MiB\n\u001B[1;30mINFO\u001B[0m [job generate_mesh_0] completed success\n\u001B[1;30mINFO\u001B[0m [step generate_mesh_0] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step convert_to_xdmf_1\n\u001B[1;30mINFO\u001B[0m [step convert_to_xdmf_1] start\n\u001B[1;30mINFO\u001B[0m [job convert_to_xdmf_1] /tmp/q47niux0$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/pd0inpeh/stg928009b1-efc5-45d7-9746-e1af8d96f0c5/workflow.py \\\n --function=workflow.convert_to_xdmf \\\n --arg_gmsh_output_file=/tmp/pd0inpeh/stg03750262-f7c4-4bb9-b558-0b7ea0a54cf3/result.pickle\n\u001B[1;30mINFO\u001B[0m [job convert_to_xdmf_1] Max memory used: 69MiB\n\u001B[1;30mINFO\u001B[0m [job convert_to_xdmf_1] completed success\n\u001B[1;30mINFO\u001B[0m [step convert_to_xdmf_1] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step poisson_2\n\u001B[1;30mINFO\u001B[0m [step poisson_2] start\n\u001B[1;30mINFO\u001B[0m [job poisson_2] /tmp/e045_wvq$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/rsa0i3o4/stgfd3d067f-0d0e-46b6-8b1c-e63a936cca99/workflow.py \\\n --function=workflow.poisson \\\n --arg_meshio_output_xdmf=/tmp/rsa0i3o4/stgebd53ba4-dbbc-4769-b359-f1f95611d02d/xdmf_file.pickle \\\n --arg_meshio_output_h5=/tmp/rsa0i3o4/stga4bcfd98-5bf0-4da3-b1e6-a4d55a00cc30/h5_file.pickle \\\n --arg_source_directory=/tmp/rsa0i3o4/stg003e0d81-81e0-4138-9e97-103a910c925e/source_directory.pickle\n\u001B[1;30mINFO\u001B[0m [job poisson_2] Max memory used: 81MiB\n\u001B[1;30mINFO\u001B[0m [job poisson_2] completed success\n\u001B[1;30mINFO\u001B[0m [step poisson_2] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step plot_over_line_3\n\u001B[1;30mINFO\u001B[0m [step plot_over_line_3] start\n\u001B[1;30mINFO\u001B[0m [job plot_over_line_3] /tmp/0tov09ih$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/z9a5oiuc/stgcde66eb7-274f-4ae3-833b-c8d99fe3bbb1/workflow.py \\\n --function=workflow.plot_over_line \\\n --arg_poisson_output_vtu_file=/tmp/z9a5oiuc/stg144f313e-e286-4800-a3d2-86a6cae36a98/vtu_file.pickle \\\n --arg_source_directory=/tmp/z9a5oiuc/stg224c0c40-7cb8-4f1c-921c-013809fb5cbf/source_directory.pickle \\\n --arg_poisson_output_pvd_file=/tmp/z9a5oiuc/stgc4c75ca6-aab0-40be-bfbd-f432d3523d2b/pvd_file.pickle\n\u001B[1;30mINFO\u001B[0m [job plot_over_line_3] Max memory used: 68MiB\n\u001B[1;30mINFO\u001B[0m [job plot_over_line_3] completed success\n\u001B[1;30mINFO\u001B[0m [step plot_over_line_3] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step substitute_macros_4\n\u001B[1;30mINFO\u001B[0m [step substitute_macros_4] start\n\u001B[1;30mINFO\u001B[0m [job substitute_macros_4] /tmp/08mfe8f0$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/5valbs4a/stg15353150-ebc3-49de-93a8-564a25f33270/workflow.py \\\n --function=workflow.substitute_macros \\\n --arg_domain_size=/tmp/5valbs4a/stg0734ab21-1efd-480b-ba2b-cea9ed24012e/domain_size.pickle \\\n --arg_source_directory=/tmp/5valbs4a/stg0c79cde6-159f-4b82-bdb9-aabef8678f9f/source_directory.pickle \\\n --arg_ndofs=/tmp/5valbs4a/stg770f9cb7-d2c3-4ce5-bc00-7a9cdb11d64d/numdofs.pickle \\\n --arg_pvbatch_output_file=/tmp/5valbs4a/stgdd978452-6c84-4308-abf6-7ae2609fc91d/result.pickle\n\u001B[1;30mINFO\u001B[0m [job substitute_macros_4] Max memory used: 60MiB\n\u001B[1;30mINFO\u001B[0m [job substitute_macros_4] completed success\n\u001B[1;30mINFO\u001B[0m [step substitute_macros_4] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step compile_paper_5\n\u001B[1;30mINFO\u001B[0m [step compile_paper_5] start\n\u001B[1;30mINFO\u001B[0m [job compile_paper_5] /tmp/hjjeng16$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/3bq2mxdl/stg34a5d902-97c3-43f1-95eb-5e14d0c68386/workflow.py \\\n --function=workflow.compile_paper \\\n --arg_macros_tex=/tmp/3bq2mxdl/stg184e0acf-5138-4b37-97f0-ba406ed71433/result.pickle \\\n --arg_plot_file=/tmp/3bq2mxdl/stg2360c6e9-26ec-43f5-99a2-26c05668b2fa/result.pickle \\\n --arg_source_directory=/tmp/3bq2mxdl/stg7194b36e-38db-4ef7-af5d-368e9575f15f/source_directory.pickle\n\u001B[1;30mINFO\u001B[0m [job compile_paper_5] Max memory used: 265MiB\n\u001B[1;30mINFO\u001B[0m [job compile_paper_5] completed success\n\u001B[1;30mINFO\u001B[0m [step compile_paper_5] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] completed success\n{\n \"result_file\": {\n \"location\": \"file:///home/jovyan/example_workflows/nfdi/result.pickle\",\n \"basename\": \"result.pickle\",\n \"class\": \"File\",\n \"checksum\": \"sha1$a23617d3f4b4e6970b7f4fb9eb4d2148f9888e58\",\n \"size\": 53,\n \"path\": \"/home/jovyan/example_workflows/nfdi/result.pickle\"\n }\n}\u001B[1;30mINFO\u001B[0m Final process status is success\n" + "text": "/srv/conda/envs/notebook/bin/cwltool:11: DeprecationWarning: Nesting argument groups is deprecated.\n sys.exit(run())\n\u001b[1;30mINFO\u001b[0m /srv/conda/envs/notebook/bin/cwltool 3.1.20250110105449\n\u001b[1;30mINFO\u001b[0m Resolved 'workflow.cwl' to 'file:///home/jovyan/example_workflows/nfdi/workflow.cwl'\n\u001b[1;30mINFO\u001b[0m [workflow ] start\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step generate_mesh_0\n\u001b[1;30mINFO\u001b[0m [step generate_mesh_0] start\n\u001b[1;30mINFO\u001b[0m [job generate_mesh_0] /tmp/xjzxjjxg$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/lint8hha/stgaa9fa1cf-dae6-4402-aa25-3da9289e3937/workflow.py \\\n --function=workflow.generate_mesh \\\n --arg_domain_size=/tmp/lint8hha/stg94834a09-0454-4ad4-bc93-b8f3a1dc72f4/domain_size.pickle \\\n --arg_source_directory=/tmp/lint8hha/stg03fa6508-a0ea-4379-97b1-e3815dfe7395/source_directory.pickle\n\u001b[1;30mINFO\u001b[0m [job generate_mesh_0] Max memory used: 60MiB\n\u001b[1;30mINFO\u001b[0m [job generate_mesh_0] completed success\n\u001b[1;30mINFO\u001b[0m [step generate_mesh_0] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step convert_to_xdmf_1\n\u001b[1;30mINFO\u001b[0m [step convert_to_xdmf_1] start\n\u001b[1;30mINFO\u001b[0m [job convert_to_xdmf_1] /tmp/q47niux0$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/pd0inpeh/stg928009b1-efc5-45d7-9746-e1af8d96f0c5/workflow.py \\\n --function=workflow.convert_to_xdmf \\\n --arg_gmsh_output_file=/tmp/pd0inpeh/stg03750262-f7c4-4bb9-b558-0b7ea0a54cf3/result.pickle\n\u001b[1;30mINFO\u001b[0m [job convert_to_xdmf_1] Max memory used: 69MiB\n\u001b[1;30mINFO\u001b[0m [job convert_to_xdmf_1] completed success\n\u001b[1;30mINFO\u001b[0m [step convert_to_xdmf_1] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step poisson_2\n\u001b[1;30mINFO\u001b[0m [step poisson_2] start\n\u001b[1;30mINFO\u001b[0m [job poisson_2] /tmp/e045_wvq$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/rsa0i3o4/stgfd3d067f-0d0e-46b6-8b1c-e63a936cca99/workflow.py \\\n --function=workflow.poisson \\\n --arg_meshio_output_xdmf=/tmp/rsa0i3o4/stgebd53ba4-dbbc-4769-b359-f1f95611d02d/xdmf_file.pickle \\\n --arg_meshio_output_h5=/tmp/rsa0i3o4/stga4bcfd98-5bf0-4da3-b1e6-a4d55a00cc30/h5_file.pickle \\\n --arg_source_directory=/tmp/rsa0i3o4/stg003e0d81-81e0-4138-9e97-103a910c925e/source_directory.pickle\n\u001b[1;30mINFO\u001b[0m [job poisson_2] Max memory used: 81MiB\n\u001b[1;30mINFO\u001b[0m [job poisson_2] completed success\n\u001b[1;30mINFO\u001b[0m [step poisson_2] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step plot_over_line_3\n\u001b[1;30mINFO\u001b[0m [step plot_over_line_3] start\n\u001b[1;30mINFO\u001b[0m [job plot_over_line_3] /tmp/0tov09ih$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/z9a5oiuc/stgcde66eb7-274f-4ae3-833b-c8d99fe3bbb1/workflow.py \\\n --function=workflow.plot_over_line \\\n --arg_poisson_output_vtu_file=/tmp/z9a5oiuc/stg144f313e-e286-4800-a3d2-86a6cae36a98/vtu_file.pickle \\\n --arg_source_directory=/tmp/z9a5oiuc/stg224c0c40-7cb8-4f1c-921c-013809fb5cbf/source_directory.pickle \\\n --arg_poisson_output_pvd_file=/tmp/z9a5oiuc/stgc4c75ca6-aab0-40be-bfbd-f432d3523d2b/pvd_file.pickle\n\u001b[1;30mINFO\u001b[0m [job plot_over_line_3] Max memory used: 68MiB\n\u001b[1;30mINFO\u001b[0m [job plot_over_line_3] completed success\n\u001b[1;30mINFO\u001b[0m [step plot_over_line_3] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step substitute_macros_4\n\u001b[1;30mINFO\u001b[0m [step substitute_macros_4] start\n\u001b[1;30mINFO\u001b[0m [job substitute_macros_4] /tmp/08mfe8f0$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/5valbs4a/stg15353150-ebc3-49de-93a8-564a25f33270/workflow.py \\\n --function=workflow.substitute_macros \\\n --arg_domain_size=/tmp/5valbs4a/stg0734ab21-1efd-480b-ba2b-cea9ed24012e/domain_size.pickle \\\n --arg_source_directory=/tmp/5valbs4a/stg0c79cde6-159f-4b82-bdb9-aabef8678f9f/source_directory.pickle \\\n --arg_ndofs=/tmp/5valbs4a/stg770f9cb7-d2c3-4ce5-bc00-7a9cdb11d64d/numdofs.pickle \\\n --arg_pvbatch_output_file=/tmp/5valbs4a/stgdd978452-6c84-4308-abf6-7ae2609fc91d/result.pickle\n\u001b[1;30mINFO\u001b[0m [job substitute_macros_4] Max memory used: 60MiB\n\u001b[1;30mINFO\u001b[0m [job substitute_macros_4] completed success\n\u001b[1;30mINFO\u001b[0m [step substitute_macros_4] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step compile_paper_5\n\u001b[1;30mINFO\u001b[0m [step compile_paper_5] start\n\u001b[1;30mINFO\u001b[0m [job compile_paper_5] /tmp/hjjeng16$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/3bq2mxdl/stg34a5d902-97c3-43f1-95eb-5e14d0c68386/workflow.py \\\n --function=workflow.compile_paper \\\n --arg_macros_tex=/tmp/3bq2mxdl/stg184e0acf-5138-4b37-97f0-ba406ed71433/result.pickle \\\n --arg_plot_file=/tmp/3bq2mxdl/stg2360c6e9-26ec-43f5-99a2-26c05668b2fa/result.pickle \\\n --arg_source_directory=/tmp/3bq2mxdl/stg7194b36e-38db-4ef7-af5d-368e9575f15f/source_directory.pickle\n\u001b[1;30mINFO\u001b[0m [job compile_paper_5] Max memory used: 265MiB\n\u001b[1;30mINFO\u001b[0m [job compile_paper_5] completed success\n\u001b[1;30mINFO\u001b[0m [step compile_paper_5] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] completed success\n{\n \"result_file\": {\n \"location\": \"file:///home/jovyan/example_workflows/nfdi/result.pickle\",\n \"basename\": \"result.pickle\",\n \"class\": \"File\",\n \"checksum\": \"sha1$a23617d3f4b4e6970b7f4fb9eb4d2148f9888e58\",\n \"size\": 53,\n \"path\": \"/home/jovyan/example_workflows/nfdi/result.pickle\"\n }\n}\u001b[1;30mINFO\u001b[0m Final process status is success\n" } ], "execution_count": 9 }, + { + "cell_type": "markdown", + "id": "c55f089b", + "source": "The result is unpickled below to confirm it is the expected `paper.pdf` path.", + "metadata": {} + }, { "id": "2942dbba-ea0a-4d20-be5c-ed9992d09ff8", "cell_type": "code", @@ -156,4 +168,4 @@ "execution_count": null } ] -} +} \ No newline at end of file diff --git a/example_workflows/nfdi/executorlib.ipynb b/example_workflows/nfdi/executorlib.ipynb index b1a0f41..8a2e0ba 100644 --- a/example_workflows/nfdi/executorlib.ipynb +++ b/example_workflows/nfdi/executorlib.ipynb @@ -4,19 +4,13 @@ "cell_type": "markdown", "id": "106ded66-d202-46ac-82b0-2755ca309bdd", "metadata": {}, - "source": [ - "# executorlib\n", - "\n", - "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/BAMresearch/NFDI4IngScientificWorkflowRequirements" - ] + "source": "# executorlib\n\nhttps://github.com/BAMresearch/NFDI4IngScientificWorkflowRequirements\n\nThis notebook defines the NFDI4Ing file-based workflow benchmark with [`executorlib`](https://github.com/pyiron/executorlib) and then loads the resulting `workflow.json` into `aiida`, `jobflow`, `pyiron_base` and `pyiron_workflow`. Every stage of the pipeline shells out to an external command-line tool (`gmsh`, `meshio`, a FEniCS-based Poisson solver, ParaView's `pvbatch`, `tectonic`) in its own conda environment and passes file paths between stages rather than in-memory Python objects — see [`workflow.py`](workflow.py)." }, { "cell_type": "markdown", "id": "91dd48ea-aa7e-4937-a68e-59fc5017eb1e", "metadata": {}, - "source": [ - "## Define workflow with executorlib" - ] + "source": "## Define workflow with executorlib\n\n`executorlib` provides a `concurrent.futures`-compatible `SingleNodeExecutor`: each `exe.submit(func, ...)` call schedules a function call and immediately returns a `Future`. Since `convert_to_xdmf` and `poisson` return a `dict`, `get_item_from_future` is used to pick a single key (e.g. `\"xdmf_file\"`) out of a future's eventual result before passing it on as an input to the next stage." }, { "cell_type": "code", @@ -78,6 +72,12 @@ "workflow_json_filename = \"executorlib_nfdi.json\"" ] }, + { + "cell_type": "markdown", + "id": "57d670cb", + "source": "Passing `export_workflow_filename` to `SingleNodeExecutor` makes it record every `submit` call as a side effect and write them out as a PWD `workflow.json` once the `with` block exits — no separate `write_workflow_json` call is needed. The six `submit` calls below mirror the six pipeline stages: mesh generation, XDMF conversion, the Poisson solve, line-plot postprocessing, LaTeX macro substitution and paper compilation.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 7, @@ -301,9 +301,7 @@ "cell_type": "markdown", "id": "d789971e-8f41-45fa-832a-11fd72dea96e", "metadata": {}, - "source": [ - "## Load Workflow with aiida" - ] + "source": "## Load Workflow with aiida\n\nThe JSON produced by `executorlib` is loaded into an `aiida-workgraph` `WorkGraph` via `load_workflow_json`, after connecting to a local AiiDA profile with `load_profile()`. `wg.run()` then re-executes the whole pipeline under `aiida`." }, { "cell_type": "code", @@ -412,9 +410,7 @@ "cell_type": "markdown", "id": "55dc8d12-dfe6-4465-a368-b7e590ae6800", "metadata": {}, - "source": [ - "## Load Workflow with jobflow" - ] + "source": "## Load Workflow with jobflow\n\nThe same `workflow.json` is reconstructed as a jobflow `Flow` and executed locally with `run_locally`, independently of the `executorlib` objects used to define it." }, { "cell_type": "code", @@ -511,9 +507,7 @@ "cell_type": "markdown", "id": "7ecc0861-6fc8-4788-b05b-fa5ae06b96e3", "metadata": {}, - "source": [ - "# Load Workflow with pyiron_base" - ] + "source": "## Load Workflow with pyiron_base\n\n`load_workflow_json` turns the graph into a list of delayed pyiron jobs; `.draw()` visualizes the dependency chain and `.pull()` triggers execution, returning the path to the final `paper.pdf`." }, { "cell_type": "code", @@ -832,9 +826,7 @@ "cell_type": "markdown", "id": "385acbf585763632", "metadata": {}, - "source": [ - "## Load Workflow with pyiron_workflow" - ] + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the graph is loaded into a `pyiron_workflow` `Workflow`. `.draw()` shows the node graph and `.run()` executes it end to end." }, { "cell_type": "code", @@ -2126,4 +2118,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file diff --git a/example_workflows/nfdi/jobflow.ipynb b/example_workflows/nfdi/jobflow.ipynb index 2cd73f2..77c5041 100644 --- a/example_workflows/nfdi/jobflow.ipynb +++ b/example_workflows/nfdi/jobflow.ipynb @@ -30,13 +30,13 @@ { "id": "106ded66-d202-46ac-82b0-2755ca309bdd", "cell_type": "markdown", - "source": "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/BAMresearch/NFDI4IngScientificWorkflowRequirements", + "source": "/BAMresearch/NFDI4IngScientificWorkflowRequirements\n\nThis notebook defines the NFDI4Ing file-based workflow benchmark with [`jobflow`](https://github.com/materialsproject/jobflow) and then loads the resulting `workflow.json` into `aiida`, `pyiron_base` and `pyiron_workflow`. Every stage of the pipeline shells out to an external command-line tool (`gmsh`, `meshio`, a FEniCS-based Poisson solver, ParaView's `pvbatch`, `tectonic`) in its own conda environment and passes file paths between stages rather than in-memory Python objects — see [`workflow.py`](workflow.py).", "metadata": {} }, { "id": "856b2ba2-93d5-4516-93e1-a1eac49c48f2", "cell_type": "markdown", - "source": "## Define workflow with jobflow", + "source": "## Define workflow with jobflow\n\nThe `job` decorator turns each of the six pipeline functions into a jobflow `Job` that delays execution until the `Flow` is run. Because `convert_to_xdmf` and `poisson` return a `dict`, their outputs are accessed downstream as `.output.` (e.g. `meshio_output_dict.output.xdmf_file`) to wire a single dictionary entry into the next job's input.", "metadata": {} }, { @@ -89,6 +89,12 @@ "outputs": [], "execution_count": 5 }, + { + "cell_type": "markdown", + "id": "1cd3d59e", + "source": "With `source_directory` and `domain_size` fixed, the calls below chain the six jobs together: mesh generation, XDMF conversion, the Poisson solve, line-plot postprocessing, LaTeX macro substitution and paper compilation. Each job runs in its own subdirectory and conda environment (see `source/envs/`), so what actually flows from one job to the next are file paths, not in-memory results.", + "metadata": {} + }, { "id": "8d911f98-3b80-457f-a0f4-3cb37ebf1691", "cell_type": "code", @@ -174,6 +180,12 @@ "outputs": [], "execution_count": 13 }, + { + "cell_type": "markdown", + "id": "71c4ff70", + "source": "The six jobs are collected into a `Flow`, which `write_workflow_json` then serializes to the PWD `workflow.json` format for the other engines to load.", + "metadata": {} + }, { "id": "fb23ad9c-76fd-4c0b-b546-e305d6c49796", "cell_type": "code", @@ -197,7 +209,7 @@ { "id": "11a829e2-face-469f-b343-2c95763b1f13", "cell_type": "markdown", - "source": "## Load Workflow with aiida", + "source": "## Load Workflow with aiida\n\n`load_workflow_json` rebuilds the pipeline as an `aiida-workgraph` `WorkGraph` from `jobflow_nfdi.json`, after connecting to a local AiiDA profile with `load_profile()`. `wg.run()` re-executes the whole pipeline under `aiida`.", "metadata": {} }, { @@ -272,7 +284,7 @@ { "id": "397b16a2-e1ec-4eec-8562-1c84f585c347", "cell_type": "markdown", - "source": "## Load Workflow with pyiron_base", + "source": "## Load Workflow with pyiron_base\n\nThe same JSON is loaded into a list of delayed pyiron jobs; `.draw()` visualizes their dependencies and `.pull()` triggers execution, returning the final `paper.pdf` path.", "metadata": {} }, { @@ -331,7 +343,7 @@ { "metadata": {}, "cell_type": "markdown", - "source": "## Load Workflow with pyiron_workflow", + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the graph is loaded into a `pyiron_workflow` `Workflow`. `.draw()` shows the node graph and `.run()` executes it end to end.", "id": "8e73fa07cebd7b72" }, { @@ -367,4 +379,4 @@ "id": "447506cdca5080e6" } ] -} +} \ No newline at end of file diff --git a/example_workflows/nfdi/pyiron_base.ipynb b/example_workflows/nfdi/pyiron_base.ipynb index f4feda6..164b5af 100644 --- a/example_workflows/nfdi/pyiron_base.ipynb +++ b/example_workflows/nfdi/pyiron_base.ipynb @@ -24,13 +24,13 @@ { "id": "106ded66-d202-46ac-82b0-2755ca309bdd", "cell_type": "markdown", - "source": "# pyiron\n\nhttps://github.com/BAMresearch/NFDI4IngScientificWorkflowRequirements", + "source": "# pyiron\n\nhttps://github.com/BAMresearch/NFDI4IngScientificWorkflowRequirements\n\nThis notebook defines the NFDI4Ing file-based workflow benchmark with [`pyiron_base`](https://github.com/pyiron/pyiron_base) and then loads the resulting `workflow.json` into `aiida`, `jobflow` and `pyiron_workflow`. Every stage of the pipeline shells out to an external command-line tool (`gmsh`, `meshio`, a FEniCS-based Poisson solver, ParaView's `pvbatch`, `tectonic`) in its own conda environment and passes file paths between stages rather than in-memory Python objects — see [`workflow.py`](workflow.py).", "metadata": {} }, { "id": "91dd48ea-aa7e-4937-a68e-59fc5017eb1e", "cell_type": "markdown", - "source": "## Define workflow with pyiron_base", + "source": "## Define workflow with pyiron_base\n\nThe `job` decorator turns a plain Python function into a delayed pyiron job. Since `convert_to_xdmf` and `poisson` return a `dict`, they are decorated with `output_key_lst=[...]` so each dictionary key becomes a separately addressable output (e.g. `meshio_output_dict.output.xdmf_file`) that can be wired into the next job's input.", "metadata": {} }, { @@ -93,6 +93,12 @@ "outputs": [], "execution_count": 6 }, + { + "cell_type": "markdown", + "id": "b80c6588", + "source": "The calls below chain the six jobs together: mesh generation, XDMF conversion, the Poisson solve, line-plot postprocessing, LaTeX macro substitution and paper compilation. Since each stage runs an external tool in its own conda environment (see `source/envs/`), what is actually passed between jobs are file paths, not in-memory Python values.", + "metadata": {} + }, { "id": "71d411b6-cbec-489e-99e3-ba71680bcb5b", "cell_type": "code", @@ -158,6 +164,12 @@ "outputs": [], "execution_count": 12 }, + { + "cell_type": "markdown", + "id": "4d1ce825", + "source": "Because pyiron builds a delayed call graph rather than an explicit `Flow`/`WorkGraph` object, `write_workflow_json` only needs the final delayed object (`paper_output`) — it walks the graph backwards to discover every upstream job.", + "metadata": {} + }, { "id": "63f29646-3846-4a97-a033-20e9df0ac214", "cell_type": "code", @@ -181,7 +193,7 @@ { "id": "d789971e-8f41-45fa-832a-11fd72dea96e", "cell_type": "markdown", - "source": "## Load Workflow with aiida", + "source": "## Load Workflow with aiida\n\n`load_workflow_json` rebuilds the pipeline as an `aiida-workgraph` `WorkGraph` from `pyiron_base_nfdi.json`, after connecting to a local AiiDA profile with `load_profile()`. `wg.run()` re-executes the whole pipeline under `aiida`.", "metadata": {} }, { @@ -256,7 +268,7 @@ { "id": "55dc8d12-dfe6-4465-a368-b7e590ae6800", "cell_type": "markdown", - "source": "## Load Workflow with jobflow", + "source": "## Load Workflow with jobflow\n\nThe same JSON is reconstructed as a jobflow `Flow` and executed locally with `run_locally`, independently of the `pyiron_base` objects used to define it.", "metadata": {} }, { @@ -316,7 +328,7 @@ { "metadata": {}, "cell_type": "markdown", - "source": "## Load Workflow with pyiron_workflow", + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the graph is loaded into a `pyiron_workflow` `Workflow`. `.draw()` shows the node graph and `.run()` executes it end to end.", "id": "385acbf585763632" }, { @@ -352,4 +364,4 @@ "id": "fff9513c4a127a96" } ] -} +} \ No newline at end of file diff --git a/example_workflows/nfdi/pyiron_workflow.ipynb b/example_workflows/nfdi/pyiron_workflow.ipynb index db1df09..887894b 100644 --- a/example_workflows/nfdi/pyiron_workflow.ipynb +++ b/example_workflows/nfdi/pyiron_workflow.ipynb @@ -1 +1,428 @@ -{"metadata":{"kernelspec":{"display_name":"Python 3 (ipykernel)","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.12.8","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":5,"nbformat":4,"cells":[{"id":"106ded66-d202-46ac-82b0-2755ca309bdd","cell_type":"markdown","source":"# pyiron\n\nhttps://github.com/BAMresearch/NFDI4IngScientificWorkflowRequirements","metadata":{}},{"id":"91dd48ea-aa7e-4937-a68e-59fc5017eb1e","cell_type":"markdown","source":"## Define workflow with pyiron_workflow","metadata":{}},{"id":"2c9622f5-ab7e-460e-b8e4-8d21413eda77","cell_type":"code","source":"import os","metadata":{"trusted":true},"outputs":[],"execution_count":1},{"id":"d265bb5aa6af79d6","cell_type":"code","source":"from workflow import (\n generate_mesh as _generate_mesh, \n convert_to_xdmf as _convert_to_xdmf,\n poisson as _poisson,\n plot_over_line as _plot_over_line,\n substitute_macros as _substitute_macros,\n compile_paper as _compile_paper,\n)","metadata":{"trusted":true},"outputs":[],"execution_count":2},{"id":"2dced28725813fc1","cell_type":"code","source":"from pyiron_workflow import Workflow, to_function_node\n\nfrom python_workflow_definition.pyiron_workflow import write_workflow_json","metadata":{"trusted":true},"outputs":[],"execution_count":3},{"id":"549ecf27-88ef-4e77-8bd4-b616cfdda2e4","cell_type":"code","source":"generate_mesh = to_function_node(\"generate_mesh\", _generate_mesh, \"generate_mesh\")\nconvert_to_xdmf = to_function_node(\"convert_to_xdmf\", _convert_to_xdmf, \"convert_to_xdmf\")\npoisson = to_function_node(\"poisson\", _poisson, \"poisson\")\nplot_over_line = to_function_node(\"plot_over_line\", _plot_over_line, \"plot_over_line\")\nsubstitute_macros = to_function_node(\"substitute_macros\", _substitute_macros, \"substitute_macros\")\ncompile_paper = to_function_node(\"compile_paper\", _compile_paper, \"compile_paper\")","metadata":{"trusted":true},"outputs":[],"execution_count":4},{"id":"3a75428e-18c7-49cf-8256-23cff58b9d6e","cell_type":"code","source":"wf = Workflow(\"my_workflow\")","metadata":{"trusted":true},"outputs":[],"execution_count":5},{"id":"8d911f98-3b80-457f-a0f4-3cb37ebf1691","cell_type":"code","source":"wf.domain_size = 2.0","metadata":{"trusted":true},"outputs":[],"execution_count":6},{"id":"c6ea980b-6761-4191-8407-7b1f78a4c3ea","cell_type":"code","source":"wf.source_directory = os.path.abspath(os.path.join(os.curdir, \"source\"))","metadata":{"trusted":true},"outputs":[],"execution_count":7},{"id":"71d411b6-cbec-489e-99e3-ba71680bcb5b","cell_type":"code","source":"wf.gmsh_output_file = generate_mesh(\n domain_size=wf.domain_size,\n source_directory=wf.source_directory,\n)","metadata":{"tags":[],"trusted":true},"outputs":[],"execution_count":8},{"id":"1d0d9804-f250-48b3-a5d0-a546d520f79b","cell_type":"code","source":"wf.meshio_output_dict = convert_to_xdmf(\n gmsh_output_file=wf.gmsh_output_file,\n)","metadata":{"tags":[],"trusted":true},"outputs":[],"execution_count":9},{"id":"7b69bcff-e2b1-4d4a-b62c-6a1c86eeb590","cell_type":"code","source":"wf.poisson_dict = poisson(\n meshio_output_xdmf=wf.meshio_output_dict[\"xdmf_file\"], \n meshio_output_h5=wf.meshio_output_dict[\"h5_file\"],\n source_directory=wf.source_directory,\n)","metadata":{"tags":[],"trusted":true},"outputs":[],"execution_count":10},{"id":"3c4a29b0-eb1e-490a-8be0-e03cfff15e0a","cell_type":"code","source":"wf.pvbatch_output_file = plot_over_line(\n poisson_output_pvd_file=wf.poisson_dict[\"pvd_file\"], \n poisson_output_vtu_file=wf.poisson_dict[\"vtu_file\"],\n source_directory=wf.source_directory,\n)","metadata":{"tags":[],"trusted":true},"outputs":[],"execution_count":11},{"id":"a0a4c233-322d-4723-9627-62ca2487bfa9","cell_type":"code","source":"wf.macros_tex_file = substitute_macros( \n pvbatch_output_file=wf.pvbatch_output_file, \n ndofs=wf.poisson_dict[\"numdofs\"], \n domain_size=wf.domain_size,\n source_directory=wf.source_directory,\n)","metadata":{"tags":[],"trusted":true},"outputs":[],"execution_count":12},{"id":"c281408f-e63d-4380-a7e6-c595d49fbb8f","cell_type":"code","source":"wf.paper_output = compile_paper(\n macros_tex=wf.macros_tex_file, \n plot_file=wf.pvbatch_output_file,\n source_directory=wf.source_directory,\n)","metadata":{"trusted":true},"outputs":[],"execution_count":13},{"id":"db4f1e15-3710-4e24-abc2-0fef417043e8","cell_type":"code","source":"wf.draw(size=(10,10))","metadata":{"trusted":true},"outputs":[{"execution_count":14,"output_type":"execute_result","data":{"image/svg+xml":"\n\n\n\n\n\nclustermy_workflow\n\nmy_workflow: Workflow\n\nclustermy_workflowInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowgmsh_output_file\n\n\n\n\n\n\n\ngmsh_output_file: generate_mesh\n\n\nclustermy_workflowgmsh_output_fileInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowgmsh_output_fileOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowmeshio_output_dict\n\n\n\n\n\n\n\nmeshio_output_dict: convert_to_xdmf\n\n\nclustermy_workflowmeshio_output_dictInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366\n\n\n\n\n\n\n\ninjected_GetItem_m7197572853956972366: GetItem\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366Inputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366OutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092\n\n\n\n\n\n\n\ninjected_GetItem_5401042856615209092: GetItem\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092Inputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092OutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowpoisson_dict\n\n\n\n\n\n\n\npoisson_dict: poisson\n\n\nclustermy_workflowpoisson_dictInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowpoisson_dictOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609\n\n\n\n\n\n\n\ninjected_GetItem_5839915180513303609: GetItem\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609Inputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609OutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100\n\n\n\n\n\n\n\ninjected_GetItem_m3497348724979863100: GetItem\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100Inputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100OutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowpvbatch_output_file\n\n\n\n\n\n\n\npvbatch_output_file: plot_over_line\n\n\nclustermy_workflowpvbatch_output_fileInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458\n\n\n\n\n\n\n\ninjected_GetItem_m6105838930489235458: GetItem\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458Inputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458OutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowmacros_tex_file\n\n\n\n\n\n\n\nmacros_tex_file: substitute_macros\n\n\nclustermy_workflowmacros_tex_fileInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowmacros_tex_fileOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowpaper_output\n\n\n\n\n\n\n\npaper_output: compile_paper\n\n\nclustermy_workflowpaper_outputInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowpaper_outputOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\n\nclustermy_workflowInputsrun\n\nrun\n\n\n\nclustermy_workflowOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowInputsgmsh_output_file__domain_size\n\ngmsh_output_file__domain_size: float\n\n\n\nclustermy_workflowgmsh_output_fileInputsdomain_size\n\ndomain_size: float\n\n\n\nclustermy_workflowInputsgmsh_output_file__domain_size->clustermy_workflowgmsh_output_fileInputsdomain_size\n\n\n\n\n\n\nclustermy_workflowInputsgmsh_output_file__source_directory\n\ngmsh_output_file__source_directory: str\n\n\n\nclustermy_workflowgmsh_output_fileInputssource_directory\n\nsource_directory: str\n\n\n\nclustermy_workflowInputsgmsh_output_file__source_directory->clustermy_workflowgmsh_output_fileInputssource_directory\n\n\n\n\n\n\nclustermy_workflowInputsinjected_GetItem_m7197572853956972366__item\n\ninjected_GetItem_m7197572853956972366__item\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366Inputsitem\n\nitem\n\n\n\nclustermy_workflowInputsinjected_GetItem_m7197572853956972366__item->clustermy_workflowinjected_GetItem_m7197572853956972366Inputsitem\n\n\n\n\n\n\nclustermy_workflowInputsinjected_GetItem_5401042856615209092__item\n\ninjected_GetItem_5401042856615209092__item\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092Inputsitem\n\nitem\n\n\n\nclustermy_workflowInputsinjected_GetItem_5401042856615209092__item->clustermy_workflowinjected_GetItem_5401042856615209092Inputsitem\n\n\n\n\n\n\nclustermy_workflowInputspoisson_dict__source_directory\n\npoisson_dict__source_directory: str\n\n\n\nclustermy_workflowpoisson_dictInputssource_directory\n\nsource_directory: str\n\n\n\nclustermy_workflowInputspoisson_dict__source_directory->clustermy_workflowpoisson_dictInputssource_directory\n\n\n\n\n\n\nclustermy_workflowInputsinjected_GetItem_5839915180513303609__item\n\ninjected_GetItem_5839915180513303609__item\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609Inputsitem\n\nitem\n\n\n\nclustermy_workflowInputsinjected_GetItem_5839915180513303609__item->clustermy_workflowinjected_GetItem_5839915180513303609Inputsitem\n\n\n\n\n\n\nclustermy_workflowInputsinjected_GetItem_m3497348724979863100__item\n\ninjected_GetItem_m3497348724979863100__item\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100Inputsitem\n\nitem\n\n\n\nclustermy_workflowInputsinjected_GetItem_m3497348724979863100__item->clustermy_workflowinjected_GetItem_m3497348724979863100Inputsitem\n\n\n\n\n\n\nclustermy_workflowInputspvbatch_output_file__source_directory\n\npvbatch_output_file__source_directory: str\n\n\n\nclustermy_workflowpvbatch_output_fileInputssource_directory\n\nsource_directory: str\n\n\n\nclustermy_workflowInputspvbatch_output_file__source_directory->clustermy_workflowpvbatch_output_fileInputssource_directory\n\n\n\n\n\n\nclustermy_workflowInputsinjected_GetItem_m6105838930489235458__item\n\ninjected_GetItem_m6105838930489235458__item\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458Inputsitem\n\nitem\n\n\n\nclustermy_workflowInputsinjected_GetItem_m6105838930489235458__item->clustermy_workflowinjected_GetItem_m6105838930489235458Inputsitem\n\n\n\n\n\n\nclustermy_workflowInputsmacros_tex_file__domain_size\n\nmacros_tex_file__domain_size: float\n\n\n\nclustermy_workflowmacros_tex_fileInputsdomain_size\n\ndomain_size: float\n\n\n\nclustermy_workflowInputsmacros_tex_file__domain_size->clustermy_workflowmacros_tex_fileInputsdomain_size\n\n\n\n\n\n\nclustermy_workflowInputsmacros_tex_file__source_directory\n\nmacros_tex_file__source_directory: str\n\n\n\nclustermy_workflowmacros_tex_fileInputssource_directory\n\nsource_directory: str\n\n\n\nclustermy_workflowInputsmacros_tex_file__source_directory->clustermy_workflowmacros_tex_fileInputssource_directory\n\n\n\n\n\n\nclustermy_workflowInputspaper_output__source_directory\n\npaper_output__source_directory: str\n\n\n\nclustermy_workflowpaper_outputInputssource_directory\n\nsource_directory: str\n\n\n\nclustermy_workflowInputspaper_output__source_directory->clustermy_workflowpaper_outputInputssource_directory\n\n\n\n\n\n\nclustermy_workflowOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowOutputsWithInjectionpaper_output__compile_paper\n\npaper_output__compile_paper: str\n\n\n\nclustermy_workflowgmsh_output_fileInputsrun\n\nrun\n\n\n\nclustermy_workflowgmsh_output_fileOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowgmsh_output_fileInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowgmsh_output_fileOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowgmsh_output_fileOutputsWithInjectiongenerate_mesh\n\ngenerate_mesh: str\n\n\n\nclustermy_workflowmeshio_output_dictInputsgmsh_output_file\n\ngmsh_output_file: str\n\n\n\nclustermy_workflowgmsh_output_fileOutputsWithInjectiongenerate_mesh->clustermy_workflowmeshio_output_dictInputsgmsh_output_file\n\n\n\n\n\n\nclustermy_workflowmeshio_output_dictInputsrun\n\nrun\n\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowmeshio_output_dictInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjectionconvert_to_xdmf\n\nconvert_to_xdmf: dict\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366Inputsobj\n\nobj\n\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjectionconvert_to_xdmf->clustermy_workflowinjected_GetItem_m7197572853956972366Inputsobj\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092Inputsobj\n\nobj\n\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjectionconvert_to_xdmf->clustermy_workflowinjected_GetItem_5401042856615209092Inputsobj\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366Inputsrun\n\nrun\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366OutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366Inputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366OutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366OutputsWithInjectiongetitem\n\ngetitem\n\n\n\nclustermy_workflowpoisson_dictInputsmeshio_output_xdmf\n\nmeshio_output_xdmf: str\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366OutputsWithInjectiongetitem->clustermy_workflowpoisson_dictInputsmeshio_output_xdmf\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092Inputsrun\n\nrun\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092OutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092Inputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092OutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092OutputsWithInjectiongetitem\n\ngetitem\n\n\n\nclustermy_workflowpoisson_dictInputsmeshio_output_h5\n\nmeshio_output_h5: str\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092OutputsWithInjectiongetitem->clustermy_workflowpoisson_dictInputsmeshio_output_h5\n\n\n\n\n\n\nclustermy_workflowpoisson_dictInputsrun\n\nrun\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowpoisson_dictInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionpoisson\n\npoisson: dict\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609Inputsobj\n\nobj\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionpoisson->clustermy_workflowinjected_GetItem_5839915180513303609Inputsobj\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100Inputsobj\n\nobj\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionpoisson->clustermy_workflowinjected_GetItem_m3497348724979863100Inputsobj\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458Inputsobj\n\nobj\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionpoisson->clustermy_workflowinjected_GetItem_m6105838930489235458Inputsobj\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609Inputsrun\n\nrun\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609OutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609Inputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609OutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609OutputsWithInjectiongetitem\n\ngetitem\n\n\n\nclustermy_workflowpvbatch_output_fileInputspoisson_output_pvd_file\n\npoisson_output_pvd_file: str\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609OutputsWithInjectiongetitem->clustermy_workflowpvbatch_output_fileInputspoisson_output_pvd_file\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100Inputsrun\n\nrun\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100OutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100Inputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100OutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100OutputsWithInjectiongetitem\n\ngetitem\n\n\n\nclustermy_workflowpvbatch_output_fileInputspoisson_output_vtu_file\n\npoisson_output_vtu_file: str\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100OutputsWithInjectiongetitem->clustermy_workflowpvbatch_output_fileInputspoisson_output_vtu_file\n\n\n\n\n\n\nclustermy_workflowpvbatch_output_fileInputsrun\n\nrun\n\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowpvbatch_output_fileInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjectionplot_over_line\n\nplot_over_line: str\n\n\n\nclustermy_workflowmacros_tex_fileInputspvbatch_output_file\n\npvbatch_output_file: str\n\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjectionplot_over_line->clustermy_workflowmacros_tex_fileInputspvbatch_output_file\n\n\n\n\n\n\nclustermy_workflowpaper_outputInputsplot_file\n\nplot_file: str\n\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjectionplot_over_line->clustermy_workflowpaper_outputInputsplot_file\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458Inputsrun\n\nrun\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458OutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458Inputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458OutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458OutputsWithInjectiongetitem\n\ngetitem\n\n\n\nclustermy_workflowmacros_tex_fileInputsndofs\n\nndofs: int\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458OutputsWithInjectiongetitem->clustermy_workflowmacros_tex_fileInputsndofs\n\n\n\n\n\n\nclustermy_workflowmacros_tex_fileInputsrun\n\nrun\n\n\n\nclustermy_workflowmacros_tex_fileOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowmacros_tex_fileInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowmacros_tex_fileOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowmacros_tex_fileOutputsWithInjectionsubstitute_macros\n\nsubstitute_macros: str\n\n\n\nclustermy_workflowpaper_outputInputsmacros_tex\n\nmacros_tex: str\n\n\n\nclustermy_workflowmacros_tex_fileOutputsWithInjectionsubstitute_macros->clustermy_workflowpaper_outputInputsmacros_tex\n\n\n\n\n\n\nclustermy_workflowpaper_outputInputsrun\n\nrun\n\n\n\nclustermy_workflowpaper_outputOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowpaper_outputInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowpaper_outputOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowpaper_outputOutputsWithInjectioncompile_paper\n\ncompile_paper: str\n\n\n\nclustermy_workflowpaper_outputOutputsWithInjectioncompile_paper->clustermy_workflowOutputsWithInjectionpaper_output__compile_paper\n\n\n\n\n\n\n","text/plain":""},"metadata":{}}],"execution_count":14},{"id":"63f29646-3846-4a97-a033-20e9df0ac214","cell_type":"code","source":"workflow_json_filename = \"pyiron_workflow_nfdi.json\"","metadata":{"trusted":true},"outputs":[],"execution_count":15},{"id":"f62111ba-9271-4987-9c7e-3b1c9f9eae7a","cell_type":"code","source":"write_workflow_json(graph_as_dict=wf.graph_as_dict, file_name=workflow_json_filename)","metadata":{"trusted":true},"outputs":[],"execution_count":16},{"id":"d789971e-8f41-45fa-832a-11fd72dea96e","cell_type":"markdown","source":"## Load Workflow with aiida","metadata":{}},{"id":"a6e85e89-5d7a-40eb-809c-ac44974e3fd7","cell_type":"code","source":"from aiida import load_profile\n\nload_profile()","metadata":{"trusted":true},"outputs":[{"execution_count":17,"output_type":"execute_result","data":{"text/plain":"Profile"},"metadata":{}}],"execution_count":17},{"id":"3de84fb7-b01b-4541-868a-92e881eb6e77","cell_type":"code","source":"from python_workflow_definition.aiida import load_workflow_json","metadata":{"trusted":true},"outputs":[],"execution_count":18},{"id":"b33f5528-10cd-47c8-8723-622902978859","cell_type":"code","source":"wg = load_workflow_json(file_name=workflow_json_filename)\nwg","metadata":{"trusted":true},"outputs":[{"execution_count":19,"output_type":"execute_result","data":{"text/plain":"NodeGraphWidget(settings={'minimap': True}, style={'width': '90%', 'height': '600px'}, value={'name': 'WorkGra…","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":1,"model_id":"6641ced7603742b8b28900587d764b7c"}},"metadata":{}}],"execution_count":19},{"id":"15282ca1-d339-40e7-ad68-8a7613ed08da","cell_type":"code","source":"wg.run()","metadata":{"trusted":true},"outputs":[{"name":"stderr","output_type":"stream","text":"05/24/2025 09:30:18 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: generate_mesh1\n05/24/2025 09:30:20 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: generate_mesh1, type: PyFunction, finished.\n05/24/2025 09:30:20 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: convert_to_xdmf2\n05/24/2025 09:30:22 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: convert_to_xdmf2, type: PyFunction, finished.\n05/24/2025 09:30:22 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: poisson3\n05/24/2025 09:30:30 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: poisson3, type: PyFunction, finished.\n05/24/2025 09:30:30 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: plot_over_line4\n05/24/2025 09:30:33 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: plot_over_line4, type: PyFunction, finished.\n05/24/2025 09:30:33 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: substitute_macros5\n05/24/2025 09:30:34 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: substitute_macros5, type: PyFunction, finished.\n05/24/2025 09:30:34 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: compile_paper6\n05/24/2025 09:30:50 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: compile_paper6, type: PyFunction, finished.\n05/24/2025 09:30:50 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: \n05/24/2025 09:30:50 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|finalize]: Finalize workgraph.\n"}],"execution_count":20},{"id":"55dc8d12-dfe6-4465-a368-b7e590ae6800","cell_type":"markdown","source":"## Load Workflow with jobflow","metadata":{}},{"id":"dff46eb8-e0e7-49bb-8c40-0db2df133124","cell_type":"code","source":"from python_workflow_definition.jobflow import load_workflow_json","metadata":{"trusted":true},"outputs":[],"execution_count":21},{"id":"6a189459-84e4-4738-ada1-37ee8c65b2ab","cell_type":"code","source":"from jobflow.managers.local import run_locally","metadata":{"trusted":true},"outputs":[],"execution_count":22},{"id":"6e7f3614-c971-4e2d-83f0-96f0d0fc04de","cell_type":"code","source":"flow = load_workflow_json(file_name=workflow_json_filename)","metadata":{"trusted":true},"outputs":[],"execution_count":23},{"id":"2d87ed45-f5d9-403f-a03a-26be4a47a3ef","cell_type":"code","source":"result = run_locally(flow)\nresult","metadata":{"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":"2025-05-24 09:31:15,165 INFO Started executing jobs locally\n2025-05-24 09:31:15,364 INFO Starting job - generate_mesh (4cc4e3d9-39aa-4c20-a047-a029b6812d61)\n2025-05-24 09:31:16,480 INFO Finished job - generate_mesh (4cc4e3d9-39aa-4c20-a047-a029b6812d61)\n2025-05-24 09:31:16,481 INFO Starting job - convert_to_xdmf (8527c563-c0b1-4ae5-a6c3-00ae25ad2508)\n2025-05-24 09:31:17,834 INFO Finished job - convert_to_xdmf (8527c563-c0b1-4ae5-a6c3-00ae25ad2508)\n2025-05-24 09:31:17,836 INFO Starting job - poisson (58b31d24-51f9-4b9a-ac3e-88a7f01a09ad)\n2025-05-24 09:31:20,318 INFO Finished job - poisson (58b31d24-51f9-4b9a-ac3e-88a7f01a09ad)\n2025-05-24 09:31:20,318 INFO Starting job - plot_over_line (b095dc38-f858-4c34-876c-df8f8b851a5f)\n2025-05-24 09:31:21,717 INFO Finished job - plot_over_line (b095dc38-f858-4c34-876c-df8f8b851a5f)\n2025-05-24 09:31:21,718 INFO Starting job - substitute_macros (656e820d-20d5-4665-bb8a-43807ed6b2e9)\n2025-05-24 09:31:22,542 INFO Finished job - substitute_macros (656e820d-20d5-4665-bb8a-43807ed6b2e9)\n2025-05-24 09:31:22,543 INFO Starting job - compile_paper (52de4e6d-7574-401b-bcdb-ae6c298cc6b9)\n2025-05-24 09:31:24,556 INFO Finished job - compile_paper (52de4e6d-7574-401b-bcdb-ae6c298cc6b9)\n2025-05-24 09:31:24,557 INFO Finished executing jobs locally\n"},{"execution_count":24,"output_type":"execute_result","data":{"text/plain":"{'4cc4e3d9-39aa-4c20-a047-a029b6812d61': {1: Response(output='/home/jovyan/example_workflows/nfdi/preprocessing/square.msh', detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))},\n '8527c563-c0b1-4ae5-a6c3-00ae25ad2508': {1: Response(output={'xdmf_file': '/home/jovyan/example_workflows/nfdi/preprocessing/square.xdmf', 'h5_file': '/home/jovyan/example_workflows/nfdi/preprocessing/square.h5'}, detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))},\n '58b31d24-51f9-4b9a-ac3e-88a7f01a09ad': {1: Response(output={'numdofs': 357, 'pvd_file': '/home/jovyan/example_workflows/nfdi/processing/poisson.pvd', 'vtu_file': '/home/jovyan/example_workflows/nfdi/processing/poisson000000.vtu'}, detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))},\n 'b095dc38-f858-4c34-876c-df8f8b851a5f': {1: Response(output='/home/jovyan/example_workflows/nfdi/postprocessing/plotoverline.csv', detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))},\n '656e820d-20d5-4665-bb8a-43807ed6b2e9': {1: Response(output='/home/jovyan/example_workflows/nfdi/postprocessing/macros.tex', detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))},\n '52de4e6d-7574-401b-bcdb-ae6c298cc6b9': {1: Response(output='/home/jovyan/example_workflows/nfdi/postprocessing/paper.pdf', detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))}}"},"metadata":{}}],"execution_count":24},{"id":"ebd3248b-ff75-40bf-a488-b5e338342671","cell_type":"markdown","source":"## Load Workflow with pyiron_base","metadata":{}},{"id":"c7e1047f-0d42-4777-911d-ab405396401a","cell_type":"code","source":"from python_workflow_definition.pyiron_base import load_workflow_json","metadata":{"trusted":true},"outputs":[],"execution_count":25},{"id":"6aeba274-10d3-4e79-8ee4-4d870f163939","cell_type":"code","source":"delayed_object_lst = load_workflow_json(file_name=workflow_json_filename)\ndelayed_object_lst[-1].draw()","metadata":{"trusted":true},"outputs":[{"output_type":"display_data","data":{"text/plain":"","image/svg+xml":"\n\n\n\n\ncreate_function_job_ba0bf2668d8f7b42ff717c2cb79b5920\n\ncreate_function_job=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d9280>\n\n\n\nplot_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\nplot_file=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8da0>\n\n\n\nplot_file_5007b1d67345934b2ffcdeaaad6ea2ab->create_function_job_ba0bf2668d8f7b42ff717c2cb79b5920\n\n\n\n\n\npoisson_output_vtu_file_51141ca0ec0a07922807865d8a067793\n\npoisson_output_vtu_file=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8a40>\n\n\n\npoisson_output_vtu_file_51141ca0ec0a07922807865d8a067793->plot_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\npvbatch_output_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\npvbatch_output_file=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8da0>\n\n\n\npoisson_output_vtu_file_51141ca0ec0a07922807865d8a067793->pvbatch_output_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\nmacros_tex_02d82cedf90ad323f7eef57ddf383160\n\nmacros_tex=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d9070>\n\n\n\npvbatch_output_file_5007b1d67345934b2ffcdeaaad6ea2ab->macros_tex_02d82cedf90ad323f7eef57ddf383160\n\n\n\n\n\nmeshio_output_h5_0af06cb14e9ddc7585647c44e4d4ef88\n\nmeshio_output_h5=<pyiron_base.project.delayed.DelayedObject object at 0x76dba3cb2600>\n\n\n\nmeshio_output_h5_0af06cb14e9ddc7585647c44e4d4ef88->poisson_output_vtu_file_51141ca0ec0a07922807865d8a067793\n\n\n\n\n\npoisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513\n\npoisson_output_pvd_file=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8a10>\n\n\n\nmeshio_output_h5_0af06cb14e9ddc7585647c44e4d4ef88->poisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513\n\n\n\n\n\nndofs_ac7427db28c0527371c0e9b27a57024b\n\nndofs=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8e30>\n\n\n\nmeshio_output_h5_0af06cb14e9ddc7585647c44e4d4ef88->ndofs_ac7427db28c0527371c0e9b27a57024b\n\n\n\n\n\npoisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513->plot_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\npoisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513->pvbatch_output_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\nndofs_ac7427db28c0527371c0e9b27a57024b->macros_tex_02d82cedf90ad323f7eef57ddf383160\n\n\n\n\n\ngmsh_output_file_6ef41eb684984daa8337ee65c2aa9f18\n\ngmsh_output_file=<pyiron_base.project.delayed.DelayedObject object at 0x76dba89319d0>\n\n\n\ngmsh_output_file_6ef41eb684984daa8337ee65c2aa9f18->meshio_output_h5_0af06cb14e9ddc7585647c44e4d4ef88\n\n\n\n\n\nmeshio_output_xdmf_c383e7d9f83d7cdc8ae04ebf9c264173\n\nmeshio_output_xdmf=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8710>\n\n\n\ngmsh_output_file_6ef41eb684984daa8337ee65c2aa9f18->meshio_output_xdmf_c383e7d9f83d7cdc8ae04ebf9c264173\n\n\n\n\n\nmeshio_output_xdmf_c383e7d9f83d7cdc8ae04ebf9c264173->poisson_output_vtu_file_51141ca0ec0a07922807865d8a067793\n\n\n\n\n\nmeshio_output_xdmf_c383e7d9f83d7cdc8ae04ebf9c264173->poisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513\n\n\n\n\n\nmeshio_output_xdmf_c383e7d9f83d7cdc8ae04ebf9c264173->ndofs_ac7427db28c0527371c0e9b27a57024b\n\n\n\n\n\ndomain_size_f12a7f1986b9dd058dfc666dbe230b20\n\ndomain_size=2.0\n\n\n\ndomain_size_f12a7f1986b9dd058dfc666dbe230b20->gmsh_output_file_6ef41eb684984daa8337ee65c2aa9f18\n\n\n\n\n\ndomain_size_f12a7f1986b9dd058dfc666dbe230b20->macros_tex_02d82cedf90ad323f7eef57ddf383160\n\n\n\n\n\nmacros_tex_02d82cedf90ad323f7eef57ddf383160->create_function_job_ba0bf2668d8f7b42ff717c2cb79b5920\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696\n\nsource_directory=/home/jovyan/example_workflows/nfdi/source\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->create_function_job_ba0bf2668d8f7b42ff717c2cb79b5920\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->plot_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->poisson_output_vtu_file_51141ca0ec0a07922807865d8a067793\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->pvbatch_output_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->poisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->ndofs_ac7427db28c0527371c0e9b27a57024b\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->gmsh_output_file_6ef41eb684984daa8337ee65c2aa9f18\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->macros_tex_02d82cedf90ad323f7eef57ddf383160\n\n\n\n\n"},"metadata":{}}],"execution_count":26},{"id":"ea92c7ad-afef-423c-b8fa-9d24c1267c73","cell_type":"code","source":"delayed_object_lst[-1].pull()","metadata":{"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":"The job generate_mesh_47725c16637f799ac042e47468005db3 was saved and received the ID: 1\nThe job convert_to_xdmf_d6a46eb9a4ec352aa996e783ec3c785f was saved and received the ID: 2\nThe job poisson_3c147fc86db87cf0c0f94bda333f8cd8 was saved and received the ID: 3\nThe job plot_over_line_ef50933291910dadcc8311924971e127 was saved and received the ID: 4\nThe job substitute_macros_63766eafd6b1980c7832dd8c9a97c96e was saved and received the ID: 5\nThe job compile_paper_128d1d58374953c00e95b8de62cbb10b was saved and received the ID: 6\n"},{"execution_count":27,"output_type":"execute_result","data":{"text/plain":"'/home/jovyan/example_workflows/nfdi/postprocessing/paper.pdf'"},"metadata":{}}],"execution_count":27},{"id":"32287743-e59a-467c-92fe-8586f199e36e","cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]} \ No newline at end of file +{ + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.12.8", + "mimetype": "text/x-python", + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "pygments_lexer": "ipython3", + "nbconvert_exporter": "python", + "file_extension": ".py" + } + }, + "nbformat_minor": 5, + "nbformat": 4, + "cells": [ + { + "id": "106ded66-d202-46ac-82b0-2755ca309bdd", + "cell_type": "markdown", + "source": "# pyiron\n\nhttps://github.com/BAMresearch/NFDI4IngScientificWorkflowRequirements\n\nThis notebook defines the NFDI4Ing file-based workflow benchmark with [`pyiron_workflow`](https://github.com/pyiron/pyiron_workflow) and then loads the resulting `workflow.json` into `aiida`, `jobflow` and `pyiron_base`. Every stage of the pipeline shells out to an external command-line tool (`gmsh`, `meshio`, a FEniCS-based Poisson solver, ParaView's `pvbatch`, `tectonic`) in its own conda environment and passes file paths between stages rather than in-memory Python objects — see [`workflow.py`](workflow.py).", + "metadata": {} + }, + { + "id": "91dd48ea-aa7e-4937-a68e-59fc5017eb1e", + "cell_type": "markdown", + "source": "## Define workflow with pyiron_workflow\n\n`to_function_node` wraps each plain Python function into a `pyiron_workflow` node; the three arguments are the node's registered class name, the function itself, and the name of its (single, unlabeled) output. Nodes are assembled into a graph by assigning them as attributes of a `Workflow` object (`wf. = node(...)`), and dictionary outputs such as `convert_to_xdmf`'s are indexed with `wf.meshio_output_dict[\"xdmf_file\"]` when wiring the next node's input.", + "metadata": {} + }, + { + "id": "2c9622f5-ab7e-460e-b8e4-8d21413eda77", + "cell_type": "code", + "source": "import os", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 1 + }, + { + "id": "d265bb5aa6af79d6", + "cell_type": "code", + "source": "from workflow import (\n generate_mesh as _generate_mesh, \n convert_to_xdmf as _convert_to_xdmf,\n poisson as _poisson,\n plot_over_line as _plot_over_line,\n substitute_macros as _substitute_macros,\n compile_paper as _compile_paper,\n)", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 2 + }, + { + "id": "2dced28725813fc1", + "cell_type": "code", + "source": "from pyiron_workflow import Workflow, to_function_node\n\nfrom python_workflow_definition.pyiron_workflow import write_workflow_json", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 3 + }, + { + "id": "549ecf27-88ef-4e77-8bd4-b616cfdda2e4", + "cell_type": "code", + "source": "generate_mesh = to_function_node(\"generate_mesh\", _generate_mesh, \"generate_mesh\")\nconvert_to_xdmf = to_function_node(\"convert_to_xdmf\", _convert_to_xdmf, \"convert_to_xdmf\")\npoisson = to_function_node(\"poisson\", _poisson, \"poisson\")\nplot_over_line = to_function_node(\"plot_over_line\", _plot_over_line, \"plot_over_line\")\nsubstitute_macros = to_function_node(\"substitute_macros\", _substitute_macros, \"substitute_macros\")\ncompile_paper = to_function_node(\"compile_paper\", _compile_paper, \"compile_paper\")", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 4 + }, + { + "id": "3a75428e-18c7-49cf-8256-23cff58b9d6e", + "cell_type": "code", + "source": "wf = Workflow(\"my_workflow\")", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 5 + }, + { + "id": "8d911f98-3b80-457f-a0f4-3cb37ebf1691", + "cell_type": "code", + "source": "wf.domain_size = 2.0", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 6 + }, + { + "id": "c6ea980b-6761-4191-8407-7b1f78a4c3ea", + "cell_type": "code", + "source": "wf.source_directory = os.path.abspath(os.path.join(os.curdir, \"source\"))", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 7 + }, + { + "cell_type": "markdown", + "id": "7c642a24", + "source": "The assignments below build the six-stage pipeline as attributes of `wf`: mesh generation, XDMF conversion, the Poisson solve, line-plot postprocessing, LaTeX macro substitution and paper compilation. Each stage runs an external tool in its own conda environment (see `source/envs/`), so the values connecting them are file paths on disk rather than shared Python objects.", + "metadata": {} + }, + { + "id": "71d411b6-cbec-489e-99e3-ba71680bcb5b", + "cell_type": "code", + "source": "wf.gmsh_output_file = generate_mesh(\n domain_size=wf.domain_size,\n source_directory=wf.source_directory,\n)", + "metadata": { + "tags": [], + "trusted": true + }, + "outputs": [], + "execution_count": 8 + }, + { + "id": "1d0d9804-f250-48b3-a5d0-a546d520f79b", + "cell_type": "code", + "source": "wf.meshio_output_dict = convert_to_xdmf(\n gmsh_output_file=wf.gmsh_output_file,\n)", + "metadata": { + "tags": [], + "trusted": true + }, + "outputs": [], + "execution_count": 9 + }, + { + "id": "7b69bcff-e2b1-4d4a-b62c-6a1c86eeb590", + "cell_type": "code", + "source": "wf.poisson_dict = poisson(\n meshio_output_xdmf=wf.meshio_output_dict[\"xdmf_file\"], \n meshio_output_h5=wf.meshio_output_dict[\"h5_file\"],\n source_directory=wf.source_directory,\n)", + "metadata": { + "tags": [], + "trusted": true + }, + "outputs": [], + "execution_count": 10 + }, + { + "id": "3c4a29b0-eb1e-490a-8be0-e03cfff15e0a", + "cell_type": "code", + "source": "wf.pvbatch_output_file = plot_over_line(\n poisson_output_pvd_file=wf.poisson_dict[\"pvd_file\"], \n poisson_output_vtu_file=wf.poisson_dict[\"vtu_file\"],\n source_directory=wf.source_directory,\n)", + "metadata": { + "tags": [], + "trusted": true + }, + "outputs": [], + "execution_count": 11 + }, + { + "id": "a0a4c233-322d-4723-9627-62ca2487bfa9", + "cell_type": "code", + "source": "wf.macros_tex_file = substitute_macros( \n pvbatch_output_file=wf.pvbatch_output_file, \n ndofs=wf.poisson_dict[\"numdofs\"], \n domain_size=wf.domain_size,\n source_directory=wf.source_directory,\n)", + "metadata": { + "tags": [], + "trusted": true + }, + "outputs": [], + "execution_count": 12 + }, + { + "id": "c281408f-e63d-4380-a7e6-c595d49fbb8f", + "cell_type": "code", + "source": "wf.paper_output = compile_paper(\n macros_tex=wf.macros_tex_file, \n plot_file=wf.pvbatch_output_file,\n source_directory=wf.source_directory,\n)", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 13 + }, + { + "cell_type": "markdown", + "id": "cb9a923d", + "source": "`wf.draw()` renders the assembled node graph; `wf.graph_as_dict` is then handed to `write_workflow_json`, which serializes it into the PWD `workflow.json` format for the other engines to load.", + "metadata": {} + }, + { + "id": "db4f1e15-3710-4e24-abc2-0fef417043e8", + "cell_type": "code", + "source": "wf.draw(size=(10,10))", + "metadata": { + "trusted": true + }, + "outputs": [ + { + "execution_count": 14, + "output_type": "execute_result", + "data": { + "image/svg+xml": "\n\n\n\n\n\nclustermy_workflow\n\nmy_workflow: Workflow\n\nclustermy_workflowInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowgmsh_output_file\n\n\n\n\n\n\n\ngmsh_output_file: generate_mesh\n\n\nclustermy_workflowgmsh_output_fileInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowgmsh_output_fileOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowmeshio_output_dict\n\n\n\n\n\n\n\nmeshio_output_dict: convert_to_xdmf\n\n\nclustermy_workflowmeshio_output_dictInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366\n\n\n\n\n\n\n\ninjected_GetItem_m7197572853956972366: GetItem\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366Inputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366OutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092\n\n\n\n\n\n\n\ninjected_GetItem_5401042856615209092: GetItem\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092Inputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092OutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowpoisson_dict\n\n\n\n\n\n\n\npoisson_dict: poisson\n\n\nclustermy_workflowpoisson_dictInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowpoisson_dictOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609\n\n\n\n\n\n\n\ninjected_GetItem_5839915180513303609: GetItem\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609Inputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609OutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100\n\n\n\n\n\n\n\ninjected_GetItem_m3497348724979863100: GetItem\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100Inputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100OutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowpvbatch_output_file\n\n\n\n\n\n\n\npvbatch_output_file: plot_over_line\n\n\nclustermy_workflowpvbatch_output_fileInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458\n\n\n\n\n\n\n\ninjected_GetItem_m6105838930489235458: GetItem\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458Inputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458OutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowmacros_tex_file\n\n\n\n\n\n\n\nmacros_tex_file: substitute_macros\n\n\nclustermy_workflowmacros_tex_fileInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowmacros_tex_fileOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\nclustermy_workflowpaper_output\n\n\n\n\n\n\n\npaper_output: compile_paper\n\n\nclustermy_workflowpaper_outputInputs\n\n\n\n\n\n\n\nInputs\n\n\nclustermy_workflowpaper_outputOutputsWithInjection\n\n\n\n\n\n\n\nOutputsWithInjection\n\n\n\nclustermy_workflowInputsrun\n\nrun\n\n\n\nclustermy_workflowOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowInputsgmsh_output_file__domain_size\n\ngmsh_output_file__domain_size: float\n\n\n\nclustermy_workflowgmsh_output_fileInputsdomain_size\n\ndomain_size: float\n\n\n\nclustermy_workflowInputsgmsh_output_file__domain_size->clustermy_workflowgmsh_output_fileInputsdomain_size\n\n\n\n\n\n\nclustermy_workflowInputsgmsh_output_file__source_directory\n\ngmsh_output_file__source_directory: str\n\n\n\nclustermy_workflowgmsh_output_fileInputssource_directory\n\nsource_directory: str\n\n\n\nclustermy_workflowInputsgmsh_output_file__source_directory->clustermy_workflowgmsh_output_fileInputssource_directory\n\n\n\n\n\n\nclustermy_workflowInputsinjected_GetItem_m7197572853956972366__item\n\ninjected_GetItem_m7197572853956972366__item\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366Inputsitem\n\nitem\n\n\n\nclustermy_workflowInputsinjected_GetItem_m7197572853956972366__item->clustermy_workflowinjected_GetItem_m7197572853956972366Inputsitem\n\n\n\n\n\n\nclustermy_workflowInputsinjected_GetItem_5401042856615209092__item\n\ninjected_GetItem_5401042856615209092__item\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092Inputsitem\n\nitem\n\n\n\nclustermy_workflowInputsinjected_GetItem_5401042856615209092__item->clustermy_workflowinjected_GetItem_5401042856615209092Inputsitem\n\n\n\n\n\n\nclustermy_workflowInputspoisson_dict__source_directory\n\npoisson_dict__source_directory: str\n\n\n\nclustermy_workflowpoisson_dictInputssource_directory\n\nsource_directory: str\n\n\n\nclustermy_workflowInputspoisson_dict__source_directory->clustermy_workflowpoisson_dictInputssource_directory\n\n\n\n\n\n\nclustermy_workflowInputsinjected_GetItem_5839915180513303609__item\n\ninjected_GetItem_5839915180513303609__item\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609Inputsitem\n\nitem\n\n\n\nclustermy_workflowInputsinjected_GetItem_5839915180513303609__item->clustermy_workflowinjected_GetItem_5839915180513303609Inputsitem\n\n\n\n\n\n\nclustermy_workflowInputsinjected_GetItem_m3497348724979863100__item\n\ninjected_GetItem_m3497348724979863100__item\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100Inputsitem\n\nitem\n\n\n\nclustermy_workflowInputsinjected_GetItem_m3497348724979863100__item->clustermy_workflowinjected_GetItem_m3497348724979863100Inputsitem\n\n\n\n\n\n\nclustermy_workflowInputspvbatch_output_file__source_directory\n\npvbatch_output_file__source_directory: str\n\n\n\nclustermy_workflowpvbatch_output_fileInputssource_directory\n\nsource_directory: str\n\n\n\nclustermy_workflowInputspvbatch_output_file__source_directory->clustermy_workflowpvbatch_output_fileInputssource_directory\n\n\n\n\n\n\nclustermy_workflowInputsinjected_GetItem_m6105838930489235458__item\n\ninjected_GetItem_m6105838930489235458__item\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458Inputsitem\n\nitem\n\n\n\nclustermy_workflowInputsinjected_GetItem_m6105838930489235458__item->clustermy_workflowinjected_GetItem_m6105838930489235458Inputsitem\n\n\n\n\n\n\nclustermy_workflowInputsmacros_tex_file__domain_size\n\nmacros_tex_file__domain_size: float\n\n\n\nclustermy_workflowmacros_tex_fileInputsdomain_size\n\ndomain_size: float\n\n\n\nclustermy_workflowInputsmacros_tex_file__domain_size->clustermy_workflowmacros_tex_fileInputsdomain_size\n\n\n\n\n\n\nclustermy_workflowInputsmacros_tex_file__source_directory\n\nmacros_tex_file__source_directory: str\n\n\n\nclustermy_workflowmacros_tex_fileInputssource_directory\n\nsource_directory: str\n\n\n\nclustermy_workflowInputsmacros_tex_file__source_directory->clustermy_workflowmacros_tex_fileInputssource_directory\n\n\n\n\n\n\nclustermy_workflowInputspaper_output__source_directory\n\npaper_output__source_directory: str\n\n\n\nclustermy_workflowpaper_outputInputssource_directory\n\nsource_directory: str\n\n\n\nclustermy_workflowInputspaper_output__source_directory->clustermy_workflowpaper_outputInputssource_directory\n\n\n\n\n\n\nclustermy_workflowOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowOutputsWithInjectionpaper_output__compile_paper\n\npaper_output__compile_paper: str\n\n\n\nclustermy_workflowgmsh_output_fileInputsrun\n\nrun\n\n\n\nclustermy_workflowgmsh_output_fileOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowgmsh_output_fileInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowgmsh_output_fileOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowgmsh_output_fileOutputsWithInjectiongenerate_mesh\n\ngenerate_mesh: str\n\n\n\nclustermy_workflowmeshio_output_dictInputsgmsh_output_file\n\ngmsh_output_file: str\n\n\n\nclustermy_workflowgmsh_output_fileOutputsWithInjectiongenerate_mesh->clustermy_workflowmeshio_output_dictInputsgmsh_output_file\n\n\n\n\n\n\nclustermy_workflowmeshio_output_dictInputsrun\n\nrun\n\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowmeshio_output_dictInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjectionconvert_to_xdmf\n\nconvert_to_xdmf: dict\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366Inputsobj\n\nobj\n\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjectionconvert_to_xdmf->clustermy_workflowinjected_GetItem_m7197572853956972366Inputsobj\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092Inputsobj\n\nobj\n\n\n\nclustermy_workflowmeshio_output_dictOutputsWithInjectionconvert_to_xdmf->clustermy_workflowinjected_GetItem_5401042856615209092Inputsobj\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366Inputsrun\n\nrun\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366OutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366Inputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366OutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366OutputsWithInjectiongetitem\n\ngetitem\n\n\n\nclustermy_workflowpoisson_dictInputsmeshio_output_xdmf\n\nmeshio_output_xdmf: str\n\n\n\nclustermy_workflowinjected_GetItem_m7197572853956972366OutputsWithInjectiongetitem->clustermy_workflowpoisson_dictInputsmeshio_output_xdmf\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092Inputsrun\n\nrun\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092OutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092Inputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092OutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092OutputsWithInjectiongetitem\n\ngetitem\n\n\n\nclustermy_workflowpoisson_dictInputsmeshio_output_h5\n\nmeshio_output_h5: str\n\n\n\nclustermy_workflowinjected_GetItem_5401042856615209092OutputsWithInjectiongetitem->clustermy_workflowpoisson_dictInputsmeshio_output_h5\n\n\n\n\n\n\nclustermy_workflowpoisson_dictInputsrun\n\nrun\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowpoisson_dictInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionpoisson\n\npoisson: dict\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609Inputsobj\n\nobj\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionpoisson->clustermy_workflowinjected_GetItem_5839915180513303609Inputsobj\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100Inputsobj\n\nobj\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionpoisson->clustermy_workflowinjected_GetItem_m3497348724979863100Inputsobj\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458Inputsobj\n\nobj\n\n\n\nclustermy_workflowpoisson_dictOutputsWithInjectionpoisson->clustermy_workflowinjected_GetItem_m6105838930489235458Inputsobj\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609Inputsrun\n\nrun\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609OutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609Inputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609OutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609OutputsWithInjectiongetitem\n\ngetitem\n\n\n\nclustermy_workflowpvbatch_output_fileInputspoisson_output_pvd_file\n\npoisson_output_pvd_file: str\n\n\n\nclustermy_workflowinjected_GetItem_5839915180513303609OutputsWithInjectiongetitem->clustermy_workflowpvbatch_output_fileInputspoisson_output_pvd_file\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100Inputsrun\n\nrun\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100OutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100Inputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100OutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100OutputsWithInjectiongetitem\n\ngetitem\n\n\n\nclustermy_workflowpvbatch_output_fileInputspoisson_output_vtu_file\n\npoisson_output_vtu_file: str\n\n\n\nclustermy_workflowinjected_GetItem_m3497348724979863100OutputsWithInjectiongetitem->clustermy_workflowpvbatch_output_fileInputspoisson_output_vtu_file\n\n\n\n\n\n\nclustermy_workflowpvbatch_output_fileInputsrun\n\nrun\n\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowpvbatch_output_fileInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjectionplot_over_line\n\nplot_over_line: str\n\n\n\nclustermy_workflowmacros_tex_fileInputspvbatch_output_file\n\npvbatch_output_file: str\n\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjectionplot_over_line->clustermy_workflowmacros_tex_fileInputspvbatch_output_file\n\n\n\n\n\n\nclustermy_workflowpaper_outputInputsplot_file\n\nplot_file: str\n\n\n\nclustermy_workflowpvbatch_output_fileOutputsWithInjectionplot_over_line->clustermy_workflowpaper_outputInputsplot_file\n\n\n\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458Inputsrun\n\nrun\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458OutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458Inputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458OutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458OutputsWithInjectiongetitem\n\ngetitem\n\n\n\nclustermy_workflowmacros_tex_fileInputsndofs\n\nndofs: int\n\n\n\nclustermy_workflowinjected_GetItem_m6105838930489235458OutputsWithInjectiongetitem->clustermy_workflowmacros_tex_fileInputsndofs\n\n\n\n\n\n\nclustermy_workflowmacros_tex_fileInputsrun\n\nrun\n\n\n\nclustermy_workflowmacros_tex_fileOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowmacros_tex_fileInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowmacros_tex_fileOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowmacros_tex_fileOutputsWithInjectionsubstitute_macros\n\nsubstitute_macros: str\n\n\n\nclustermy_workflowpaper_outputInputsmacros_tex\n\nmacros_tex: str\n\n\n\nclustermy_workflowmacros_tex_fileOutputsWithInjectionsubstitute_macros->clustermy_workflowpaper_outputInputsmacros_tex\n\n\n\n\n\n\nclustermy_workflowpaper_outputInputsrun\n\nrun\n\n\n\nclustermy_workflowpaper_outputOutputsWithInjectionran\n\nran\n\n\n\n\nclustermy_workflowpaper_outputInputsaccumulate_and_run\n\naccumulate_and_run\n\n\n\nclustermy_workflowpaper_outputOutputsWithInjectionfailed\n\nfailed\n\n\n\nclustermy_workflowpaper_outputOutputsWithInjectioncompile_paper\n\ncompile_paper: str\n\n\n\nclustermy_workflowpaper_outputOutputsWithInjectioncompile_paper->clustermy_workflowOutputsWithInjectionpaper_output__compile_paper\n\n\n\n\n\n\n", + "text/plain": "" + }, + "metadata": {} + } + ], + "execution_count": 14 + }, + { + "id": "63f29646-3846-4a97-a033-20e9df0ac214", + "cell_type": "code", + "source": "workflow_json_filename = \"pyiron_workflow_nfdi.json\"", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 15 + }, + { + "id": "f62111ba-9271-4987-9c7e-3b1c9f9eae7a", + "cell_type": "code", + "source": "write_workflow_json(graph_as_dict=wf.graph_as_dict, file_name=workflow_json_filename)", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 16 + }, + { + "id": "d789971e-8f41-45fa-832a-11fd72dea96e", + "cell_type": "markdown", + "source": "## Load Workflow with aiida\n\n`load_workflow_json` rebuilds the pipeline as an `aiida-workgraph` `WorkGraph` from `pyiron_workflow_nfdi.json`, after connecting to a local AiiDA profile with `load_profile()`. `wg.run()` re-executes the whole pipeline under `aiida`.", + "metadata": {} + }, + { + "id": "a6e85e89-5d7a-40eb-809c-ac44974e3fd7", + "cell_type": "code", + "source": "from aiida import load_profile\n\nload_profile()", + "metadata": { + "trusted": true + }, + "outputs": [ + { + "execution_count": 17, + "output_type": "execute_result", + "data": { + "text/plain": "Profile" + }, + "metadata": {} + } + ], + "execution_count": 17 + }, + { + "id": "3de84fb7-b01b-4541-868a-92e881eb6e77", + "cell_type": "code", + "source": "from python_workflow_definition.aiida import load_workflow_json", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 18 + }, + { + "id": "b33f5528-10cd-47c8-8723-622902978859", + "cell_type": "code", + "source": "wg = load_workflow_json(file_name=workflow_json_filename)\nwg", + "metadata": { + "trusted": true + }, + "outputs": [ + { + "execution_count": 19, + "output_type": "execute_result", + "data": { + "text/plain": "NodeGraphWidget(settings={'minimap': True}, style={'width': '90%', 'height': '600px'}, value={'name': 'WorkGra…", + "application/vnd.jupyter.widget-view+json": { + "version_major": 2, + "version_minor": 1, + "model_id": "6641ced7603742b8b28900587d764b7c" + } + }, + "metadata": {} + } + ], + "execution_count": 19 + }, + { + "id": "15282ca1-d339-40e7-ad68-8a7613ed08da", + "cell_type": "code", + "source": "wg.run()", + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": "05/24/2025 09:30:18 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: generate_mesh1\n05/24/2025 09:30:20 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: generate_mesh1, type: PyFunction, finished.\n05/24/2025 09:30:20 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: convert_to_xdmf2\n05/24/2025 09:30:22 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: convert_to_xdmf2, type: PyFunction, finished.\n05/24/2025 09:30:22 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: poisson3\n05/24/2025 09:30:30 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: poisson3, type: PyFunction, finished.\n05/24/2025 09:30:30 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: plot_over_line4\n05/24/2025 09:30:33 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: plot_over_line4, type: PyFunction, finished.\n05/24/2025 09:30:33 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: substitute_macros5\n05/24/2025 09:30:34 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: substitute_macros5, type: PyFunction, finished.\n05/24/2025 09:30:34 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: compile_paper6\n05/24/2025 09:30:50 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|update_task_state]: Task: compile_paper6, type: PyFunction, finished.\n05/24/2025 09:30:50 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|continue_workgraph]: tasks ready to run: \n05/24/2025 09:30:50 AM <282> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [3|WorkGraphEngine|finalize]: Finalize workgraph.\n" + } + ], + "execution_count": 20 + }, + { + "id": "55dc8d12-dfe6-4465-a368-b7e590ae6800", + "cell_type": "markdown", + "source": "## Load Workflow with jobflow\n\nThe same JSON is reconstructed as a jobflow `Flow` and executed locally with `run_locally`, independently of the `pyiron_workflow` objects used to define it.", + "metadata": {} + }, + { + "id": "dff46eb8-e0e7-49bb-8c40-0db2df133124", + "cell_type": "code", + "source": "from python_workflow_definition.jobflow import load_workflow_json", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 21 + }, + { + "id": "6a189459-84e4-4738-ada1-37ee8c65b2ab", + "cell_type": "code", + "source": "from jobflow.managers.local import run_locally", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 22 + }, + { + "id": "6e7f3614-c971-4e2d-83f0-96f0d0fc04de", + "cell_type": "code", + "source": "flow = load_workflow_json(file_name=workflow_json_filename)", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 23 + }, + { + "id": "2d87ed45-f5d9-403f-a03a-26be4a47a3ef", + "cell_type": "code", + "source": "result = run_locally(flow)\nresult", + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": "2025-05-24 09:31:15,165 INFO Started executing jobs locally\n2025-05-24 09:31:15,364 INFO Starting job - generate_mesh (4cc4e3d9-39aa-4c20-a047-a029b6812d61)\n2025-05-24 09:31:16,480 INFO Finished job - generate_mesh (4cc4e3d9-39aa-4c20-a047-a029b6812d61)\n2025-05-24 09:31:16,481 INFO Starting job - convert_to_xdmf (8527c563-c0b1-4ae5-a6c3-00ae25ad2508)\n2025-05-24 09:31:17,834 INFO Finished job - convert_to_xdmf (8527c563-c0b1-4ae5-a6c3-00ae25ad2508)\n2025-05-24 09:31:17,836 INFO Starting job - poisson (58b31d24-51f9-4b9a-ac3e-88a7f01a09ad)\n2025-05-24 09:31:20,318 INFO Finished job - poisson (58b31d24-51f9-4b9a-ac3e-88a7f01a09ad)\n2025-05-24 09:31:20,318 INFO Starting job - plot_over_line (b095dc38-f858-4c34-876c-df8f8b851a5f)\n2025-05-24 09:31:21,717 INFO Finished job - plot_over_line (b095dc38-f858-4c34-876c-df8f8b851a5f)\n2025-05-24 09:31:21,718 INFO Starting job - substitute_macros (656e820d-20d5-4665-bb8a-43807ed6b2e9)\n2025-05-24 09:31:22,542 INFO Finished job - substitute_macros (656e820d-20d5-4665-bb8a-43807ed6b2e9)\n2025-05-24 09:31:22,543 INFO Starting job - compile_paper (52de4e6d-7574-401b-bcdb-ae6c298cc6b9)\n2025-05-24 09:31:24,556 INFO Finished job - compile_paper (52de4e6d-7574-401b-bcdb-ae6c298cc6b9)\n2025-05-24 09:31:24,557 INFO Finished executing jobs locally\n" + }, + { + "execution_count": 24, + "output_type": "execute_result", + "data": { + "text/plain": "{'4cc4e3d9-39aa-4c20-a047-a029b6812d61': {1: Response(output='/home/jovyan/example_workflows/nfdi/preprocessing/square.msh', detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))},\n '8527c563-c0b1-4ae5-a6c3-00ae25ad2508': {1: Response(output={'xdmf_file': '/home/jovyan/example_workflows/nfdi/preprocessing/square.xdmf', 'h5_file': '/home/jovyan/example_workflows/nfdi/preprocessing/square.h5'}, detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))},\n '58b31d24-51f9-4b9a-ac3e-88a7f01a09ad': {1: Response(output={'numdofs': 357, 'pvd_file': '/home/jovyan/example_workflows/nfdi/processing/poisson.pvd', 'vtu_file': '/home/jovyan/example_workflows/nfdi/processing/poisson000000.vtu'}, detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))},\n 'b095dc38-f858-4c34-876c-df8f8b851a5f': {1: Response(output='/home/jovyan/example_workflows/nfdi/postprocessing/plotoverline.csv', detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))},\n '656e820d-20d5-4665-bb8a-43807ed6b2e9': {1: Response(output='/home/jovyan/example_workflows/nfdi/postprocessing/macros.tex', detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))},\n '52de4e6d-7574-401b-bcdb-ae6c298cc6b9': {1: Response(output='/home/jovyan/example_workflows/nfdi/postprocessing/paper.pdf', detour=None, addition=None, replace=None, stored_data=None, stop_children=False, stop_jobflow=False, job_dir=PosixPath('/home/jovyan/example_workflows/nfdi'))}}" + }, + "metadata": {} + } + ], + "execution_count": 24 + }, + { + "id": "ebd3248b-ff75-40bf-a488-b5e338342671", + "cell_type": "markdown", + "source": "## Load Workflow with pyiron_base\n\nFinally, the graph is loaded into a list of delayed `pyiron_base` jobs. `.draw()` visualizes the dependency chain and `.pull()` triggers execution, returning the final `paper.pdf` path.", + "metadata": {} + }, + { + "id": "c7e1047f-0d42-4777-911d-ab405396401a", + "cell_type": "code", + "source": "from python_workflow_definition.pyiron_base import load_workflow_json", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": 25 + }, + { + "id": "6aeba274-10d3-4e79-8ee4-4d870f163939", + "cell_type": "code", + "source": "delayed_object_lst = load_workflow_json(file_name=workflow_json_filename)\ndelayed_object_lst[-1].draw()", + "metadata": { + "trusted": true + }, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": "", + "image/svg+xml": "\n\n\n\n\ncreate_function_job_ba0bf2668d8f7b42ff717c2cb79b5920\n\ncreate_function_job=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d9280>\n\n\n\nplot_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\nplot_file=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8da0>\n\n\n\nplot_file_5007b1d67345934b2ffcdeaaad6ea2ab->create_function_job_ba0bf2668d8f7b42ff717c2cb79b5920\n\n\n\n\n\npoisson_output_vtu_file_51141ca0ec0a07922807865d8a067793\n\npoisson_output_vtu_file=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8a40>\n\n\n\npoisson_output_vtu_file_51141ca0ec0a07922807865d8a067793->plot_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\npvbatch_output_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\npvbatch_output_file=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8da0>\n\n\n\npoisson_output_vtu_file_51141ca0ec0a07922807865d8a067793->pvbatch_output_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\nmacros_tex_02d82cedf90ad323f7eef57ddf383160\n\nmacros_tex=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d9070>\n\n\n\npvbatch_output_file_5007b1d67345934b2ffcdeaaad6ea2ab->macros_tex_02d82cedf90ad323f7eef57ddf383160\n\n\n\n\n\nmeshio_output_h5_0af06cb14e9ddc7585647c44e4d4ef88\n\nmeshio_output_h5=<pyiron_base.project.delayed.DelayedObject object at 0x76dba3cb2600>\n\n\n\nmeshio_output_h5_0af06cb14e9ddc7585647c44e4d4ef88->poisson_output_vtu_file_51141ca0ec0a07922807865d8a067793\n\n\n\n\n\npoisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513\n\npoisson_output_pvd_file=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8a10>\n\n\n\nmeshio_output_h5_0af06cb14e9ddc7585647c44e4d4ef88->poisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513\n\n\n\n\n\nndofs_ac7427db28c0527371c0e9b27a57024b\n\nndofs=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8e30>\n\n\n\nmeshio_output_h5_0af06cb14e9ddc7585647c44e4d4ef88->ndofs_ac7427db28c0527371c0e9b27a57024b\n\n\n\n\n\npoisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513->plot_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\npoisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513->pvbatch_output_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\nndofs_ac7427db28c0527371c0e9b27a57024b->macros_tex_02d82cedf90ad323f7eef57ddf383160\n\n\n\n\n\ngmsh_output_file_6ef41eb684984daa8337ee65c2aa9f18\n\ngmsh_output_file=<pyiron_base.project.delayed.DelayedObject object at 0x76dba89319d0>\n\n\n\ngmsh_output_file_6ef41eb684984daa8337ee65c2aa9f18->meshio_output_h5_0af06cb14e9ddc7585647c44e4d4ef88\n\n\n\n\n\nmeshio_output_xdmf_c383e7d9f83d7cdc8ae04ebf9c264173\n\nmeshio_output_xdmf=<pyiron_base.project.delayed.DelayedObject object at 0x76dba30d8710>\n\n\n\ngmsh_output_file_6ef41eb684984daa8337ee65c2aa9f18->meshio_output_xdmf_c383e7d9f83d7cdc8ae04ebf9c264173\n\n\n\n\n\nmeshio_output_xdmf_c383e7d9f83d7cdc8ae04ebf9c264173->poisson_output_vtu_file_51141ca0ec0a07922807865d8a067793\n\n\n\n\n\nmeshio_output_xdmf_c383e7d9f83d7cdc8ae04ebf9c264173->poisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513\n\n\n\n\n\nmeshio_output_xdmf_c383e7d9f83d7cdc8ae04ebf9c264173->ndofs_ac7427db28c0527371c0e9b27a57024b\n\n\n\n\n\ndomain_size_f12a7f1986b9dd058dfc666dbe230b20\n\ndomain_size=2.0\n\n\n\ndomain_size_f12a7f1986b9dd058dfc666dbe230b20->gmsh_output_file_6ef41eb684984daa8337ee65c2aa9f18\n\n\n\n\n\ndomain_size_f12a7f1986b9dd058dfc666dbe230b20->macros_tex_02d82cedf90ad323f7eef57ddf383160\n\n\n\n\n\nmacros_tex_02d82cedf90ad323f7eef57ddf383160->create_function_job_ba0bf2668d8f7b42ff717c2cb79b5920\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696\n\nsource_directory=/home/jovyan/example_workflows/nfdi/source\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->create_function_job_ba0bf2668d8f7b42ff717c2cb79b5920\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->plot_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->poisson_output_vtu_file_51141ca0ec0a07922807865d8a067793\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->pvbatch_output_file_5007b1d67345934b2ffcdeaaad6ea2ab\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->poisson_output_pvd_file_0440a581658dfd1533b4c5d08eaf8513\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->ndofs_ac7427db28c0527371c0e9b27a57024b\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->gmsh_output_file_6ef41eb684984daa8337ee65c2aa9f18\n\n\n\n\n\nsource_directory_053958014e7cd4a4df22cfa51c9fc696->macros_tex_02d82cedf90ad323f7eef57ddf383160\n\n\n\n\n" + }, + "metadata": {} + } + ], + "execution_count": 26 + }, + { + "id": "ea92c7ad-afef-423c-b8fa-9d24c1267c73", + "cell_type": "code", + "source": "delayed_object_lst[-1].pull()", + "metadata": { + "trusted": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": "The job generate_mesh_47725c16637f799ac042e47468005db3 was saved and received the ID: 1\nThe job convert_to_xdmf_d6a46eb9a4ec352aa996e783ec3c785f was saved and received the ID: 2\nThe job poisson_3c147fc86db87cf0c0f94bda333f8cd8 was saved and received the ID: 3\nThe job plot_over_line_ef50933291910dadcc8311924971e127 was saved and received the ID: 4\nThe job substitute_macros_63766eafd6b1980c7832dd8c9a97c96e was saved and received the ID: 5\nThe job compile_paper_128d1d58374953c00e95b8de62cbb10b was saved and received the ID: 6\n" + }, + { + "execution_count": 27, + "output_type": "execute_result", + "data": { + "text/plain": "'/home/jovyan/example_workflows/nfdi/postprocessing/paper.pdf'" + }, + "metadata": {} + } + ], + "execution_count": 27 + }, + { + "id": "32287743-e59a-467c-92fe-8586f199e36e", + "cell_type": "code", + "source": "", + "metadata": { + "trusted": true + }, + "outputs": [], + "execution_count": null + } + ] +} \ No newline at end of file diff --git a/example_workflows/nfdi/universal_workflow.ipynb b/example_workflows/nfdi/universal_workflow.ipynb index 5c1d889..8ceeaad 100644 --- a/example_workflows/nfdi/universal_workflow.ipynb +++ b/example_workflows/nfdi/universal_workflow.ipynb @@ -23,12 +23,12 @@ "cells": [ { "cell_type": "markdown", - "source": "# Load Simple Workflow", + "source": "# Load Simple Workflow\n\nThis notebook loads the single, pre-existing NFDI4Ing `workflow.json` — the file-based benchmark from [`workflow.py`](workflow.py) that runs `gmsh`, `meshio`, a FEniCS-based Poisson solver, ParaView's `pvbatch` and `tectonic` in turn — with several different engines. Since the JSON only describes function nodes, inputs, outputs and the edges between them, it does not matter which engine originally wrote it; any of the engines below can load and execute it, each re-running the same chain of external tools.", "metadata": {} }, { "cell_type": "markdown", - "source": "## Plot", + "source": "## Plot\n\nBefore running the workflow with any engine, `python_workflow_definition.plot.plot` renders `workflow.json` directly as a graph of function, input and output nodes — independent of any particular workflow engine.", "metadata": {} }, { @@ -60,7 +60,7 @@ }, { "cell_type": "markdown", - "source": "## Aiida ", + "source": "## Aiida\n\nLoading the JSON into `aiida-workgraph` requires connecting to an AiiDA profile first via `load_profile()`; the workflow is then reconstructed as a `WorkGraph` and executed with `.run()`, which re-runs `generate_mesh`, `convert_to_xdmf`, `poisson`, `plot_over_line`, `substitute_macros` and `compile_paper` in order.", "metadata": {} }, { @@ -130,7 +130,7 @@ }, { "cell_type": "markdown", - "source": "## executorlib", + "source": "## executorlib\n\n`python_workflow_definition.executorlib.load_workflow_json` submits each function node to the given `Executor` and returns the `Future` of the final node (the compiled `paper.pdf` path); calling `.result()` blocks until the whole chain of external tools has completed.", "metadata": {} }, { @@ -180,7 +180,7 @@ }, { "cell_type": "markdown", - "source": "## Jobflow", + "source": "## Jobflow\n\nThe JSON is reconstructed as a jobflow `Flow` and executed locally with `run_locally`.", "metadata": {} }, { @@ -235,7 +235,7 @@ }, { "cell_type": "markdown", - "source": "## pyiron", + "source": "## pyiron\n\nLoading the JSON into `pyiron_base` produces a list of delayed jobs; `.draw()` visualizes the dependency graph and `.pull()` triggers execution of the whole chain, again writing intermediate files to `preprocessing/`, `processing/` and `postprocessing/`.", "metadata": {} }, { @@ -291,7 +291,7 @@ { "metadata": {}, "cell_type": "markdown", - "source": "## Load Workflow with pyiron_workflow" + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the workflow is loaded into a `pyiron_workflow` `Workflow` object, drawn with `.draw()` and executed with `.run()`." }, { "metadata": {}, @@ -324,7 +324,7 @@ { "metadata": {}, "cell_type": "markdown", - "source": "## Python" + "source": "## Python\n\nAs a baseline, `python_workflow_definition.purepython.load_workflow_json` interprets the JSON directly in plain Python, with no workflow engine involved: it evaluates each function node in dependency order — invoking `gmsh`, `meshio`, the Poisson solver, `pvbatch` and `tectonic` via `conda_subprocess` exactly as `workflow.py` defines — and returns the final result." }, { "cell_type": "code", @@ -354,4 +354,4 @@ "execution_count": 19 } ] -} +} \ No newline at end of file diff --git a/example_workflows/quantum_espresso/aiida.ipynb b/example_workflows/quantum_espresso/aiida.ipynb index bddc516..8b297f2 100644 --- a/example_workflows/quantum_espresso/aiida.ipynb +++ b/example_workflows/quantum_espresso/aiida.ipynb @@ -3,16 +3,12 @@ { "cell_type": "markdown", "metadata": {}, - "source": [ - "# Aiida" - ] + "source": "# aiida-workgraph\n\nThis notebook builds the Quantum Espresso energy-volume-curve workflow directly with [`aiida-workgraph`](https://github.com/aiidateam/aiida-workgraph) and exports it as a PWD `workflow.json`. The second half of the notebook loads that same JSON file with `jobflow`, `pyiron_base` and `pyiron_workflow` to show that a graph built with one engine can be executed by another." }, { "cell_type": "markdown", "metadata": {}, - "source": [ - "## Define workflow with aiida" - ] + "source": "## Define workflow with aiida\n\n`aiida-workgraph` represents a workflow as a `WorkGraph` built from `task`s, each wrapping a Python function; connecting a task's output to another task's input builds up the graph. `load_profile()` connects to the local AiiDA profile/database, which is required before any AiiDA node can be created, since every task and its result are stored there for provenance." }, { "cell_type": "code", @@ -59,6 +55,11 @@ ")" ] }, + { + "cell_type": "markdown", + "source": "`calculate_qe` (defined in [`workflow.py`](workflow.py)) returns a dict with `energy`, `volume` and `structure` keys. Wrapping it with `task(outputs=[...])` tells `aiida-workgraph` to expose these as separate output sockets (`.outputs.energy`, `.outputs.volume`, `.outputs.structure`) instead of a single opaque result, so downstream tasks can connect to individual keys.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 4, @@ -84,9 +85,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": [ - "### Prepare the inputs" - ] + "source": "### Prepare the inputs\n\nScalar inputs are wrapped as AiiDA data types (`orm.Str`, `orm.Float`, `orm.Bool`, `orm.Dict`, `orm.List`) so AiiDA can track their provenance. `strain_lst` are the five volumetric strains (0.9-1.1) that will later be applied to the relaxed structure to sample the energy-volume curve." }, { "cell_type": "code", @@ -111,9 +110,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": [ - "### Actual tasks to construct the EOS workflow" - ] + "source": "### Build the task graph\n\nThe pipeline is: `get_bulk_structure` builds the initial Al structure, `calculate_qe` relaxes it (`vc-relax`), `generate_structures` applies the strains, and one `calculate_qe` (`scf`) task per strained structure computes its energy and volume. `get_dict` bundles the structure, calculation type, k-points, pseudopotentials and smearing into the `input_dict` argument `calculate_qe` expects." }, { "cell_type": "code", @@ -155,6 +152,11 @@ ")" ] }, + { + "cell_type": "markdown", + "source": "`generate_structures` applies each of the five strains to the relaxed structure, producing five strained structures (`s_0` ... `s_4`). This is the fan-out point of the workflow: the loop below adds one independent `scf` `calculate_qe` task per strained structure, and their `energy`/`volume` outputs are collected into two lists (via `get_list` tasks) for the final plot.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 9, @@ -243,6 +245,11 @@ ")" ] }, + { + "cell_type": "markdown", + "source": "Displaying `wg` renders an interactive `NodeGraphWidget` of the task graph, which is useful to check the wiring before running it. `write_workflow_json` then serializes the `WorkGraph` into the PWD `workflow.json` format (nodes = functions/inputs/outputs, edges = data connections) so it can be loaded by another engine.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 14, @@ -861,9 +868,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": [ - "## Load Workflow with jobflow" - ] + "source": "## Load Workflow with jobflow\n\n`load_workflow_json` reconstructs the same graph as a `jobflow` `Flow` of `Job`s. To show that the loaded graph behaves like a native `jobflow` object, the lattice constant `a` of the first job is overridden (4.04 -> 4.05) before the flow is executed locally with `run_locally`." }, { "cell_type": "code", @@ -1099,9 +1104,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": [ - "## Load Workflow with pyiron_base" - ] + "source": "## Load Workflow with pyiron_base\n\n`pyiron_base`'s `load_workflow_json` turns each node into a delayed pyiron job. `.draw()` renders the dependency graph, and `.pull()` triggers execution of the whole chain, saving every intermediate job (relax + 5 strained SCF calculations) to the pyiron project." }, { "cell_type": "code", @@ -1994,9 +1997,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": [ - "## Load Workflow with pyiron_workflow" - ] + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the same JSON is loaded into a `pyiron_workflow` `Workflow` object. Nodes become graph nodes reachable via attribute access (e.g. `wf.get_bulk_structure`), and `.run()` executes them in dependency order." }, { "cell_type": "code", @@ -5796,4 +5797,4 @@ }, "nbformat": 4, "nbformat_minor": 4 -} +} \ No newline at end of file diff --git a/example_workflows/quantum_espresso/cwl.ipynb b/example_workflows/quantum_espresso/cwl.ipynb index 5dc1979..4c47756 100644 --- a/example_workflows/quantum_espresso/cwl.ipynb +++ b/example_workflows/quantum_espresso/cwl.ipynb @@ -21,6 +21,12 @@ "nbformat_minor": 5, "nbformat": 4, "cells": [ + { + "cell_type": "markdown", + "id": "f9fecca7", + "source": "# CWL\n\nThis notebook executes the Quantum Espresso energy-volume-curve `workflow.json` (produced by one of the other notebooks in this directory, e.g. `aiida.ipynb`) with the [Common Workflow Language](https://www.commonwl.org) via [`cwltool`](https://github.com/common-workflow-language/cwltool). Unlike the other engines, PWD does not build a CWL graph directly in Python - instead it translates an existing `workflow.json` into native CWL files, which `cwltool` then executes as a separate command line process.", + "metadata": {} + }, { "id": "4eca79ef-1053-4f69-89ad-2bee8411068e", "cell_type": "code", @@ -41,6 +47,12 @@ "outputs": [], "execution_count": 2 }, + { + "cell_type": "markdown", + "id": "8da8470b", + "source": "## Translate workflow.json to CWL\n\n`write_workflow` reads the existing `workflow.json` and generates: one `.cwl` `CommandLineTool` per PWD function node (each one invokes `python -m python_workflow_definition.cwl --function=...` to call the matching function in `workflow.py`), a top-level `workflow.cwl` that chains these steps together following the `workflow.json` edges, and a `workflow.yml` job file listing one input file per PWD input node. Since CWL only passes files between steps, every input and every function output is exchanged as a pickle file.", + "metadata": {} + }, { "id": "92e3921b-2bb8-4333-8cfe-4bd27f785d24", "cell_type": "code", @@ -61,6 +73,12 @@ "outputs": [], "execution_count": 4 }, + { + "cell_type": "markdown", + "id": "21fc2973", + "source": "## Run the workflow with cwltool\n\nASE's Quantum Espresso writer resolves the pseudopotential directory from the `ESPRESSO_PSEUDO` environment variable when no explicit path is given. Because `cwltool` runs every step in its own isolated temporary directory, the environment variable has to be forwarded explicitly with `--preserve-environment=ESPRESSO_PSEUDO`, so that `pw.x` can find the pseudopotential file shipped in [`espresso/pseudo`](espresso/pseudo) from any step.", + "metadata": {} + }, { "id": "0192ca74-3971-464b-9435-c156e0b6e623", "cell_type": "code", @@ -82,11 +100,17 @@ { "name": "stdout", "output_type": "stream", - "text": "/srv/conda/envs/notebook/bin/cwltool:11: DeprecationWarning: Nesting argument groups is deprecated.\n sys.exit(run())\n\u001B[1;30mINFO\u001B[0m /srv/conda/envs/notebook/bin/cwltool 3.1.20250110105449\n\u001B[1;30mINFO\u001B[0m Resolved 'workflow.cwl' to 'file:///home/jovyan/example_workflows/quantum_espresso/workflow.cwl'\n\u001B[1;30mINFO\u001B[0m [workflow ] start\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_bulk_structure_0\n\u001B[1;30mINFO\u001B[0m [step get_bulk_structure_0] start\n\u001B[1;30mINFO\u001B[0m [job get_bulk_structure_0] /tmp/9sant4h9$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/va137af6/stga19aac01-d6f9-48d3-ba6b-d8a6e460f525/workflow.py \\\n --function=workflow.get_bulk_structure \\\n --arg_cubic=/tmp/va137af6/stgd2861bb7-939d-449d-9600-71f380ee058d/cubic.pickle \\\n --arg_element=/tmp/va137af6/stg93f5775e-477a-4b2e-87ac-822ff4097346/element.pickle \\\n --arg_a=/tmp/va137af6/stgcfd0bd7b-a186-4b46-80a6-663aeb5d0fa9/a.pickle\n\u001B[1;30mINFO\u001B[0m [job get_bulk_structure_0] Max memory used: 110MiB\n\u001B[1;30mINFO\u001B[0m [job get_bulk_structure_0] completed success\n\u001B[1;30mINFO\u001B[0m [step get_bulk_structure_0] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_dict_13\n\u001B[1;30mINFO\u001B[0m [step get_dict_13] start\n\u001B[1;30mINFO\u001B[0m [job get_dict_13] /tmp/_o8rxcyc$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/1h2uwpt9/stg17e865b8-f1f4-406f-a9c7-b8deddc5695d/smearing.pickle \\\n --arg_calculation=/tmp/1h2uwpt9/stgf1d25320-f3de-48f7-8087-5cdb5531765c/calculation_0.pickle \\\n --arg_kpts=/tmp/1h2uwpt9/stgc9e37196-83cf-49d9-88e1-b5acb7c53e7f/kpts.pickle \\\n --arg_pseudopotentials=/tmp/1h2uwpt9/stgcfb6bfd1-3ae7-4600-b07d-ca107c1a275c/pseudopotentials.pickle \\\n --arg_structure=/tmp/1h2uwpt9/stg0da627e7-f19e-4948-b23e-4552aa692862/result.pickle\n\u001B[1;30mINFO\u001B[0m [job get_dict_13] completed success\n\u001B[1;30mINFO\u001B[0m [step get_dict_13] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step calculate_qe_1\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_1] start\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_1] /tmp/q56p65x4$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/5hwlsb8q/stga00637b3-d27a-4b67-b4b2-5857ffc8f94a/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/5hwlsb8q/stgea0d773c-cef9-49b7-9895-09c22db6f52d/result.pickle \\\n --arg_working_directory=/tmp/5hwlsb8q/stg541840e1-d4a6-4a2c-a220-ab09e72c869d/working_directory_0.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09273] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_1] Max memory used: 273MiB\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_1] completed success\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_1] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step generate_structures_2\n\u001B[1;30mINFO\u001B[0m [step generate_structures_2] start\n\u001B[1;30mINFO\u001B[0m [job generate_structures_2] /tmp/81vwps2h$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/0d6xih7u/stg8da9bfd1-34a7-4282-8446-97a728d4747b/workflow.py \\\n --function=workflow.generate_structures \\\n --arg_structure=/tmp/0d6xih7u/stg89eb15f9-014d-4c20-8197-87c6c7345ffe/structure.pickle \\\n --arg_strain_lst=/tmp/0d6xih7u/stg80ec088c-9b6a-45bd-8565-e68fd3c76210/strain_lst.pickle\n\u001B[1;30mINFO\u001B[0m [job generate_structures_2] Max memory used: 96MiB\n\u001B[1;30mINFO\u001B[0m [job generate_structures_2] completed success\n\u001B[1;30mINFO\u001B[0m [step generate_structures_2] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_dict_25\n\u001B[1;30mINFO\u001B[0m [step get_dict_25] start\n\u001B[1;30mINFO\u001B[0m [job get_dict_25] /tmp/hj3teyvh$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/ky497mud/stgf2ab029c-fd78-4490-a4aa-2ee352fa40a2/smearing.pickle \\\n --arg_calculation=/tmp/ky497mud/stg12f82711-28cb-42e6-80c5-59e7fa946ea8/calculation_1.pickle \\\n --arg_kpts=/tmp/ky497mud/stgd35fecc6-83e5-493e-85ad-caf6acd301cb/kpts.pickle \\\n --arg_pseudopotentials=/tmp/ky497mud/stg9b98aba6-11a6-4718-bd3a-23f160518077/pseudopotentials.pickle \\\n --arg_structure=/tmp/ky497mud/stg00376f4a-a991-469a-b494-27c160e6c15d/s_2.pickle\n\u001B[1;30mINFO\u001B[0m [job get_dict_25] completed success\n\u001B[1;30mINFO\u001B[0m [step get_dict_25] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_dict_20\n\u001B[1;30mINFO\u001B[0m [step get_dict_20] start\n\u001B[1;30mINFO\u001B[0m [job get_dict_20] /tmp/y9drdoe7$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/dsu142t1/stgcba59359-36e1-4bf9-9e6f-e7cf99a23077/smearing.pickle \\\n --arg_calculation=/tmp/dsu142t1/stg56a2646f-2057-4670-bb4e-f55af62c5858/calculation_1.pickle \\\n --arg_kpts=/tmp/dsu142t1/stgc443e9a8-c9b6-4c5e-8321-6a70d347706b/kpts.pickle \\\n --arg_pseudopotentials=/tmp/dsu142t1/stg47b4b44d-c69c-4629-aaed-0e15bf14426e/pseudopotentials.pickle \\\n --arg_structure=/tmp/dsu142t1/stg0e5b3c6e-a315-4eaf-9a66-dbf5669eaada/s_0.pickle\n\u001B[1;30mINFO\u001B[0m [job get_dict_20] completed success\n\u001B[1;30mINFO\u001B[0m [step get_dict_20] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_dict_23\n\u001B[1;30mINFO\u001B[0m [step get_dict_23] start\n\u001B[1;30mINFO\u001B[0m [job get_dict_23] /tmp/9v59c4vd$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/2mc_byfs/stgace4b178-c0ff-406e-95ac-754318766886/smearing.pickle \\\n --arg_calculation=/tmp/2mc_byfs/stg793df1d8-8f8d-4bb3-9c7c-243a48dc494b/calculation_1.pickle \\\n --arg_kpts=/tmp/2mc_byfs/stga852e76a-885f-4e39-8ab4-ecebf95c656f/kpts.pickle \\\n --arg_pseudopotentials=/tmp/2mc_byfs/stgd06acdff-6e0c-476c-be57-1f21b2d03711/pseudopotentials.pickle \\\n --arg_structure=/tmp/2mc_byfs/stg02210df5-d3e2-42b8-a75e-47e2f2c9f274/s_1.pickle\n\u001B[1;30mINFO\u001B[0m [job get_dict_23] completed success\n\u001B[1;30mINFO\u001B[0m [step get_dict_23] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_dict_29\n\u001B[1;30mINFO\u001B[0m [step get_dict_29] start\n\u001B[1;30mINFO\u001B[0m [job get_dict_29] /tmp/q_svsf9g$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/8u8e7db4/stgbba611dc-3b23-4653-a293-f71d492a42af/smearing.pickle \\\n --arg_calculation=/tmp/8u8e7db4/stge9a32972-6649-4eb5-abd7-9db61d929d2d/calculation_1.pickle \\\n --arg_kpts=/tmp/8u8e7db4/stg02098ea5-7487-4a66-bb77-99a6336d7b71/kpts.pickle \\\n --arg_pseudopotentials=/tmp/8u8e7db4/stg571b6259-4599-4c39-8cb2-79a70feb78b8/pseudopotentials.pickle \\\n --arg_structure=/tmp/8u8e7db4/stg64c55a4d-8af6-40bf-abfd-3ec8a690b90b/s_4.pickle\n\u001B[1;30mINFO\u001B[0m [job get_dict_29] completed success\n\u001B[1;30mINFO\u001B[0m [step get_dict_29] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_dict_27\n\u001B[1;30mINFO\u001B[0m [step get_dict_27] start\n\u001B[1;30mINFO\u001B[0m [job get_dict_27] /tmp/vgcwzb84$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/xf32x1ld/stg3cbb6408-2383-49d0-9476-ef3fbee7c165/smearing.pickle \\\n --arg_calculation=/tmp/xf32x1ld/stgeb36ed3b-b442-43ea-b14d-1397875e8839/calculation_1.pickle \\\n --arg_kpts=/tmp/xf32x1ld/stge8854d32-32f3-4a90-8e66-8ce7b1160d8d/kpts.pickle \\\n --arg_pseudopotentials=/tmp/xf32x1ld/stg0dba3c44-e6af-46a9-9f00-c08969392ede/pseudopotentials.pickle \\\n --arg_structure=/tmp/xf32x1ld/stg4223a500-1f85-4651-9130-bd1629008770/s_3.pickle\n\u001B[1;30mINFO\u001B[0m [job get_dict_27] completed success\n\u001B[1;30mINFO\u001B[0m [step get_dict_27] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step calculate_qe_3\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_3] start\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_3] /tmp/v2e_84tz$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/n81xp6_a/stgf98a9803-3329-400e-a03c-02b51d3f3c97/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/n81xp6_a/stgd93a737b-2598-4fb8-bdcb-bb1d2048116f/result.pickle \\\n --arg_working_directory=/tmp/n81xp6_a/stg5ad2a52b-2844-45e4-b227-3d835efa01f6/working_directory_1.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09459] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_3] Max memory used: 240MiB\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_3] completed success\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_3] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step calculate_qe_4\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_4] start\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_4] /tmp/tgwdkhxx$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/l5mihx97/stgf73a20aa-bd97-46af-a923-b8d776da2b98/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/l5mihx97/stgb8c3fcbe-9277-46c4-b8a2-a1171a292580/result.pickle \\\n --arg_working_directory=/tmp/l5mihx97/stg3ae0b93a-2e0b-4f1b-b079-985ea6bbaa72/working_directory_2.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09533] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_4] Max memory used: 213MiB\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_4] completed success\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_4] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step calculate_qe_5\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_5] start\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_5] /tmp/d36x49f6$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/nvm7fsf5/stg384bab04-833c-43f2-bd00-f4b5a5c93eec/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/nvm7fsf5/stg9aa51e21-01b0-4157-b73f-a3feee8a917a/result.pickle \\\n --arg_working_directory=/tmp/nvm7fsf5/stg02ef6345-c598-4127-95c7-e92d0a977c40/working_directory_3.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09611] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_5] Max memory used: 243MiB\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_5] completed success\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_5] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step calculate_qe_6\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_6] start\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_6] /tmp/eiyg8it6$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/708xmf3f/stg791b1717-cfb1-4569-ba75-d648a84e0a72/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/708xmf3f/stg9ebf7054-080c-4173-a8b9-2247d01a0068/result.pickle \\\n --arg_working_directory=/tmp/708xmf3f/stg1afccee5-d9b6-46ed-bc2c-fa36bdb8013d/working_directory_4.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09688] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_6] Max memory used: 245MiB\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_6] completed success\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_6] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step calculate_qe_7\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_7] start\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_7] /tmp/qjgxwxnd$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/g3sp392q/stg0a51a8da-c7a7-4d90-a49c-1817002a62a6/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/g3sp392q/stg67e73a60-3cc6-46dd-b009-cfc0034aec2c/result.pickle \\\n --arg_working_directory=/tmp/g3sp392q/stgf3d265db-820b-4af3-8975-0a6faa0d7a67/working_directory_5.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09770] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_7] Max memory used: 244MiB\n\u001B[1;30mINFO\u001B[0m [job calculate_qe_7] completed success\n\u001B[1;30mINFO\u001B[0m [step calculate_qe_7] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_list_31\n\u001B[1;30mINFO\u001B[0m [step get_list_31] start\n\u001B[1;30mINFO\u001B[0m [job get_list_31] /tmp/mnovytq_$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_list \\\n --arg_0=/tmp/bz0ntajx/stgdab2f59b-b118-44c4-96ef-b4639edaee4e/energy.pickle \\\n --arg_2=/tmp/bz0ntajx/stg2e9bec4d-1824-4152-b003-aab0eac204cd/energy.pickle \\\n --arg_1=/tmp/bz0ntajx/stg9511563e-9784-48a9-8d81-74ade3a084c1/energy.pickle \\\n --arg_4=/tmp/bz0ntajx/stg17845efd-e335-421e-9891-6e627254edb4/energy.pickle \\\n --arg_3=/tmp/bz0ntajx/stg051fa3bc-832a-4d70-b9fb-b9948f51904d/energy.pickle\n\u001B[1;30mINFO\u001B[0m [job get_list_31] completed success\n\u001B[1;30mINFO\u001B[0m [step get_list_31] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step get_list_30\n\u001B[1;30mINFO\u001B[0m [step get_list_30] start\n\u001B[1;30mINFO\u001B[0m [job get_list_30] /tmp/vc0dzppy$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_list \\\n --arg_0=/tmp/mejux8b1/stg71e3a6a9-c56b-4be8-8720-0ca90ac40513/volume.pickle \\\n --arg_2=/tmp/mejux8b1/stg429d498d-cf56-4561-a653-8048c1213851/volume.pickle \\\n --arg_1=/tmp/mejux8b1/stgee1a0001-20f2-45de-ac4e-77b0d0c59957/volume.pickle \\\n --arg_4=/tmp/mejux8b1/stg0de2a6dc-57cd-4a0c-9637-ef2e63e96fce/volume.pickle \\\n --arg_3=/tmp/mejux8b1/stga4551f25-d5f6-47c6-92ea-452cf694a286/volume.pickle\n\u001B[1;30mINFO\u001B[0m [job get_list_30] completed success\n\u001B[1;30mINFO\u001B[0m [step get_list_30] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] starting step plot_energy_volume_curve_8\n\u001B[1;30mINFO\u001B[0m [step plot_energy_volume_curve_8] start\n\u001B[1;30mINFO\u001B[0m [job plot_energy_volume_curve_8] /tmp/j4v3j3mt$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/_67k4bu3/stg5be48d76-8a92-4b02-b3d6-7bdb4c35c9ce/workflow.py \\\n --function=workflow.plot_energy_volume_curve \\\n --arg_volume_lst=/tmp/_67k4bu3/stgd43e71b0-d16b-4eb2-8d14-83924dafd7d6/result.pickle \\\n --arg_energy_lst=/tmp/_67k4bu3/stg43eeacdd-5042-4f2d-8efc-d8e0d5fe1c14/result.pickle\n\u001B[1;30mINFO\u001B[0m [job plot_energy_volume_curve_8] Max memory used: 112MiB\n\u001B[1;30mINFO\u001B[0m [job plot_energy_volume_curve_8] completed success\n\u001B[1;30mINFO\u001B[0m [step plot_energy_volume_curve_8] completed success\n\u001B[1;30mINFO\u001B[0m [workflow ] completed success\n{\n \"result_file\": {\n \"location\": \"file:///home/jovyan/example_workflows/quantum_espresso/result.pickle\",\n \"basename\": \"result.pickle\",\n \"class\": \"File\",\n \"checksum\": \"sha1$dbc1aaddc8b7343d6d33b34edcf608b8f8801918\",\n \"size\": 4,\n \"path\": \"/home/jovyan/example_workflows/quantum_espresso/result.pickle\"\n }\n}\u001B[1;30mINFO\u001B[0m Final process status is success\n" + "text": "/srv/conda/envs/notebook/bin/cwltool:11: DeprecationWarning: Nesting argument groups is deprecated.\n sys.exit(run())\n\u001b[1;30mINFO\u001b[0m /srv/conda/envs/notebook/bin/cwltool 3.1.20250110105449\n\u001b[1;30mINFO\u001b[0m Resolved 'workflow.cwl' to 'file:///home/jovyan/example_workflows/quantum_espresso/workflow.cwl'\n\u001b[1;30mINFO\u001b[0m [workflow ] start\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_bulk_structure_0\n\u001b[1;30mINFO\u001b[0m [step get_bulk_structure_0] start\n\u001b[1;30mINFO\u001b[0m [job get_bulk_structure_0] /tmp/9sant4h9$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/va137af6/stga19aac01-d6f9-48d3-ba6b-d8a6e460f525/workflow.py \\\n --function=workflow.get_bulk_structure \\\n --arg_cubic=/tmp/va137af6/stgd2861bb7-939d-449d-9600-71f380ee058d/cubic.pickle \\\n --arg_element=/tmp/va137af6/stg93f5775e-477a-4b2e-87ac-822ff4097346/element.pickle \\\n --arg_a=/tmp/va137af6/stgcfd0bd7b-a186-4b46-80a6-663aeb5d0fa9/a.pickle\n\u001b[1;30mINFO\u001b[0m [job get_bulk_structure_0] Max memory used: 110MiB\n\u001b[1;30mINFO\u001b[0m [job get_bulk_structure_0] completed success\n\u001b[1;30mINFO\u001b[0m [step get_bulk_structure_0] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_dict_13\n\u001b[1;30mINFO\u001b[0m [step get_dict_13] start\n\u001b[1;30mINFO\u001b[0m [job get_dict_13] /tmp/_o8rxcyc$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/1h2uwpt9/stg17e865b8-f1f4-406f-a9c7-b8deddc5695d/smearing.pickle \\\n --arg_calculation=/tmp/1h2uwpt9/stgf1d25320-f3de-48f7-8087-5cdb5531765c/calculation_0.pickle \\\n --arg_kpts=/tmp/1h2uwpt9/stgc9e37196-83cf-49d9-88e1-b5acb7c53e7f/kpts.pickle \\\n --arg_pseudopotentials=/tmp/1h2uwpt9/stgcfb6bfd1-3ae7-4600-b07d-ca107c1a275c/pseudopotentials.pickle \\\n --arg_structure=/tmp/1h2uwpt9/stg0da627e7-f19e-4948-b23e-4552aa692862/result.pickle\n\u001b[1;30mINFO\u001b[0m [job get_dict_13] completed success\n\u001b[1;30mINFO\u001b[0m [step get_dict_13] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step calculate_qe_1\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_1] start\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_1] /tmp/q56p65x4$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/5hwlsb8q/stga00637b3-d27a-4b67-b4b2-5857ffc8f94a/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/5hwlsb8q/stgea0d773c-cef9-49b7-9895-09c22db6f52d/result.pickle \\\n --arg_working_directory=/tmp/5hwlsb8q/stg541840e1-d4a6-4a2c-a220-ab09e72c869d/working_directory_0.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09273] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_1] Max memory used: 273MiB\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_1] completed success\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_1] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step generate_structures_2\n\u001b[1;30mINFO\u001b[0m [step generate_structures_2] start\n\u001b[1;30mINFO\u001b[0m [job generate_structures_2] /tmp/81vwps2h$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/0d6xih7u/stg8da9bfd1-34a7-4282-8446-97a728d4747b/workflow.py \\\n --function=workflow.generate_structures \\\n --arg_structure=/tmp/0d6xih7u/stg89eb15f9-014d-4c20-8197-87c6c7345ffe/structure.pickle \\\n --arg_strain_lst=/tmp/0d6xih7u/stg80ec088c-9b6a-45bd-8565-e68fd3c76210/strain_lst.pickle\n\u001b[1;30mINFO\u001b[0m [job generate_structures_2] Max memory used: 96MiB\n\u001b[1;30mINFO\u001b[0m [job generate_structures_2] completed success\n\u001b[1;30mINFO\u001b[0m [step generate_structures_2] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_dict_25\n\u001b[1;30mINFO\u001b[0m [step get_dict_25] start\n\u001b[1;30mINFO\u001b[0m [job get_dict_25] /tmp/hj3teyvh$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/ky497mud/stgf2ab029c-fd78-4490-a4aa-2ee352fa40a2/smearing.pickle \\\n --arg_calculation=/tmp/ky497mud/stg12f82711-28cb-42e6-80c5-59e7fa946ea8/calculation_1.pickle \\\n --arg_kpts=/tmp/ky497mud/stgd35fecc6-83e5-493e-85ad-caf6acd301cb/kpts.pickle \\\n --arg_pseudopotentials=/tmp/ky497mud/stg9b98aba6-11a6-4718-bd3a-23f160518077/pseudopotentials.pickle \\\n --arg_structure=/tmp/ky497mud/stg00376f4a-a991-469a-b494-27c160e6c15d/s_2.pickle\n\u001b[1;30mINFO\u001b[0m [job get_dict_25] completed success\n\u001b[1;30mINFO\u001b[0m [step get_dict_25] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_dict_20\n\u001b[1;30mINFO\u001b[0m [step get_dict_20] start\n\u001b[1;30mINFO\u001b[0m [job get_dict_20] /tmp/y9drdoe7$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/dsu142t1/stgcba59359-36e1-4bf9-9e6f-e7cf99a23077/smearing.pickle \\\n --arg_calculation=/tmp/dsu142t1/stg56a2646f-2057-4670-bb4e-f55af62c5858/calculation_1.pickle \\\n --arg_kpts=/tmp/dsu142t1/stgc443e9a8-c9b6-4c5e-8321-6a70d347706b/kpts.pickle \\\n --arg_pseudopotentials=/tmp/dsu142t1/stg47b4b44d-c69c-4629-aaed-0e15bf14426e/pseudopotentials.pickle \\\n --arg_structure=/tmp/dsu142t1/stg0e5b3c6e-a315-4eaf-9a66-dbf5669eaada/s_0.pickle\n\u001b[1;30mINFO\u001b[0m [job get_dict_20] completed success\n\u001b[1;30mINFO\u001b[0m [step get_dict_20] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_dict_23\n\u001b[1;30mINFO\u001b[0m [step get_dict_23] start\n\u001b[1;30mINFO\u001b[0m [job get_dict_23] /tmp/9v59c4vd$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/2mc_byfs/stgace4b178-c0ff-406e-95ac-754318766886/smearing.pickle \\\n --arg_calculation=/tmp/2mc_byfs/stg793df1d8-8f8d-4bb3-9c7c-243a48dc494b/calculation_1.pickle \\\n --arg_kpts=/tmp/2mc_byfs/stga852e76a-885f-4e39-8ab4-ecebf95c656f/kpts.pickle \\\n --arg_pseudopotentials=/tmp/2mc_byfs/stgd06acdff-6e0c-476c-be57-1f21b2d03711/pseudopotentials.pickle \\\n --arg_structure=/tmp/2mc_byfs/stg02210df5-d3e2-42b8-a75e-47e2f2c9f274/s_1.pickle\n\u001b[1;30mINFO\u001b[0m [job get_dict_23] completed success\n\u001b[1;30mINFO\u001b[0m [step get_dict_23] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_dict_29\n\u001b[1;30mINFO\u001b[0m [step get_dict_29] start\n\u001b[1;30mINFO\u001b[0m [job get_dict_29] /tmp/q_svsf9g$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/8u8e7db4/stgbba611dc-3b23-4653-a293-f71d492a42af/smearing.pickle \\\n --arg_calculation=/tmp/8u8e7db4/stge9a32972-6649-4eb5-abd7-9db61d929d2d/calculation_1.pickle \\\n --arg_kpts=/tmp/8u8e7db4/stg02098ea5-7487-4a66-bb77-99a6336d7b71/kpts.pickle \\\n --arg_pseudopotentials=/tmp/8u8e7db4/stg571b6259-4599-4c39-8cb2-79a70feb78b8/pseudopotentials.pickle \\\n --arg_structure=/tmp/8u8e7db4/stg64c55a4d-8af6-40bf-abfd-3ec8a690b90b/s_4.pickle\n\u001b[1;30mINFO\u001b[0m [job get_dict_29] completed success\n\u001b[1;30mINFO\u001b[0m [step get_dict_29] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_dict_27\n\u001b[1;30mINFO\u001b[0m [step get_dict_27] start\n\u001b[1;30mINFO\u001b[0m [job get_dict_27] /tmp/vgcwzb84$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_dict \\\n --arg_smearing=/tmp/xf32x1ld/stg3cbb6408-2383-49d0-9476-ef3fbee7c165/smearing.pickle \\\n --arg_calculation=/tmp/xf32x1ld/stgeb36ed3b-b442-43ea-b14d-1397875e8839/calculation_1.pickle \\\n --arg_kpts=/tmp/xf32x1ld/stge8854d32-32f3-4a90-8e66-8ce7b1160d8d/kpts.pickle \\\n --arg_pseudopotentials=/tmp/xf32x1ld/stg0dba3c44-e6af-46a9-9f00-c08969392ede/pseudopotentials.pickle \\\n --arg_structure=/tmp/xf32x1ld/stg4223a500-1f85-4651-9130-bd1629008770/s_3.pickle\n\u001b[1;30mINFO\u001b[0m [job get_dict_27] completed success\n\u001b[1;30mINFO\u001b[0m [step get_dict_27] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step calculate_qe_3\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_3] start\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_3] /tmp/v2e_84tz$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/n81xp6_a/stgf98a9803-3329-400e-a03c-02b51d3f3c97/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/n81xp6_a/stgd93a737b-2598-4fb8-bdcb-bb1d2048116f/result.pickle \\\n --arg_working_directory=/tmp/n81xp6_a/stg5ad2a52b-2844-45e4-b227-3d835efa01f6/working_directory_1.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09459] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_3] Max memory used: 240MiB\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_3] completed success\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_3] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step calculate_qe_4\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_4] start\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_4] /tmp/tgwdkhxx$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/l5mihx97/stgf73a20aa-bd97-46af-a923-b8d776da2b98/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/l5mihx97/stgb8c3fcbe-9277-46c4-b8a2-a1171a292580/result.pickle \\\n --arg_working_directory=/tmp/l5mihx97/stg3ae0b93a-2e0b-4f1b-b079-985ea6bbaa72/working_directory_2.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09533] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_4] Max memory used: 213MiB\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_4] completed success\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_4] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step calculate_qe_5\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_5] start\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_5] /tmp/d36x49f6$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/nvm7fsf5/stg384bab04-833c-43f2-bd00-f4b5a5c93eec/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/nvm7fsf5/stg9aa51e21-01b0-4157-b73f-a3feee8a917a/result.pickle \\\n --arg_working_directory=/tmp/nvm7fsf5/stg02ef6345-c598-4127-95c7-e92d0a977c40/working_directory_3.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09611] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_5] Max memory used: 243MiB\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_5] completed success\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_5] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step calculate_qe_6\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_6] start\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_6] /tmp/eiyg8it6$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/708xmf3f/stg791b1717-cfb1-4569-ba75-d648a84e0a72/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/708xmf3f/stg9ebf7054-080c-4173-a8b9-2247d01a0068/result.pickle \\\n --arg_working_directory=/tmp/708xmf3f/stg1afccee5-d9b6-46ed-bc2c-fa36bdb8013d/working_directory_4.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09688] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_6] Max memory used: 245MiB\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_6] completed success\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_6] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step calculate_qe_7\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_7] start\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_7] /tmp/qjgxwxnd$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/g3sp392q/stg0a51a8da-c7a7-4d90-a49c-1817002a62a6/workflow.py \\\n --function=workflow.calculate_qe \\\n --arg_input_dict=/tmp/g3sp392q/stg67e73a60-3cc6-46dd-b009-cfc0034aec2c/result.pickle \\\n --arg_working_directory=/tmp/g3sp392q/stgf3d265db-820b-4af3-8975-0a6faa0d7a67/working_directory_5.pickle\n[jupyter-pythonworkflow-fl--x---d7231032:09770] mca_base_component_repository_open: unable to open mca_btl_openib: librdmacm.so.1: cannot open shared object file: No such file or directory (ignored)\nNote: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_7] Max memory used: 244MiB\n\u001b[1;30mINFO\u001b[0m [job calculate_qe_7] completed success\n\u001b[1;30mINFO\u001b[0m [step calculate_qe_7] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_list_31\n\u001b[1;30mINFO\u001b[0m [step get_list_31] start\n\u001b[1;30mINFO\u001b[0m [job get_list_31] /tmp/mnovytq_$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_list \\\n --arg_0=/tmp/bz0ntajx/stgdab2f59b-b118-44c4-96ef-b4639edaee4e/energy.pickle \\\n --arg_2=/tmp/bz0ntajx/stg2e9bec4d-1824-4152-b003-aab0eac204cd/energy.pickle \\\n --arg_1=/tmp/bz0ntajx/stg9511563e-9784-48a9-8d81-74ade3a084c1/energy.pickle \\\n --arg_4=/tmp/bz0ntajx/stg17845efd-e335-421e-9891-6e627254edb4/energy.pickle \\\n --arg_3=/tmp/bz0ntajx/stg051fa3bc-832a-4d70-b9fb-b9948f51904d/energy.pickle\n\u001b[1;30mINFO\u001b[0m [job get_list_31] completed success\n\u001b[1;30mINFO\u001b[0m [step get_list_31] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step get_list_30\n\u001b[1;30mINFO\u001b[0m [step get_list_30] start\n\u001b[1;30mINFO\u001b[0m [job get_list_30] /tmp/vc0dzppy$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --function=python_workflow_definition.shared.get_list \\\n --arg_0=/tmp/mejux8b1/stg71e3a6a9-c56b-4be8-8720-0ca90ac40513/volume.pickle \\\n --arg_2=/tmp/mejux8b1/stg429d498d-cf56-4561-a653-8048c1213851/volume.pickle \\\n --arg_1=/tmp/mejux8b1/stgee1a0001-20f2-45de-ac4e-77b0d0c59957/volume.pickle \\\n --arg_4=/tmp/mejux8b1/stg0de2a6dc-57cd-4a0c-9637-ef2e63e96fce/volume.pickle \\\n --arg_3=/tmp/mejux8b1/stga4551f25-d5f6-47c6-92ea-452cf694a286/volume.pickle\n\u001b[1;30mINFO\u001b[0m [job get_list_30] completed success\n\u001b[1;30mINFO\u001b[0m [step get_list_30] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] starting step plot_energy_volume_curve_8\n\u001b[1;30mINFO\u001b[0m [step plot_energy_volume_curve_8] start\n\u001b[1;30mINFO\u001b[0m [job plot_energy_volume_curve_8] /tmp/j4v3j3mt$ python \\\n -m \\\n python_workflow_definition.cwl \\\n --workflowfile=/tmp/_67k4bu3/stg5be48d76-8a92-4b02-b3d6-7bdb4c35c9ce/workflow.py \\\n --function=workflow.plot_energy_volume_curve \\\n --arg_volume_lst=/tmp/_67k4bu3/stgd43e71b0-d16b-4eb2-8d14-83924dafd7d6/result.pickle \\\n --arg_energy_lst=/tmp/_67k4bu3/stg43eeacdd-5042-4f2d-8efc-d8e0d5fe1c14/result.pickle\n\u001b[1;30mINFO\u001b[0m [job plot_energy_volume_curve_8] Max memory used: 112MiB\n\u001b[1;30mINFO\u001b[0m [job plot_energy_volume_curve_8] completed success\n\u001b[1;30mINFO\u001b[0m [step plot_energy_volume_curve_8] completed success\n\u001b[1;30mINFO\u001b[0m [workflow ] completed success\n{\n \"result_file\": {\n \"location\": \"file:///home/jovyan/example_workflows/quantum_espresso/result.pickle\",\n \"basename\": \"result.pickle\",\n \"class\": \"File\",\n \"checksum\": \"sha1$dbc1aaddc8b7343d6d33b34edcf608b8f8801918\",\n \"size\": 4,\n \"path\": \"/home/jovyan/example_workflows/quantum_espresso/result.pickle\"\n }\n}\u001b[1;30mINFO\u001b[0m Final process status is success\n" } ], "execution_count": 6 }, + { + "cell_type": "markdown", + "id": "7ddd08ec", + "source": "`cwltool` runs one step per PWD function node - `get_bulk_structure`, the `vc-relax` `calculate_qe`, `generate_structures`, the five strained `scf` `calculate_qe` steps and finally `plot_energy_volume_curve` - each reading its input pickles and writing its output pickle(s), exactly mirroring the fan-out of one relax followed by five parallel SCF calculations seen in the other notebooks. The workflow's final output is declared as `result_file`, the pickle produced by the last step.", + "metadata": {} + }, { "id": "2942dbba-ea0a-4d20-be5c-ed9992d09ff8", "cell_type": "code", @@ -103,6 +127,12 @@ ], "execution_count": 7 }, + { + "cell_type": "markdown", + "id": "e9ec76ee", + "source": "`plot_energy_volume_curve` has no return value, so `result_file` unpickles to `None` here - that is expected. The actual output of this workflow is the `evcurve.png` plot, written as a side effect into the working directory of the last CWL step.", + "metadata": {} + }, { "id": "60e909ee-d0d0-4bd1-81c8-dd5274ae5834", "cell_type": "code", @@ -114,4 +144,4 @@ "execution_count": null } ] -} +} \ No newline at end of file diff --git a/example_workflows/quantum_espresso/executorlib.ipynb b/example_workflows/quantum_espresso/executorlib.ipynb index b1f6069..e4ebd6f 100644 --- a/example_workflows/quantum_espresso/executorlib.ipynb +++ b/example_workflows/quantum_espresso/executorlib.ipynb @@ -4,17 +4,13 @@ "cell_type": "markdown", "id": "be2d61b0-0d47-4349-b4b0-1b767c961644", "metadata": {}, - "source": [ - "# executorlib" - ] + "source": "# executorlib\n\nThis notebook builds the Quantum Espresso energy-volume-curve workflow with [`executorlib`](https://github.com/pyiron/executorlib), a lightweight `concurrent.futures`-style executor. Tasks are submitted individually with `.submit()`, and the resulting graph is exported as a PWD `workflow.json` as a side effect. The second half of the notebook loads that same JSON with `aiida-workgraph`, `jobflow` and `pyiron_base`/`pyiron_workflow`." }, { "cell_type": "markdown", "id": "0bad2a57-1bd2-4837-94fe-f8c60e211fae", "metadata": {}, - "source": [ - "## Define workflow with executorlib" - ] + "source": "## Define workflow with executorlib\n\n`SingleNodeExecutor` runs submitted functions in local worker processes. Passing `export_workflow_filename` makes it record every `.submit()` call and its dependencies and write them out as `workflow_json_filename` once the `with` block exits - no separate `write_workflow_json` call is needed, unlike the other engines." }, { "cell_type": "code", @@ -57,6 +53,12 @@ "pseudopotentials = {\"Al\": \"Al.pbe-n-kjpaw_psl.1.0.0.UPF\"}" ] }, + { + "cell_type": "markdown", + "id": "9ce84a62", + "source": "Each `exe.submit()` call schedules one function call and returns a `Future`. Since `calculate_qe` returns a dict, `get_item_from_future` extracts a single key (e.g. `\"structure\"`, `\"energy\"`, `\"volume\"`) from a future's eventual result so it can be passed as an argument to the next `submit()` call. The relaxed structure is strained into `number_of_strains` (5) structures by `generate_structures`, and each strained structure is submitted as its own independent `scf` `calculate_qe` call - the fan-out that produces the 5 points of the energy-volume curve.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 5, @@ -688,9 +690,7 @@ "cell_type": "markdown", "id": "7d75a2f6-6fad-49c8-bd29-37cca1b84441", "metadata": {}, - "source": [ - "## Load Workflow with aiida" - ] + "source": "## Load Workflow with aiida\n\n`load_workflow_json` reconstructs the graph as an `aiida-workgraph` `WorkGraph`. To demonstrate that the loaded graph can still be edited like one built natively with `aiida-workgraph`, the lattice constant `a` is overridden (4.04 -> 4.05) before running it with `wg.run()`." }, { "cell_type": "code", @@ -828,9 +828,7 @@ "cell_type": "markdown", "id": "c4f5c047-c6da-4b54-9007-415faca7a448", "metadata": {}, - "source": [ - "## Load Workflow with jobflow" - ] + "source": "## Load Workflow with jobflow\n\nThe same JSON is loaded as a `jobflow` `Flow` and executed locally with `run_locally`, again after overriding the lattice constant `a`." }, { "cell_type": "code", @@ -966,9 +964,7 @@ "cell_type": "markdown", "id": "9a956c9e", "metadata": {}, - "source": [ - "## Load Workflow with pyiron_base" - ] + "source": "## Load Workflow with pyiron_base\n\n`pyiron_base`'s loader turns the graph into a list of delayed jobs. `.draw()` visualizes the dependency graph and `.pull()` executes the whole chain (relax + 5 strained SCF calculations), after the lattice constant override." }, { "cell_type": "code", @@ -1774,9 +1770,7 @@ "cell_type": "markdown", "id": "406b0429e65b9760", "metadata": {}, - "source": [ - "## Load Workflow with pyiron_workflow" - ] + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the graph is loaded into a `pyiron_workflow` `Workflow`, its lattice constant overridden, drawn with `.draw()`, and executed with `.run()`." }, { "cell_type": "code", @@ -5561,4 +5555,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file diff --git a/example_workflows/quantum_espresso/jobflow.ipynb b/example_workflows/quantum_espresso/jobflow.ipynb index 4e04b74..d8b1e1f 100644 --- a/example_workflows/quantum_espresso/jobflow.ipynb +++ b/example_workflows/quantum_espresso/jobflow.ipynb @@ -181,9 +181,7 @@ "outputs_hidden": false } }, - "source": [ - "Next, for the \"Energy vs. Volume\" curve, we meed to specify the number of strained structures and save them into a list object. For each of the strained structures, we will carry out a QE calculation." - ] + "source": "Next, for the \"Energy vs. Volume\" curve, we need to specify the number of strained structures and save them into a list object. This is the fan-out point of the workflow: for each of the `number_of_strains` (5) strained structures, an independent `scf` QE calculation is carried out below." }, { "cell_type": "code", @@ -1027,11 +1025,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": [ - "## Load Workflow with pyiron_base\n", - "\n", - "And we can repeat the same process using `pyiron`." - ] + "source": "## Load Workflow with pyiron_base\n\nAnd we can repeat the same process using `pyiron_base`: `load_workflow_json` turns the graph into a list of delayed jobs, `.draw()` visualizes the dependency graph, and `.pull()` executes the whole chain (relax + 5 strained SCF calculations) after overriding the lattice constant `a`." }, { "cell_type": "code", @@ -1924,9 +1918,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": [ - "## Load Workflow with pyiron_workflow" - ] + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the same JSON is loaded into a `pyiron_workflow` `Workflow`; nodes become graph nodes reachable via attribute access (e.g. `wf.get_bulk_structure`). The lattice constant `a` is overridden again before drawing the graph with `.draw()` and executing it with `.run()`." }, { "cell_type": "code", @@ -5744,4 +5736,4 @@ }, "nbformat": 4, "nbformat_minor": 4 -} +} \ No newline at end of file diff --git a/example_workflows/quantum_espresso/pyiron_base.ipynb b/example_workflows/quantum_espresso/pyiron_base.ipynb index 38703d8..1369193 100644 --- a/example_workflows/quantum_espresso/pyiron_base.ipynb +++ b/example_workflows/quantum_espresso/pyiron_base.ipynb @@ -4,17 +4,13 @@ "cell_type": "markdown", "id": "be2d61b0-0d47-4349-b4b0-1b767c961644", "metadata": {}, - "source": [ - "# pyiron" - ] + "source": "# pyiron\n\nThis notebook builds the Quantum Espresso energy-volume-curve workflow with [`pyiron_base`](https://github.com/pyiron/pyiron_base). The `job` decorator turns plain functions into delayed pyiron jobs that are only executed once `.pull()` is called on the final result, and the whole chain of jobs is exported as a PWD `workflow.json`. The second half of the notebook loads that same JSON with `aiida-workgraph`, `jobflow` and `pyiron_workflow`." }, { "cell_type": "markdown", "id": "0bad2a57-1bd2-4837-94fe-f8c60e211fae", "metadata": {}, - "source": [ - "## Define workflow with pyiron_base" - ] + "source": "## Define workflow with pyiron_base\n\nWe import the `job` decorator from `pyiron_base` and the PWD writer for `pyiron_base`, together with the Quantum Espresso functions from `workflow.py`." }, { "cell_type": "code", @@ -59,6 +55,12 @@ "workflow_json_filename = \"pyiron_base_qe.json\"" ] }, + { + "cell_type": "markdown", + "id": "d73d32d1", + "source": "`calculate_qe` returns a dict with `energy`, `volume` and `structure` keys. `job(..., output_key_lst=[...])` tells `pyiron_base` which keys to expose as separate delayed outputs (e.g. `calc_mini.output.structure`), so a single downstream job can depend on just one of them instead of the whole dict.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 4, @@ -123,6 +125,12 @@ ")" ] }, + { + "cell_type": "markdown", + "id": "1e0aace0", + "source": "`generate_structures` fans out the relaxed structure into `number_of_strains` (5) strained structures. Since `pyiron_base` builds the graph lazily, it cannot know how many items a delayed dict result will contain, so `list_length` is passed explicitly - this lets `structure_lst` be iterated with `enumerate()` below to create one independent `scf` `calculate_qe` job per strained structure, before anything has actually run.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 8, @@ -179,6 +187,12 @@ ")" ] }, + { + "cell_type": "markdown", + "id": "e7e789dd", + "source": "`write_workflow_json` only needs the single terminal delayed object (`plot`, the plotting job) - `pyiron_base` walks its dependency chain backwards to discover the full graph, unlike `jobflow`'s `Flow` which needs an explicit list of jobs.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 11, @@ -773,9 +787,7 @@ "cell_type": "markdown", "id": "7d75a2f6-6fad-49c8-bd29-37cca1b84441", "metadata": {}, - "source": [ - "## Load Workflow with aiida" - ] + "source": "## Load Workflow with aiida\n\n`load_workflow_json` reconstructs the graph as an `aiida-workgraph` `WorkGraph`. The lattice constant `a` is overridden (4.04 -> 4.05) to show the loaded graph can still be edited like one built natively with `aiida-workgraph`, before running it with `wg.run()`." }, { "cell_type": "code", @@ -925,9 +937,7 @@ "cell_type": "markdown", "id": "c4f5c047-c6da-4b54-9007-415faca7a448", "metadata": {}, - "source": [ - "## Load Workflow with jobflow" - ] + "source": "## Load Workflow with jobflow\n\nThe same JSON is loaded as a `jobflow` `Flow` and executed locally with `run_locally`, again after overriding the lattice constant `a`." }, { "cell_type": "code", @@ -1163,9 +1173,7 @@ "cell_type": "markdown", "id": "406b0429e65b9760", "metadata": {}, - "source": [ - "## Load Workflow with pyiron_workflow" - ] + "source": "## Load Workflow with pyiron_workflow\n\nFinally, the graph is loaded into a `pyiron_workflow` `Workflow`, its lattice constant overridden, drawn with `.draw()`, and executed with `.run()`." }, { "cell_type": "code", @@ -4990,4 +4998,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file diff --git a/example_workflows/quantum_espresso/pyiron_workflow.ipynb b/example_workflows/quantum_espresso/pyiron_workflow.ipynb index 9f7e084..687be88 100644 --- a/example_workflows/quantum_espresso/pyiron_workflow.ipynb +++ b/example_workflows/quantum_espresso/pyiron_workflow.ipynb @@ -4,10 +4,7 @@ "cell_type": "markdown", "id": "760acc89-8c02-4bc9-a8f6-2572506b7085", "metadata": {}, - "source": [ - "# pyiron_workflow\n", - "## Define workflow with pyiron_workflow" - ] + "source": "# pyiron_workflow\n## Define workflow with pyiron_workflow\n\nThis notebook builds the Quantum Espresso energy-volume-curve workflow with [`pyiron_workflow`](https://github.com/pyiron/pyiron_workflow). Plain functions are converted to graph nodes with `to_function_node`, assembled into a `Workflow` via attribute access, and exported as a PWD `workflow.json`. The second half of the notebook loads that same JSON with `aiida-workgraph`, `jobflow` and `pyiron_base`." }, { "cell_type": "code", @@ -42,6 +39,12 @@ ")" ] }, + { + "cell_type": "markdown", + "id": "2d3f3ba6", + "source": "`pyiron_workflow` has no built-in equivalent of the `get_list` helper the other engines use to collect several outputs into a list, so `get_values_from_dict` is defined here as a custom node with `as_function_node` to turn a dict of energies/volumes into a plain list for the final plot.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 3, @@ -83,6 +86,12 @@ "plot_energy_volume_curve = to_function_node(\"plot_energy_volume_curve\", _plot_energy_volume_curve, \"plot_energy_volume_curve\", validate_output_labels=False)" ] }, + { + "cell_type": "markdown", + "id": "be84f115", + "source": "`to_function_node` wraps each plain Python function into a `pyiron_workflow` node type. `plot_energy_volume_curve` passes `validate_output_labels=False` since it has no return value, so its output cannot be matched to an explicit output label.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 6, @@ -107,6 +116,12 @@ "wf.pseudopotentials = {\"Al\": \"Al.pbe-n-kjpaw_psl.1.0.0.UPF\"}" ] }, + { + "cell_type": "markdown", + "id": "c117be40", + "source": "Nodes are added to the `Workflow` by assigning their result to an attribute of `wf` (e.g. `wf.structure = get_bulk_structure(...)`), which both registers the node and gives it a name in the graph. Since `calculate_qe` expects a single `input_dict` argument, `inputs_to_dict` is used to bundle the structure, pseudopotentials, k-points, calculation type and smearing into one dict-valued node before the first (`vc-relax`) QE calculation.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 8, @@ -164,6 +179,12 @@ ")" ] }, + { + "cell_type": "markdown", + "id": "6d21ce04", + "source": "`generate_structures` fans out the relaxed structure into `number_of_strains` (5) strained structures (`wf.structure_lst[\"s_0\"]` ... `[\"s_4\"]`). The loop below builds one `input_dict` and one independent `scf` `calculate_qe` node per strained structure, so five QE calculations run to sample the energy-volume curve.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 11, @@ -207,6 +228,12 @@ " job_strain_lst.append(getattr(wf, \"calc_strain_\" + str(i)))" ] }, + { + "cell_type": "markdown", + "id": "edc600a3", + "source": "The `energy` and `volume` outputs of the five strained calculations are bundled with `inputs_to_dict` and then flattened into two plain lists with the `get_values_from_dict` node defined above, ready to be passed to `plot_energy_volume_curve`.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 13, @@ -246,6 +273,12 @@ "wf.plot = plot_energy_volume_curve(volume_lst=wf.volume_lst, energy_lst=wf.energy_lst)" ] }, + { + "cell_type": "markdown", + "id": "e5408a80", + "source": "`.draw()` renders the assembled node graph, and `write_workflow_json` exports it - using `wf.graph_as_dict` - to the PWD `workflow.json` format so it can be loaded by another engine.", + "metadata": {} + }, { "cell_type": "code", "execution_count": 15, @@ -4688,9 +4721,7 @@ "cell_type": "markdown", "id": "fef7107a-5af3-4434-ae4c-a8d45e1d9b61", "metadata": {}, - "source": [ - "## Load Workflow with aiida" - ] + "source": "## Load Workflow with aiida\n\n`load_workflow_json` reconstructs the graph as an `aiida-workgraph` `WorkGraph`. The lattice constant `a` is overridden (4.04 -> 4.05) to show the loaded graph can still be edited like one built natively with `aiida-workgraph`, before running it with `wg.run()`." }, { "cell_type": "code", @@ -4840,9 +4871,7 @@ "cell_type": "markdown", "id": "9226d966-d90a-4ab2-9776-df1b50bd6a49", "metadata": {}, - "source": [ - "## Load Workflow with jobflow" - ] + "source": "## Load Workflow with jobflow\n\nThe same JSON is loaded as a `jobflow` `Flow` and executed locally with `run_locally`, again after overriding the lattice constant `a`." }, { "cell_type": "code", @@ -5084,9 +5113,7 @@ "cell_type": "markdown", "id": "e0c78db2-2ead-45d3-92ed-d087e19952ba", "metadata": {}, - "source": [ - "## Load Workflow with pyiron_base" - ] + "source": "## Load Workflow with pyiron_base\n\nFinally, `pyiron_base`'s loader turns the graph into a list of delayed jobs. `.draw()` visualizes the dependency graph and `.pull()` executes the whole chain (relax + 5 strained SCF calculations), after overriding the lattice constant `a`." }, { "cell_type": "code", @@ -6012,4 +6039,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file diff --git a/example_workflows/quantum_espresso/universal_workflow.ipynb b/example_workflows/quantum_espresso/universal_workflow.ipynb index 12568d2..6e834b7 100644 --- a/example_workflows/quantum_espresso/universal_workflow.ipynb +++ b/example_workflows/quantum_espresso/universal_workflow.ipynb @@ -23,12 +23,12 @@ "cells": [ { "cell_type": "markdown", - "source": "# Load Quantum Espresso Energy Volume Curve Workflow", + "source": "# Load Quantum Espresso Energy Volume Curve Workflow\n\nThis notebook loads the same, already-exported [`workflow.json`](workflow.json) with six different backends in turn - `aiida-workgraph`, `executorlib`, `jobflow`, `pyiron_base`, `pyiron_workflow` and plain Python - to demonstrate that the PWD format is interoperable regardless of which engine originally produced it.", "metadata": {} }, { "cell_type": "markdown", - "source": "## Plot", + "source": "## Plot\n\n`python_workflow_definition.plot.plot` renders the raw node/edge structure of `workflow.json` as a graph, independent of any workflow engine - useful to inspect the relax -> strain -> 5x scf -> plot pipeline before executing it.", "metadata": {} }, { @@ -60,7 +60,7 @@ }, { "cell_type": "markdown", - "source": "## Aiida ", + "source": "## Aiida\n\n`load_workflow_json` reconstructs the graph as an `aiida-workgraph` `WorkGraph`, after connecting to the local AiiDA profile with `load_profile()`. `wg.run()` executes the relax step followed by the 5 parallel strained SCF calculations and the final plot.", "metadata": {} }, { @@ -138,7 +138,7 @@ }, { "cell_type": "markdown", - "source": "## executorlib", + "source": "## executorlib\n\nHere `load_workflow_json` takes a live `exe` (a `SingleNodeExecutor`) and submits the whole graph to it, returning a `Future` for the final result; `.result()` blocks until every node - including the 5 fanned-out SCF calculations - has finished executing.", "metadata": {} }, { @@ -185,7 +185,7 @@ }, { "cell_type": "markdown", - "source": "## jobflow", + "source": "## jobflow\n\nThe graph is loaded as a `jobflow` `Flow` and executed locally with `run_locally`, which runs the relax job, the 5 strained SCF jobs and the plotting job in dependency order.", "metadata": {} }, { @@ -318,7 +318,7 @@ }, { "cell_type": "markdown", - "source": "## pyiron_base", + "source": "## pyiron_base\n\n`load_workflow_json` turns the graph into a list of delayed pyiron jobs; `.draw()` visualizes the dependency graph (including the fan-out into 5 strained SCF jobs) and `.pull()` executes the whole chain.", "metadata": {} }, { @@ -434,7 +434,7 @@ { "metadata": {}, "cell_type": "markdown", - "source": "## Load Workflow with pyiron_workflow" + "source": "## Load Workflow with pyiron_workflow\n\nThe graph is loaded into a `pyiron_workflow` `Workflow`, drawn with `.draw()`, and executed with `.run()`." }, { "metadata": {}, @@ -466,7 +466,7 @@ }, { "cell_type": "markdown", - "source": "## Python", + "source": "## Python\n\n`python_workflow_definition.purepython.load_workflow_json` is a minimal reference implementation with no external workflow engine: it topologically sorts the nodes by their edges and calls each function directly in plain Python, returning the final result.", "metadata": {} }, { @@ -502,4 +502,4 @@ "execution_count": 19 } ] -} +} \ No newline at end of file