Repository navigation
Sphinx gallery scraper is not parallel-safe #4959
Description
Activity
For example this comes up frequently in scikit-image because we build the gallery in parallel e.g., see https://github.com/scikit-image/scikit-image/actions/runs/12680820918/job/35343334532#step:6:2316:
Details
Extension error: Here is a summary of the problems encountered when running the examples: Unexpected failing examples (1): ../examples/applications/plot_3d_interaction.py failed leaving traceback: Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/sphinx_gallery/scrapers.py", line 387, in save_figures rst = scraper(block, block_vars, gallery_conf) File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/plotly/io/_sg_scraper.py", line 59, in plotly_sg_scraper shutil.move(png, this_image_path_png) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/shutil.py", line 876, in move copy_function(src, real_dst) ~~~~~~~~~~~~~^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/shutil.py", line 468, in copy2 copyfile(src, dst, follow_symlinks=follow_symlinks) ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/shutil.py", line 260, in copyfile with open(src, 'rb') as fsrc: ~~~~^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/Users/runner/work/scikit-image/scikit-image/doc/examples/applications/plot_3d_interaction.png'Thanks, @gvwilson.
Unfortunately, we haven't been able to get our CI to work with the new Kaleido, but I'll try the latest version today. Once a new rc of plotly 6 is out, we'll attempt to switch to that, which may also resolve our problem.
Sphinx Gallery now allows parallel execution of gallery examples. The plotly Orca renderer correctly outputs PNGs with the example filename as root, e.g.,
plot_my_example.png.However, the scraper grabs all PNGs from the example directory. So, if other plotly examples execute simultaneously, those PNGs may be "stolen" by the scraper.
To fix the issue, the scraper would have to be modified from:
to
Note the more selective filter on the globbed PNGs.