Repository navigation
Request for Data Preparation Scripts #16
Description
Dear Authors,
Thank you for your great work. I wanted to ask whether you plan to release the scripts for WSI data preparation (segmentation, patching, and feature extraction). I have been trying to follow the data processing pipelines provided in the CLAM repo, but I encountered a few dataset-specific issues:
Segmentation and filtering hyperparameters.
It seems that the optimal hyperparameters can vary across datasets. For example, the CLAM repo provides presets for TCGA, BWH_biopsy, and BWH_resection, but not for others. I noticed that using unsuitable hyperparameters sometimes leads to stalled feature extraction jobs and zombie processes without raising explicit errors.
Thank you very much for your time and consideration.
Best,
Wei
Activity
wlin-at commentedon Aug 29, 2025
I think I figured out the reason of the feature extraction job stalling. The Dataloader num_workers is harded as 8 in the script.
PyTorch DataLoader with multiple workers (num_workers > 0) is notorious for stalling silently, especially when the dataset (Whole_Slide_Bag_FP) uses libraries like OpenSlide and h5py that aren’t fully multiprocessing-safe. The worker processes get stuck waiting on locks (e.g., HDF5 file handles).
wlin-at commentedon Aug 30, 2025
When processing WSI datasets (e.g., BRACS), I noticed that the default segmentation and filter parameters sometimes result in zero valid tissue contours being detected. In these cases, the function WSI_object.process_contours(**kwargs) does not generate an .h5 patch file for the corresponding slide.
This leads to situations where:
Could you share the hyperparameters used for segmentation and patching for these WSI datasets?
Thanks in advance!