I need some advice on how to proceed on loading (large) pyramidal microscopy data for deep learning in Python. This is my scenario:
Training: I opened and annotated some czi images in QuPath and use the Tile Exporter to create some 4k (.tif) tiles with raw data and respective annotation. I do the training not at full resolution (20k x 20k, 0.44µm pixel size), but at a lower resolution level ( 3k x 3k, 1.8µm pixel size). I read the .tif tiles with python bio-formats.
Inference: To apply my model to all raw images, I have to open the czi images, downsample them to the correct resolution and run them through my model. I currently use python bioformats for this task, as it appears to be the only python loader that supports pyramidal image data - and I would like to avoid loading the whole image at full resolution, if possible.
Here’s the catch: Using bioformats to read the tiles is massively slower than other python packages (e.g. tifffile, cv2, PIL), and hence a huge bottleneck for the training process. On the other hand, using other readers for the training messes up the data types/color scaling badly so that my model doesn’t work at all in the inference mode. But since bio-formats appears to be the only python package that supports reading pyramidal czi images, I’m tied to using it in this step. The way out is either digging deeper into the statistics and data type conversion of different file readers, ooor simply using the same reader during testing and inference.
TL;DR: I’m looking for a python image reader that’s fast (i.e. faster than bioformats) and supports pyramidal czi images.
Thanks in advance!