Using QuPath 0.2.0-m11 I’m adding jpg images via a file list. Some of them, seemingly randomly, appear with their name as “Image null”. They also have no thumbnail and have image.hasImageData() return false. Despite this you can click on them and view open like any other image.
Another perhaps key detail is that inside the QuPath project folder the entries corresponding to these images are missing. So we’ll have for examples images 99 and 101 having folders containing the thumbnail, a data.qpdata file, and a summary.json. But this whole folder seems missing for these “Image null” images.
Thanks, I haven’t heard of this problem (at least not recently) but we’ll try to reproduce it.
Out of curiosity, do the same images have problems if you re-import them? If so, I’d suspect some file issue. If not, I suspect a multithreading issue.
It appears that it’s random. So yeah multithreading or some OS issue sounds likely. If I make a smaller image list, including some for which this happened and just load that then they all appear to work fine.
Thanks, I’ve just imported >800 JPEGs and TIFFs (separately) multiple times and could see no problems with either. I did it through drag & drop, but I don’t think it should make a difference.
Can you give any more details to help me reproduce it? For example:
- What operating system are you using?
- Do you change any settings during the import? E.g. provider, rotate, image type
- Approximately what size are the JPEGs (in terms of pixels)?
- Approximately how big is the smallest batch that results in problems?
- When you open one image, what is the ‘Server type’ under the ‘Image’ tab?
- Are the images local on your computer, or are some on a server elsewhere?
- OS:Ubuntu 19.10
- Settings: Occurs with defaults, of if changing imagetype to HE.
- JPEG size: ~12000 x 12000 pixels +/- 4 thousand-ish
- Smallest problematic batch: Lowest I’ve seen is 38, but seems semi-random. Sometimes 40 they will all be ok. Sometimes it has 2 nulls.
- Server type: For "Image null"s it’s “Bio-Formats”, otherwise it’s "Generated pyramid (Bio-Formats).
- Storage: Local ext4 drive mounted via fstab
I also get warnings of the sort:
[7546.708s][warning][gc,alloc] project-import56: Retried waiting for GCLocker too often allocating 29628860 words
Which seem to occur during imports where this happens. The number of warnings seems unrelated to the number of images which come through as null.
Ah ok, the JPEGs I tested were much smaller. Do see any errors reported under View → Show log?
And how much RAM is available to QuPath? And how many CPUs…?
Having just checked the code again, QuPath is using a cached thread pool to load images more quickly in parallel - but this does not inherently limit the number of threads. Generally that doesn’t really matter, since reading a small / pyramidal image doesn’t use a huge amount of memory. But for reading multiple large, non-pyramidal images I guess that it may be problematic, since the entire image will need to be read (at least temporarily) to be able to generate the thumbnail.
6 physical cores (12 virtual), QuPath appears to be allowed access to 10. I have 32GB of physical RAM and QuPath reports the current maximum memory as 15.65GB (-Xmx16384).
No message in the log, but the warnings I mentioned are going to the terminal from which QuPath was started.
Thanks, I can now replicate the issue (including the GCLocker message) on my Mac is I try to import JPEGs of a similar size. It is also horrendously slow.
This commit now limits the thread pool, and that’s enough for the import to work correctly for me. It also means that the QuPath preferences can be used to restrict the threadpool size even further if needed. Still slow, but it gets there in the end.
Would it be feasible to convert your images to a pyramidal format? This would be the best way to get reasonable performance with QuPath even once this fix is incorporated. You could potentially do the conversion with VIPS, or even the
convert-ome command line tool in QuPath itself.
That appears to fix the issue! Thank you very much.
I could definitely convert the images yeah, and may do in the future. For now I’m just using QuPath for a fairly straightforward automated task.
Great, thanks for the prompt & detailed responses to help get to the bottom of it quickly!