Loading images / creating sequence is SLOW - any workaround?

fmp · April 7, 2021, 10:39am

dear all,

In a plugin under development (multiCAFE) using Java/Eclipse, we process stacks of relatively small images (typically 1280 x 720 pixels) to analyze time-lapse data. Loading a stack of 320 images using a “Loader” function takes about 30 s on a reasonably sized computer running under Windows 10 ( AMD Rhyzen 5 with 6 core, 3.6 GHz, 32 Gb, 500 Gb SSD).

Is there a way to make it quicker (besides by calling it within a separate thread)?

Thank you in advance for your help,
Frederic

PS code snippet:

protected Sequence loadSequenceOfImagesFromList(List myListOfFilesNames, boolean testFilenameDecoratedwithLR)
{
seq = null;

	long startTime = System.nanoTime();
	if (seqFileImporter == null) 
	{
		seqFileImporter = Loader.getSequenceFileImporter(myListOfFilesNames.get(0), true);
		long endTime = System.nanoTime();
		System.out.println("*** create file importer - duration: "+((endTime-startTime)/ 1000000000f) + " s");
		startTime = endTime;
	}
	seq = Loader.loadSequence(seqFileImporter, myListOfFilesNames, false);
	long endTime = System.nanoTime();
	System.out.println("loading sequence done - duration: "+((endTime-startTime)/ 1000000000f) + " s");		
	
	
	
	return seq;
}

Stephane · April 9, 2021, 8:42am

Dear Frederic,

The image loading speed is really tied to the image format itself, can you give us more information about the image format you’re using here ? So we can verify if we get the same magnitude of performance here or if there is something wrong. Still Icy internally (almost) always loads images through Bio-format (so it correctly get attached metadata) and it’s true that Bio-Format is generally slow to load or save dataset.
If you don’t use any fancy image format and don’t require the specific OME metadata then it may be better to use the Java Readers here instead.

Best,

– Stephane

fmp · April 9, 2021, 9:01am

Thank you Stephane.

These files are simple jpg files, with not particular information attached to them. They are coming from a program (iSpy.exe), capturing images from a standard webcam in RGB 24 bits.

May be we do not need the bio-format. If so, how can we bypass the standard routines?

Best
Fred

Stephane · April 9, 2021, 1:16pm

I made some tests using 1280x720 RGB JPG images, a timelaps of 300 images takes about 30s to load indeed.Just because of the file metadata parsing from Bio-Formats (not even loading the data) but in your case you don’t need that. So yeah that is slow…
I think using the Java reader will perform much better in that case.
You can try using the ImageUtil.load(..) method for that :

        Sequence seq = new Sequence();
        seq.beginUpdate();
        try
        {
            for (int t = 0; t < filePaths.size(); t++)
            {
                BufferedImage img = ImageUtil.load(filePaths.get(t));

                if (img != null)
                    seq.setImage(t, 0, IcyBufferedImage.createFrom(img));
            }
        }
        finally
        {
            seq.endUpdate();
        }

Best,

– Stephane

fmp · April 9, 2021, 1:33pm

Thank you so much, Stephane. I’ll post how it goes on our computers.

One more question however - I don’t need to have all images available at once in memory. Will this load the images physically or will it just fill in a list of files? This is critical for our application because our timelapse series may run over several days, which means thousand and thousand of images.

Fred

Stephane · April 9, 2021, 1:56pm

In that case you could adapt the code like this :

        Sequence seq = new Sequence();
        seq.setVolatile(true);
        seq.beginUpdate();
        try
        {
            for (int t = 0; t < filePaths.size(); t++)
            {
                BufferedImage img = ImageUtil.load(filePaths.get(t));

                if (img != null)
                {
                    IcyBufferedImage icyImg = IcyBufferedImage.createFrom(img);
                    // enable cachable image
                    icyImg.setVolatile(true);
                    // set image
                    seq.setImage(t, 0, icyImg);
                }
            }
        }
        finally
        {
            seq.endUpdate();
        }

But for sure that will have an impact on the performance as image will be loaded to RAM then written to disk. The advantage of the Loader is that it will load data on demand, here all images will be first loaded in RAM then stored on disk cache… For very large timelaps I’m not sure how much faster that will be in the end.

– Stephane

fmp · April 9, 2021, 2:33pm

Thank you, Stephane. This seems better indeed, and I will try to implement what you propose here.

It might be less than optimal for long data series. In our case, a long experiment runs over 2 weeks, i.e. sampling 1 image per minute during 2 weeks make 20160 images. As 1 image is about 134 Kb, that would make something like 2.6 Gb in the cache? This seems an overkill as most of the time, we analyze only 1 image at a time.

Stephane · April 13, 2021, 7:32am

You’re right, it’s a bit wasteful in that case…
What we really need to make that simpler is to have a javaIO reader which handle basic format when metadata aren’t required so you could still use Loader but providing the JavaIO Reader instead of the Bio-Formats reader. In the meantime we can also try to improve the performance during the metadata parsing process by using multi(threading as the Bio-Format importer support it.

– Stephane

fmp · April 13, 2021, 8:17am

Actually, in our case, the time lapse experiments generate (numerous) stacks of (numerous) images. These timelapse stacks may warrant a different type of “sequence” class as it is a bit different of what you have to deal usually, i.e. large files containing images in several dimensions (T, Z, etc) for which you do not need to handle such “lists of files”.

But even these big files do not need to be loaded in memory at once, as it might suffice to load single images once at a time. This would keep the memory use quite low. It is not possible with the current implementation of the class sequence?

May be the most flexible way for me to go would be to subclass the viewers so that they interface to custom sequence types?

Frederic

fmp · April 13, 2021, 1:29pm

Tested under Eclipse / Icy plug on a list of 432 jpg images (each 135-142 KB):

With Loader: 22.33 s
With non-volatile images: 11.8 s
With volatile images: 11.7 s

***************Code with loader:
SequenceFileImporter seqFileImporter = Loader.getSequenceFileImporter(imagesList.get(0), true);
Sequence seq = Loader.loadSequence(seqFileImporter, imagesList, false);

***************Code with non-volatile images:
Sequence seq = new Sequence();
seq.beginUpdate();
try
{
for (int t = 0; t < imagesList.size(); t++)
{
BufferedImage img = ImageUtil.load(imagesList.get(t));

            if (img != null)
                seq.setImage(t, 0, IcyBufferedImage.createFrom(img));
        }
    }
    finally
    {
        seq.endUpdate();
    }

***************Code with volatile images:
Sequence seq = new Sequence();
seq.setVolatile(true);
seq.beginUpdate();
try
{
for (int t = 0; t < imagesList.size(); t++)
{
BufferedImage img = ImageUtil.load(imagesList.get(t));
if (img != null)
{
IcyBufferedImage icyImg = IcyBufferedImage.createFrom(img);
// enable cachable image
icyImg.setVolatile(true);
// set image
seq.setImage(t, 0, icyImg);
}
}
}
finally
{
seq.endUpdate();
}

Stephane · April 13, 2021, 4:36pm

Thanks for the detailed performance comparison ! And finally this is not that much faster (only a factor of x2)… I wonder if we can just speed up drastically the metadata parsing in case of JPG/PNG files where obviously meta data are not as prominent. I will do some tests and let you know the results !

– Stephane

fmp · April 13, 2021, 4:50pm

Thank you, Stephane.

As I need a Sequence to be defined, I could not use the simplest approach :

 SequenceFileImporter seqFileImporter = Loader.getSequenceFileImporter(imagesList.get(0), true);
 Sequence seq = Loader.loadSequence(seqFileImporter, imagesList, false);

What I am trying to implement now, is to read 1 image, and return the sequence created to the caller, while adding images in the background (code below). Is this acceptable / good practice? Does Sequence send a message to a SequenceListener when we call “endUpdate()”? or should I call seq.dataChanged() to let sequenceListener know that something happened?

Best
Frederic

 public static Sequence loadSequenceFromImagesList(List <String> imagesList) 
 {
	  System.out.println("load list of "+ imagesList.size() +" files "+ imagesList.get(0));
	  SequenceFileImporter seqFileImporter = Loader.getSequenceFileImporter(imagesList.get(0), true);
	  Sequence seq = Loader.loadSequence(seqFileImporter, imagesList.get(0), 0, false);
	  ThreadUtil.bgRun( new Runnable() { 
		@Override public void run() 
		{
			long startTime2 =  System.nanoTime();
			seq.setVolatile(true);
			seq.beginUpdate();
			try
			{
				for (int t=1; t< imagesList.size(); t++)
				{
					BufferedImage img = ImageUtil.load(imagesList.get(t));
					if (img != null)
					{
						IcyBufferedImage icyImg = IcyBufferedImage.createFrom(img);
						// enable cachable image
						icyImg.setVolatile(true);
						// set image
						seq.setImage(t, 0, icyImg);
					}
				}
			}
			finally
			{
				seq.endUpdate();
				System.out.println("end update...............");
			}
			long endTime2 = System.nanoTime();
			System.out.println("loading images for sequence done - duration: "+((endTime2-startTime2)/ 1000000000f) + " s");
		}});	
	return seq;

Stephane · April 13, 2021, 5:24pm

I think i found a workaround In future version of Icy we will avoid reading metadata for image format which do not contains any meaningful as JPG so initial loading should be much faster in that case.
Still the solution you’re proposing is perfectly valid and yes, the endUpdate() will propagate automatically change event to the Sequence so that should work !

Best,

– Stephane