Icy ec-CLEM: image size and forma requirements and workstation config

Hi!

I am a new Icy user who is particularly interested in the ec-CLEM/ec-CLEM Autofinder plug in. I have been using the ec-CLEM for a few months and had some inspiring successes.

I observed multiple instances where ec-CLEM falls into "eternal transformation finding” - its neither finding a pretty straight forward transformation (low magnification grid map in EM vs. that in FM) nor reporting "transformation not found” after some time, but running endlessly. Only program restart with images reload helps. This fall into "eternal transformation finding” could happen a few times in a raw. When reloaded, same areas are usually chosen for correlative points selection and in one of the reloads the correlation works. Is there any suggestion on plugin tweaking for such instances?

Running the ec-CLEM plugin I noticed that the size of images I am trying to correlate matters a lot. In addition, it seems that the performance also depends on the file format - e.g. in a comparison when used .mrc vs. tiff, where the .tiff is produces from the .mrc). Could you advise what would be the best file format to use with the ec-CLEM and what is the file size limitation I should not overcome?

It occurred to me to run the ec-CLEM with a very similar input data on a few different computers. I could not help but notice that the performance differs from one computer to another. All of them are quite powerful iMacs and a PC workstation. As we are currently planning to purchase a new image processing workstation we were wondering if you could give us an advice what hardware parameters are crucial for Icy and ec-CLEM plugin in particular. Do you have any suggestions on configuration?

Thank you,

Sergii

Dear Sergii,

I am inviting @PerrineGilloteaux, the developer of ec-CLEM, to join this thread. I am also inviting @Stephane, regarding questions related to file formats and performances with Icy.

Could you provide some more details regarding

  • the version of Icy you used (bottom right corner of the graphical interface, also in the Output tab of the side panel, next to History and Layer)
  • the version of the ec-CLEM and ec-CLEM autofinder plugins you used (Preferences menu, in Plugin -> Local Plugin)
  • the typical size or range of size of images you are processing
  • the maximum memory you allow Icy to use (Preferences menu, in General, value for “Max memory”)
  • and the amount of RAM you have on the different computers you tested

I add a few screenshots to help you find these information:

Many thanks in advance!
Best regards,
Marion

2 Likes

Dear Marion, Perrine and Stephan,

Thank you for looking into my issue. Answering your questions: We have autoupdate “ON” for Icy and its plug-ins on all computers, so presumably we always run latest versions. All mentioned issues happened in 1.8.6 and 2.1.0.1.

  1. Current Icy version is 2.1.0.1

  2. ec-CLEM version is 1.0.1.5

  3. ec -CLEM Autofinder version is 1.0.0.3

  4. RAM amount. We used 3 computers: a) PC with 64Gb DDR3 RAM (max memory), 32Gb allocated for Icy; b) an iMac with 32Gb DDR3, 31Gb allocated for Icy; c) MacBookPro with 16Gb DDR4 RAM and 12Gb allocated for Icy. PC has SATA7200 spinning drive. Macs are SSD based. PC demonstrated poorest Icy performance among them, whereas iMac - the highest.

  5. File size. Initially we were using *.mrc’s of up to 200 and 900Mb for low- and medium-magnification EM maps, respectively. Those were very heavy to work with, so we optimized binning and file sizes are now less than 50Mb (typically 20-30Mb). We convert them to tiffs and work with tiffs. On the other hand, FM images are usually in the range 400-700Mb. Max intensity projection or slice extraction creates an image of ~50Mb we work with. With these small size files the low-mag map correction usually works well. However, medium-mag correlation still often stalls and goes into "endless transformation mode” described earlier.

Thank you,

Sergii

1 Like

Hi @Sergii
sorry about the delay in answering you, I was on holidays. Thanks for providing this information.
First of all, according to the size of your images and of your computers, you should have no problem, and any problem you are encountering may come from a bug from my side.

Also it sounds like you are using ec-clem autofinder and not the ec-clem plugin, am I correct?

Please find below some answers or hints to your questions (assuming you are talking about autofinder). Please note also that a new release of ec-clem is supposed to be proposed this month (but I am myself encountering some vtk problem and the release is getting late, I will do an open a thread for that to the icy team myself :blush: ) .

In any case, could you send me the log from the output panel in icy when you run the autofinder?

I observed multiple instances where ec-CLEM falls into "eternal transformation finding”

-> Could you send the log from the output panel in icy when you run the autofinder? I do not think I had an eternal transfo finding event before :thinking:. For autofinder, the performance depends on the number of detection/ points rather than on the image size. The image size will influence mainly the step of the application of the transformation, not of its computation.

the performance also depends on the file format

–> I can not see why it would depend on the file format because all images are converted internally to a specific type for java and icy (Icy Buffered Image) (maybe rather on 8 bit vs 16 bit vs 32 bits) @Stephane any clue about that?
Regarding your binnning, you can do it directly when you open the image (right panel in open image, you can select the binning), without having to save them on disk (that is may be what you are already doing, I am mentionning this nice feature in case you did not see it) . Then if you use ec-clem or autofinder on this binned image, you can use ApplyTransfoToScaledImage (or aif using the ec-clem advanced menu Apply a reduced Scale transfo to … ) using the transfo computed on the full size image, and it will consume less memory to simply apply it.

For the autofinder use, there is a tutorial in a book and you can access the chapter describing the autofinder on research gate) ( (by requesting a private copy)

Myself I use the 32Gb RAM Win 10 computer, and java oracle 8, and do not encounter usually any problem. One importnat point is that vtk should be supported by your graphical card, but I am starting another thread about that as specified above. In its current version you need 4 times the size of the biggest image in RAM for ec-clem and ec-clem autofinder, but this has been drastically reduced in the new release (to come very soon …), and not all code was optimized to take advantage of several CPUS, but this has also been fixed.
Waiting for your logs :slight_smile:
Best regards,
Perrine

1 Like