Not enough RAM to export Ilastik probabilities as tiff sequence

ilastik
#1

I am using Pixel Classification in Ilastik to export for segmentation in Cell Profiler. We recently began taking larger images, and I was working with an image in Ilastik that was about 500 x 9000 pixels. I am working on a laptop in Windows 10 with 64GB RAM. When I tried to export probabilities as a tiff sequence, as we usually do, I received an error that there was not enough RAM to export to the selected format (it only worked when I cropped the image to a lower size). There were no other significant programs running in the background, and only about 10% of memory was being used. However, when I tried to export the exact same file on my personal 2011 Macbook Pro with only 16GB RAM, it was able to export the entire file.

This problem remained in both Ilastik 1.3.2 and 1.3.0 after reinstallation. I was wondering if there is some issue with a Mac vs PC in terms of how RAM is allocated to Ilastik, or if there is any way to allocate more RAM to Ilastik or some other solution to this problem.

Thank you

#2

This is very interesting, thank you very much for the report. It has to be an issue in the tiff libraries we use, or how we use them. We’ll investigate!

#3

Zavi,
How much ram is actually allocated to ImageJ? You can change it in Edit > options > memory & threads to see if that changes anything.

Bob

#4

I am working in Ilastik. Is there a similar setting to allocate memory in Ilastik?

#5

For ilastik you can achieve that using a configuration file. It should be located in your userhome:

  • for windows: C:\Users\<YOURUSERNAME>\.ilastikrc
  • for mac, and linux you would put it to the equivalent locations.

if the file doesn’t exist, just create it and add the following contents:

[lazyflow]
lazyflow_total_ram=<MEMORY_IN_MB>

so say, you would want to limit ram usage to 8 gigabyte, you’d replace <MEMORY_IN_MB> with 8000.

That is not a guarantee that RAM usage doesn’t exceed this value, but it encourages it :wink:

#6

I don’t see a file with the name .ilastikrc
To clarify, do I just create a text file with those 2 lines?

Is it possible that, for some reason, Ilastik’s RAM usage is currently limited, and I could increase by writing a larger value? For example, if the computer has 64GB of RAM, could I create that text file and write 64000 for <MEMORY_IN_MB> to actually increase the RAM allocated to Ilastik?

Zach

#7

Hi @Zavi,

I think I have misread your first post. So you really had only one image with 500 x 9000? That should, in any case not use more that 1 GB of your RAM.

Could you do us a favor and, on your windows machine delete the log file (found at C:\Users\<YourUserName>\ilastik_log.txt), start ilastik again, do the export that results in the error, and send us the log file to team@ilastik.org?

Cheers
Dominik

#8

Just sent it to the team. Thank you for your help, let me know if there is any other information I can provide.

Zach

#9

Hi Dominik,

I was curious if there has been any progress on this question since I sent the log file to your team.

Thanks,
Zach

#10

Hi @Zavi
thank you very much for sending in the logs! We started looking into it, but it is not something obvious…

#11

Hey @Zavi, I looked into it some more and could you please give me a bit more information on the image you are using? Is it’s size really just 500x9000? Furthermore, to which data type are you exporting? float, or uint16…? Also are you reshuffeling the axis order to cyx before you export?

#12

In any case we’ve figured out the problem (it is related to default number types that are used by numpy, the numerical library we use, which are different on windows…)
Will be fixed in the next release - thank you very much for discovering/reporting this issue!!

#13

Hi Dominik,

For more information: the size is 511 x 9063. I tried cropping a few sizes to see what would export, and it worked once I got the width lower than about 5500 (i.e. 511 x 5500). For our export, I was using unsigned 8 bit (though I think I had the same issue attempting with unsigned 16 bit), transposing axis order to cyx, and formatting as a tiff sequence.

Glad to hear you figured out the problem, looking forward to the fix in the next release! Any sense of the timing on that release?

Thanks,
Zach

#14

Hi Dominik,

I was wondering if you knew when this release would come out to fix this issue of exporting larger images on a PC. Thanks!
Zach

#15

Hi @Zavi,

sorry I got distracted away from this issue and left it in a half finished state. Will pick it up again and keep you in the loop

Cheers
Dominik

#16

Hi Dominik,
I just got an email from you about the new stable release of Ilastik 1.3.2. I was just wondering if the new release addresses this RAM issue on PCs.
Thanks,
Zach

#17

Hey Zavi,

this didn’t make it, unfortunately. However, we’re releasing an update of our fiji import/export plugin that is much faster, and hence, you can just export to h5 and convert in fiji, if tiff stacks is what you need.
Will post about this update, here, too.

Cheers