Is it possible to conduct image analysis using a HPC cluster?

Hello,

I want to analyse the structure of open cell iron foams using micro-CT data. I am considering to use fiji to do so. However, the size of my data is so big that I am not able to load so many slices in fiji for further analysis, and it is very hard to construct the 3D model. That made me think about using supercomputers (High Performance Computing or HPC) to do my analysis on fiji. Does anybody know if we can implement fiji and its different plugins (such as bonej) on a HPC cluster ? If yes, my challenge is that fiji is not installed on their system and I need to do it myself. I know that a Linux version of fiji is available, but I need help to install it in the first place. I would appreciate any input helping me to install and implement fiji on an HPC cluster.

Thanks,
Reza

3 Likes

As long as your HPC machine has a Java virtual machine (JVM) that can ‘see’ all the hardware resources (RAM, CPU cores), Fiji and BoneJ will simply scale to use them. We usually develop BoneJ in an Ubuntu GNU/Linux environment so the OS element should not be a problem.

BoneJ1 is still the stable production version but that comes with a limitation which is that it requires/prefers Java 6 (although, in our hands it can work OK with Java 8 on Linux, but not elsewhere e.g. Mac OS).

The bigger challenge may be running plugins ‘headless’, i.e. without a display. If you are able to have a remote desktop e.g. NoMachine on your HPC, you may find it easier to get set up your image processing environment.

An alternative may be to set up a container e.g. Docker that has everything you need on your local machine, then run the container on the HPC cluster.

Good luck and please let us know how you get on.

2 Likes

Hello,

yes, Fiji can in principle be setup and used on a HPC cluster. Simply running Fiji in a virtual frame buffer will do.

Here is the link to the publication: https://academic.oup.com/bioinformatics/article/32/7/1112/1744153

The Fiji wiki:

The github page:

You do not need to implement anything within the snakemake workflow. Simple bash scripts are sufficient. Here is also a guide to install Fiji on the cluster: https://imagej.net/SPIM_Registration_on_cluster_(deprecated)#Original_SPIM_registration_pipeline

Just be aware that you will not have access to any interaction via the graphical user interface. So any error messages or panels that Fiji is creating you will not be able to see or use.

3 Likes

Thank you for your response. I will check out the resources you mentioned. One question: after installing fiji on a HCP, how should I implement it if I will not have access to GUI. Should I write the code for each command I want to implement on fiji ?

Thanks,
Reza

1 Like

Hi,

yes you need to write scripts. In this case we used beanshell scripts. But in principle it could be also a macro. If you want to parallelize the jobs on a HPC you will need to pass arguments from the individual cluster jobs to the Fiji script. Which can be done in macro as well: https://imagej.nih.gov/ij/developer/macro/macros.html#cli
But I did not try that.

Cheers,
Christopher

1 Like

I’m not sure why everyone tells you that you can’t have a GUI (maybe I misunderstand something). Using X forwarding you can easily “forward” the Fiji GUI from the cluster to your own computer. You need to login using the -X flag:

ssh -X your_login_name@your_cluster

Then when you start Fiji by typing:

./folder_where_you_downloaded_fiji/Fiji.app/ImageJ-linux64

the Fiji GUI will just open on your computer if you have an X server installed on it (e.g. Xquarz for Mac or Xming for Windows). You can find some more infos on many university cluster web pages, for example here: https://www.hoffman2.idre.ucla.edu/access/x11_forwarding/

You might not be allowed and probably should not run Fiji directly on the login node of your cluster. It’s better to create an interactive job that you submit to the queuing system of your cluster just like any other job. How to do that depends on your queuing system and you can probably ask the admins for help. As an example, with SLURM you would type something like:

srun --mem-per-cpu=2G --cpus-per-task=2 --time=01:00:00 --pty --x11 bash

This opens a new bash terminal session as an interactive job which will remain open for one hour with two CPUs and 4Gb of RAM. Within that terminal you can then start Fiji and use X-forwarding to work with the GUI.

Good luck!

Guillaume

4 Likes

Hi,

cool I did not know that was possible. If you want to run it as a single machine then this makes sense to do.

Cheers,
Christopher