ImageJ hanging in bioconda Fji installation in headless mode

Hi,

I am a novice with image analysis and this forum so apologies for any naivety!

I am trying to run ImageJ from within a conda installation of Fiji on Linux (https://github.com/bioconda/bioconda-recipes/tree/master/recipes/fiji). I am trying to run the images from multiple samples through ImageJ on a cluster in parallel. The command I am using is:

ImageJ --headless -macro fiji_merge_filter.ijm --debug "id_1.tiff id_2.tiff ... ..."

The output files seem to be generated as expected, however the process seems to hang after launching the update process. I am unable to attach the debug trace but I have included the final bits of the output below. I have tried tirelessly to find a solution on the web but to no avail…I would be really grateful if you can help.

These are the commands I used to install and test within the conda environment:

conda create -n fiji fiji -c bioconda
source activate fiji

Please let me know if you would require any further information.

Thanks in advance.

###########################

....
process roi_2_metal_131Xe.tiff
process roi_2_metal_134Ba.tiff
process roi_2_metal_134Xe.tiff
process roi_2_metal_140Ce.tiff
process roi_2_metal_141Pr_aSMA.tiff
process roi_2_metal_142Nd_MHCcII.tiff
process roi_2_metal_143Nd_Vimentin.tiff
process roi_2_metal_144Sm_MHCcI.tiff
process roi_2_metal_145Nd_CD4.tiff
process roi_2_metal_146Nd_F480.tiff
process roi_2_metal_147Sm_CD68.tiff
process roi_2_metal_148Nd_CD11b.tiff
process roi_2_metal_149Sm_CD19.tiff
process roi_2_metal_150Sm_CD44.tiff
process roi_2_metal_151Eu_Ly6G.tiff
process roi_2_metal_152Sm_CD3e.tiff
process roi_2_metal_153Eu_PDL1.tiff
process roi_2_metal_154Gd_CD73.tiff
process roi_2_metal_155Gd_pSTAT1.tiff
process roi_2_metal_156Gd_CD24.tiff
process roi_2_metal_157Gd.tiff
process roi_2_metal_158Gd_FoxP3.tiff
process roi_2_metal_159Tb_TCRgd.tiff
process roi_2_metal_160Gd_pSTAT3.tiff
process roi_2_metal_161Dy_CD103.tiff
process roi_2_metal_162Dy_Tim3.tiff
process roi_2_metal_163Dy_pAKT.tiff
process roi_2_metal_164Dy_EpCAM.tiff
process roi_2_metal_165Ho_PECAM.tiff
process roi_2_metal_166Er_PD1.tiff
process roi_2_metal_167Er_Nkp46.tiff
process roi_2_metal_168Er_CD8a.tiff
process roi_2_metal_169Tm_CD206.tiff
process roi_2_metal_170Er_pS6.tiff
process roi_2_metal_171Yb_pERK.tiff
process roi_2_metal_172Yb_casp3.tiff
process roi_2_metal_173Yb_Ki67.tiff
process roi_2_metal_174Yb_Langerin.tiff
process roi_2_metal_175Lu_CD80.tiff
process roi_2_metal_176Yb_B220.tiff
process roi_2_metal_190Pt.tiff
process roi_2_metal_191Ir_DNA1.tiff
process roi_2_metal_192Pt.tiff
process roi_2_metal_193Ir_DNA2.tiff
process roi_2_metal_194Pt.tiff
process roi_2_metal_195Pt.tiff
process roi_2_metal_196Pt.tiff
process roi_2_metal_198Pt.tiff
process roi_2_metal_204Pb.tiff
process roi_2_metal_209Bi_CD11c.tiff
process roi_2_metal_80ArAr.tiff
process roi_2_metal_89Y_CD45.tiff
[DEBUG] unregister legacy image: img[roi_2_metal_142Nd_MHCcII.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_150Sm_CD44.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_193Ir_DNA2.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_127I.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_131Xe.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_134Ba.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_134Xe.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_140Ce.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_141Pr_aSMA.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_143Nd_Vimentin.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_144Sm_MHCcI.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_145Nd_CD4.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_146Nd_F480.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_147Sm_CD68.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_148Nd_CD11b.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_149Sm_CD19.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_151Eu_Ly6G.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_152Sm_CD3e.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_153Eu_PDL1.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_154Gd_CD73.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_155Gd_pSTAT1.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_156Gd_CD24.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_157Gd.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_158Gd_FoxP3.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_159Tb_TCRgd.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_160Gd_pSTAT3.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_161Dy_CD103.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_162Dy_Tim3.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_163Dy_pAKT.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_164Dy_EpCAM.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_165Ho_PECAM.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_166Er_PD1.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_167Er_Nkp46.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_168Er_CD8a.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_169Tm_CD206.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_170Er_pS6.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_171Yb_pERK.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_172Yb_casp3.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_173Yb_Ki67.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_174Yb_Langerin.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_175Lu_CD80.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_176Yb_B220.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_190Pt.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_191Ir_DNA1.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_192Pt.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_194Pt.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_195Pt.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_196Pt.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_198Pt.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_204Pb.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_209Bi_CD11c.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_80ArAr.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[roi_2_metal_89Y_CD45.tiff (1041x1036x1x1x1)]
[DEBUG] unregister legacy image: img[merge_filter.tiff (1041x1036x3x1x1)]
[DEBUG] Discovered user interface: net.imagej.legacy.ui.LegacyUI
[DEBUG] Discovered user interface: org.scijava.ui.swing.sdi.SwingSDIUI
[DEBUG] Discovered user interface: org.scijava.ui.awt.AWTUI
[DEBUG] Discovered user interface: org.scijava.ui.swing.mdi.SwingMdiUI
[DEBUG] Launching user interface: net.imagej.legacy.ui.LegacyUI
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        ui = legacy,null,null), called from non-EDT Thread:null
file:/camp/stp/babs/working/patelh/code/conda/envs/fiji_20151222-0/share/java/linux-amd64/jdk1.6.0_24/jre/lib/rt.jar
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        progress = -1
        maximum = -1
        status = Running command: Up-to-date check
        warning = false,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates,null,null), called from non-EDT Thread:null
[DEBUG] Executing module: net.imagej.updater.CheckForUpdates@5d3f03b
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = org.scijava.module.process.DebugPreprocessor@34a2970f,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = net.imagej.legacy.plugin.MacroPreprocessor@20c00612,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = org.scijava.module.process.ValidityPreprocessor@d809bd3,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = net.imagej.display.process.ActiveChannelCollectionPreprocessor@76764091,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = net.imagej.display.process.ActiveDataViewPreprocessor@4d892c59,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = net.imagej.display.process.ActiveDatasetPreprocessor@68beaed3,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = net.imagej.display.process.ActiveDatasetViewPreprocessor@3ec7d45e,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = net.imagej.display.process.ActiveImageDisplayPreprocessor@31478785,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = net.imagej.display.process.ActiveOverlayPreprocessor@4f124609,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = net.imagej.display.process.ActivePositionPreprocessor@38b4216d,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = net.imagej.legacy.plugin.ActiveImagePlusPreprocessor@39fe9830,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = org.scijava.display.ActiveDisplayPreprocessor@79b9950e,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = org.scijava.module.process.DefaultValuePreprocessor@7709c5c,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = org.scijava.module.process.GatewayPreprocessor@72295fd4,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = org.scijava.module.process.ServicePreprocessor@7a5291d9,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = org.scijava.ui.UIPreprocessor@42e57c4c,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = net.imagej.ops.NamespacePreprocessor@684383e3,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = org.scijava.module.process.InitPreprocessor@70e98812,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = org.scijava.module.process.LoadInputsPreprocessor@73e48fa7,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = org.scijava.ui.FilePreprocessor@22935862,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = net.imagej.legacy.ui.LegacyInputHarvester@778f9ba9,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = org.scijava.ui.awt.widget.AWTInputHarvester@7534b191,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = org.scijava.ui.swing.widget.SwingInputHarvester@5675e2b4,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = org.scijava.ui.swing.widget.SwingMdiInputHarvester@5576c3c4,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = org.scijava.module.process.SaveInputsPreprocessor@682ea4b1,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = org.scijava.module.process.CheckInputsPreprocessor@4e3e95e6,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates,null,null), called from non-EDT Thread:null
[DEBUG] INPUTS:
[DEBUG] OUTPUTS:
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = org.scijava.module.process.DebugPostprocessor@112062da,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = net.imagej.legacy.plugin.MacroRecorderPostprocessor@6539defe,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates
        processor = org.scijava.display.DisplayPostprocessor@3a5f299d,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        module = net.imagej.updater.CheckForUpdates,null,null), called from non-EDT Thread:null
[DEBUG] publish(
        context = org.scijava.Context@3b626c6d
        consumed = false
        progress = -1
        maximum = -1
        status = Command finished: Up-to-date check
        warning = false,null,null), called from non-EDT Thread:null

###########################

Wow, I didn’t know a fiji recipe for bioconda existed :slight_smile:

Unfortunately, this hasn’t been updated for two years now, and actually still greps a life-line version of Fiji from Dec. 2015:

Can you maybe test if it works for you with an up-to-date version of Fiji downloaded from here:

It’s a portable application, so no need to install anything; just run your command with the ImageJ-linux64 executable.

Thanks alot for the quick response @imagejan , and for the editing…it was late and I didnt realise you could use Markdown :face_with_raised_eyebrow:

I will attempt to add an up-to-date recipe to bioconda. Would you mind just having a look at the build files for the old recipe so you can maybe advise me on what is now necessary. It just means I can strip it back to the bare essentials.

Also, is it better to pull the Fiji package from GitHub for version control purposes because the links on the downloads page will always point to latest. Ideally, the bioconda recipe should be flexible enough to allow builds on Linux and MacOS.

Ill try and get something together with some testing over the next couple of days. It would be great if you can have a look when I have finished it.

Thanks again for your help.

Harshil

@drpatelh please note that there’s also pyimagej on conda-forge:

I’m not sure, @ctrueden might have some advice here…

What is the policy for bioconda packages? Do they need to be built from source? The Java philosophy is to build once, and then distribute the binaries to “run everywhere”, which by itself is very nice. But it tends to interact badly with some package managers, particularly source-driven Linux distributions: Fiji is highly componentized across many repositories on GitHub, so rebuilding it all from source requires some legwork—but doing so should almost never be necessary technically.

About conda channels: the ImageJ/Fiji core developers are trying to publish things on conda-forge rather than bioconda, but I admit to being ignorant of the technical differences. Are there compelling reasons to prefer bioconda? ImageJ is used by people outside of life sciences, so I’m hesitant to publish a general-purpose imagej conda package there… but Fiji does indeed target life sciences, so I suppose bioconda could be OK for fiji.

Regarding debugging headless behavior: have you read over these pages?

I notice that the latter link has the following hack at the conclusion of one of the example macros:

eval("script", "System.exit(0);");

You could try that to force the JVM to shutdown after doing your processing.

1 Like

Thanks for your responses @imagejan @ctrueden

To be honest, I am not entirely sure whether there is a preference to build from source. I only started uploading to bioconda a couple of weeks ago. Its been quite a learning curve! Im assuming it should be fine to at least try to use a binary appropriate for the OS. However, I suspect the version control aspect will be quite important for the conda repositories.

As you pointed out, I would say that ImageJ and Fiji would be appropriate for conda-forge and bioconda, respectively. Also, there is an existing Fiji recipe on bioconda so it probably makes sense to keep it there.

Having said all of that, if you havent noticed already, I am a complete novice with image analysis. I was roped into it because we had an internal data challenge/hackathon at the beginning of the week, and the project I was working on involved scripting an image analysis pipeline for Image Mass Cytometry. I managed to just about hack a nextflow pipeline together that ran through, and I am now trying to formalise everything in order to make things completely portable and reproducible. The main script is below:

For the reproducibility aspect I will need to be able to run the pipeline using software hosted on conda, and it essentially needs to be able to run entirely on the command-line in parallel on a cluster. This is why I asked the initial question about Fiji - to be honest Id rather not use the old conda recipe if its so out of date.

I had a play around with a conda environment containing pyimagej last night, and to be honest if its easier to run this then I dont need to create a separate Fiji recipe. However, I noticed that the Fiji plugins have to be downloaded at runtime and that the appropriate files are placed in .jgo and .m2 in the user home directory. This may be a problem with nextflow in the way that it interacts with the file system. Is this configurable? So from what I gather I would have to run the plugin download once, and then execute the actual ImageJ commands separately to avoid any overwriting? I am getting some debug errors but from what I read on the interwebnet they arent overly serious:

11:14:56.750 [SciJava-78d6447a-Thread-0] DEBUG loci.formats.ClassList - Could not find loci.formats.in.ScreenReader
java.lang.ClassNotFoundException: loci.formats.in.ScreenReader

It would be really helpful if you can have a look at the pipeline I have stitched together to see if all of this is even necessary. Your expert input may save me alot of time and effort. I may be going about this in completely the wrong way!

Thank you very much.

Harshil