Run script headlessly

Hi all,

A user came to me with a script which he wants to be able to run headlessly.

Here’s the script :

// v2

// Macro to load nd2 files and convert them to hd5 files using the fiji > ilastik plugin

// Author: Alex (but with help from Loic Sauteur)
// version1
// Friday 10th July 2020
#@File (label="Select input folder", style="directory") input_path
#@File (label="Select output folder", style="directory") export_path

// variables
input_path += File.separator;
export_path += File.separator;
suffix = ".nd2";

fileList = getFileList(input_path);
for (i = 0; i < fileList.length; i++) {
	if (endsWith(fileList[i], suffix)) {
		run("Bio-Formats Importer", "open=[" + input_path + fileList[i] +"] color_mode=Default concatenate_series open_all_series rois_import=[ROI manager] view=Hyperstack stack_order=XYCZT");
		title = replace(fileList[i], suffix, ".h5");
		print(title);
		exportPath = export_path + title;
		print(exportPath);
		//everything above here runs without an error. Error could therefore be in line 28.
		run("Export HDF5", "select=["+exportPath+"] exportpath=["+exportPath+"] datasetname=data compressionlevel=0 input=[" + getTitle() + "]" ); ;
		close();
	}
}

exit();

Here’s the command used to run it :

/Path/to/Fiji --ij2 --headless --console --run nd2_to_hd5_batchconver_v2.ijm "input_path='Raw_nd2/',export_path='Export/'"  

It seems that the export HDF5 command fails as an image is not open (I guess visible in the GUI).

What’s the best way to solve this ?

Thanks !

1 Like

Hi @lguerard,

interesting… Do you get something like an error message? Also, does getTitle() work?

Cheers
Dominik

Hi Dominik,

Thank you for your (very) fast answer ! :slight_smile:

The print(title) and print(export_path) work and are printed in the console.

Here is the complete log from the console where it complains for line 25 that there aren’t any images open.

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
java.lang.VerifyError: Bad type on operand stack
Exception Details:
  Location:
    loci/plugins/in/MainDialog.rebuildDialog(Lij/gui/GenericDialog;)V @1926: invokestatic
  Reason:
    Type 'ij/gui/GenericDialog' (current frame, stack[0]) is not assignable to 'java/awt/Container'
  Current Frame:
    bci: @1926
    flags: { }
    locals: { 'loci/plugins/in/MainDialog', 'ij/gui/GenericDialog', 'java/util/List', 'java/util/List', 'java/util/List', 'java/awt/Label', 'java/awt/Label', 'java/awt/Label', 'java/awt/Label', '[Ljava/awt/Component;', 'java/lang/String', 'java/lang/String', 'com/jgoodies/forms/builder/PanelBuilder', 'com/jgoodies/forms/layout/CellConstraints', integer }
    stack: { 'ij/gui/GenericDialog' }
  Bytecode:
    0x0000000: 014d 014e 013a 0401 3a05 013a 0601 3a07
    0x0000010: 013a 082b b600 533a 0919 09c6 0266 bb00
    0x0000020: 5459 b700 554d bb00 5459 b700 554e bb00
    0x0000030: 5459 b700 553a 0403 360a 150a 1909 bea2
    0x0000040: 008a 1909 150a 32c1 0056 9900 2b19 0915
    0x0000050: 0a32 c000 563a 0b19 0b2a b600 5719 0b2a
    0x0000060: b600 5819 0b2a b600 592c 190b b900 5a02
    0x0000070: 0057 a700 5119 0915 0a32 c100 5b99 002b
    0x0000080: 1909 150a 32c0 005b 3a0b 190b 2ab6 005c
    0x0000090: 190b 2ab6 005d 190b 2ab6 005e 2d19 0bb9
    0x00000a0: 005a 0200 57a7 001e 1909 150a 32c1 005f
    0x00000b0: 9900 1319 0419 0915 0a32 c000 5fb9 005a
    0x00000c0: 0200 5784 0a01 a7ff 7403 360a 0336 0b03
    0x00000d0: 360c 2a2c 150a 840a 01b9 0060 0200 c000
    0x00000e0: 56b5 0061 2a2d 150b 840b 01b9 0060 0200
    0x00000f0: c000 5bb5 0062 1904 150c 840c 01b9 0060
    0x0000100: 0200 c000 5f3a 052a 2c15 0a84 0a01 b900
    0x0000110: 6002 00c0 0056 b500 632a 2c15 0a84 0a01
    0x0000120: b900 6002 00c0 0056 b500 642a 2c15 0a84
    0x0000130: 0a01 b900 6002 00c0 0056 b500 652a 2c15
    0x0000140: 0a84 0a01 b900 6002 00c0 0056 b500 662a
    0x0000150: 2c15 0a84 0a01 b900 6002 00c0 0056 b500
    0x0000160: 6784 0a01 2a2c 150a 840a 01b9 0060 0200
    0x0000170: c000 56b5 0068 2a2c 150a 840a 01b9 0060
    0x0000180: 0200 c000 56b5 0069 2a2c 150a 840a 01b9
    0x0000190: 0060 0200 c000 56b5 006a 2a2d 150b 840b
    0x00001a0: 01b9 0060 0200 c000 5bb5 006b 1904 150c
    0x00001b0: 840c 01b9 0060 0200 c000 5f3a 062a 2c15
    0x00001c0: 0a84 0a01 b900 6002 00c0 0056 b500 6c2a
    0x00001d0: 2c15 0a84 0a01 b900 6002 00c0 0056 b500
    0x00001e0: 6d2a 2c15 0a84 0a01 b900 6002 00c0 0056
    0x00001f0: b500 6e2a 2c15 0a84 0a01 b900 6002 00c0
    0x0000200: 0056 b500 6f2a 2d15 0b84 0b01 b900 6002
    0x0000210: 00c0 005b b500 4a19 0415 0c84 0c01 b900
    0x0000220: 6002 00c0 005f 3a07 2a2d 150b 840b 01b9
    0x0000230: 0060 0200 c000 5bb5 0070 1904 150c 840c
    0x0000240: 01b9 0060 0200 c000 5f3a 082a 2c15 0a84
    0x0000250: 0a01 b900 6002 00c0 0056 b500 712a 2c15
    0x0000260: 0a84 0a01 b900 6002 00c0 0056 b500 722a
    0x0000270: 2c15 0a84 0a01 b900 6002 00c0 0056 b500
    0x0000280: 732a 01b7 0049 2abb 0074 59b7 0075 b500
    0x0000290: 3d2a b400 3d2a b400 612a b400 20b6 0076
    0x00002a0: b900 7703 0057 2ab4 003d 2ab4 0062 2ab4
    0x00002b0: 0020 b600 78b9 0077 0300 572a b400 3d19
    0x00002c0: 052a b400 20b6 0078 b900 7703 0057 2ab4
    0x00002d0: 003d 2ab4 0063 2ab4 0020 b600 79b9 0077
    0x00002e0: 0300 572a b400 3d2a b400 642a b400 20b6
    0x00002f0: 007a b900 7703 0057 2ab4 003d 2ab4 0065
    0x0000300: 2ab4 0020 b600 7bb9 0077 0300 572a b400
    0x0000310: 3d2a b400 662a b400 20b6 007c b900 7703
    0x0000320: 0057 2ab4 003d 2ab4 0067 2ab4 0020 b600
    0x0000330: 7db9 0077 0300 572a b400 3d2a b400 682a
    0x0000340: b400 20b6 007e b900 7703 0057 2ab4 003d
    0x0000350: 2ab4 0069 2ab4 0020 b600 7fb9 0077 0300
    0x0000360: 572a b400 3d2a b400 6a2a b400 20b6 0080
    0x0000370: b900 7703 0057 2ab4 003d 2ab4 006b 2ab4
    0x0000380: 0020 b600 81b9 0077 0300 572a b400 3d19
    0x0000390: 062a b400 20b6 0081 b900 7703 0057 2ab4
    0x00003a0: 003d 2ab4 006c 2ab4 0020 b600 82b9 0077
    0x00003b0: 0300 572a b400 3d2a b400 6d2a b400 20b6
    0x00003c0: 0083 b900 7703 0057 2ab4 003d 2ab4 006e
    0x00003d0: 2ab4 0020 b600 84b9 0077 0300 572a b400
    0x00003e0: 3d2a b400 6f2a b400 20b6 0085 b900 7703
    0x00003f0: 0057 2ab4 003d 2ab4 004a 2ab4 0020 b600
    0x0000400: 86b9 0077 0300 572a b400 3d19 072a b400
    0x0000410: 20b6 0086 b900 7703 0057 2ab4 003d 2ab4
    0x0000420: 0070 2ab4 0020 b600 87b9 0077 0300 572a
    0x0000430: b400 3d19 082a b400 20b6 0087 b900 7703
    0x0000440: 0057 2ab4 003d 2ab4 0071 2ab4 0020 b600
    0x0000450: 88b9 0077 0300 572a b400 3d2a b400 722a
    0x0000460: b400 20b6 0089 b900 7703 0057 2ab4 003d
    0x0000470: 2ab4 0073 2ab4 0020 b600 8ab9 0077 0300
    0x0000480: 5712 8b3a 0a12 8c3a 0bbb 008d 59bb 008e
    0x0000490: 5919 0a19 0bb7 008f b700 903a 0cbb 0091
    0x00004a0: 59b7 0092 3a0d 0436 0e19 0c12 9319 0d04
    0x00004b0: 150e 06b6 0094 b600 9557 840e 0219 0c19
    0x00004c0: 0719 0d04 150e b600 96b6 0097 5719 0c2a
    0x00004d0: b400 4a19 0d06 150e b600 96b6 0097 5784
    0x00004e0: 0e02 190c 1908 190d 0415 0eb6 0096 b600
    0x00004f0: 9757 190c 2ab4 0070 190d 0615 0eb6 0096
    0x0000500: b600 9757 840e 0619 0c12 9819 0d04 150e
    0x0000510: 06b6 0094 b600 9557 840e 0219 0c2a b400
    0x0000520: 652a 190d 0415 0e06 b700 99b6 0097 5784
    0x0000530: 0e02 190c 2ab4 0066 2a19 0d04 150e 06b7
    0x0000540: 0099 b600 9757 840e 0219 0c2a b400 712a
    0x0000550: 190d 0415 0e06 b700 99b6 0097 5784 0e02
    0x0000560: 190c 2ab4 0067 2a19 0d04 150e 06b7 0099
    0x0000570: b600 9757 840e 0219 0c2a b400 632a 190d
    0x0000580: 0415 0e06 b700 99b6 0097 5784 0e02 190c
    0x0000590: 2ab4 0073 2a19 0d04 150e 06b7 0099 b600
    0x00005a0: 9757 840e 0219 0c12 9a19 0d04 150e 06b6
    0x00005b0: 0094 b600 9557 840e 0219 0c19 0519 0d04
    0x00005c0: 150e b600 96b6 0097 5719 0c2a b400 6219
    0x00005d0: 0d06 150e b600 96b6 0097 5784 0e02 190c
    0x00005e0: 2ab4 0061 2a19 0d04 150e 06b7 0099 b600
    0x00005f0: 9757 840e 0204 360e 190c 129b 190d 0815
    0x0000600: 0e06 b600 94b6 0095 5784 0e02 190c 2ab4
    0x0000610: 0068 2a19 0d08 150e 06b7 0099 b600 9757
    0x0000620: 840e 0219 0c2a b400 692a 190d 0815 0e06
    0x0000630: b700 99b6 0097 5784 0e02 190c 2ab4 006a
    0x0000640: 2a19 0d08 150e 06b7 0099 b600 9757 840e
    0x0000650: 0219 0c19 0619 0d08 150e b600 96b6 0097
    0x0000660: 5719 0c2a b400 6b19 0d10 0715 0eb6 0096
    0x0000670: b600 9757 840e 0219 0c12 9c19 0d08 150e
    0x0000680: 06b6 0094 b600 9557 840e 0219 0c2a b400
    0x0000690: 722a 190d 0815 0e06 b700 99b6 0097 5784
    0x00006a0: 0e02 190c 2ab4 006c 2a19 0d08 150e 06b7
    0x00006b0: 0099 b600 9757 840e 0219 0c2a b400 642a
    0x00006c0: 190d 0815 0e06 b700 99b6 0097 5784 0e04
    0x00006d0: 190c 129d 190d 0815 0e06 b600 94b6 0095
    0x00006e0: 5784 0e02 190c 2ab4 006f 2a19 0d08 150e
    0x00006f0: 06b7 0099 b600 9757 840e 0219 0c2a b400
    0x0000700: 6d2a 190d 0815 0e06 b700 99b6 0097 5784
    0x0000710: 0e02 190c 2ab4 006e 2a19 0d08 150e 06b7
    0x0000720: 0099 b600 9757 190c 129e 190d 1009 04b6
    0x0000730: 0096 b600 9557 2abb 009f 59b7 00a0 b500
    0x0000740: 402a b400 4012 a1b6 00a2 2ab4 0040 03b6
    0x0000750: 00a3 2ab4 0040 12a5 b600 4619 0cbb 00a6
    0x0000760: 592a b400 40b7 00a7 190d 1009 0604 150e
    0x0000770: b600 a8b6 0097 572b b600 a92b 190c b600
    0x0000780: aab6 00ab 572b b800 ac2b b200 adb6 00ae
    0x0000790: b1                                     
  Stackmap Table:
    full_frame(@58,{Object[#164],Object[#4],Object[#309],Object[#309],Object[#309],Object[#95],Object[#95],Object[#95],Object[#95],Object[#310],Integer},{})
    same_frame(@117)
    same_frame(@168)
    same_frame(@195)
    chop_frame(@201,1)
    same_frame_extended(@641)

	at loci.plugins.in.ImporterPrompter.promptMain(ImporterPrompter.java:132)
	at loci.plugins.in.ImporterPrompter.statusUpdated(ImporterPrompter.java:79)
	at loci.plugins.in.ImportProcess.notifyListeners(ImportProcess.java:476)
	at loci.plugins.in.ImportProcess.step(ImportProcess.java:772)
	at loci.plugins.in.ImportProcess.execute(ImportProcess.java:140)
	at loci.plugins.in.Importer.showDialogs(Importer.java:140)
	at loci.plugins.in.Importer.run(Importer.java:76)
	at loci.plugins.LociImporter.run(LociImporter.java:78)
	at ij.IJ.runUserPlugIn(IJ.java:235)
	at ij.IJ.runPlugIn(IJ.java:198)
	at ij.Executer.runCommand(Executer.java:150)
	at ij.Executer.run(Executer.java:68)
	at ij.IJ.run(IJ.java:317)
	at ij.IJ.run(IJ.java:328)
	at ij.macro.Functions.doRun(Functions.java:686)
	at ij.macro.Functions.doFunction(Functions.java:98)
	at ij.macro.Interpreter.doStatement(Interpreter.java:278)
	at ij.macro.Interpreter.doBlock(Interpreter.java:712)
	at ij.macro.Interpreter.doStatement(Interpreter.java:323)
	at ij.macro.Interpreter.doIf(Interpreter.java:1090)
	at ij.macro.Interpreter.doStatement(Interpreter.java:299)
	at ij.macro.Interpreter.doBlock(Interpreter.java:712)
	at ij.macro.Interpreter.doStatement(Interpreter.java:323)
	at ij.macro.Interpreter.doFor(Interpreter.java:634)
	at ij.macro.Interpreter.doStatement(Interpreter.java:305)
	at ij.macro.Interpreter.doStatements(Interpreter.java:264)
	at ij.macro.Interpreter.run(Interpreter.java:160)
	at ij.macro.Interpreter.run(Interpreter.java:93)
	at ij.macro.Interpreter.run(Interpreter.java:104)
	at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:161)
	at ij.IJ.runMacro(IJ.java:153)
	at ij.IJ.runMacro(IJ.java:142)
	at net.imagej.legacy.IJ1Helper$3.call(IJ1Helper.java:1148)
	at net.imagej.legacy.IJ1Helper$3.call(IJ1Helper.java:1144)
	at net.imagej.legacy.IJ1Helper.runMacroFriendly(IJ1Helper.java:1095)
	at net.imagej.legacy.IJ1Helper.runMacro(IJ1Helper.java:1144)
	at net.imagej.legacy.plugin.IJ1MacroEngine.eval(IJ1MacroEngine.java:145)
	at org.scijava.script.ScriptModule.run(ScriptModule.java:157)
	at org.scijava.module.ModuleRunner.run(ModuleRunner.java:165)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:124)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:63)
	at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

well_C3_point_1__Channel_x40 Brightfield_Cy3,x40 Cy3_Bf.h5
/media/BZ/guerard/Projects/UserName/FolderName/Export/well_C3_point_1__Channel_x40 Brightfield_Cy3,x40 Cy3_Bf.h5
Macro Error: There are no images open in line 25
 
run ( "Export HDF5" , "select=[" + exportPath + "] exportpath=[" + exportPath + "] datasetname=data compressionlevel=0 ...

Hope that helps !

sure,

but you do print(title). Later in the macro call you do getTitle().

Thanks for the error stack :smiley:

1 Like

I have the feeling like this getTitle() might really be a problem, at least it was for someone else in a similar situation

1 Like

Indeed, I changed getTitle() to fileList[i] and the error is gone. It seems that the script is not working however, and the error is not shown in the console, but this is something to fix in the script itself so I’ll look for it.

Thanks a lot for your sharp eyes ! :wink:

1 Like

MMMhh cool. Maybe enlighten us all with the solution once you have it running! Fingers crossed :slight_smile:

Which version of ij.jar are you using? The error message indicates that the GenericDialog class (which is used by the Bio-Formats ImporterPrompter dialog) wasn’t patched to be compatible with headless mode.

Explanation why GenericDialog needs to be patched

The ij.gui.GenericDialog class extends java.awt.Dialog, a class of the Java AWT framework. Classes in the java.awt package cannot be instantiated with a graphical user interface (GUI), so cannot be used in headless mode. That is the reason why in Fiji, the GenericDialog is hacked at runtime by ij1-patcher to replace its parent java.awt.Dialog by a headless-compatible dummy class.

Usally, ij1-patcher takes care of making everything headless-compatible, but the current version of ij1-patcher works with ij version 1.53b and earlier only. There were breaking changes in ij versions 1.53c and 1.53d that are not yet accounted for in ij1-patcher, which might lead to GenericDialog not being patched at all, and therefore might be the reason for the error you’re seeing… (all just speculation, since I do not know your system details).

If my suspicion above is correct, then you might indeed not have any images open when running the Export HDF5 command in line 25.

2 Likes

Will do ! :slight_smile:

This was on a brand new Fiji which I updated and only activated the Ilastik update site so I guess it could explain the GenericDialog.

2 Likes

Hi again @imagejan,

Just another question, in the script I’m running the BioFormats Importer to concatenate all series in the file but unfortunately this gives a very long name for the file.

I need the title of the image to use the Export HDF5 plugin but I tried getTitle() and rename() and both seem to require the image open and don’t work headless. Do you know any workaround for that ?

Thanks !

Hi @lguerard and all

It seems that also the Ilastik Export HDF5 plugin call requires an opened image…

Below an adapted groovy script based on the Alex’s original one.
Not the most elegant one, but it does the job headlessly.
Any improvements are welcome (i.e. I am not sure how to get the logServices and statusService properly and if the image type handling is “clean”)!

import groovy.io.FileType
import loci.plugins.BF
import loci.plugins.in.ImporterOptions
import net.imagej.ImageJ
import net.imglib2.img.ImagePlusAdapter
import org.apache.poi.ss.formula.functions.T
import org.ilastik.ilastik4ij.hdf5.Hdf5DataSetWriter
import org.scijava.app.StatusService
import org.scijava.log.LogService



#@File (label="Select input folder", style="directory") input_path
#@File (label="Select output folder", style="directory") export_path

context = new ImageJ().getContext()
logService = context.getService(LogService.class)
statusService = context.getService(StatusService.class)

suffix = ".nd2"


list = []
count = 0
def dirFile = new File(input_path.getAbsolutePath())
dirFile.eachFileRecurse(FileType.FILES) { file ->
    if (file.name.endsWith(suffix)) {
        count++
        list << file
    }
}
list.sort()

for (i in 0..<list.size()) {
    imp = openImageWithBF(list[i].getAbsolutePath(), true)
    // convert to ImgPlus
    img = ImagePlusAdapter.wrapImgPlus(imp[0])
    title = list[i].name.replace(" ", "_")
    title1 = title.replace("nd2", "h5")
    exportPath = export_path.getAbsolutePath() + File.separator + title1

    new Hdf5DataSetWriter<T>(img, exportPath, "exported_data", 0, logService, statusService).write()
    println("${i+1} out of ${list.size()} images done.")

}
println("Processing done.")




/**
* open image with bio formats
* will only return the first image, if multi image file
* opens with composite display mode
* @param file = String path
* @param allSeries = to return only one image or array of all images
* @return = ImagePlus object
*/
def openImageWithBF(String file, Boolean allSeries) {
    options = new ImporterOptions()
    // import options
    options.setId(file)
    options.setAutoscale(false)
    //options.setSeriesOn(2, true) // 0-based
    options.setCrop(false)
    options.setOpenAllSeries(allSeries)
    options.setConcatenate(true)

    // open the image
    imp = BF.openImagePlus(options) // returns an array of image references
    if (allSeries) return imp
    else return imp[0]
}

Best,
Loïc

1 Like

@lguerard & @loicsauteur thank you both so much for your help!