Increase the number of trees in the Random Forest Classifier to improve performance?

Hello everybody,

I have a very basic understanding of ML topics so bear with me. After increasing the number of trees in Weka (up to 800), it seems to me that the segmentation output is much more fine grained and on point than what I’m getting in Ilastik.
Ilastik has a default of 100 trees I was wondering how would I go about changing this parameter.

I scoured the ilastik files looking for a variable where the number is set and I found something relevant in Ilastik-1.3.2-Linux/ilastik-meta/ilastik/ilastik/workflows/pixelClassification/pixelClassificationWorkflow.py

I changed the row self.tree_count = parsed_args.tree_count in self.tree_count = 400

Will that work for the pixel classification workflow and/or the autocontext worflow?

Doing this modification will change it for both workflows. This setting is also preserved in the project file, so if you share it with someone else, they will have the same setting there (default is not changed for them of course). The performance takes some hit of course if you use a greater number of trees. We usually don’t see the need to change this parameter (that’s also why it is not exposed in the gui).

There is also no need to hard-code this setting if you really intend on changing it. If you start ilastik from the ccmmand line with ./run_ilastik.sh --tree-count 800 and create a new pixel classification project, this project will have 800 trees for the random forest.

One last note, since I see you’re working with ilastik 1.3.2. We have released 1.3.3 recently, with lots of fixes and other niceties.

1 Like

Hi Dominik, thank you for the quick reply! I keep a close eye on the new developments of the program so I was aware of your progress, but kept on postponing the update. I am downloading the latest version as I type.

I’m segmenting for the first time some RGB pictures from H&E stained kidney sections and I’m having a much harder time than what I’m used to (two photon fluorescence microscopy).
Part of my problems derive from the uneven staining of the sections that I corrected by using a z-score standardization for each color channel, discarded values outside the ±3 sigma range and then converting back to 8 bit RGB. After the normalization they objectively look very similar. I’m trying to get any advantage on the segmentation front because I would like to isolate faint remnants of tubular cell cytoplasm from the surrounding structures.They are very hard to distinguish even by human observers.

PS I have been using Ilastik since 2017 and I must say that the program has become much more stable over time. Thank you for your work.

Hi again,

I tried opening a previously saved auto-context project after launching Ilastik with the argument --tree-count 400 (or 800) and it fails with the following message:

ERROR 2019-11-18 17:13:23,682 log_exception 28761 140173141407552 Could not load project file.
Could not create instance of '<class 'ilastik.workflows.newAutocontext.newAutocontextWorkflow.AutocontextTwoStage'>'
args   = (<ilastik.shell.gui.ilastikShell.IlastikShell object at 0x7f7c7d284318>, False, ['--tree-count', '400'], [])
kwargs = {}
The exception was:
Could not find one or more input files.  See logged errors.
Traceback:
Traceback (most recent call last):
  File "/home/crom/ilastik-1.3.3post1-Linux/ilastik-meta/lazyflow/lazyflow/operator.py", line 118, in __call__
    instance = ABCMeta.__call__(cls, *args, **kwargs)
  File "/home/crom/ilastik-1.3.3post1-Linux/ilastik-meta/ilastik/ilastik/workflows/newAutocontext/newAutocontextWorkflow.py", line 716, in __init__
    super(AutocontextTwoStage, self).__init__(*args, n_stages=2, **kwargs)
  File "/home/crom/ilastik-1.3.3post1-Linux/ilastik-meta/ilastik/ilastik/workflows/newAutocontext/newAutocontextWorkflow.py", line 158, in __init__
    self._batch_input_args, unused_args = self.batchProcessingApplet.parse_known_cmdline_args(unused_args)
  File "/home/crom/ilastik-1.3.3post1-Linux/ilastik-meta/ilastik/ilastik/applets/batchProcessing/batchProcessingApplet.py", line 56, in parse_known_cmdline_args
    parsed_args, unused_args = DataSelectionApplet.parse_known_cmdline_args(cmdline_args, self.role_names)
  File "/home/crom/ilastik-1.3.3post1-Linux/ilastik-meta/ilastik/ilastik/applets/dataSelection/dataSelectionApplet.py", line 200, in parse_known_cmdline_args
    raise RuntimeError("Could not find one or more input files.  See logged errors.")
RuntimeError: Could not find one or more input files.  See logged errors.

After hard-coding the setting the file opens without issue which makes me think that something is wrong with my launching or importing projects with a different tree number or my hard-coded setting is ignored…

Hi @DS4242,

that’s interesting. Thought I tried that. From the error message it looks like you have moved your project file without the data, and thus ilastik could not find it. In order to overcome this, you could either:

  • Per default a relative link to the data is stored in the project file, so you’d have to move the data along with the project (with the same relative location), or
  • Copy the data to the project file (you can do that in the data selection applet by double clicking the line with the data and changing the storage field to copy to project file, see also our docs.

Please let me know if that resolves the issue. Otherwise I’ll take a deeper look into this treecount thing (but it works for me).

Hi, i was able to replicate the issue without opening any existing file, I think that the issue is with the auto-context workflow when tweaking the tree number.

These were the steps I followed on a freshly unzipped Ilastik 1.3.3 folder:

  1. launch Ilastik with the cli argument --tree-count 3200 (I know it’s a lot and probably will lead to overfitting)
  2. select auto-context workflow
  3. try to save the project file in any location
  4. the project file is created and the error window pops up

This is my log output:

ERROR 2019-11-21 18:05:30,373 log_exception 12636 139855201916736 Could not load project file.
Could not create instance of '<class 'ilastik.workflows.newAutocontext.newAutocontextWorkflow.AutocontextTwoStage'>'
args   = (<ilastik.shell.gui.ilastikShell.IlastikShell object at 0x7f327c100318>, False, ['--tree-count', '3200'], ["b'--tree-count'", "b'3200'"])
kwargs = {}
The exception was:
Could not find one or more input files.  See logged errors.
Traceback:
Traceback (most recent call last):
  File "/home/crom/ilastik-1.3.3post1-Linux/ilastik-meta/lazyflow/lazyflow/operator.py", line 118, in __call__
    instance = ABCMeta.__call__(cls, *args, **kwargs)
  File "/home/crom/ilastik-1.3.3post1-Linux/ilastik-meta/ilastik/ilastik/workflows/newAutocontext/newAutocontextWorkflow.py", line 716, in __init__
    super(AutocontextTwoStage, self).__init__(*args, n_stages=2, **kwargs)
  File "/home/crom/ilastik-1.3.3post1-Linux/ilastik-meta/ilastik/ilastik/workflows/newAutocontext/newAutocontextWorkflow.py", line 158, in __init__
    self._batch_input_args, unused_args = self.batchProcessingApplet.parse_known_cmdline_args(unused_args)
  File "/home/crom/ilastik-1.3.3post1-Linux/ilastik-meta/ilastik/ilastik/applets/batchProcessing/batchProcessingApplet.py", line 56, in parse_known_cmdline_args
    parsed_args, unused_args = DataSelectionApplet.parse_known_cmdline_args(cmdline_args, self.role_names)
  File "/home/crom/ilastik-1.3.3post1-Linux/ilastik-meta/ilastik/ilastik/applets/dataSelection/dataSelectionApplet.py", line 200, in parse_known_cmdline_args
    raise RuntimeError("Could not find one or more input files.  See logged errors.")
RuntimeError: Could not find one or more input files.  See logged errors.

Just to remove a possible source of error, I checked whether my user (kubuntu 18.04) had write permission on its own home folder and apparently it does.

crom@VivoBook:~$ getfacl /home/crom/
getfacl: Removing leading '/' from absolute path names
# file: home/crom/
# owner: crom
# group: crom
user::rwx
group::r-x
other::r-x

Please let me know if I can do anything to help you in any way.

Hey @DS4242,

thanks for clearing that up. I can reproduce it with the autocontext workflow - will look into it.

Cheers
Dominik