Weka segmentation error after update 29.09.16

I am working on a script that will take a whole .lif stack, find a region of interest (soma first of all, will add dendrites later), align the stack and apply intensity measurement to all regions of interest. This was working fine the trainable weka plugin untill today, it crashes consistently.

This is my current python prototype, which suddenly stopped working.

  • I use tresholding to separate the different classes (soma/dendrites/background) after creating the segmentation with Weka. Is there an easier way to do this?

PS. I am working on dentate gyrus organotypic slices.

"""ROI selection jython script - z-stack to ROI"""

from ij import IJ, WindowManager, ImagePlus
from ij.process import ImageConverter
from ij.process import ImageProcessor
from ij.gui import GenericDialog
from ij.plugin import ZProjector
from loci.plugins import BF
from loci.common import Region
from loci.plugins.in import ImporterOptions
from trainableSegmentation import WekaSegmentation
from trainableSegmentation import Weka_Segmentation
from fiji.threshold import Auto_Threshold
from ij.plugin.filter import ThresholdToSelection
from ij.plugin.filter import EDM
from ij.plugin.filter import Binary
import Watershed_Irregular_Features

from net.imglib2.img.display.imagej import ImageJFunctions
from jarray import array

def imageprocessing():
	path = "\\\\hallingskeid.uib.no\\sgr073\\Settings\\Desktop\\sgerre.lif"
	options = ImporterOptions()
	options.setId(path)
	options.setOpenAllSeries(False)
	options.setSplitChannels(False)
	imps = BF.openImagePlus(options)

	for imp in imps:
   		project = ZProjector()
		project.setMethod(ZProjector.MAX_METHOD)
		project.setImage(imps[0])
		project.doProjection()
		impout = project.getProjection()
		projection = impout.getTitle()
	
	   	weka = WekaSegmentation()
	   	weka.setTrainingImage(impout)
	   	weka.loadClassifier("\\\\hallingskeid.uib.no\sgr073\Settings\Desktop\Imageprocessing\classifier 2.0 .model")
	   	weka.applyClassifier(False)
	   	result = weka.getClassifiedImage()
	   	
	   	ImageConverter(result).convertToRGB()
	   	ImageConverter(result).convertToGray8()
	   	
	    
	   	hist = result.getProcessor().getHistogram()
	   	lowth = Auto_Threshold.Shanbhag(hist)
	   	result.getProcessor().threshold(lowth)
	   	result.getProcessor().setThreshold(0, 0, ImageProcessor.NO_LUT_UPDATE)
		result.getProcessor().invert()
	
		IJ.run(result, "Watershed Irregular Features", "erosion=20 convexity_treshold=0 separator_size=0-Infinity")
		result.show()
   		
   		
	   	roi = ThresholdToSelection.run(result)
		result.setRoi(roi)

   		

imageprocessing()

Thanks for the bug report @Sverre!

I released a new version of the Trainable Weka Segmentation two days ago so it might have caused your problem. Can you please send me the two files you are using (sgerre.lif and “classifier 2.0 .model”) so I can reproduce the error?

Also, can you post here the output of your Log window?

Thanks a lot!

1 Like

Hello and thanks for the quick reply, I love your plugin! The lif file is 263 mb, how can I share this with you? And the forum won’t let me upload the classifier file type.

Here is a screenshot of the log.

1 Like

Thank you for helping me improving it!

I see! Can you share them using a system like Dropbox or Google drive?

I see, can you please check in your code if the following command returned true or false?

It is probably not the reason of the NullPointerException later, but just in case.

1 Like

Dropbox

False.

Is there some way to roll back my version in the meantime?

Thanks again.

1 Like

Great! I’ll work with these files and be back to you soon.

If loading the classifier returned false, there must be a problem between versions of the reading method… or your file is not in the right place…

Regarding rolling back to the old version, you can replace the corresponding jar in the plugins folder of your Fiji by the one you need from the plugin releases page.

2 Likes

And for pre-built binaries, you can download old versions from the ImageJ Maven repository:

http://maven.imagej.net/#nexus-search;gav~sc.fiji~Trainable_Segmentation~~~~kw,versionexpand

2 Likes

@Sverre did you create your model using the latest plugin version or the previous one?

I created the model using the previous version of the plugin, and it crashes when I run it on the new version. Do I need to build a new classifier? I intented to do so anyways, the current classifier is more of a test for the script.

A new classifier should definitely work. I made some tests and it seems even the former version of Weka (now we are using Weka 3.9.0) is unable to open your classifier. I’m trying to figure out if the problem is on my code or theirs…

1 Like

@Sverre, @ctrueden I confirm the problem comes from the Weka upgrade (3.7.11 to 3.9.0). I will try to figure out in the next days if there is a solution.

Ok, this solves the problem for me then at least. I’ll just make a new classifier. But will this classifier be compatible with the next update? Are they usually compatible?

I appreciate the quick help! I will probably be able to finish before I get data to analyse :slight_smile:

Yes, this is the first time I encounter this problem. I’m trying to figure out if there is a way to keep the compatibility between these two versions. Hopefully, I’ll make a new release fixing the problem soon!

1 Like

@ctrueden I have narrowed down the origin of the problem to the transition from weka 3.7.11 to 3.7.12 but I don’t know if there is any easy solution we can apply.

Let me go into the details. When saving a model (classifier) two things need to be actually saved: the serialized AbstractClassifier object containing the details of the model, and the serialized train header stored in an Instances object, which contains information about the attributes (name and number of all them, class index, possible class values, etc). In Weka 3.7.12 the Instances class has changed some fields (what seems to break backward compatibility) and that’s why the error occurs when loading the file with both objects.

You can reproduce the error with the following script and Sverre’s model file:

import weka.core.SerializationHelper;
import weka.core.Instances;
import hr.irb.fastRandomForest.FastRandomForest;

// read both model and train header
obj = SerializationHelper.readAll( "classifier 2.0 .model" );
// cast first object to FastRandomForest
frf = (FastRandomForest) obj[ 0 ];
// cast second object to Instances
data = (Instances) obj[ 1 ];

I don’t know if there is a way to read the old Instances format. What puzzles me is the fact that I couldn’t find anybody complaining about this problem in the Weka forums…

PS: the save/load model code is the same in TWS and the Weka core.

1 Like

2 posts were split to a new topic: Choosing training features in weka segmentation

@ctrueden After more searching I discovered they discussed the issue before in the Weka mailing list and it seems they broke backward compatibility to improve memory consumption. I have wrote to them to see if there is any solution…

1 Like

It seems they do not expect backward compatibility between those weka versions. This is what they said:

Users of Weka 3.6 will find that serialized models created in 3.6 cannot be used in 3.8. Unfortunately, there is no workaround for this. Models will need to be recreated in Weka 3.8. Similarly, developers using 3.6 will find that there are some small changes that they need to make to their code in order to compile against 3.8. A quick check of the javadoc for 3.8 will hopefully show what is necessary.

Yes, serialization is not intended to be used as a long-term storage mechanism. I did not realize that TWS was doing that. The proper solution is to persist the models in some more robust data structure. It would be nice if the Weka library provided such a structure, instead of every consumer of the library needing to invent their own.

However, the cat is already out of the bag. Presumably, many people have existing data saved using the serialization approach. I see two possible solutions here:

  1. Downgrade to Weka 3.7.11, at least for now.

  2. Create an upgrade tool which can convert old serialized data to the new format. This could easily live as a standalone tool, but it would be much nicer to do this transparently within ImageJ when someone opens old data. This could be done with a custom class loader which loads the old Weka 3.7 classes dynamically, but is not exactly trivial to code. Unfortunately, I cannot make time to work on such a project this year.

What do you think?

I have talked with some of the Weka developers and they have some upgrade tools already implemented. I’m going to test them and see how it goes. I hope we won’t need to downgrade Weka…

Thanks for your help!

1 Like

@Sverre, talking with the Weka developers I got a way of migrating old models into the new system, so you don’t need to recalculate yours. Here you are the new file for your model.

Cheers!

3 Likes