Unable to save trained model

Hello,

thank you for making this fantastic tool available in Fiji. I have used it mainly on SBF SEM stacks and it worked wonders. During the easter holidays I started a rather long 3D training session, which went fine, looking at the previews and log info. However, for some reason I am unable to save the file now and also can not predict any images.

[INFO] Saving model to C:\Users\admin-em\Desktop\n2v-831865674021231162.bioimage.io.zip
Exception in thread "Thread-222" java.util.zip.ZipError: zip END header not found
	at com.sun.nio.zipfs.ZipFileSystem.zerror(ZipFileSystem.java:1651)
	at com.sun.nio.zipfs.ZipFileSystem.findEND(ZipFileSystem.java:1040)
	at com.sun.nio.zipfs.ZipFileSystem.initCEN(ZipFileSystem.java:1049)
	at com.sun.nio.zipfs.ZipFileSystem.<init>(ZipFileSystem.java:130)
	at com.sun.nio.zipfs.ZipFileSystemProvider.newFileSystem(ZipFileSystemProvider.java:139)
	at java.nio.file.FileSystems.newFileSystem(FileSystems.java:390)
	at net.imagej.modelzoo.io.ModelZooIOPlugin.save(ModelZooIOPlugin.java:187)
	at net.imagej.modelzoo.io.DefaultModelZooIOService.save(DefaultModelZooIOService.java:63)
	at net.imagej.modelzoo.display.SwingModelArchiveDisplayViewer.saveModelTo(SwingModelArchiveDisplayViewer.java:439)
	at net.imagej.modelzoo.display.SwingModelArchiveDisplayViewer.access$000(SwingModelArchiveDisplayViewer.java:104)
	at net.imagej.modelzoo.display.SwingModelArchiveDisplayViewer$1.lambda$actionPerformed$0(SwingModelArchiveDisplayViewer.java:208)
	at java.lang.Thread.run(Thread.java:748)

It appears that there is a problem with the zip-file, when trying to predict images with the model.

[INFO] Saving model to C:\Users\admin-em\Desktop\n2v-831865674021231162.bioimage.io.zip
Exception in thread "Thread-222" java.util.zip.ZipError: zip END header not found
	at com.sun.nio.zipfs.ZipFileSystem.zerror(ZipFileSystem.java:1651)
	at com.sun.nio.zipfs.ZipFileSystem.findEND(ZipFileSystem.java:1040)
	at com.sun.nio.zipfs.ZipFileSystem.initCEN(ZipFileSystem.java:1049)
	at com.sun.nio.zipfs.ZipFileSystem.<init>(ZipFileSystem.java:130)
	at com.sun.nio.zipfs.ZipFileSystemProvider.newFileSystem(ZipFileSystemProvider.java:139)
	at java.nio.file.FileSystems.newFileSystem(FileSystems.java:390)
	at net.imagej.modelzoo.io.ModelZooIOPlugin.save(ModelZooIOPlugin.java:187)
	at net.imagej.modelzoo.io.DefaultModelZooIOService.save(DefaultModelZooIOService.java:63)
	at net.imagej.modelzoo.display.SwingModelArchiveDisplayViewer.saveModelTo(SwingModelArchiveDisplayViewer.java:439)
	at net.imagej.modelzoo.display.SwingModelArchiveDisplayViewer.access$000(SwingModelArchiveDisplayViewer.java:104)
	at net.imagej.modelzoo.display.SwingModelArchiveDisplayViewer$1.lambda$actionPerformed$0(SwingModelArchiveDisplayViewer.java:208)
	at java.lang.Thread.run(Thread.java:748)
java.util.zip.ZipException: zip file is empty
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.<init>(ZipFile.java:230)
	at java.util.zip.ZipFile.<init>(ZipFile.java:160)
	at java.util.zip.ZipFile.<init>(ZipFile.java:131)
	at net.imagej.modelzoo.io.ModelZooIOPlugin.open(ModelZooIOPlugin.java:102)
	at net.imagej.modelzoo.io.ModelZooIOPlugin.open(ModelZooIOPlugin.java:141)
	at net.imagej.modelzoo.io.DefaultModelZooIOService.open(DefaultModelZooIOService.java:47)
	at net.imagej.modelzoo.io.DefaultModelZooIOService.open(DefaultModelZooIOService.java:53)
	at net.imagej.modelzoo.consumer.command.AbstractSingleImagePredictionCommand.getArchive(AbstractSingleImagePredictionCommand.java:115)
	at net.imagej.modelzoo.consumer.command.DefaultSingleImagePredictionCommand.createPrediction(DefaultSingleImagePredictionCommand.java:85)
	at net.imagej.modelzoo.consumer.command.AbstractSingleImagePredictionCommand.run(AbstractSingleImagePredictionCommand.java:93)
	at org.scijava.command.CommandModule.run(CommandModule.java:196)
	at org.scijava.module.ModuleRunner.run(ModuleRunner.java:165)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:124)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:63)
	at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

It would be a shame if I were to lose the results, maybe you have an idea. I am using Win10 btw.
Best,
Leo

Hi @lbreitsprech

I took the liberty to open a new topic with this issue.

@jdeschamps could you have a look at this? Maybe something happened to the modelzoo version in between?

1 Like

Hi @lbreitsprech,

Unfortunately I did not manage to reproduce the error (3D stack, training, saving model, then prediction). Could you check if your Fiji is up to date (Help / Update…)? Could you tell us the imagej-modelzoo and n2v versions (in the update menu, click on “Advanced menu” and search the items)?

It is surprising because I believe N2V saves the best and the last model throughtout the training, so if it did manage to save them many times over, I don’t expect a problem within the code itself or incompatible versions. When you say that you don’t manage to predict, is it from the model window that opens at the end of the training?

As I said, N2V is supposed to save the models along the way. So basically, somewhere on your computer there should be files ending with *.bioimage.io.zip saved in temporary (maybe hidden) folder. Under Windows I would guess that they are in the AppData folder (in user folder?). But that’s a non-Windows user guess. One way to maybe figure it out would be to check what is the default folder when you try to save the model. Otherwise, it could be that it can be discovered by the Windows search!

If you find such models (here is an example of the kind of name to expect: n2v-992915525714050783.bioimage.io.zip) with dates that fits your training, save them aside and try to predict on a small image/stack with another Fiji instance to check the integrity of the zip file.

2 Likes

Hi @jdeschamps ,

I have checked the default saving folder (AppData) and mysteriously there are no folders with the respective date of the training. I have checked older training sessions folders and they all seem to be empty for some reason (with the structure looking like that C:\Users\admin-em\AppData\Local\Temp\n2v-latest-3878865978550891943\tf_saved_model_bundle\variables, but containing no files whatsoever).

I now suspect that Windows deleted the temporary files during my easter absence… I thought I had everything unchecked in this regard, but I found an activated function “deleting files that my apps aren’t using” in the Storage Sense configuration.

Anyhow, thanks for the quick reply and keep up the great work.

Cheers,
Leo

1 Like

Oh damn… Basically, the saving function you get at the end of the training is just loading the best or last model and saving it somewhere else. So unfortunately that might explain your problem.