Can't reopen Ilastik carving projects

Dear @ilastik_team,

I am having trouble re-opening carving project files after closing them, and am getting an error message:

Could not load project file.
index 1108889 is out of bounds for axis 1 with size 1108887

Is there a way to get the annotations out of this project and reuse them, and to stop this error in the future?

I am running the ilastik windows binary 1.3.3post3. The following is the full ilastik_log.txt output:

Thanks very much for the help!

INFO 2021-04-27 10:46:58,633 projectManager 119568 128656 Opening Project: C:/Users/quickep/tmp_data/PQ_good_carving-largefilt.ilp
WARNING 2021-04-27 10:46:58,634 warnings 119568 128656 dataset.py(313): H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead.
WARNING 2021-04-27 10:46:58,738 warnings 119568 128656 slot.py(1287): UserWarning: Deprecated use slot.operator property instead
INFO 2021-04-27 10:47:58,277 carvingSerializer 119568 128656 loading object with name=‘Vessel Lumen’
INFO 2021-04-27 10:47:58,295 carvingSerializer 119568 128656 [CarvingSerializer] de-serializing Vessel Lumen, with opCarving=2133736868272, mst=2128740181552
INFO 2021-04-27 10:47:58,295 carvingSerializer 119568 128656 203606 voxels labeled with green seed
INFO 2021-04-27 10:47:58,295 carvingSerializer 119568 128656 34268 voxels labeled with red seed
INFO 2021-04-27 10:47:58,295 carvingSerializer 119568 128656 object is made up of 13029 supervoxels
INFO 2021-04-27 10:47:58,295 carvingSerializer 119568 128656 bg priority = 0.950000
INFO 2021-04-27 10:47:58,295 carvingSerializer 119568 128656 no bias below = 64
INFO 2021-04-27 10:47:58,296 carvingSerializer 119568 128656 loading object with name=‘Vessel Lumen 2’
INFO 2021-04-27 10:47:58,297 carvingSerializer 119568 128656 [CarvingSerializer] de-serializing Vessel Lumen 2, with opCarving=2133736868272, mst=2128740181552
INFO 2021-04-27 10:47:58,298 carvingSerializer 119568 128656 1218 voxels labeled with green seed
INFO 2021-04-27 10:47:58,298 carvingSerializer 119568 128656 609 voxels labeled with red seed
INFO 2021-04-27 10:47:58,298 carvingSerializer 119568 128656 object is made up of 1343 supervoxels
INFO 2021-04-27 10:47:58,298 carvingSerializer 119568 128656 bg priority = 0.950000
INFO 2021-04-27 10:47:58,298 carvingSerializer 119568 128656 no bias below = 64
INFO 2021-04-27 10:47:58,298 carvingSerializer 119568 128656 loading object with name=‘Vitreous’
INFO 2021-04-27 10:47:58,350 carvingSerializer 119568 128656 [CarvingSerializer] de-serializing Vitreous, with opCarving=2133736868272, mst=2128740181552
INFO 2021-04-27 10:47:58,350 carvingSerializer 119568 128656 697494 voxels labeled with green seed
INFO 2021-04-27 10:47:58,350 carvingSerializer 119568 128656 61744 voxels labeled with red seed
INFO 2021-04-27 10:47:58,350 carvingSerializer 119568 128656 object is made up of 621318 supervoxels
INFO 2021-04-27 10:47:58,350 carvingSerializer 119568 128656 bg priority = 0.950000
INFO 2021-04-27 10:47:58,350 carvingSerializer 119568 128656 no bias below = 64
INFO 2021-04-27 10:47:58,487 opCarving 119568 128656 Writing seeds to label array
INFO 2021-04-27 10:48:11,259 opCarving 119568 128656 Writing seeds to label array took 12.771857 seconds
INFO 2021-04-27 10:48:12,400 opCarving 119568 128656 Writing seeds to MST
INFO 2021-04-27 10:48:30,563 opCarving 119568 128656 Writing seeds to MST took 18.163746 seconds
INFO 2021-04-27 10:48:30,563 carvingSerializer 119568 128656 restored seeds
INFO 2021-04-27 10:48:30,563 opCarving 119568 128656 building ‘done’ lut
ERROR 2021-04-27 10:48:30,573 log_exception 119568 128656 Traceback (most recent call last):
File “C:\Program Files\ilastik-1.3.3post3\ilastik-meta\ilastik\ilastik\shell\projectManager.py”, line 460, in _loadProject
serializer.deserializeFromHdf5(self.currentProjectFile, projectFilePath, self._headless)
File “C:\Program Files\ilastik-1.3.3post3\ilastik-meta\ilastik\ilastik\applets\base\appletSerializer.py”, line 1155, in deserializeFromHdf5
self._deserializeFromHdf5(topGroup, groupVersion, hdf5File, projectFilePath, headless)
File “C:\Program Files\ilastik-1.3.3post3\ilastik-meta\ilastik\ilastik\workflows\carving\carvingSerializer.py”, line 200, in _deserializeFromHdf5
opCarving._buildDone()
File “C:\Program Files\ilastik-1.3.3post3\ilastik-meta\ilastik\ilastik\workflows\carving\opCarving.py”, line 229, in _buildDone
self._done_seg_lut[objectSupervoxels] = self._mst.object_names[name]
IndexError: index 1108889 is out of bounds for axis 1 with size 1108887

ERROR 2021-04-27 10:48:30,575 log_exception 119568 128656 Project could not be loaded due to the exception shown above.
Aborting Project Open Action
ERROR 2021-04-27 10:48:30,581 log_exception 119568 128656 Traceback (most recent call last):
File “C:\Program Files\ilastik-1.3.3post3\ilastik-meta\ilastik\ilastik\shell\gui\ilastikShell.py”, line 1542, in _loadProject
self.projectManager._loadProject(hdf5File, projectFilePath, readOnly)
File “C:\Program Files\ilastik-1.3.3post3\ilastik-meta\lazyflow\lazyflow\utility\timer.py”, line 153, in wrapper
return func(*args, **kwargs)
File “C:\Program Files\ilastik-1.3.3post3\ilastik-meta\ilastik\ilastik\shell\projectManager.py”, line 460, in _loadProject
serializer.deserializeFromHdf5(self.currentProjectFile, projectFilePath, self._headless)
File “C:\Program Files\ilastik-1.3.3post3\ilastik-meta\ilastik\ilastik\applets\base\appletSerializer.py”, line 1155, in deserializeFromHdf5
self._deserializeFromHdf5(topGroup, groupVersion, hdf5File, projectFilePath, headless)
File “C:\Program Files\ilastik-1.3.3post3\ilastik-meta\ilastik\ilastik\workflows\carving\carvingSerializer.py”, line 200, in _deserializeFromHdf5
opCarving._buildDone()
File “C:\Program Files\ilastik-1.3.3post3\ilastik-meta\ilastik\ilastik\workflows\carving\opCarving.py”, line 229, in _buildDone
self._done_seg_lut[objectSupervoxels] = self._mst.object_names[name]
IndexError: index 1108889 is out of bounds for axis 1 with size 1108887

Hello @peq10,

welcome to the image.sc community! :slight_smile:

Sorry that you ran in an issue in carving :grimacing:
just so I will be able to reproduce it - did you by any chance, segment some objects, and then went back to the watershed, changed parameters there and went back to segmenting more objects?

Hi @k-dominik thanks for your reply! The carving is a great tool - I appreciate all the work put into it!

Unfortunately that is exactly what I did - I was trying to see if I got a better result with a wider filter. Is that a big no?

If I can’t re-open this project is it possible to import annotations into a new project with a different preprocessing step?

Cheers,

Peter

great that Carving is useful for you.

well… the worklfow was not designed to enable this going back an forth. The segmentaiton results point to the superpixels generated in the first step - and there is no notion of having multiple of those.

On the positive side, it should be possible to get your annotations out of ilastik. the .ilp file is an hdf5 file and can be manipulated/read with the any hdf5 library.

so in the key carving/objects you’ll find a group for each of your saved objects. Inside each of those there will be the dataset fg_voxels, and bg_voxels (foreground, and background voxels). Those are arrays of coordinates in 3D. So in order to get them back in to ilastik, the idea would be to save those as images.
So this could be done in a Python script

Thanks for all the help!

Are the fg_voxels in the superpixel coordinates? So if I copied them into a different .ilp file I would run into the same error? Or are they in the original pixel coordinates?

those are the pixel coordinates of the seeds, so you would not run into the same error. I’m currently investigating if deleting the supervoxels is not a better idea…

Fantastic thanks! I will have a go extracting the annotations and transferring them to a new project in the meantime.

okay, I’d suggest the following (not really tested)…

import h5py
import numpy

project_file = "myprojectfile.ilp"  # well.. please backup your file first

with h5py.File(project_file, "a") as f:
    for obj_group in f["carving/objects"].values():
        del obj_group["sv"]
        obj_group.create_dataset("sv", data=numpy.ones((1, 0), dtype="int64"))

then open this project file in carving and load the objects one by one…

1 Like

I just tried this and it opens so it looks like it did for me?

it even kept the supervoxels so I didn’t have to re-run the preprocessing which I thought I would have to!

so you ran the python snipped that i had there? And the objects came back for you? did you load, and then save them? Does the segmented objects layer look like you’d expect it?

I ran the python and then could load the project. The objects look like they are coming back - I’m just looking through them all now but they are so far good!

1 Like

I can confirm they are all there and working - thanks so much for the help!

I marked your snippet as the answer as I saw you restored it :slight_smile: