Error message when exporting prediction maps in pixel classification

Hi all,
Since having updated my pc from Windows 7 to Windows 10 I’m having problems with exporting a batch of prediction maps in my ilastik pixel classification project. When starting the batch processing I get thousands of Warnings like

'Warning: TIFFDecoder: no TIFFTAG_SAMPLEFORMAT or TIFFTAG_DATATYPE, guessing pixeltype 'UINT16'.'

which seem inexplicable to me as I’ve always been working with the same TIFF datafiles ove the past months.
Ususally the batch export works for few data files and then stops somewhere returning certain error messages such as

ERROR 2020-02-05 23:16:42,358 request 5492 6768 Traceback (most recent call last):
  File "C:\Program Files\ilastik-1.3.3post2\ilastik-meta\lazyflow\lazyflow\request\request.py", line 395, in _post_execute
    self._sig_finished(self._result)
  File "C:\Program Files\ilastik-1.3.3post2\ilastik-meta\lazyflow\lazyflow\request\request.py", line 80, in __call__
    f(*args, **kwargs)
  File "C:\Program Files\ilastik-1.3.3post2\ilastik-meta\lazyflow\lazyflow\utility\roiRequestBatch.py", line 245, in _handleCompletedRequest
    self.resultSignal(roi, result)
  File "C:\Program Files\ilastik-1.3.3post2\ilastik-meta\lazyflow\lazyflow\utility\orderedSignal.py", line 80, in __call__
    f(*args, **kw)
  File "C:\Program Files\ilastik-1.3.3post2\ilastik-meta\lazyflow\lazyflow\operators\ioOperators\ioOperators.py", line 648, in handle_block_result
    self.d.write_direct(data.view(numpy.ndarray), dest_sel=slicing)
  File "C:\Program Files\ilastik-1.3.3post2\lib\site-packages\h5py\_hl\dataset.py", line 758, in write_direct
    self.id.write(mspace, fspace, source, dxpl=self._dxpl)
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py\h5d.pyx", line 221, in h5py.h5d.DatasetID.write
  File "h5py\_proxy.pyx", line 132, in h5py._proxy.dset_rw
  File "h5py\_proxy.pyx", line 93, in h5py._proxy.H5PY_H5Dwrite
OSError: Can't write data (file write failed: time = Wed Feb  5 23:16:42 2020
, filename = '____________.h5', file descriptor = 5, errno = 22, error message = 'Invalid argument', buf = 0000022CB2F16C48, total write size = 511224, bytes this sub-write = 511224, bytes actually written = 18446744073709551615, offset = 1627957456)

Does somebody know what to do?

Hi @apoptosis,

the Warning you get means that we don’t fully understand the type of tiff you are loading (ilastik’s tiff support is not the best in the world). In general you could also consider to convert your images to h5 in fiji first using the ilastik import/export plugin to get rid of those issues. Also h5 should be faster in ilastik, especially for larger data.

For the error, can you maybe isolate a single image where this is happening. Ideally it would always happen at the same image which could make it easy for us to reproduce it ;).

Hi @k-dominik,
Thank you for answering. I will test your advice.
Here’s a snapshot of the tiff file where the prediction map export stopped:


Unfortunately the error occurs for more than this tiff file.

Hi @apoptosis, thank you! I guess more interesting than the screenshot would be the original tif file, so we could get a better understanding on what is going wrong, which would help future development. In the meantime the best advice really is converting the files via fiji…

Yes, of course. Here it is:
frame-0001.tif (2.0 MB)
Thank you :slight_smile:

1 Like

thank you very much :slight_smile:

Hi @k-dominik,
Well, I tried your idea using hdf5 files for the batch processing but after a few movies were exported the export stopped with these error messages:

RuntimeError: Can't decrement id ref count (file write failed: time = Thu Feb 13 14:16:39 2020
, filename = '__(path_to_my_data)___.h5', file descriptor = 6, errno = 22, error message = 'Invalid argument', buf = 00000227B87F6018, total write size = 511224, bytes this sub-write = 511224, bytes actually written = 18446744073709551615, offset = 883599536)

ERROR 2020-02-13 14:16:58,804 log_exception 5644 460 Error encountered during batch processing:
Can't decrement id ref count (file write failed: time = Thu Feb 13 14:16:39 2020
, filename = '__(path_to_my_data)____.h5', file descriptor = 6, errno = 22, error message = 'Invalid argument', buf = 00000227B87F6018, total write size = 511224, bytes this sub-write = 511224, bytes actually written = 18446744073709551615, offset = 883599536)

Do you know what’s the problem?
Thanks in advance.

hmm, is it possible that you are running out of disk space?

actually there’s still a lot of disk space left :frowning:

Recently I noticed that the ilastik fiji plugin swaps the dimensions when converting a tif file to a hdf5 file. You can adjust the dimensions in the input menu of the pixel classification workflow but there is no possibility to do this in batch processing. Maybe this is the point?

The plugin should only rearrange the dimensions - and ilastik should understand this.
Is it failing randomly for different images, or will it fail always on the same one?

It fails for all files the same way. Always when loading my hdf5-files into ‘raw data’ I will have to reorder the dimensions from tzyxc to ztyxc.

this is really unfortunate. There is already an issue about the plugin not adding axis information :confused:

Unfortunately you cannot add this information in batch processing.

Alternatively you could use the headless mode, where you can specify the input axes order with --input_axes=tzyxc

That would be the best solution! Thanks :slight_smile:

Cool, let us know if you run into any problems setting up batch processing in headless mode!

1 Like