SCIFIO needs your sample data

Hi everyone,

I have been working on the next release of scifio:


It includes support for reading from remote locations, bugfixes and more. During development I had to rely on private sample data which is limited and not suitable for public automated testing. By using real data from the public we ensure that scifio actually works in the field.

While I was able to procure and create test data for many formats, I am still looking for sample data from the formats listed bellow. This is where I hope the image.sc community can help me:

Desired Formats
  • DICOM
  • FITS
  • MicroManager 1 & 2
  • MNG
  • NRRD
  • OBF
  • PCX
  • PGM
Sample file requirements

Sample files should be as small as possible while staying representative for the format. For example a 200x200x3x5 image volume is completely sufficient to demonstrate that a certain file type can be read.

All example files will be published on https://scif.io/images/, so they can’t be encumbered by licensing or usage restrictions.

Thank you very much!
Gabriel

5 Likes

Hi Gabriel,

https://scif.io/images/ looks slick! Do any of the images under https://downloads.openmicroscopy.org/images do what you need? We’ve begun asking all bug submitters to provide their images under an open license (see COPYING in each folder) and this is the subset of images used for regression testing of #bio-formats that we can make public.

~Josh

2 Likes

Thank you @joshmoore, looks like I will find a lot of what I need there :slight_smile:
best,
Gabriel

I’d be happy to contribute (to any or both the above repositories) a few Metamorph/VisiView datasets that currently fail with bio-formats (since two years actually):

See the following commit and related issue for how they fail:

Let me know if I need to do anything to have them integrated.

Hi @gab1one,
Thanks for pushing SCIFIO dev forward.
Does this release address the performance issues when reading from ZIP files?
E.g. any 3D stack, saved as a ZIP file a la ImageJ (one 2D slice per ZIP entry), is read fast in ImageJ and several orders of magnitude slower with SCIFIO.
Would be nice to incude any such ZIP file into the SCIFIO test suite.
Thanks,
Albert

1 Like

Hi @imagejan,
there is currently no reader for that format in scifio, so we would need to add one there to fix this issue for you. I do not know when I will have time for that though.
best,
Gabriel

Sadly not, increasing the performance of SCIFIO is planed for the next iteration though, thank you for pointing out this issue.
best,
Gabriel

1 Like

hi @gab1one

I pushed 2 DICOM images which has problem to be read by SCIFIO


And a problem with tag sequences in DICOM

i’m using a lot of DICOM, SCIFIO is of interest for us because it is our hope to see compressed images supported by Fiji.
As now these bugs are preventing us to use SCIFIO as reader but I will do more test when I read DICOM and report to you everytime I see problems

Thanks for your work

Best regards,

Salim

Hi, Do you really mean this PCX format? https://en.wikipedia.org/wiki/PCX
Pretty much replaced by PNG long long time ago. Is anybody still using it?

Thank you for posting them, I will take a look at these and try to get them fixed.

Explains why I was not able to find any examples for it :slight_smile:

I’ve got a QuickBasic (!) version of the 8bit palette PCX reader and writer, from 1995. I do not think that you want to see that. :smile:
PCX (RLE compression) and GIFs (LZW compression) were quickly replaced by the PNG format (which had better compression and was free) after Unysis decided to start charging to use the compression algorithm in the GIF format.
There are plenty of converters around (ImageMagick, Gwenview, Gimp) so maybe it is not worth spending too much time as it has been a long deprectated format.

I agree, I did not spend much time on updating that format anyway, so we should be good.

Hi @gab1one,

Thanks for working to improve scifio! Somehow I missed this post the first time around. Do you have any sample data for micromanager? I recently noticed with the update that the micromanager scifio reader was not working any more. I made a separate post about the issue Fixes and improvements to SCIFIO MicromanagerFormat and filed a pull request fixing the display and plane reading issue https://github.com/scifio/scifio/pull/439

But there are many more issues we have encountered in attempting to get it working for our data. I have outlined some of the changes in the other post linked above.

To help with this effort we have collected some sample micromanager data both version 1 and version 2 gamma. They are just small videos as you suggested for testing. You are free to use them as you like and post them on the very nice scifio images section. But the videos just show beads on glass slides, so they are not the most exciting sample but they will work well for testing.

I am happy to file another pull request to further update the micromanager format for scifio if you can provide some feedback on my other post.

How were you planning to deal with the differences between micromanager version 1 and 2. Should a separate format be created or should the current reader be modified to accept both formats?

200831_256x256_BeadSample_1.4_20frames_1.zip (3.6 MB) 200831_256x256_BeadSample_2.0gamma_20frames_1.zip (3.6 MB)

2 Likes

@marktsuchida do you happen to have any pre-micromanager v1.4 sample data you could share here for testing? Thank you!

1 Like

@hinerm I should be able to create, say, Micro-Manager 1.3 datasets, if they don’t need to be real data (binaries are still available for download). Is there a specific feature you need to test, or just an assortment of a few simple files?

Just a few simple files would be great! Nothing in particular… just compatibility.

@marktsuchida That would be very helpful if you could create some Micro-Manager 1.3 datasets. We would like to add compatibility for MM 1.4 and MM 2.0 to the SCIFIO micromanager format. As you can see above, I have sample data for the newer version, but we need to make sure the updates doesn’t break reading of MM 1.3 datasets. That is what we need the sample data to test and ensure.

Hi @gab1one,
Have you tested e.g. TIFF stacks compressed in a ZIP file, or BZIP2 files? Last I checked, SCIFIO took minutes to open compressed files that ImageJ takes seconds to open. Optimizing opening compressed files would be useful.
Thanks,
Albert