Support for Philips TIFF and conversion to OME-TIFF

Is Philips TIFF (created via Philips DP v1.0 software) supported by Bio-Formats (using 6.2.1)? If so is it possible to convert it to OME-TIFF? For example, I tried to convert a file 1.7G into OME-TIFF via bfconvert VOA-TEST.tiff VOA-TEST.ome.tiff and bfconvert just kept writing output VOA-TEST.ome.tiff without stopping. I halted the program when VOA-TEST.ome.tiff became 27G large!

Below is the metadata as obtained via showinfo -nopix VOA-TEST.tiff (must use -nopix since file to large). I assume “SizeZ = 1” means it is not pyramidal or consists of only one level?

Checking file format [Tagged Image File Format]
Initializing reader
TiffDelegateReader initializing VOA-TEST.tiff
Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
Initialization took 0.198s

Reading core metadata
filename = VOA-TEST.tiff
Series count = 1
Series #0 :
	Image count = 1
	RGB = true (3) 
	Interleaved = false
	Indexed = false (false color)
	Width = 181248
	Height = 99328
	SizeZ = 1
	SizeT = 1
	SizeC = 3 (effectively 1)
	Tile size = 512 x 512
	Thumbnail size = 128 x 70
	Endianness = intel (little)
	Dimension order = XYCZT (certain)
	Pixel type = uint8
	Valid bits per pixel = 8
	Metadata complete = true
	Thumbnail series = false
	-----
	Plane #0 <=> Z 0, C 0, T 0


Reading global metadata
?xml version: "1.0" encoding="UTF-8" ?
Attribute Name: "DP_COLOR_MANAGEMENT" Group="0x301D" Element="0x1013" PMSVR="IDataObjectArray"
BitsPerSample: 8
Comment: 	/Attribute
	/Attribute
		Array
					Array
						/DataObject
					/Array
				/Attribute
					Array
								Array /
							/Attribute
						/DataObject
								Array /
							/Attribute
						/DataObject
					/Array
				/Attribute
					Array
						/DataObject
						/DataObject
						/DataObject
						/DataObject
						/DataObject
						/DataObject
					/Array
				/Attribute
			/DataObject
			/DataObject
					Array
						/DataObject
					/Array
				/Attribute
			/DataObject
		/Array
	/Attribute
/DataObject

Compression: JPEG
DataObject ObjectType: "DPColorManagement"
ImageLength: 99328
ImageWidth: 181248
MetaDataPhotometricInterpretation: RGB
MetaMorph: no
NumberOfChannels: 3
Orientation: 1st row - top; 1st column - left
PhotometricInterpretation: RGB
PlanarConfiguration: Chunky
SamplesPerPixel: 3
Software: Philips DP v1.0
TileLength: 512
TileWidth: 512

Reading metadata

Not sure I can help with the rest, but ZTC are standard for Zstacks, time series, and channels (RGB in this case).

If you can get the image to open in QuPath, you might try that ome-tiff writer as well. I think it is in File->Export in the newer Milestone releases.

That’s would be a good suggestion, unfortunately I’m unable to open any of my lab’s Philip TIFF files in the newest QuPath version (0.2.0 Milestone 4) so I’m not able to try that export method.

Interestingly QuPath version 0.1.2 is able to open the same files.

What server is being used in 1.2? Openslide? ImageJ? (Image tab)

It uses Openslide (server type).

FYI ImageJ fails with

15:39:49.674 [JavaFX Application Thread] [WARN ] q.i.i.servers.ImageJServerBuilder - Error opening
/home/omero/file/slides/tiff/VOA-TEST.tiff with ImageJ: Could not open /home/omero/file/slides/tiff/VOA-TEST.tiff with ImageJ

Actually is there a way to change server for QuPath 0.2.0?

You should be able to select the server when you import the image.
image
Try selecting OpenSlide in M4.

1 Like

Update: Thanks @Research_Associate, that really helped! I can now open the Philips TIFF images. However I’m still having trouble converting the file to OME-TIFF:

I lanched QuPath 0.2.0 Milestone 4 from terminal with sudo with 8G of memory. I then opened a Philips TIFF image using the project import image method @Research_Associate suggests. When I export the image I get the same experience as my first post: QuPath just keeps writing output VOA-TEST.ome.tiff without stopping. The VOA-TEST.ome.tiff is now 17G large! The below is from stdout(?) from terminal running QuPath.

16:03:07.440 [main] [INFO ] qupath.QuPath - Launching QuPath with args: 
16:03:07.705 [JavaFX Application Thread] [INFO ] qupath.lib.gui.prefs.PathPrefs - Locale FORMAT set to en_US
16:03:07.707 [JavaFX Application Thread] [INFO ] qupath.lib.gui.prefs.PathPrefs - Locale DISPLAY set to en_US
16:03:07.791 [JavaFX Application Thread] [WARN ] qupath.lib.gui.QuPathGUI - No directory set for log files! None will be written.
16:03:07.795 [JavaFX Application Thread] [INFO ] qupath.lib.gui.QuPathGUI - QuPath build: Version: 0.2.0-m4
Build time: 2019-08-20, 20:38
Latest commit tag: 'e9b60579'
16:03:07.798 [JavaFX Application Thread] [INFO ] qupath.lib.gui.prefs.PathPrefs - Setting tile cache size to 1994.50 MB (25.0% max memory)

(QuPath:17248): Gdk-WARNING **: 16:03:08.400: XSetErrorHandler() called with a GDK error trap pushed. Don't do that.
16:03:08.771 [JavaFX Application Thread] [INFO ] q.l.i.s.b.BioFormatsOptionsExtension - Bio-Formats version 6.2.0
16:03:08.783 [JavaFX Application Thread] [INFO ] qupath.lib.gui.QuPathGUI - Loaded extension Bio-Formats server options (Bio-Formats 6.2.0) (21 ms)
16:03:08.790 [JavaFX Application Thread] [INFO ] qupath.lib.gui.QuPathGUI - Loaded extension Experimental commands (7 ms)
16:03:08.834 [JavaFX Application Thread] [INFO ] qupath.lib.gui.QuPathGUI - Loaded extension ImageJ extension (44 ms)
16:03:08.848 [JavaFX Application Thread] [INFO ] qupath.lib.gui.QuPathGUI - Loaded extension JPen extension (14 ms)
16:03:08.851 [JavaFX Application Thread] [INFO ] qupath.lib.gui.QuPathGUI - Loaded extension OpenCV extensions (3 ms)
Oct 21, 2019 4:03:08 PM jpen.provider.NativeLibraryLoader$4 run
INFO: loading JPen 2-150301 JNI library: jpen-2-4-x86_64 ...
Oct 21, 2019 4:03:08 PM jpen.provider.NativeLibraryLoader$4 run
INFO: jpen-2-4-x86_64 loaded
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/home/omero/prog/QuPath-0.2.0-m4/app/groovy-2.5.7.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
16:03:09.107 [JavaFX Application Thread] [INFO ] qupath.lib.gui.QuPathGUI - Loaded extension Rich script editor extension (256 ms)
16:03:09.112 [JavaFX Application Thread] [INFO ] q.l.i.s.o.OpenslideServerBuilder - OpenSlide version 3.4.1
16:03:09.171 [JavaFX Application Thread] [INFO ] qupath.lib.gui.QuPathGUI - Selected style: null
16:03:09.171 [JavaFX Application Thread] [INFO ] qupath.lib.gui.QuPathGUI - Performing update check...
16:03:09.175 [JavaFX Application Thread] [INFO ] qupath.lib.gui.QuPathApp - Starting QuPath with parameters: []

(QuPath:17248): Gdk-WARNING **: 16:03:09.175: XSetErrorHandler() called with a GDK error trap pushed. Don't do that.
16:03:40.426 [JavaFX Application Thread] [INFO ] qupath.lib.gui.QuPathGUI - Project set to Project: qupath-project
16:03:43.116 [JavaFX Application Thread] [WARN ] q.l.i.s.o.OpenslideImageServer - Openslide: Property 'openslide.objective-power' not available, will return default value NaN
16:03:45.573 [JavaFX Application Thread] [INFO ] qupath.lib.gui.viewer.QuPathViewer - Image data set to ImageData: Not set, VOA-TEST.tiff
16:04:24.090 [JavaFX Application Thread] [INFO ] q.lib.gui.helpers.DisplayHelpers - OME Pyramid writer: Exporting to /home/omero/file/slides/tiff/VOA-TEST.ome.tiff
16:04:24.139 [ome-pyramid-export1] [INFO ] q.l.i.writers.ome.OMEPyramidWriter - Writing VOA-TEST.tiff to /home/omero/file/slides/tiff/VOA-TEST.ome.tiff with compression Uncompressed
16:04:24.187 [ome-pyramid-export1] [INFO ] q.l.i.writers.ome.OMEPyramidWriter - Writing resolution 1 of 5 (downsample=1.0, 274703 tiles)
16:04:24.188 [ome-pyramid-export1] [INFO ] q.l.i.writers.ome.OMEPyramidWriter - Writing plane 1/1
16:10:44.948 [ForkJoinPool.commonPool-worker-13] [INFO ] q.l.i.writers.ome.OMEPyramidWriter - Written 5% tiles
16:17:22.573 [ForkJoinPool.commonPool-worker-31] [INFO ] q.l.i.writers.ome.OMEPyramidWriter - Written 10% tiles
16:23:50.388 [ForkJoinPool.commonPool-worker-21] [INFO ] q.l.i.writers.ome.OMEPyramidWriter - Written 15% tiles
16:29:53.080 [ForkJoinPool.commonPool-worker-7] [INFO ] q.l.i.writers.ome.OMEPyramidWriter - Written 20% tiles
16:35:57.039 [ForkJoinPool.commonPool-worker-27] [INFO ] q.l.i.writers.ome.OMEPyramidWriter - Written 25% tiles
16:35:57.039 [ForkJoinPool.commonPool-worker-27] [INFO ] q.l.i.writers.ome.OMEPyramidWriter - Written 30% tiles

Ah well, sorry, and thanks for posting the results. It looks like, based on the pixel count, the file should be around 18 gigabytes at the first layer, and depending on how many layers/tile sizes you are using would grow from there. At least without any compression.

For clarity =181248*99328=18003001344 or 18x10^9 bytes. Whoops, *3. R, plus G and B.

Few questions:

  1. Is the large file-size normal? Is there a setting that I’m supposed to use that I’m not?
  2. Is it possible to check if the Philips TIFF is pyramidal and how many levels it has? Does this affect the OME-TIFF output?
  3. Is there a setting to convert TIFF -> OME-TIFF so that my output isn’t so large?

If I can do TIFF -> OME-TIFF (data loss is not pref but OK) by either compression or settings that doesn’t increase the size, that would be golden.

  1. File size seems normal for something uncompressed. I don’t know anything about compression when writing out omes. Maybe @petebankhead would know more.
  2. In QuPath, there should be a Pyramid entry under Server where you saw OpenSlide
    image
    Sorry for the black on black. If there is only a 1 there, your image is not pyramidal (or at least not that QuPath can tell).
  3. No clue.

Thanks for your help so far. Also I just noticed my VOA-TEST.tiff uncompressed is ~50G with 6 levels whereas I was under the impression it was 1 level uncompressed. There is a -compression setting for bfconvert. Trying that…

Well, that certainly makes the size of your files so far seem pretty normal! Good luck.

Bio-Formats can read the Philips TIFF but appears not to see the pyramidal levels, so in QuPath tries to open everything in one go… which doesn’t end well.

I’ve just made some changes to the logic in QuPath to give a more meaningful error, and fall back to trying OpenSlide again when this happens (which is what v0.1.2 would have been using):

In QuPath under Edit → Preferences… there are options to customize the export - type OME in the search box at the top to find them. Here you can change the compression type, tile size and also request that the export use parallelization where possible (some info about that here).

it can still be rather slow though… can you say a bit more about why the conversion to OME-TIFF is needed? i ask in case some other form of export from QuPath of only part of the image might make things easier (and faster).

2 Likes

It does look as if experimenting with setting the compression type could help. From what you paste showinf mentions Compression: JPEG but your bfconvert run is writing “with compression Uncompressed”. Just run bfconvert without any options to see a list of available compression types.

@petebankhead thank you for the changes. I hope this can help other people who have trouble running QuPath! That’s a good question. My reasoning is two-fold:

  1. The lab has different WSI file types (Philips TIFF, SVS, etc) and I want to see if using unified format (e.g. OME-TIFF) is viable. Also want to see if converting to OME-TIFF will make it show up in OMERO.iviewer.

  2. The lab wishes to use a secure central repository like OMERO.server to handle WSI images for this annotation web-app we have. However neither OMERO.iviewer or OMERO.insight is able to view them. Earlier I thought it has something to due with how the TIFF is encoded ( https://github.com/openslide/openslide/issues/137 ), but inferring from our conversation I guess this can be fixed by using OpenSlide as a server rendering engine? I can create a new Issue / Topic for this if needed.

I work with software, and quite new to pathology so still familiarizing myself with the stack that’s used here.
FYI most WSI are compressed Philips TIFF of 1-4G and 40-50G uncompressed and ~6 levels pyramidal. They can be viewed using the OpenSlide-python example server https://github.com/openslide/openslide-python.

@fireofearth ah, those sound like very good reasons - and then it seems very much a Bio-Formats/OMERO question.

It sounds like the ideal solution for you might be if Bio-Formats were to support Philips TIFF (or, even better, if Philips would write OME-TIFFs…), and therefore no conversion is necessarily. Alternatively, since both SVS and Philips TIFF are TIFF-based in some way, if it might be possible to convert to OME-TIFF more efficiently then decompressing and recompressing the data that would be very helpful… but I’m not aware of any way to do this. The decompression/recompression is both slow and will lose (at least a little bit of) information.

I’d expect that converting through QuPath should be faster than with bfconvert, but I don’t know for sure if this is the case. The reason I expect it is because of the parallelization and the fact that OpenSlide can read the pyramid levels directly - while bfconvert will (presumably) need to recompute them from the full-resolution image each time.

In any case, if I were you I would probably be looking for a solution for your central repository that uses Bio-Formats and OMERO because these are active and well-supported. As far as I’m aware OpenSlide is no longer being actively developed, and is inherently limited to 2D RGB images (not multiplexed or z-stacks) and so relying on it might be troublesome in the longer term.

1 Like

Unfortunately that change will take some time as we’re just the team that does image processing, and not the ones with the tissues :sweat_smile: .

Unfortunately Philips TIFF support and Philips TIFF -> OME-TIFF conversion does not work Bio-Formats.

I’ve put my experiments in an Issue:

1 Like

Hi,

trying to summarize the position of OME on a few threads mentioned above:

  1. in the general sense, we strongly encourage and support the usage of open community file formats whenever possible. The latest version of the OME-TIFF specification is an implementation of a multi-resolution format which supports WSI data among other modalities. It is fully compatible with the full OME software stack, from Bio-Formats to the OMERO environment - see https://idr.openmicroscopy.org/webclient/img_detail/8343617/?dataset=7808 for an example of multi-resolution OME-TIFF rendered by the OMERO iviewer client.

  2. you are both correct that Bio-Formats does not currently support WSI data stored under the proprietary Philips TIFF file format. I assume with the current implementation, the library falls back to reading such files as basic TIFF files ignoring all sub-resolutions preventing the faithful conversion to a pyramidal file formats like OME-TIFF. As mentioned by Pete, OpenSlide has probably the reference open-source reader for this proprietary format. In QuPath, his latest implementation should help selecting the most appropriate library.

  3. our position about adding support for proprietary file formats has been expressed under a few formats - see our blog post or our image.sc post. Briefly, the onus of writing, testing and maintaining readers for proprietary file formats can no longer exclusively fall on the shoulders of an academically funded project.

  4. this does not mean that new proprietary file formats cannot be added to Bio-Formats. We have established successful partnerships with instrument manufacturers including Zeiss, Olympus or PerkinElmer to add support for their newest file format. More generally, we are happy to discuss integration of new readers with any committed third party.

4 Likes

Fantastic blog post in #3, and I encourage everyone to read it (especially those with purchasing power)!

3 Likes