Dealing with BigTiff files (open, edit, save, etc)

Hello everyone,

I am currently working with some BigTiff bio-images that are all more than 8 gbs. All of them need some transformation editings like flip, trim and free transformation. Photoshop was able to read and edit, but fail to save them due to file size limitation (psd wont work because we need the tiff format, and I want to keep the original resolution due to the need to later analysis). If anyone know any image processing software that can save BigTiff file with no size limitation, please let me know, and any other advice are welcome as well!

Thank you
Jonathan

Just to clarify, have you already tried using BioFormats with Fiji/ImageJ?

Thanks for the reply, I did try that, but wasn´t not successful in loading the image (the image is about 11 gbs), tried in both hyperstack and standard ImageJ option. Keep getting the arrayindexoutofboundsexception error…

Thanks!
Jonathan

I see. Unfortunately, I don’t have experience with truly large data sets (although I suppose “large” is a relative term). I have seen BigDataViewer suggested now and then. However, I suspect that might have view-only capabilities. I don’t know if it would be possible to open with that and save with something else (like BioFormats). There is a list of I/O plugins but I have no idea if any of them will be helpful.

It seems that bioformats should support BigTiff: https://docs.openmicroscopy.org/bio-formats/5.7.3/formats/tiff.html

Which file ending do you use for your images? According to the wiki (https://imagej.net/Bio-Formats) Tiff files with the file ending .tif or .tiff are loaded with the standard 32-bit Tiff format. Whereas .tf2 , .tf8, and .btf is loaded with the 64-bit Tiff format. Could you test that?

If that does not help could you specify what is writing those files?

1 Like

Try libvips for out-of-core processing of huge two-dimensional images.

Python’s numpy, scipy.ndimage, and tifffile libraries should be able to read, transform, and write practically any dimensional and size of images given enough RAM. Tifffile can also directly memory map ndarrays in (Big)TIFF files if the image data is stored contiguously such as in uncompressed ImageJ hyperstacks.

Hi @Yuheng_Chen, do you have the stack trace for the exception you are seeing when opening with Bio-Formats? It should be able to handle the reading and writing of BigTiff images.

Dear all, I have some problem with tiff or big tiff from .ndpi
If I create Tiff from NDPItools with LZW compression and I tried to open the file with NDPI tools or Bio-format but it failed for the 40x magnification (14gb compressed) . if I use 10x magnification (1 Gb file compressed) I have error (see below). And if I create Big-tiff I can open my 2gb file but with error see the print screen. Do you know if imageJ have a limit in pixel in 2D ?

Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
[ERROR] Module threw exception
java.lang.NegativeArraySizeException
at loci.formats.tiff.TiffParser.getSamples(TiffParser.java:1063)
at loci.formats.tiff.TiffParser.getSamples(TiffParser.java:851)
at loci.formats.in.MinimalTiffReader.openBytes(MinimalTiffReader.java:312)
at loci.formats.in.TiffDelegateReader.openBytes(TiffDelegateReader.java:71)
at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:156)
at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:148)
at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:200)
at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:348)
at loci.formats.DimensionSwapper.openBytes(DimensionSwapper.java:249)
at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:348)
at loci.formats.MinMaxCalculator.openBytes(MinMaxCalculator.java:269)
at loci.formats.MinMaxCalculator.openBytes(MinMaxCalculator.java:260)
at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:334)
at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:334)
at loci.plugins.util.ImageProcessorReader.openProcessors(ImageProcessorReader.java:186)
at loci.plugins.in.ImagePlusReader.readProcessors(ImagePlusReader.java:422)
at loci.plugins.in.ImagePlusReader.readPlanes(ImagePlusReader.java:387)
at loci.plugins.in.ImagePlusReader.readImage(ImagePlusReader.java:282)
at loci.plugins.in.ImagePlusReader.readImages(ImagePlusReader.java:243)
at loci.plugins.in.ImagePlusReader.readImages(ImagePlusReader.java:221)
at loci.plugins.in.ImagePlusReader.openImagePlus(ImagePlusReader.java:116)
at loci.plugins.in.Importer.readPixels(Importer.java:149)
at loci.plugins.in.Importer.run(Importer.java:86)
at loci.plugins.LociImporter.run(LociImporter.java:78)
at ij.IJ.runUserPlugIn(IJ.java:230)
at ij.IJ.runPlugIn(IJ.java:193)
at ij.IJ.runPlugIn(IJ.java:182)
at net.imagej.legacy.command.LegacyCommand.run(LegacyCommand.java:59)
at org.scijava.command.CommandModule.run(CommandModule.java:199)
at org.scijava.module.ModuleRunner.run(ModuleRunner.java:168)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:127)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:66)
at org.scijava.thread.DefaultThreadService$3.call(DefaultThreadService.java:238)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

It looks likely that you are running into the limit for a single plane size. If you select Crop on Import and try to view a smaller region do you see the same exception?

if I crop on import I don’t have this problem. But I would like to avoid to crop the file.
Do you know if this limitation is a software or a hardware limitation?

Should I stop with ImageJ and start a script on Qupath ?

It does sound like you are running into the 2GB limit for displaying a single frame, which would be a software limitation: https://imagej.net/Frequently_Asked_Questions#What_is_the_largest_size_image_that_ImageJ_can_open.3F

I found a way to crop during my macro but I have issue with Bio-format
If i try to open the files with bio-format in the macro with 2 files it’s ok but with more files in my folder it take ages to open or freeze ?
I also have this error sometime when I drag and drop my macro.

Exception in thread “AWT-EventQueue-0” java.lang.Error: Error: could not match input
at org.scijava.ui.swing.script.highliters.ImageJMacroTokenMaker.zzScanError(ImageJMacroTokenMaker.java:2624)
at org.scijava.ui.swing.script.highliters.ImageJMacroTokenMaker.yylex(ImageJMacroTokenMaker.java:2883)
at org.scijava.ui.swing.script.highliters.ImageJMacroTokenMaker.getTokenList(ImageJMacroTokenMaker.java:2474)
at org.fife.ui.rsyntaxtextarea.RSyntaxDocument.getTokenListForLine(RSyntaxDocument.java:431)
at org.fife.ui.rsyntaxtextarea.SyntaxView.paint(SyntaxView.java:728)
at javax.swing.plaf.basic.BasicTextUI$RootView.paint(BasicTextUI.java:1434)
at javax.swing.plaf.basic.BasicTextUI.paintSafely(BasicTextUI.java:737)
at org.fife.ui.rtextarea.RTextAreaUI.paintSafely(RTextAreaUI.java:539)
at javax.swing.plaf.basic.BasicTextUI.paint(BasicTextUI.java:881)
at javax.swing.plaf.basic.BasicTextUI.update(BasicTextUI.java:860)
at org.fife.ui.rtextarea.RTextAreaBase.paintComponent(RTextAreaBase.java:735)
at org.fife.ui.rsyntaxtextarea.RSyntaxTextArea.paintComponent(RSyntaxTextArea.java:2072)
at javax.swing.JComponent.paint(JComponent.java:1056)
at javax.swing.JComponent.paintChildren(JComponent.java:889)
at javax.swing.JComponent.paint(JComponent.java:1065)
at javax.swing.JViewport.paint(JViewport.java:728)
at javax.swing.JComponent.paintChildren(JComponent.java:889)
at javax.swing.JComponent.paint(JComponent.java:1065)
at javax.swing.JComponent.paintChildren(JComponent.java:889)
at javax.swing.JSplitPane.paintChildren(JSplitPane.java:1047)
at javax.swing.JComponent.paint(JComponent.java:1065)
at javax.swing.JComponent.paintChildren(JComponent.java:889)
at javax.swing.JComponent.paint(JComponent.java:1065)
at javax.swing.JComponent.paintChildren(JComponent.java:889)
at javax.swing.JSplitPane.paintChildren(JSplitPane.java:1047)
at javax.swing.JComponent.paint(JComponent.java:1065)
at javax.swing.JComponent.paintChildren(JComponent.java:889)
at javax.swing.JComponent.paint(JComponent.java:1065)
at javax.swing.JComponent.paintChildren(JComponent.java:889)
at javax.swing.JComponent.paint(JComponent.java:1065)
at javax.swing.JLayeredPane.paint(JLayeredPane.java:586)
at javax.swing.JComponent.paintChildren(JComponent.java:889)
at javax.swing.JComponent.paintToOffscreen(JComponent.java:5217)
at javax.swing.RepaintManager$PaintManager.paintDoubleBuffered(RepaintManager.java:1579)
at javax.swing.RepaintManager$PaintManager.paint(RepaintManager.java:1502)
at javax.swing.RepaintManager.paint(RepaintManager.java:1272)
at javax.swing.JComponent.paint(JComponent.java:1042)
at java.awt.GraphicsCallback$PaintCallback.run(GraphicsCallback.java:39)
at sun.awt.SunGraphicsCallback.runOneComponent(SunGraphicsCallback.java:79)
at sun.awt.SunGraphicsCallback.runComponents(SunGraphicsCallback.java:116)
at java.awt.Container.paint(Container.java:1978)
at java.awt.Window.paint(Window.java:3906)
at javax.swing.RepaintManager$4.run(RepaintManager.java:842)
at javax.swing.RepaintManager$4.run(RepaintManager.java:814)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)
at javax.swing.RepaintManager.paintDirtyRegions(RepaintManager.java:814)
at javax.swing.RepaintManager.paintDirtyRegions(RepaintManager.java:789)
at javax.swing.RepaintManager.prePaintDirtyRegions(RepaintManager.java:738)
at javax.swing.RepaintManager.access$1200(RepaintManager.java:64)
at javax.swing.RepaintManager$ProcessingRunnable.run(RepaintManager.java:1732)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758)
at java.awt.EventQueue.access$500(EventQueue.java:97)
at java.awt.EventQueue$3.run(EventQueue.java:709)
at java.awt.EventQueue$3.run(EventQueue.java:703)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:728)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:205)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)

my macro work if I use 2 files but with multiple files it fail. And If I create a second macro to run the second part with Bioformat it’s ok. Did I write a mistake or the problem is the memory ??

I start the macro last night and it was ok with 6 files but t take a very long time to open the Bigtiff with LZW compression.
Do you know if bioformat have a problem ?
Because If I run the extraction part of the macro and then the conversion part in a second macro it’s fast 44 second to open a file with Bioformat. But in the same macro it take a lot of minutes to open the file.

Do you have a link to the macros you are using?

Hello, I will send you 2 macros the first one have a problem but I don’t know why and the second is ok because I split the batch

input = getDirectory("choose the input directory"); //ask for the input directory
output= input + "/Images/"; 
File.makeDirectory(output);

function processFolder(input);
list = getFileList(input);   
for (i=0; i<list.length; i++) {
			if (endsWith(list[i],".ndpi")){
			inputPath= input + list[i];
			//only big data superior to 1go in .ndpi
			length = File.length(inputPath);
				if (length > 1000000000){
					run("Preview NDPI...", "ndpitools=[inputPath]");
					waitForUser("draw a box");
					print("Extraction NDPI to tiff:  " + inputPath +" please wait");
					run("Custom extract to TIFF / Mosaic...", "label=[] format_of_split_images=[TIFF with LZW compression] store_split_images_in_bigtiff_files_rather_than_tiff make_mosaic=never mosaic_pieces_format=[TIFF with JPEG compression] requested_jpeg_compression=75 mosaic_pieces_overlap=0.000000 mosaic_pieces_overlap_unit=pixels size_limit_on_each_mosaic_piece=0 width_of_each_mosaic_piece_in_pixels=0 height_of_each_mosaic_piece_in_pixels=0 save_these_choices_in_imagej's_preferences_file extract_images_at_magnification_10x extract_images_with_z-offset_0");
					close("*");
					print("Extraction NDPI to tiff:  " + inputPath +" OK");
					
				}
				//small data save in tiff see if need to save in Bigtiff
				else {
					print("Extraction NDPI to tiff:  " + inputPath +" please wait");
					run("Custom extract to TIFF / Mosaic...", "no=[inputPath] format_of_split_images=[TIFF with LZW compression] store_split_images_in_bigtiff_files_rather_than_tiff make_mosaic=never mosaic_pieces_format=[TIFF with JPEG compression] requested_jpeg_compression=75 mosaic_pieces_overlap=0.000000 mosaic_pieces_overlap_unit=pixels size_limit_on_each_mosaic_piece=0 width_of_each_mosaic_piece_in_pixels=0 height_of_each_mosaic_piece_in_pixels=0 save_these_choices_in_imagej's_preferences_file extract_images_at_magnification_10x extract_images_with_z-offset_0");
					close("*");
					print("Extraction NDPI to tiff:  " + inputPath +" OK"); 
				}
			}
			//.tif
			else {
				list2 = getFileList(input); 
				for (j=0; j<list2.length; j++) {
					if (endsWith(list2[j],".tif")){
					inputPath2= input +  list2[j];
					print("Conversion: " + inputPath2 +" please wait");
					//only big data superior to 200KB in .ndpi
					length = File.length(inputPath2);
					if (length > 700000){
							run("Bio-Formats", "open=[inputPath2] color_mode=Default rois_import=[ROI manager] view=Hyperstack stack_order=XYCZT");
							wait(2000);
							imagesName=getTitle();
							saveAs("Tiff", output + File.separator + imagesName);
							close("*");
							File.delete(inputPath2);
						}
						//else delete file because NDPItools create map and small files during the preview
						else {
							File.delete(inputPath2);
							}
													}
									}
				}
}	
print("\\Clear");
print("Macro ends");		


and the second

input = getDirectory("choose the input directory"); //ask for the input directory
output= input + "/Images/"; 
File.makeDirectory(output);

function processFolder(input);
list = getFileList(input);   
for (i=0; i<list.length; i++) {
			if (endsWith(list[i],".ndpi")){
			inputPath= input + list[i];
			//only big data superior to 1go in .ndpi
			length = File.length(inputPath);
				if (length > 1000000000){
					run("Preview NDPI...", "ndpitools=[inputPath]");
					waitForUser("draw a box");
					print("Extraction NDPI to tiff:  " + inputPath +" please wait");
					run("Custom extract to TIFF / Mosaic...", "label=[] format_of_split_images=[TIFF with LZW compression] store_split_images_in_bigtiff_files_rather_than_tiff make_mosaic=never mosaic_pieces_format=[TIFF with JPEG compression] requested_jpeg_compression=75 mosaic_pieces_overlap=0.000000 mosaic_pieces_overlap_unit=pixels size_limit_on_each_mosaic_piece=0 width_of_each_mosaic_piece_in_pixels=0 height_of_each_mosaic_piece_in_pixels=0 save_these_choices_in_imagej's_preferences_file extract_images_at_magnification_10x extract_images_with_z-offset_0");
					close("*");
					print("Extraction NDPI to tiff:  " + inputPath +" OK");
					
				}
				//small data save in tiff see if need to save in Bigtiff
				else {
					print("Extraction NDPI to tiff:  " + inputPath +" please wait");
					run("Custom extract to TIFF / Mosaic...", "no=[inputPath] format_of_split_images=[TIFF with LZW compression] store_split_images_in_bigtiff_files_rather_than_tiff make_mosaic=never mosaic_pieces_format=[TIFF with JPEG compression] requested_jpeg_compression=75 mosaic_pieces_overlap=0.000000 mosaic_pieces_overlap_unit=pixels size_limit_on_each_mosaic_piece=0 width_of_each_mosaic_piece_in_pixels=0 height_of_each_mosaic_piece_in_pixels=0 save_these_choices_in_imagej's_preferences_file extract_images_at_magnification_10x extract_images_with_z-offset_0");
					close("*");
					print("Extraction NDPI to tiff:  " + inputPath +" OK"); 
				}
			}
			
							}
//continue the macro but check for tiff files
function processFolder(input);
list2 = getFileList(input); 
	for (j=0; j<list2.length; j++) {
		if (endsWith(list2[j],".tif")){
			inputPath2= input +  list2[j];
			print("Conversion: " + inputPath2 +" please wait");
	//only big data superior to 200KB in .ndpi
			length = File.length(inputPath2);
			if (length > 700000){
				run("Bio-Formats Importer", "open=inputPath2 color_mode=Default rois_import=[ROI manager] view=Hyperstack stack_order=XYCZT");
				imagesName=getTitle();
				saveAs("Tiff", output + File.separator + imagesName);
				close("*");
				File.delete(inputPath2);
						}
	//else delete file because NDPItools create map and small files during the preview
				else {
					File.delete(inputPath2);	
					}
								}
				
}
					
print("\\Clear");
print("Macro ends");	

It looks like this bug which was posted here a while ago, see:

However it seems to be fixed in OpenJDK 13 (tested on Windows only with the example of @Wayne).

I think it could be the same because in my case I have some problem only with the files bigger than 880 Mb tiff LZW compressed.

Which JRE/JDK are you running?

Sorry I’m very New in the image analysis field and I’m not sure if I understand correctly your question.
I just download Fiji app then my plugin (NDPITools) and then I wrote a macro and I run the macro with fiji. (Fiji.app\java\win64\jdk1.8.0_172)

I have java on my computer Java 8 build 1.8.0_231-b11

It could be that the bug is still present in those Java versions. Just create a huge random example image in ImageJ (File->New Image) for a test. Unfortunately the link to the bug report is not valid anymore.