How to in-memorily read multi-file images?

Development
,
#1

Olympus OIR is a kind of multi-file image format, with each file a max size of 1㎇. Now I have an OIR list of 72 files, that is 72㎇. I found it unreasonably slow reading it with common readers, 1~2㎆ per second (measured by the Task Manager), leaving my SSD mostly idle.
Then I tried in-memory reading. However, it cannot detect the following files, even though I set the fileName in Location.mapFile with the same nomenclature for all following files, which cost huge RAM. The reader just consumed the first oir and done, refusing to read following files.
Is there a way to in-memorily read such multi-file images? Or any other solution for quicker reading?
And another serious problem is how to release the memory used? I didn’t find any function in loci.common.Location doing this.

#2

It seems that I found a tricky way. When calling Location.mapFile, use the virtual fileName just the same as the real fileName. However I’m not sure whether the reader will read the memory or the disk.
And most importantly, I can’t find a way to release the memory! Closing the ByteArrayHandle does not release the memory as reported by the Task Manager and the java heap overflow from time to time.

#3

Hi @EbolaChan, do you have a link to your code to see exactly what calls you are making?

#4

This project, in short, is to slice a super large oir file sequences into channels or z-stacks and write each channel/Z into individual Tiff files.
https://github.com/Silver-Fang/Oir2TiffMatlab
Memory mapping calls, coded in MATLAB, in Oir2OmeTiff.mlx line 40~64:

Fid=fopen(OirPath);
PathBase=fullfile(Directory,FileName);
MemoryMap=PathBase+".oir";
NoParts=numel(OirList);
BAHs=cell(NoParts,1);

%Read and map the first file

BAH=ByteArrayHandle(fread(Fid,Inf,"uint8=>uint8"));
Location.mapFile(MemoryMap,BAH); %The map file has a name exactly the same as the real file
BAHs{1}=BAH;
fclose(Fid);

%Read and map extra files

for a=2:NoParts
	MemoryMap=OirList(a);
	Fid=fopen(MemoryMap);
	MemoryMap=PathBase+sprintf("_%05u",a-1);
	try
		BAH=ByteArrayHandle(fread(Fid,Inf,"uint8=>uint8"));
	catch
		error("Java堆内存不足。请在"+fullfile(matlabroot,computer("arch"))+"下创建一个java.opts文本文件，其内容为-Xmx131072m。如果仍然出现同样的错误，您可以尝试将数值设为更大。");
	end
	Location.mapFile(MemoryMap,BAH); %Extra files are also mapped to the same name as the real file
	fclose(Fid);
	BAHs{a}=BAH;
end
BAHs=vertcat(BAHs{:});
Cleaner=onCleanup(@()arrayfun(@close,BAHs));
MemoryMap=PathBase+".oir";

Then the Reader is created in private/GetBfOirReader:

import loci.formats.*;
BfOirReader=Memoizer(ChannelSeparator(ChannelFiller));
BfOirReader.setMetadataStore(ome.OMEPyramidStore);
if MapPath==""
	BfOirReader.setId(OirHeaderPath);
else
	BfOirReader.setId(MapPath);
end

The read/write operations are done in many different functions to support different user requirements. A typical one is in private/OCOF.m:

%Oir formats arrange planes in an XYCZT order, so we can calculate the linear index with CZT indices
ZIndex=C-1;
if OneZOneFile
	Output=TiffPaths+"."+Metadata.DeviceNames(C)+".Z"+string(1:SizeZ)+".tif";
	for Z=1:SizeZ
%This function GetTiffSubWriter returns a MATLAB (not OME, which has been tested to be much slower) Tiff writer
		[TiffWriter,TagStruct]=GetTiffSubWriter(OirReader,Output(Z),Cs=C,Zs=Z);
		TIndex=ZIndex;
		FirstPlane=true;
		for T=1:SizeT
			if FirstPlane
				FirstPlane=false;
			else
				TiffWriter.setTag(TagStruct);
			end
%This reshape(typecast()) conversion is required by the MATLAB Tiff writer
			TiffWriter.write(reshape(typecast(OirReader.openBytes(TIndex),"uint16"),SizeX,SizeY)');
			TiffWriter.writeDirectory;
			TIndex=TIndex+TDelta;
		end
		ZIndex=ZIndex+SizeC;
		TiffWriter.close;
	end
end
#5

The only obvious thing I can see there is that the reader does not appear to be getting closed. Otherwise it may be a MATLAB specific issue, I will carry out some further tests to check this.

Overall do you find this is providing much speed up compared to just using the standard readers?