BigDataProcessor

This thread collects feedback on using BDP2: https://github.com/tischi/fiji-plugin-bigDataProcessor2#bigdataprocessor2

@Agamanad suggests:

  • It would be great for BDP2 to be macro record-able such that can be run automated
  • Add magnification and rotation correction to the chromatic corrections
  • For changing the time-point it would be good to be able enter the timepoint rather than using the slider

@LThomas suggests:

  • Implement other binning modes than average.

@ue-hd suggests:

  • For opening Luxendo files the hdf5 dataset name Data could be used automatically, without asking the user to confirm.

Other users suggested:

  • Enable entering calibrated (not voxel unit) chromatic shift correction values.
1 Like

Hi there. Also some kind of automated ROI detection for cropping would be great, either implemented into the plugin, or by allowing some user script to determine cropping coordinates instead of manual selection. For simple structures, I’m thinking of blurring, center of mass determination and thresholding to create a bounding box.

1 Like

I’d also love the plugin to work with files from Zeiss Z1 (.czi files). For output, options for tiff volumes would be great (e.g. multiple channels in one file).

1 Like

@Christian_Tischer thanks for the great workshop introducing your tool.

For cropping data in batch it would be cool to just mark some positions and do allways the same crop dimension on this positions

1 Like

Hello, I would also like to be updated regarding BigDataProcessor related developments
Thanks!!

1 Like

Pleas keep me updated about the BDP as well, thank you!

1 Like

Hello, I would also like to be updated regarding BigDataProcessor related developments!
Thanks!

1 Like

adding @seadaisy, too.

@ischneid @seadaisy @Ashna_Alladin

I am thinking about simplifying the saving dialog. Do you think it would be ok if the volumes and projections would end up in directories with default names.

So, the user would only select one root saving directory, e.g.

/Users/Data/Output/

and then the software would automatically create the sub-folders

/Users/Data/Output/volumes
/Users/Data/Output/projections

?!

Dear All,
I like Tischis suggestion. I think it would make the data structure more clean.

Hello

@seadaisy @metavibor @ischneid @Ashna_Alladin @T-Zobel @Agamanad @Cova @eguren @LThomas @ue-hd @NicoKiaru @haesleinhuepf @schorb @nornil @Alex_H

I hope you are all doing well, given the circumstances!

I took some days to work on BigDataProcessor, mainly to enable macro recording. It would be amazing if you could test the new version and provide feedback and bug reports :smiley:


Install

  • If you have not done so yet, please enable the Fiji Update Site: EMBL-CBA
  • Update your Fiji

Record

Before you start BigDataProcessor, enable the Recorder in Fiji:


Open

You have to scroll very far down in the Plugins Menu:

Process

Click on the BigDataProcessor window and use the items in the BigDataProcessor menu, e.g. like this:

Note: There are also similar menu items in the Fiji plugins menu:
[ Plugins > BigDataProcessor > Process > ... ]
I recommend not using them, there are mainly there to enable macro recording.

Save

Saving is also done via the BigDataProcessor menu:


Test data

If you need some data for testing, here is the data that we used at the TiM2020 conference (~20 GB): https://oc.embl.de/index.php/s/nJAqBi8KuXTNXZF

Feedback

It would be awesome to get some feedback :slight_smile:


Thanks a lot and stay healthy!

3 Likes

cc @VolkerH @matthewboucher @LucasC

@seadaisy thank you very much for reporting two issues by mail. Both are addressed if you update your Fiji.

  1. Opening of Luxendo files did not work on Windows. This was a bug and is fixed now.
  2. Navigation of Leica DSL Tiff Planes along the z-axis was not working. This is more tricky. The issue is that one cannot know the z-spacing from the Tiff planes. The default z-spacing in BDP was far off, rendering the z-browsing in BigDataViewer, which is based on physical units, practically impossible. As a work-around I now set the z-spacing to the same distance as the x/y pixel size. This make z-browsing possible, but, of course is still wrong. Take home message: after loading your data immediately go to Calibrate… and check/ fix your voxel sizes!
    image

Hello, I would also like to be updated regarding BigDataProcessor related developments.

1 Like

I am also adding @LuxSK to this thread.

Hi @Christian_Tischer - Thanks for setting up the BigDataProcessor2! It works really great with the macro set up, which I am just finishing up.

I have two questions, both regarding macro recording and saving:

For reference, I have attached my code as it stands. BigDataProcessor.ijm.ijm (1.9 KB)

1). I want the plugin to close during the macro, because otherwise I will end up with lots of different BigDataProcessor interface windows (for a single experiment, it would have to open 12 times, which I’m sure would make it crash). Is there a way to automate this in the code?

  1. I would preferably like my files to save in the same format as I have been doing it so far. Which is as follows:

Position001_Target–bin-2-2-2–C01-T00045 (as an example). As I understand it, each individual file contains the whole Z-stack.

Currently, I can only work out how to save it as follows:

volume–C00–T00000-Z00000

Previously, I would have all the files from a single experiment in one folder, but here the files are all in individual folders. If you could show me a way of making it save in the old way, and how to put this into the macro, would be fantastic.

Hope that makes sense. Let me know if any questions

Matt

1 Like
  1. Could you explain your workflow in more detail? Why do you have to open 1 experiment 12 times?
  2. When do you want to close BDP? I could imagine that you would like it to close once the saving is done? If that’s the main use-case I could add a check-box to the saving dialog:
    close when finished.
    What do you think?

Sure - so in a typical light-sheet experiment, I have 4 different positions. For each position, there are 3 versions of the image, Target, which is the combined data from each light-sheet mirror, and then the individual light-sheet mirrors (LSEA00 and LSEA01). I like to save all of these - the Target files are what I do the analysis on, and the LSEA00/01 files are just to validate that there’s little disparity between the two individual mirrors. So overall, 3 versions for 4 positions = 12.

So because of this, I want to open each set of files (e.g. Position 1 Target, or Position 3 LSEA01), do the binning, and then save it. At the moment, I can’t work out how to close and reopen (or refresh) the big data processor with the new data set, so it just builds up after saving each one. I could imagine with larger data sets, having the big data processor open so many times is going to cause problems.

You are correct! I think having the check box would solve the problem (assuming it can be automated in the macro)

Hi Tischi,

Unfortunately, opening Luxendo files is still not working, see the message from the console window:

[INFO] Reading available sites from https://imagej.net/

Exception in thread “AWT-EventQueue-0” java.util.regex.PatternSyntaxException: Unmatched closing ‘)’ near index 0

d)+.h5$

^

at java.util.regex.Pattern.error(Pattern.java:1957)

at java.util.regex.Pattern.compile(Pattern.java:1702)

at java.util.regex.Pattern.(Pattern.java:1351)

at java.util.regex.Pattern.compile(Pattern.java:1028)

at de.embl.cba.bdp2.open.files.FileInfosHelper.sortAndFilterFileList(FileInfosHelper.java:509)

at de.embl.cba.bdp2.open.files.FileInfosHelper.getFilesInFolder(FileInfosHelper.java:602)

at de.embl.cba.bdp2.open.files.FileInfosHelper.getFilesInFolders(FileInfosHelper.java:417)

at de.embl.cba.bdp2.open.files.FileInfosHelper.setFileInfos(FileInfosHelper.java:186)

at de.embl.cba.bdp2.open.files.FileInfos.(FileInfos.java:106)

at de.embl.cba.bdp2.BigDataProcessor2.openImage(BigDataProcessor2.java:91)

at de.embl.cba.bdp2.open.OpenLuxendoChannelsFromFoldersCommand.lambda$run$0(OpenLuxendoChannelsFromFoldersCommand.java:37)

at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311)

at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758)

at java.awt.EventQueue.access$500(EventQueue.java:97)

at java.awt.EventQueue$3.run(EventQueue.java:709)

at java.awt.EventQueue$3.run(EventQueue.java:703)

at java.security.AccessController.doPrivileged(Native Method)

at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)

at java.awt.EventQueue.dispatchEvent(EventQueue.java:728)

at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:205)

at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)

at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)

at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)

at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)

at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)

1 Like

Hi Tischi,
the Macro recording does not work for me. I will send you a screen recording, to double check, that is was not a mistake by me.

1 Like