ExportToExcel

cellprofiler

#1

Question ;
Is the module ExportToExcel work on Cpcluster ?
thanks.
Marie Laure


#2

I do not believe so. If it did, it would give you excel files for each batch that you run and you would have to merge them all together! The best thing to do is use the MergeOutputFiles data tool to merge all of your .mat output files together. Then, export the MergedOut.mat using the ExportData data tool (which is another way of exporting data to Excel).

Hope that helps!
martha


#3

Hello Martha,

Thank you for your answer, but I am not sure to understand very well.

You make you pipeline on a computer with at last the module CreateBatchfiles.
Then we close our Cellprofiler.
Then we submit batch files to our cluster with an in-house script.

Then we have to do use MergeOutputFiles data tool to merge all of your .mat output files together. How do we can do this ?
Is it possible to load .mat files ?
Is it possible to merge output files and then to export Data without the cellprofiler GUI ?

Thank you for your answer.

Marie Laure

Then you sen the
Have you an example of pipeline ?


#4

Hello Marie,

I think what Martha intended was to use CellProfiler (not CPCluster) DataTools to process your ExportToExcel batch outputs. However, as she hinted, this is not our usual workflow. We expect users most of CPCluster to be dealing with amounts of data which are too large for Excel to handle, and would thus suggest using a pipeline like this:

Module1

ModuleN
ExportToDatabase
CreateBatchFiles

However, to save time, even on smaller projects, one could utilize a cluster as Martha mentioned with the use of CP DataTools afterward. To be more explicit:
(1) Run CP without any Export modules, but with CreateBatchFiles at the end
(2) Submit your batch jobs on your cluster using CPCluster
(3) With Cellprofiler (GUI), use MergeOutputFiles DataTool on the cluster output files
(4) Run ExportData(base) DataTool or CalculateStatisticsDataTool to output and/or analyze your data

Hope this helps,
David


#5

Hi David,

Thank you very much for your explanation.
You suggest to use ExportToDatabase on the pipeline.

My question are :
Where is your the database on your informatic system ?
On the master, on one node, on a dedicated server ?

Which type of DB do you use ? SQL, Oracle ?
a preference ?

Thank you for your help.

Marie Laure


#6

Hi Marie,

The database here at the Broad Inst. is on a dedicated server. And we use MySQL, since it’s free and has the functionality that we need. We recommend MySQL over Oracle, only because we have used MySQL extensively with CellProfilerAnalyst and only spot-tested Oracle. We have tested the output of CP to Oracle and, following the help in ExportToDatabase, you should be able to import into Oracle fine. But we haven’t tested CPAnalyst much at all with Oracle.

Hope this helps,
David