ExportToDatabase output folder problem

Dear all,
I’m having a problem with the ExportToDatabase module, setting the output folder for an SQLite database.
In the “Output file location” field I choose “Elsewhere…” and the text area allows me to input metadata tags (right-click). However, the path constructed in this way is ignored and the output defaults to another path. Obviously the path constructed with metadata tags is unique and unambiguous.
I would assume that since the right-click input of metadata tags is enabled for the “Elsewhere…” these would be available for the path definition. Is that not so?
Thanks a lot for any help
Best
Carlos

Hi Carlos,

Are you extracting metadata from image file headers in the Metadata module? That option seems to always populate the metadata fields like “Well” and “Plate” in the right-click context menus where you can define particular folders (like in ExportToDatabase) regardless of whether they are in the headers or not (i.e. columns listed at the bottom of the Metadata module). If you switch to another metadata extraction method, this does not seem to be an issue. It does seem like a bug to me to list metadata that has not been extracted per se, but I will ask our software engineer if there is a good reason to do this. Does the right behavior occur if you create an output folder with a metadata tag from a “good” metadata that you can actually see in Metadata?

Thanks,
David

Hi David,
Thanks for your reply.
I’m extracting metadata from folders and image names. They seem to work well both in exported tables and in saved images.
My folder/metadata structure is:
Root\Folder\Group\Treatment\ImageName\Position<actual tif files> (they’re from Micromanager so image names are folders and position comes from Micromanager as well, but it doesn’t refer to a plate well).
Root and Folder are common to all images (they start to vary from Group onwards).
I also have some masks, generated in a separate project, where Treatment and ImageName are in the file name, and Root, Folder and Group in the folders. All metadata seem to match perfectly well both through the pipeline, in the output overlays and in the resulting databases.

I ran a few tests and found the following. If I use “Elsewhere…” Root\Folder (right-clicked from tags) the outputs are:
From ExportToDatabase selecting MySQL/CSV (4 images, SQL_ as prefix):
SQL_1_4_Object.CSV goes to C:\Program Files\CellProfiler
SQL_1_4_Image.CSV and SQL_SETUP.SQL go to Root\Folder

From ExportToSpreadsheet (MyExpt_ as prefix):
MyExpt_Experiment.csv goes to C:\Program Files\CellProfiler
all other files go to Root\Folder

In all cases, if explicitly set the output folder in Elsewhere… everything correctly goes to the specified folder. So, it does work Ok, of course, it’s just that I’d rather not have to remember to specify the folder (just overwrote a database by accident!).
In the module SaveImages Root\Folder works perfectly well.
Thanks very much for your help
Best
Carlos

Hi Carlos,

I think the issue is that your Default Output Folder, as specified in the “View Output settings” button at the bottom of your pipeline, is likely set “C:\Program Files\CellProfiler” which is our default for output that does not have an obvious output folder. This is the case for the ‘Experiment’ table when you have metadata tags – please see the discussion here for our current thinking, since I had the same question awhile back! github.com/CellProfiler/CellPro … ssues/1110

[quote]I ran a few tests and found the following. If I use “Elsewhere…” Root\Folder (right-clicked from tags) the outputs are:
From ExportToDatabase selecting MySQL/CSV (4 images, SQL_ as prefix):
SQL_1_4_Object.CSV goes to C:\Program Files\CellProfiler
SQL_1_4_Image.CSV and SQL_SETUP.SQL go to Root\Folder[/quote]

It would seem to me that the Object.CSV should also go to {Root}{Folder} as well. I’ll file a Github issue for this.

[quote]From ExportToSpreadsheet (MyExpt_ as prefix):
MyExpt_Experiment.csv goes to C:\Program Files\CellProfiler
all other files go to Root\Folder[/quote]

As discussed here, this is the expected behavior, though I can understand why this is confusing. With metadata tags there may not be a true “root” folder though and so we default to your Default Output Folder.

Does that make sense?
David

Actually, I just mocked up my own test and the Object.CSV seems to go to the application root folder, not the Default Output Folder. That looks like a bug to me. Filed here: github.com/CellProfiler/CellPro … ssues/1276

Thanks for reporting!
Cheers,
David

That’s great David, thanks very much.
Best
Carlos

It clearly is a bug in ExportToDatabase - the metadata fields don’t work as they should. But ExportToSpreadsheet has much the same functionality and you can successfully use metadata fields to create separate .csvs based on the metadata values.

Would it be possible to switch from ExportToDatabase to ExportToSpreadsheet?

Hi Lee,
As I mentioned in my original post, ExportToSpreadsheet also sends one file to the CP folder when destination folder is specified through metadata (admittedly, the more important files go to the metadata-specified folder).
Everything works Ok if the output folder is left to default in both modules and export types.
It’s just that it would be very convenient to be able just to “drag and forget”, letting all the folders sort themselves out through metadata without further user input (especially when the user forgets to check things, like me! - to be honest, I decided to post after I overwrote a previous processing…)
With respect to the earlier discussion that David pointed to, that was my initial thinking when I found that the output didn’t work, but was surprised to find that it went to the CP folder. However, I’d imagine that a sort of manual validation (a button enabled when “Elsewhere…” is selected) or an automatic validation at the start of the analysis would work here. If the metadata combination as populated by each run’s input doesn’t point to a unique destination folder it’d be flagged as error.
It’s just that the whole metadata handling works so well throughout all modules and is such a great feature of CP that I’m getting greedy!
Thanks again for your help
Best regards
Carlos