Headless multicore CP batches randomly skip images

I have been recently testing a script to execute CP headless as 4 separate processes on a Win7 core I7 computer. It appears to randomly skip images. I think that if a CP process happen to be writing to the sqlite.db file at a time when the file is locked it seems that the CP process just shuts down without waiting and trying again. So, it is probably a file write race condition since other CP processes don’t seem to wait when the sqlite file is locked during a write. I ran the same pipeline on a single core with the GUI and all images completed correctly but it takes over an hour when using only one processor. Would it help me to use the SQL/CSV database output option instead? I posted the batch script below. Has anyone seen this problem before?

cd \Program Files\CellProfiler

START CellProfiler -p F:\3262_MMR_batch\output\Batch_data.mat -i F:\3262_MMR_batch output1.mat -c -r -f 1 -l 49

START CellProfiler -p F:\3262_MMR_batch\output\Batch_data.mat -i F:\3262_MMR_batch output2.mat -c -r -f 50 -l 98

START CellProfiler -p F:\3262_MMR_batch\output\Batch_data.mat -i F:\3262_MMR_batch output3.mat -c -r -f 99 -l 147

START CellProfiler -p F:\3262_MMR_batch\output\Batch_data.mat -i F:\3262_MMR_batch output4.mat -c -r -f 148 -l 196

I have just run across this issue myself :smile: You are correct; according to this page, SQLite does not support multiple simultaneous writes.

If you are using the measurements for CellProfiler Analyst, then yes, since CPA will create a SQLite database from per-image and per-object CSVs if the properties file is directed to them. Otherwise, you would need to the resultant SQL setup script to upload the measurements to an MySQL database.

Regards,
-Mark

Could you put a conditional (if busy-then wait-else write sqlite db entries) wait timer call into the SQLITE_BUSY routine to test when the sqlite file is locked before trying to write to it? It seems to take about 5-8 seconds at the moment for the write to db module to complete with most of my images. Maybe this would stop the process from prematurely terminating before completion of its other image processing jobs. I will try to use the MySQL/CSV output option in the SaveToDatabase module for the time being.

Hi Derek,

Sorry for the long delay on this, but we in fact go ahead and insert a longer timeout into the SQLite write (30 seconds). Also, for anyone else viewing this, please look at the caveats for SQLite multiprocessing - the locking may not work for NFS file systems: sqlite.org/faq.html#q5.

The modification is not yet officially released, but you can get this addition with our latest public build from source code here. The trunk build is not as heavily vetted as our releases, so you should take that into account; please note the caveats mentioned on the page linked.

Regards,
-Mark