Different row counts in the per-object tables

Is there a plausible reason why the number of rows in the _Per_Cell table and the _Per_Cytoplasm table could be higher than the number of rows in the _Per_Nuclei table?

I have a project (CDP2) where 15 batches of images were processed, presumably with the same pipeline. For 13 of the batches, all three per-object tables (…_Per_Cells, …_Per_Cytoplasm, …_Per_Nuclei) have the same number of rows, but for two of them, they are different:

week3: 7,891,608 nuclei, 8,023,715 cells, 7,987,520 cytoplasms
week4: 5,550,908 nuclei, 5,550,458 cells, 5,550,892 cytoplasms

Should I assume that something went wrong and that these weeks will have to be reanalyzed?

Is there any way to discern whether those batches partially completed and then restarted, with some kind of strange over-writing or lack thereof?

(This post is mostly intended for users at the Broad Inst., sorry)

Using BatchProfiler (our in-house batch processing system), a user can check the Batch output via ViewBatch.py, however certain errors are not always picked up. I have also written a shell script to do some further testing, which has helped me sometimes, and lives here:

Here’s the script, commented at top:

[code]#!/bin/bash

This script checks the output of BatchRunner.py jobs, to see if:

(1) They do not contain the characters ‘[Ee]rror’

(2) the number of ./status/DONE.mat entries equals the number of ./txt_output/[0-9].txt entries

It should be executed from the Batch_data.mat directory, which is expected to have txt_output and status directories beneath it

error_msgs_length=grep \[Ee\]rror ./txt_output/* | wc -l
if $error_msgs_length -ne 0 ]
then grep [Ee]rror ./txt_output/*
else echo No errors found in txt_output. Good!
fi

nLinesDONE=ls -l ./status/*DONE.mat | wc -l
nLinesTxt=ls -l ./txt_output/[0-9]*.txt | wc -l

if $nLinesDONE -ne $nLinesTxt ]
then echo Number of DONE files $nLinesDONE does not equal number of txt files $nLinesTxt. Batches may still be running, or they may have been an error.
else echo Number of DONE files $nLinesDONE equals number of txt files $nLinesTxt. Good!
fi

nLinesImageCSV=ls -1 *_image.CSV | wc -l

if $nLinesImageCSV -ne $nLinesTxt ]
then echo Number of PerImage CSV files $nLinesImageCSV does not equal number of txt files $nLinesTxt. Batches may still be running.
else echo Number of PerImage CSV files $nLinesDONE equals number of txt files $nLinesTxt. Good!
fi
[/code]