CP2.1.0 linux grid - tmp error

Dear developers,
we would like to run CP on a grid infrastructure providing computing (around 10000 CPUs), storage and cloud facilities

I am user, but cooperating with IT experts and they went through installation successfully with version CP 2.1.0 rev f6e3a5a.
However we blocked on running whole analysis.

The Test mode runs without problems, but start analysis generates problem

IO Error: [Errno 2] No such file or directory: ‘/tmp/Cpmeasurementskakq92.hdf5’

Could you please give us some advice, what to change? How to start using CP on cluster?
Is it better to install non-developers version from here? cellprofiler.org/linux.shtml

Thank you very much.
Best wishes.
Josef

Hi Josef,

Setting up CellProfiler to run on a cluster is non-trivial! This is our general help for batch processing: cellprofiler.org/CPmanual/He … ssing.html
Is this what you follow?

You say Test Mode, so that indicates to me that you are logging into, and running this on the cluster directly? Our setup (and suggested setup, as detailed on the Manual page above) is to rather:
(1) Run CellProfiler locally, on your desktop machine, and optimize your pipeline
(2) Use ExportToDatabase (we use a MySQL server)
(3) Still in CP locally, add a CreateBatchFiles module and ‘translate’ the path names of your local machine to paths that the cluster has access to (usually Windows -> Linux, or Mac -> Linux). Run your pipeline once, and it will run through the first cycle only and produce a file that sets up the batches for processing.
(4) Submit your batches to the cluster - more detail is on theBatch Processing Help page I linked to above. This step is quite variable depending on your cluster, how you access it, and where CellProfiler resides, etc. But note that CP should be running headless (no GUI) on the cluster. The nodes may not have access to Java, and you don’t want display windows popping up uselessly there anyway. So on the cluster itself, there ought to be no Test Mode, per se.

As for the /tmp/CPmeas… error, likely following the above will solve this, but it is a temporary measurement file that the cluster(?) cannot access. Possibly ran out of space on /tmp? But writing to a database may preclude this, too. If the above does not help, please provide some more detail on exactly how you are implementing CP and when you get this error.

Hope that helps!
David

Hi Josef,

This may be a silly question but does the /tmp folder exist, and do you have write-access to it? If yes to the former, but no to the latter, you can change the location of the temporary folder under File > Preferences.

Also, that revision is quite old (from Oct '13, if I’m not mistaken) and you might be better served by retrieving the actual release: d2abe90fc2 from the release_2.1.0 branch; there may be have been bugs from back then that might have been fixed. And yes, obtaining the Linux install from that page is always an option; it corresponds to the 2.1.0 release as well.
-Mark

Dear all,

let me slightly depict the described compilation problems – actually, I’m installing the josefjar’s CP within our grid environment – in more detail.

Our infrastructure runs Debian 6/7 clusters – for the first attempt, I’ve tried to use the Makefile.CP2.ubuntu.64 and follow the compilation steps described at github.com/CellProfiler/CellPro … -for-Linux . And using this, I run into the problems described by JosefJar (there were also several compilation problems, which I managed to overcome). In fact, the file CpmeasurementsXXXXX was created, however with no content (and that coused the error).

Nevertheless, based on your reactions, I’ve tried the Makefile.CP2.64 compilation – now, the compilation went more smoothly, and I managed to prepare much more stable/better version.

Now, I’m facing hopefully a last problem – when starting CP (Version: 2014-06-14T17:02:06 4e6ec2c / 20140614170206), it shows:

ilastik import: failed to import the ilastik. Please follow the instructions on "http://www.ilastik.org" to install ilastik Traceback (most recent call last): File "/afs/ics.muni.cz/software/cellprofiler/2.1.0/CellProfiler/cellprofiler/modules/ilastik_pixel_classification.py", line 81, in <module> import ilastik_main ImportError: No module named ilastik_main

Does anybody know where the problem could be? I’ve tried different versions of ilastik with no luck. It seems to me that there’s something wrong with the ilastik_pixel_classification.py script…

Thanks a lot for your hints.
Tom.

PS: For David: in the future, we’ll definitely prepare the workflow based on batch jobs. However, since our infrastructure supports interactive jobs as well (including the ones with GUI apps), we’re currently testing the installation this way. Once we overcome all problems, we’ll try to optimize the whole processing workflow – so, thank you very much for your hints, we’ll definitely use them…

The Ilastik 1.0 issue should be ignored. Our builds are for the previous version (0.5) and ClassifyPixels, which uses that version should work. You’re getting that error because the newly-contributed ilastik_pixel_classification uses the Ilastik 1.0 codebase which isn’t built. The error can safely be ignored, unless you want to use Ilastik 1.0 instead of Ilastik 0.5 (or more likely, if you don’t know what Ilastik is, you can ignore the error altogether).

If you want to use ilastik_pixel_classification, you can contact the Ilastik team at www.ilastik.org.

Excellent. As the final user, I want to say Thank you very much!
Great cooperation. Running!
Best wishes.
Josef

Very good! Thanks for the reports on the Debian build, and glad you are running.
David

Dear Lee,

let me also thank you – thanks a lot for the information? I thought that the problem is something like that…

Regarding the Ilastik v. 1.0 – since Josef is not currently using it, will wait for the final release…

Tom.