How to install cellprofiller2.0 in a Linux Cluster

How was python installed on your system? What distribution are you using?

Well, the system have python 2.4.2 on a SUSE® Linux® Enterprise Server 10 - SLES10SP2.
The Cluster is a SGI Altix XE 1300.

The system python instalation is default from distribution, but the Makefile don’t install a “python” with “PYTHONPATH” approach?

Hi adamoski, sorry for ignoring your post, I was on holidays! Have you tried again, any news on this? From the error message, python seems to miss ‘zlib’ support. That is very strange, can you check that you have zlib installed? Its one of the most basic packages! Also check if there is an zlib-dev package, and install if necessary. Let me know if that helps!
All the best, Mario

Hi,

Download from: data.marssoft.de/cp/Makefile.CP2

I have updated the Makefile for CellProfiler 2 including its prerequisites, to use more recent versions of several thirdparty libraries. It is currently in alpha status, but can be quite useful.
The Makefile is now even more convenient to use. First, download the Makefile from here: data.marssoft.de/cp/Makefile.CP2 to an empty directory. Then set the environment in your .bashrc (or whatever shell you are using), i.e. if you want to install to TGTDIR = ${HOME}/usr/CellProfiler2 then the following should work:

# CP2, python-2.5 and many other CP2 prerequisites
export  PATH="${HOME}/usr/CellProfiler2/bin:${PATH}"
export  LD_LIBRARY_PATH="${HOME}/usr/CellProfiler2/lib:$HOME/usr/CellProfiler2/lib64:${LD_LIBRARY_PATH}"
export  PYTHONPATH="${HOME}/usr/CellProfiler2/lib/python2.5/site-packages:${PYTHONPATH}"
# java-1.6 for 64bit
export  JAVA_HOME="${HOME}/usr/jdk-6u20-linux-x86_64"
export  PATH="$JAVA_HOME/bin:${PATH}"
export  LD_LIBRARY_PATH="${JAVA_HOME}/jre/lib/amd64:${JAVA_HOME}/jre/lib/amd64/server:${LD_LIBRARY_PATH}"

Then, open a shell, change to the Makefile.CP2 directory, and invoke make:

make -f Makefile.CP2 TGTDIR=${HOME}/usr/CellProfiler2

That should finish the installation of all prerequisites, including CellProfiler 2 itself. You should then be able to run CellProfiler 2 by invoking:

python CellProfiler.py

The Makefile should provide good performance in SciPy and NumPy by using ATLAS. However, it is still a preliminary version that could use some fine-tuning. Below is a list of possible quirks:

  • You have to take care to disable CPU frequency scaling before compilation, else ATLAS can not tune its algorithms for your CPU

  • You need a fairly recent version of gcc. I have used gcc-4.4 successfully. If you require a Makefile for gcc as well, please contact me.

  • Java is not downloaded or installed automatically. Please download and unpack it yourself, and adjust your JAVA_HOME variable accordingly.

  • MySQL is not downloaded or installed automatically. Please download and install it yourself, and adjust your PATH variable accordingly.

I am more than happy about feedback and reports. I would like to improve the Makefile over time, to have a stable and reliable foundation for CP2 installation on a Linux cluster - any help is highly appreciated.

All the best, Mario Emmenlauer, Biozentrum Basel

This is pretty impressive, emmenlau. I’m going to put it on my schedule this week to try out the build and run the unit tests. After that, we’ll check the script in and update directions on the wiki to use it.

We are planning to integrate the Ilastik pixel classifier (hci.iwr.uni-heidelberg.de/download3/ilastik.php) into CellProfiler and this will add a few dependencies to the build. There is already code in CellProfiler to use Ilastik if the libraries are built and the appropriate packages are installed (see cellprofiler/modules/classifypixels.py). I am guessing that we’d get around to updating the build to support Ilastik within a month unless someone else does it for us first (have to warn you, it’s a lot to ask).

Thanks,

–Lee

Hi Lee, thanks!
Its my goal to reach a state where all our cooperating institutes (InfectX in Switzerland) can install CP2 as simple as possible, so I’m prepared to invest more time into CP2 compilation. My next step would be the MySQL dependency, then can look into Ilastik. From a quick look, I could see it requires

  • vigra

  • PyQt4

  • h5py

What did I miss? :smile:

Hi-
I was having a weird problem with the Makefile.CP2 and I just worked it out, so I figured I’d post what was wrong. In the rule to build ATLAS, there are a couple of lines that look like this:

cd "./ATLAS/" && \ mkdir -p "./$(HOSTTYPE)" && \ cd "./$(HOSTTYPE)/" && \ export PATH="$(PYTHONPREFIX)/bin:$(PATH)" && \ export LD_LIBRARY_PATH="$(PYTHONPREFIX)/lib:$(LD_LIBRARY_PATH)" && \ ../configure -Fa alg -fPIC -b 64 \ -Ss flapack "$(SRCDIR)/$(LAPACKVERSION)/lapack_LINUX.a" \ -Ss pmake "make -j4" -Si cputhrchk 0 \ --prefix="$(ATLASPREFIX)" && \

For some reason, the HOSTTYPE was not working properly - it was returning a blank value, so the actual lines that were being called were:

cd "./ATLAS/" mkdir -p "./" cd ".//"

This made the following “…/configure” call point to the wrong spot, and everything stopped. I added a “HOSTTYPE = x86_64” line at the top of the makefile, and it’s now compiling again.
I’m confused as to why this didn’t work, as running “echo $HOSTTYPE” returns “x86_64”.
Further information - I’m compiling this on Centos 5.5.

Thanks for putting together this makefile, by the way. It’s pretty slick.

Thanks for posting this, Mario.

I made a few modifications and checked in the build file to trunk/CellProfiler/Makefile.CP2

For my machine, I needed to explicitly supply the fortran compiler as gfortran in Numpy and Scipy to match the one used by LAPACK.

–Lee

Hi Lee,

thanks for your input, I have tested your changes and can confirm they work for me on CentOS-5.5 with gcc-4.4.4. Based on Lee’s and orozco’s findings I have added better documentation and robustness to the Makefile. It now includes a pre-check, that should detect most common errors before starting the compile. The pre-check can be disabled by adding DISABLE_PRECHECKS=“YES” on the command line. Otherwise, the check is executed by default (by the ‘all’ target), and currently tests for:

  • HOSTTYPE variable is exported from the shell

  • Subversion is available, and version is >= 1.4.0

  • gfortran, gcc and g++ are available, and version is >= 4.2.0

Lee, could you update the SVN version if you find time? As usually, please download from: data.marssoft.de/cp/Makefile.CP2

All the best, Mario Emmenlauer

Hi all,

Download from: data.marssoft.de/cp/Makefile.CP2

Again, here is a new/updated version of the buildfile for CP2 and its prerequisites. This update to 0.1.0 includes a fullblown GTK. To build CellProfiler2 against the included gtk, use the (optional) “gtk” target before the (default) “all” target:

make -f Makefile.CP2 TGTDIR="${HOME}/usr/CellProfiler2" gtk all

Also, this version brings an improved check for libjvm.so based on Lee Kamentsky’s code. Please report back any problems/issues you might be having, and happy compiling!

All the best, Mario Emmenlauer

Hi all,

Download from: data.marssoft.de/cp/Makefile.CP2

I’m almost sorry for the frequent updates - here is another one, but after this I’ll let the Makefile.CP2 rest for a while :smiley: ! It has finally reached a mature state, and all basic requirements are met. Please find the newest version of the Makefile. This version adds MySQL 5.1.50 and its prerequisites. Another important change is the introduction of variables TMPDIR, SRCDIR and PREFIX:

  • mandatory: PREFIX replaces TGTDIR. All packages are installed in PREFIX/ :exclamation:

  • optional: temporary compilation is performed in TMPDIR/ which requires enough space (~2GB) :exclamation:

  • optional: source archives are expected in SRCDIR/

The Makefile can for example be invoked as follows:

make -f Makefile.CP2 PREFIX="${HOME}/usr/cp2" TMPDIR="/tmp/$USER" gtk all

All the best, Mario Emmenlauer

Hello!

…it seems i am almost done installing it on the Cluster, however when running:

python CellProfiler.py

i still get some errors (although CellProfiler seems to run anyway):

running build_ext
Using jdk_home = None
running build_ext
building ‘javabridge’ extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/g/almf/software/usr/cp2/lib/python2.5/site-packages/numpy/core/include -I/g/almf/software/bin/java/jre1.6.0_21/include -I/g/almf/software/bin/java/jre1.6.0_21/include/linux -I/g/almf/software/usr/cp2/include/python2.5 -c javabridge.c -o build/temp.linux-x86_64-2.5/javabridge.o
javabridge.c:204:17: error: jni.h: No such file or directory
javabridge.c:383: error: expected specifier-qualifier-list before ‘jmethodID’
javabridge.c:398: error: expected specifier-qualifier-list before ‘jclass’
javabridge.c:411: error: expected specifier-qualifier-list before ‘JNIEnv’
javabridge.c:425: error: expected specifier-qualifier-list before ‘jobject’
javabridge.c:440: error: expected specifier-qualifier-list before ‘JavaVM’
javabridge.c:453: error: expected specifier-qualifier-list before ‘jfieldID’

my set-up is like follows:

142 setenv JAVA_HOME /g/almf/software/bin/java/jre1.6.0_21
143 setenv JDK_HOME /g/almf/software/bin/java/jdk1.6.0_22
144 setenv jdk_home /g/almf/software/bin/java/jdk1.6.0_22
145 setenv LD_LIBRARY_PATH /g/almf/software/bin/java/jre1.6.0_21:/g/almf/software/bin/java/jdk1.6.0_22
146 setenv PATH $JAVA_HOME/bin:$JDK_HOME/bin:/g/almf/software/usr/cp2/bin/:$PATH
147 setenv LD_LIBRARY_PATH /g/almf/software/usr/cp2/lib:/g/almf/software/usr/cp2/lib64:${LD_LIBRARY_PATH}.
148 setenv PYTHONPATH /g/almf/software/usr/cp2/lib/python2.5

any ideas anyone?

Thanks, Tischi

Hi Tischi,

it seems there might be a problem with your JAVA_HOME? Can you try searching for “jni.h”?

find $JAVA_HOME -name jni.h

on my machine, JAVA_HOME is “/usr/lib/jvm/java-6-openjdk/”, so this returns:

/usr/lib/jvm/java-6-openjdk/include/jni.h

If jni.h is indeed not there for you, then your java would be broken. All the best!

Hi Mario,

indeed, i had to point JAVA_HOME to the JDK instead of the JRE, because the jni.h only exists in the JDK.

Thanks!
Tischi