How can I use the yacudecu deconvolution code in FIJI?

fiji
imagej
scripting
deconvolution

#21

Hi @GoranL

If you are interested in trying to build it, you could start by simply entering the following at a command prompt (assuming you have git and maven on your system).

git clone https://github.com/imagej/ops-experiments.git
cd ops-experiments-cuda
mvn

Some errors will likely happen, and we can discuss them. I don’t know anything about compiling cuda code with a Mac. However the build uses CMake, which automatically generates the right build files for each system. So we may be able to get it to work with a minimal amount of effort. I expect we may have to direct it to the location of Cuda, and potentially install the right version of the Cuda.


#22

Hi @bnorthan
I’ll definitely give it a try. I’ve never used compilers before, your help will be precious and usefull.
I just need to clone my system first, then I’ll try to install everything required.


#23

I gave it a quick try, to see how many obstacles there might be on macOS. Unfortunately, the road looks rough. Firstly, I installed the CUDA Toolkit for macOS from here. That dropped a bunch of goodies into /usr/local/cuda.

First attempt to build the ops-experiments-cuda directory:

mvn clean install
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Detecting the operating system and CPU architecture
[INFO] ------------------------------------------------------------------------
[INFO] os.detected.name: osx
[INFO] os.detected.arch: x86_64
[INFO] os.detected.version: 10.14
[INFO] os.detected.version.major: 10
[INFO] os.detected.version.minor: 14
[INFO] os.detected.classifier: osx-x86_64
[INFO]
[INFO] ------------------< net.imagej:ops-experiments-cuda >-------------------
[INFO] Building ops-experiments-cuda 0.1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.0.0:clean (default-clean) @ ops-experiments-cuda ---
[INFO] Deleting /Users/curtis/code/imagej/ops-experiments/ops-experiments-cuda/target
[INFO]
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-rules) @ ops-experiments-cuda ---
[INFO] Skipping Rule Enforcement.
[INFO]
[INFO] --- build-helper-maven-plugin:1.12:regex-property (sanitize-version) @ ops-experiments-cuda ---
[INFO]
[INFO] --- buildnumber-maven-plugin:1.4:create (default) @ ops-experiments-cuda ---
[INFO] Executing: /bin/sh -c cd '/Users/curtis/code/imagej/ops-experiments/ops-experiments-cuda' && 'git' 'rev-parse' '--verify' 'HEAD'
[INFO] Working directory: /Users/curtis/code/imagej/ops-experiments/ops-experiments-cuda
[INFO] Storing buildNumber: 7b49ca54c707000538d0a15710f47b999cf88d9c at timestamp: 1542207290472
[INFO] Storing buildScmBranch: master
[INFO]
[INFO] --- scijava-maven-plugin:1.0.0:set-rootdir (set-rootdir) @ ops-experiments-cuda ---
[INFO] Setting rootdir: /Users/curtis/code/imagej/ops-experiments
[INFO]
[INFO] --- exec-maven-plugin:1.5.0:exec (cppbuild) @ ops-experiments-cuda ---
Detected platform "macosx-x86_64"
Building for platform "macosx-x86_64"
Installing "YacuDecu"
~/code/imagej/ops-experiments/ops-experiments-cuda/native/YacuDecu/cppbuild ~/code/imagej/ops-experiments/ops-experiments-cuda/native
TODO
~/code/imagej/ops-experiments/ops-experiments-cuda/native
[INFO]
[INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ ops-experiments-cuda ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/curtis/code/imagej/ops-experiments/ops-experiments-cuda/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ ops-experiments-cuda ---
[INFO] Compiling 8 source files to /Users/curtis/code/imagej/ops-experiments/ops-experiments-cuda/target/classes
[INFO]
[INFO] --- javacpp:1.3:build (process-classes) @ ops-experiments-cuda ---
[INFO] Detected platform "macosx-x86_64"
[INFO] Building for platform "macosx-x86_64"
[INFO] Generating /Users/curtis/code/imagej/ops-experiments/ops-experiments-cuda/target/classes/net/imagej/ops/experiments/filter/deconvolve/jniYacuDecuRichardsonLucyWrapper.cpp
[INFO] Compiling /Users/curtis/code/imagej/ops-experiments/ops-experiments-cuda/target/classes/net/imagej/ops/experiments/filter/deconvolve/macosx-x86_64/libjniYacuDecuRichardsonLucyWrapper.dylib
[INFO] clang++ -I/Users/curtis/code/imagej/ops-experiments/ops-experiments-cuda/src/main/java -I/Users/curtis/code/imagej/ops-experiments/ops-experiments-cuda/native/YacuDecu/src/ -I/Library/Java/JavaVirtualMachines/jdk1.8.0_172.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/jdk1.8.0_172.jdk/Contents/Home/include/darwin /Users/curtis/code/imagej/ops-experiments/ops-experiments-cuda/target/classes/net/imagej/ops/experiments/filter/deconvolve/jniYacuDecuRichardsonLucyWrapper.cpp -march=x86-64 -m64 -O3 -Wl,-rpath,@loader_path/. -Wall -fPIC -dynamiclib -undefined dynamic_lookup -o libjniYacuDecuRichardsonLucyWrapper.dylib -lcufft -lcudart -lYacuDecu -framework JavaVM
ld: library not found for -lcufft
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Oops, no CUDA libraries on the library path. Let’s try adding them:

Manual clang++ invocation
$ clang++ -I/Users/curtis/code/imagej/ops-experiments/ops-experiments-cuda/src/main/java -I/Users/curtis/code/imagej/ops-experiments/ops-experiments-cuda/native/YacuDecu/src/ -I/Library/Java/JavaVirtualMachines/jdk1.8.0_172.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/jdk1.8.0_172.jdk/Contents/Home/include/darwin /Users/curtis/code/imagej/ops-experiments/ops-experiments-cuda/target/classes/net/imagej/ops/experiments/filter/deconvolve/jniYacuDecuRichardsonLucyWrapper.cpp -march=x86-64 -m64 -O3 -Wl,-rpath,@loader_path/. -Wall -fPIC -dynamiclib -undefined dynamic_lookup -o libjniYacuDecuRichardsonLucyWrapper.dylib -lcufft -lcudart -lYacuDecu -framework JavaVMclang++ -I/Users/curtis/code/imagej/ops-experiments/ops-experiments-cuda/src/main/java -I/Users/curtis/code/imagej/ops-experiments/ops-experiments-cuda/native/YacuDecu/src/ -I/Library/Java/JavaVirtualMachines/jdk1.8.0_172.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/jdk1.8.0_172.jdk/Contents/Home/include/darwin -L/usr/local/cuda/lib /Users/curtis/code/imagej/ops-experiments/ops-experiments-cuda/target/classes/net/imagej/ops/experiments/filter/deconvolve/jniYacuDecuRichardsonLucyWrapper.cpp -march=x86-64 -m64 -O3 -Wl,-rpath,@loader_path/. -Wall -fPIC -dynamiclib -undefined dynamic_lookup -o libjniYacuDecuRichardsonLucyWrapper.dylib -lcufft -lcudart -lYacuDecu -framework JavaVM
ld: library not found for -lYacuDecu
clang: error: linker command failed with exit code 1 (use -v to see invocation)

OK, so I forgot YacuDecu. No problem, let’s build from source!

Hmm, the YacuDecu README says “The software has been tested on 64bit Linux and Windows 7.” Uh oh. Well, let’s try anyway!

Download and build YacuDecu
$ git clone git://github.com/bobpepin/YacuDecu
...
$ cd YacuDecu
$ export PATH=/usr/local/cuda/bin:$PATH
$ make -f Makefile.linux
nvcc -O3 -Xcompiler=-fPIC -c -o deconv.o deconv.cu
nvcc fatal   : The version ('10.0') of the host compiler ('Apple clang') is not supported
make: *** [deconv.o] Error 1

After some research, I believe the solution is to downgrade the version of clang being used, by installing an older Command Line Tools for Xcode from here. According to the CUDA Installation Guide for Mac OS X, CUDA 10.0 (the current release as of this writing) works with macOS 10.13.6 with Xcode 9.4 + Apple LLVM 9.0.0. Unfortunately, I am running macOS 10.14, and there is no available “Command Line Tools (macOS 10.14) for Xcode 9.x”—the oldest available for macOS 10.14 is Xcode 10. So that’s where I got stuck.

In summary, I believe at least the following will be necessary:


#24

Hi Curtis

Other than installing the right version of Cuda (currently the code is set up to link to Cuda 9.0) the build (theoretically) doesn’t have any extra steps.

You shouldn’t have to build YacuDecu.lib manually.

A modified version (with some bug fixes and the addition of non-circulant edge normalization) of the YacuDecu (deconv.cu) code is in the ops-experiments code base here.

The pom.xml (thanks to @hadim) has a section which invokes a native build script [cppbuild.sh] which in turn invokes lower level build scripts also called cppbuild.sh.

Then a Makefile is called if you are using linux. If you are on Mac it will just say “Todo”

@GoranL basically I need to implement a section in the cppbuild.sh to build the YacuDecu library on a Mac (probably a few lines similar to the command Curtis used to try and build YacuDecu).

However we may still get stuck because of the incompatibility between CUDA 10.0 and macOS 10.14/Xcode10.

(as a side note there are similar issues with Windows, some versions of CUDA do not work with the latest version of Visual Studio Community, and sadly it’s not trivial to install old versions of Visual Studio because everything is done through web installers that only seem to allow you to go one version back).

There are a few of things we can do to continue to make progress

  1. I can take a crack at adding build instructions for Mac in the cppbuild.sh file(s).
  2. We can try and track down someone who is actively developing CUDA applications for macOS and is willing to help us. If anyone knows of someone point them to this thread!
  3. You can let us know if you have access to another operating system, either linux or Windows 10.0.

Brian


#25

As a side note and in case it can be helpful, various Visual compilers version are available on conda forge: https://github.com/conda-forge/staged-recipes/wiki/VC-features

How to integrate this with the cppbuild.sh file is another story :slight_smile:


#26

Hi @hadim

If using the Microsoft Visual c++ compiler one option is is to start mingw64 from a “x64 native tools VS” terminal, thus you will have a bash prompt set up with the compiler paths. Alternatively you could write a script to set up the paths.

For example if git bash is installed call “C:\Program Files\Git\bin\sh.exe”.

Then cppbuild.sh will be called properly and you can use nmake from there. See this commit

(It’s not integrated into master yet but will be soon).