Using pyinstaller on a program that uses pyimagej

I have a program that uses pyimagej in a python script for multiple image processing steps. The program runs fine and I want to make it available to the users on windows as a .exe program, bundled with all its dependencies.

I can create such a bundle on macOS by using:

pyinstaller -p fork-stitcher --add-data '/miniconda3/envs/fork_stitcher/share/pyjnius/pyjnius.jar:./share/pyjnius/' fork-stitcher/run_stitching_batches.py

(I need to manually add the pyjnius JAR file, then it runs). This program then runs and performs the task. But I need this to run on Windows, which is where I run into difficulties.

The python script itself runs fine on my Windows machine, but when I package it up with pyinstaller, I run into issues.

  1. By default, it can’t find the jre folder. So I tried to add this as a datafile, like this:
C:\\Users\\j.luethi\\AppData\\Local\\Continuum\\miniconda3\\envs\\fork_stitcher\\Library\\jre; .\\share\\jre

And set the javapath in my pythonscript to it: os.environ['JAVA_HOME'] = '.\\share'

Then, it has an issue because maven is not found on the PATH:
jgo.jgo.ExecutableNotFound: mvn not found on path ....

So I also add Maven to the files that are added (this bin folder is where the script gets maven from when run directly, so I included the whole folder):

C:\\Users\\j.luethi\\AppData\\Local\\Continuum\\miniconda3\\envs\\fork_stitcher\\Library\\bin; '\\share\\bin

And add it to the PATH variable: os.environ['PATH'] += os.pathsep + '.\\share\\bin'

After that, it doesn’t complain about not finding maven anymore, but now has the following:

INFO Failed to bootstrap the artifact.
INFO Possible solutions:
INFO * Double check the endpoint for correctness (https://search.maven.org/).
INFO * Add needed repositories to ~/.jrunrc [repositories] block (see README).
INFO * Try with an explicit version number (release metadata might be wrong).
Traceback
....
File "site-packages\jgo\jgo.py", line 203, in run_and_combine_outputs
...
subprocessed.CalledProcessError: Command mvn.CMD -B -f pathToFijiInstance (in .jgo folder) dependenc:resolve returned a non-zero exit status 1.

Does anyone have an idea on how I get around this? Or how I correctly package up a script using pyimage with all its dependencies? I’m open to alternative approaches, but could only find pyinstaller and py2exe (which does not support Python 3.7). Is there a way to install the java dependencies correctly to that package instead of my approach of adding the things that made the program crash?

@ctrueden Maybe you could come to the rescue once more? I saw this issue that also has the Failed to bootstrap the artifact. message, so maybe that’s related? But I can’t follow the solution as discussed there.

(sorry, I can’t copy the error message, because it crashes just after showing it. Here’s a picture of it)

1 Like

@jluethi The shell script version of jgo dumps the output of the Maven command when verbose mode is set. This would be useful to understand more precisely what is going wrong in the invocation. Without that output, the problem could be anything from “the path to the mvn executable is wrong” to “a network error occurred while downloading some artifact” to “the installed version of Maven is too old” to “your backslashes are too escaped, or not escaped enough”…

Here is a proposed (untested) patch that adds this feature to the Python version of the program as well:

diff --git jgo/jgo.py jgo/jgo.py
index f27d641..9018836 100644
--- jgo/jgo.py
+++ jgo/jgo.py
@@ -448,13 +448,15 @@ def resolve_dependencies(
         mvn     = executable_path_or_raise('mvn')
         mvn_out = run_and_combine_outputs(mvn, *mvn_args)
     except subprocess.CalledProcessError as e:
-        _logger.info("Failed to bootstrap the artifact.")
-        _logger.info("")
-        _logger.info("Possible solutions:")
-        _logger.info("* Double check the endpoint for correctness (https://search.maven.org/).")
-        _logger.info("* Add needed repositories to ~/.jgorc [repositories] block (see README).")
-        _logger.info("* Try with an explicit version number (release metadata might be wrong).")
-        print()
+        _logger.error("Failed to bootstrap the artifact.")
+        _logger.error("")
+        _logger.error("Possible solutions:")
+        _logger.error("* Double check the endpoint for correctness (https://search.maven.org/).")
+        _logger.error("* Add needed repositories to ~/.jgorc [repositories] block (see README).")
+        _logger.error("* Try with an explicit version number (release metadata might be wrong).")
+        _logger.error("")
+        _logger.debug("Here is the Maven log:")
+        _logger.debug(mvn_out)
         raise e

Give it a shot and see if any more details of the failure emerge. (The log level will need to be set to DEBUG, of course.)

1 Like

Great idea @ctrueden! Unfortunately, mvn_out is only assigned in mvn_out = run_and_combine_outputs(mvn, *mvn_args) (just a few lines above in the try), which is the line that actually fails. Thus, if I try to log this output, I get a UnboundLocalError: local variable mvn_out referenced before assignment .

But following this logic that the command line gives more information, I tried calling mvn.CMD directly from the command line. When I call it in its original location, it runs fine. But when I call it from the location where pyinstaller puts it, it gives the following error message: Error: Could not find or load main class org.codehaus.plexus.classworlds.launcher.Launcher

I don’t really understand Maven that well. I saw this thread on StackOverflow about it mentioning the problem can be the M2_HOME variable. I checked an it’s not set for me (as they say it should). They also mention that the issue can be with the JAVAPATH and maybe the location of the m2 folder. Would I need to place them in a specific location in my pyinstaller bundle? Or set this variable to something specific?

1 Like

@jluethi I’m sorry, I don’t regularly use Windows, so my knowledge here is limited. But I have used Maven on Windows many times successfully in the past. I never set M2_HOME or M3_HOME so I think that’s a red herring. All I did was add the bin directory of the Maven installation to the system PATH. Maven should not require any magic—you can just download the ZIP and unzip it and run it, even on Windows, no need to run an installer. So I’m surprised that pyinstaller copying the Maven installation somewhere else is creating problems. Are you sure that pyinstaller is copying the entire installation? Perhaps there are some files missing?

You can try running Maven with -X to get extended debugging output. Maybe it gives some more details? Also try diffing the directories. Manually copying your Maven directory to another location and running from there (cutting out pyinstaller from the equation).

1 Like

Thanks @ctrueden
I now downloaded Maven freshly and moved the whole unzipped directory into my package (works with or without doing via pyinstaller). Plus I set

os.environ['M2_HOME'] = '.\\share\\apache-maven-3.6.1\\bin'
os.environ['MAVEN_HOME'] = '.\\share\\apache-maven-3.6.1\\bin'

because I got error messages otherwise.

Now, maven seems to be installed correctly and if I manually call it, it works. But still, I get the following error message when I run ij = imagej.init('sc.fiji:fiji:2.0.0-pre-10+ch.fmi:faim-ij2-visiview-processing:0.0.1') (or ij = imagej.init('sc.fiji:fiji:2.0.0-pre-10'):

subprocess.CalledProcessError: Command '('.\\share\\apache-maven-3.6.1\\bin\\mvn.CMD', 
'-B', '-f', 'C:\\Users\\j.luethi\\.jgo\\sc\fiji\\fiji\\2.0.0-pre-10+ch.fmi-faim-ij2-visiview-processing-0.0.1+net.imglib-imglib2-imglyb-0.3.0\\pom.xml',
'dependency:resolve')' returned non-zero exit status 1.

(added newlines for readability, see screenshot for exact formatting and info before that)
The weird thing is, if I run this exact command manually in the console (without the ‘’ and the double ), I get a BUILD SUCCESS. Also, if I don’t initialize my java VM, the whole thing runs fine as well.

@ctrueden Any idea why maven successfully builds when manually called but returns a exit status one when called from within my .exe program? Anything I could print from within jgo to get relevant information? Because I seem to be getting the exact call that is being made and when I manually make it, it works.

Some details on the error message:

A couple of thoughts:

  • Are you sure your current working directory is what you expect? The command being invoked uses a relative path. Try setting the Maven variables to use an absolute path instead, and see if that alleviates the issue.
  • We really need to get the output of that command. See if you can modify the Python code to capture the output and feed it to the logger, as I suggested before.

@ctrueden I managed to catch the error message of this call in python (see this pull request for a suggestion on how to integrate this into jgo).

The actual error message I get is:

The JAVA_HOME environment variable is not defined correctly
This environment variable is needed to run this program
NB: JAVA_HOME should point to a JDK not a JRE

EDIT: My bad. I seem to slowly be learning about Java JDKs & JREs now. I made a wrong assumption before and only copied the jre folder from the JAVA_PATH. If I copy everything, I don’t get this issue (but apparently other things I now need to go through).

1 Like

Hmm, it seems that because we lean on Maven, a full JDK needs to be available. Maven assumes you’ll want to do things like compile code—even though in this case, you won’t actually need to do that. I have no idea whether it would be possible to convince the Maven command line tool to be satisfied with a JRE. Probably we could figure out how to call Maven programmatically from other Java code, but this would be more work. For now, using a full JDK is the easiest way forward.

@ctrueden Ok, using the openjdk, I can deal with that, I think.

Now Maven starts to run, but I always get the following error message (even when it’s pointed to the native library without this one being moved):

Error occurred during initialization of VM
Unable to load native library: Can't find dependent libraries

This only happens when I import pyimagej. I searched and found this stackoverflow thread where people recommend to use dependency walker. I used it at it finds some missing dependencies:

Error: At least one module has an unresolved import due to a missing export function in an implicitly dependent module.
Error: Modules with different CPU types were found.
Warning: At least one delay-load dependency module was not found.

Specifically, this was for the api-ms-win-service-private-l1-1-1.dll file. Any idea what I could be exporting wrongly? Or how I would need to adapt paths?

05

@jluethi It sounds like some pieces of the JDK are not being included in the bundle you are creating? Which OpenJDK build are you using? Did you try the AdoptOpenJDK Windows x64 bundled as a ZIP? I advise staying away from installers for a project like this, since you are repackaging things yourself.

There is also the version of openjdk distributed on conda-forge (based on the Zulu build). Maybe they already solved similar problems?

@ctrueden Thanks for the idea with using AdpotOpenJDK directly. It does not seem to make a difference. But it’s a cleaner installation procedure and I’m going with it.

The actual issue seems to be with the jvm.dll that is copied by pyinstaller to the same directory as the .exe file. I’ve seen people recommending that no jvm.dll should be in the folder where the .exe file goes (see here). If the PATH is set correctly (both to jre/bin and jre/bin/server), I can manually remove this jvm.dll and my application still starts (because the jvm.dll in jre/bin/server is now on the path).

After that, my application seems to run fine, as far as I can judge it. Currently running some tests to see if everything goes smooth :smile:

PS: The part that dependency walker found missing likely was a red herring, as dependency walker is unmaintained and gives these messages by mistake (see here).

1 Like

@jluethi There are some brief notes relating to this in the pyjnius documentation:

https://pyjnius.readthedocs.io/en/stable/packaging.html

That page recommends that end users have their own Java installed, available on the PATH, which pyjnius will then be able to find. There is no discussion of actually bundling the JVM with the application as you have been trying to do, although the implication is that it won’t work?

If you feel strongly about wanting to do this, I recommend filing an issue in the pyjnius repository here:

Just be aware that the pyinstaller packaging instructions linked above were added in response to this closed PR:

Thanks @ctrueden
Unfortunately, for my use case, I cannot assume users will have Java installed. But adding the jdk by having it in the datas part of the pyinstaller Analysis and adding that to the PATH variable and deleting the jvm.dll automatically included solves the issue for me.

With the jdk bundled & the PATH modified, my application now runs stably from the .exe
(well, at least the directory version of it runs, I haven’t gotten the --onefile to work). Therefore, this issue is now solved for my use case.

1 Like

@jluethi Glad you got it working! :smile:

Would it make sense to write up your process a little more explicitly and contribute it as an addition to that pyjnius packaging guide? As written, the guide does not make it clear that the approach you are using is viable.

1 Like