Depending on outside JAR

fiji
imagej
bigdataviewer
micro-manager
#1

@ctrueden,

I would like to add a JAR for Micro-Magellan to the main FIJI distribution. Magellan is a Micro-Manager plugin for high throughput imaging with widefield, confocal, and multiphoton microscopes. There’s not much about it online yet, but that will change in the near future. It is under a BSD license.

Tobias Pietzsch has assisted in making image data written by Magellan readable by the BigDataViewer. The code for this reader lives within the Magellan jar. This project is not mavenized or on GitHub yet. Is it possible to add a static copy of the Magellan jar to the ImageJ maven repository so that it can be included in distributions of FIJI, and users will not have to manually add it when opening data with the BDV?

Henry

#2

In principle this could be possible, but there may be some challenges.

So presumably it depends on one or more Micro-Manager components. MMCoreJ perhaps?

As a rule, if we ship a JAR library, we also ship its dependencies. To do otherwise creates technical hurdles, and also produces a non-working system (w.r.t. that library) at runtime.

Great!

Is that planned? If you push the code to GitHub, I can help with the Mavenization. Or do you have reasons you would prefer not to use Maven for this project?

One of the Fiji contribution requirements is that the project must be deployed as a Maven artifact. That doesn’t mean it needs to use Maven to build—but doing so is the easiest way to meet that requirement. Other Maven-friendly build systems like Gradle are also possible, but the core team has little experience with them and thus cannot troubleshoot related problems as effectively.

Yes, we have done this with some third-party libraries which are not otherwise available from Maven repositories, such as Mines JTK. This lets Fiji components properly depend on those libraries.

Again, one question is what dependencies it brings in. We could potentially add the Magellan dependency to Fiji while excluding its Micro-Manager dependencies. But that might bring trouble in unexpected places (e.g., ClassNotFoundException).

#3

For acquiring data, yes. But by design, all of the parts used in reading data data used by the BDV are free of external dependencies.

It’s certainly something I would like to do down the road, but unfortunately I don’t have time for active development at the moment. I’d like to get a static copy up to make Magellan as useful as possible in the meantime.

I think this is the way to go. Would the mechanism of including it be to add a dependency in the BDV project?

#4

Note that to include Magellan with Fiji, it is required for the source to live on GitHub somewhere.

That depends. There is no library in the BDV realm that actually would depend on Magellan, right? I suppose you could add it to BDV as a runtime dependency, so it gets brought in to Fiji transitively. That is the intended scope, right? Just augmenting the functionality of BDV?

#5

Okay, done.

Correct on all counts. How would one go about adding it as a runtime dependency?

#6

So, I took at look at Magellan’s dependencies.

Firstly, it seems that Magellan is actually part of the core Micro-Manager SVN repository. Sorry, I did not realize that. But it doesn’t appear to be distributed with the Micro-Manager download?

Anyway, I started trying to grok your dependencies. For me, the easiest way to do this is to Mavenize a project, so I started doing it:

https://github.com/henrypinkard/micromagellan/compare/master...ctrueden:maven

It almost works, except for a couple of still-missing dependencies that I could not find: delaunay_triangulation and ij3d.image3d. Any ideas where those live?

I already uploaded MMCoreJ and MMJ_ version 1.4.22 (extracted from the official distribution) to the ImageJ Maven repository, so those dependencies are OK for now (if improper since they lack dependency declarations of their own).

<dependency>
	<groupId>org.micromanager</groupId>
	<artifactId>micromagellan</artifactId>
	<version>0.1.0-SNAPSHOT</version>
	<scope>runtime</scope>
</dependency>

But in order to guarantee reproducible builds, we will first need to cut a release version.

After going through all that, I have some misgivings about this plan. All of Fiji’s dependencies have been painstakingly organized over the past 2+ years as the project was Mavenized, and I don’t really like throwing a dirty project (incomplete dependencies) into the mix.

But I don’t have a concrete reason not to move forward, beyond the issue with the delaunay_triangulation and ij3d.image3d packages. If we can sort those, we can go ahead and add Magellan as a dependency of BDV.

There is an open question whether you want Magellan’s other dependencies besides Micro-Manager itself to be uploaded to Fiji also. In particular, commons-math (v2) is not currently on the Fiji update site.

#7

I think what would make me happier, IIUC, is for Magellan to be split into multiple components. Because it is the case that you want the BDV to leverage the file format-related portion of Magellan, but nothing related to the acquisition portion, right?

If so, it seems the acquisition part should be a separate component from the file format part, no?

#8

That will change in the very near future (potentially as early as tomorrow)

These will be distributed with micromanager once magellan is added to the build. They can also be found elsewhere online but are not mavenized as far as I know.

Almost, but not exactly. All of Magellan’s external dependencies come from the acquisition related portion, but the acquisition and format portions depend on a good deal of common internal code. So splitting them apart would require splitting into at least 3 components. I’m not particularly eager to do this. It would be a good deal of work for minimal payoff, and I’m not sure what the consequences would be on the micromanager side of things where magellan is actually built. Hence my desire to simply place a static version of the magellan jar in FIJI for use with BDV.

#9

Magellan should not be added as a BDV dependency! Existing file-format backends are picked up at runtime using the scijava annotations framework.

There are other independently developed backends already, e.g. for the KLB lightsheet format and these are not BDV dependencies either. I expect that over time there may be more and I don’t want to keep track and add all of them as dependencies. That was the reason to use scijava annotations in the first place.

Regarding the discussion about putting it in the Fiji update: At this point I think it would make most sense to add it as a personal update site? Then none of the restrictions (must be maven artifact, clean dependencies, etc.) would apply, Henry could upload whatever he wants, and users would just have to tick one checkbox to activate it. We could move it into the main update site if at some point in the future micromanager is mavenized and splitted into components. This seems like the least amount of work for everyone.

#10

Yeah, I agree that the main point of the extensibility mechanism is so that you don’t have to keep track of all possible extensions from the core project.

However, if there are some extensions which are considered “blessed” or curated somehow, then it’s not necessarily a horrible scheme. This is what we do with ImageJ: the net.imagej:imagej artifact has several imagej-plugins-foo dependencies with runtime scope. To be completely correct, they should probably be declared with <optional>true</optional>, but then they aren’t brought in automatically via the dependency mechanism, which is what 95+% of people probably want. (You can always exclude them if you don’t want them.)

That said, I’m not necessarily arguing that it makes sense for BDV to do that with any of its own file format plugins. Just pointing out the existing use case.

This sounds like the best option to me. What do you think, @henrypinkard?

#11

I agree with this as well, since there’s nothing in BDV that actually depends on Magellan.

Yes, this is probably the most logical way that Magellan could be included in FIJI. Unfortunately, I’m not sure if there are immediate plans to add Micro-Manager. Last I heard there were difficulties with versioning ImageJ1 that had the potential to introduce unexpected bugs in Micro-Manager.

I’d hoped there might be a way to include it without the extra step of activating, but if you guys both agree this is the most logical option, then I’ll move ahead with this plan.

#12

It is the path of least effort (by far). And since it sounds like you don’t have a lot of time to devote to the project right now, almost certainly the best route.

Ultimately I would love it if someone (@Chris? Mark T? :wink:) would champion a cleanup of Micro-Manager’s build system and project structure on the Java side. Johannes and the OpenSPIM team and Mark T already spent many hours moving things in the right direction, but my understanding is that more still needs to be done—and no one is currently working on it. If there are ever proper Micro-Manager Maven artifacts and releases and properly curated update site, it will become much easier to “do the right thing” with components like Magellan which are built on top of it.

#13

Hi guys,

Improving µManager’s build system is one of those things that Mark knows more about than I do. The big problem from our perspective is that we’re right now doing everything we can just to make certain that we still have jobs in six months, which unfortunately doesn’t leave a lot of time for working on “non-essential” (even if very welcome) things like Mavenizing the build.

Hopefully things will stabilize sooner rather than later and we’ll be able to take a look at this, but right now our lives are too uncertain. We don’t like it any more than you do. :frowning:

#14

Yeah, I fully understand and sympathize. I mostly mentioned you just to clue you guys in to the discussion, since it is directly relevant to Micro-Manager. I don’t realistically expect any work on this front from you in the near future. Let the ImageJ team know if there is anything we can do to help you succeed, especially during this crucial stage.

#15

Sure, and thanks for thinking of us! Henry, good luck with Magellan!