Bio-Formats 6 dependencies not present on Java-8 update site

dependencies
bio-formats
amazon-s3
#1

With Bio-Formats 6, several new dependencies are now required. In particular, the ome-common library now depends on minio, which has several dependencies. From a quick audit I did:

  • annotations-3.0.1.jar
  • google-http-client-1.20.0.jar
  • google-http-client-xml-1.20.0.jar
  • httpclient-4.0.1.jar
  • httpcore-4.0.1.jar
  • jackson-annotations-2.9.6.jar
  • jackson-core-2.9.6.jar
  • jackson-databind-2.9.6.jar
  • jcip-annotations-1.0.jar
  • jsr305-3.0.1.jar
  • minio-5.0.2.jar
  • okhttp-3.7.0.jar
  • okio-1.12.0.jar
  • xpp3-1.1.4c.jar

Without these artifacts in place, the BigStitcher emits serialization errors as follows:

KryoException stack trace
com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Unable to create serializer "com.esotericsoftware.kryo.serializers.FieldSerializer" for class: loci.common.services.S3ClientServiceImpl
Serialization trace:
services (loci.common.services.ServiceFactory)
factory (loci.formats.in.TiffReader)
nativeReader (loci.formats.in.TiffDelegateReader)
readers (loci.formats.ImageReader)
	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:82)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
	at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599)
	at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:348)
	at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:289)
	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:505)
	at loci.formats.Memoizer$KryoDeser.saveReader(Memoizer.java:206)
	at loci.formats.Memoizer.saveMemo(Memoizer.java:968)
	at loci.formats.Memoizer.setId(Memoizer.java:697)
	at net.preibisch.mvrecon.fiji.spimdata.imgloaders.LegacyFileMapImgLoaderLOCI.openImg(LegacyFileMapImgLoaderLOCI.java:223)
	at net.preibisch.mvrecon.fiji.spimdata.imgloaders.LegacyFileMapImgLoaderLOCI.getImage(LegacyFileMapImgLoaderLOCI.java:169)
	at mpicbg.spim.data.legacy.LegacyImgLoaderWrapper$SetupImgLoaderWrapper.getImage(LegacyImgLoaderWrapper.java:79)
	at bdv.export.WriteSequenceToHdf5.writeHdf5PartitionFile(WriteSequenceToHdf5.java:444)
	at bdv.export.WriteSequenceToHdf5.writeHdf5File(WriteSequenceToHdf5.java:157)
	at net.preibisch.mvrecon.fiji.plugin.resave.Generic_Resave_HDF5.writeHDF5(Generic_Resave_HDF5.java:243)
	at net.preibisch.mvrecon.fiji.datasetmanager.FileListDatasetDefinition.createDataset(FileListDatasetDefinition.java:1118)
	at net.preibisch.mvrecon.fiji.plugin.Define_Multi_View_Dataset.defineDataset(Define_Multi_View_Dataset.java:141)
	at net.preibisch.mvrecon.fiji.plugin.queryXML.LoadParseQueryXML.queryXML(LoadParseQueryXML.java:122)
	at net.preibisch.mvrecon.fiji.plugin.queryXML.LoadParseQueryXML.queryXML(LoadParseQueryXML.java:109)
	at net.preibisch.stitcher.plugin.BigStitcher.run(BigStitcher.java:74)
	at org.scijava.command.CommandModule.run(CommandModule.java:199)
	at org.scijava.module.ModuleRunner.run(ModuleRunner.java:168)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:127)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:66)
	at org.scijava.thread.DefaultThreadService$3.call(DefaultThreadService.java:238)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Unable to create serializer "com.esotericsoftware.kryo.serializers.FieldSerializer" for class: loci.common.services.S3ClientServiceImpl
	at com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:48)
	at com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:26)
	at com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:351)
	at com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:344)
	at com.esotericsoftware.kryo.util.DefaultClassResolver.registerImplicit(DefaultClassResolver.java:56)
	at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:461)
	at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
	at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:488)
	at com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.write(DefaultSerializers.java:239)
	at com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.write(DefaultSerializers.java:232)
	at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:577)
	at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:91)
	at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:21)
	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
	... 37 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
	at sun.reflect.GeneratedConstructorAccessor186.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:35)
	... 51 common frames omitted
Caused by: java.lang.NoClassDefFoundError: io/minio/errors/MinioException
	at java.lang.Class.getDeclaredFields0(Native Method)
	at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
	at java.lang.Class.getDeclaredFields(Class.java:1916)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.rebuildCachedFields(FieldSerializer.java:193)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.rebuildCachedFields(FieldSerializer.java:156)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.<init>(FieldSerializer.java:133)
	... 55 common frames omitted
Caused by: java.lang.ClassNotFoundException: io.minio.errors.MinioException
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 61 common frames omitted

The issue relates to loci.common.services.S3ClientServiceImpl not being able to load because the MinIO library and its dependencies are not currently distributed with Fiji.

@ome What is the correct fix? Upload all the needed dependencies to the Java-8 update site? Or did you intend for this feature to be optional, and for Bio-Formats to work without these libraries in place?

#2

Hi @ctrueden, thanks for bringing this to our attention. The expectation was that the Minio dependency should be optional. We look at resolving this issue as a priority.

1 Like
#3

@dgault Thanks a lot for pursuing a fix!

In the short term, what do you think we should do to get the BigStitcher working again? As things stand, that plugin (and possibly other functionality in Fiji) does not work, due to this issue.

Possible solutions:

  • Downgrade version of ome-common shipped with Fiji.
  • Make a quick patch release of ome-common temporarily removing the S3 functionality.
  • Ship the needed dependencies on the Java-8 update site.

Thoughts?

#4

FYI this is the PR that should’ve made Minio an optional dependency:

#5

We have been testing a solution for this today. The plan would be to release a new version of ome-common with the upcoming Bio-Formats 6.1.0 release.

1 Like
#6

Based on datestamps, it looks like ome-common 6.0.3, which includes the fix, was released and uploaded to the Java-8 update site yesterday. I was not able to reproduce the BigStitcher kryo issue on an up-to-date Fiji with BigStitcher update site enabled. So this issue is now fixed! :tada: Thank you @ome—and please correct me if I’m wrong.

1 Like