Plugins for reading and writing compressed video

maven
video
imagej
plugins
ffmpeg

#1

Hello everyone!
While developing a pair of specialized plugins for certain objects tracking in a stack I often missed a possibility to use various compressed video files as sources for such stacks.
AFAIK ImageJ can only uncompressed video. So I decided to write my own plugin which could open compressed video files as virtual stacks.

I used FFmpeg library for this task (trough javacv - a bundle of jna interfaced libraries by bytedeco) and I succeeded. Ok, there is some complicated story with dependencies because ImageJ updater cannot distinguish different parts of the library with platform specifications in their names (or may be this is me who cannot…). But finally I’ve managed with this somehow, more or less (although I still have to ignore some updater’s warnings from time to time).
The plugin allows loading video files supported by FFmpeg decoders into a virtual stack and frame by frame linear reading of the video. It is more or less similar to the AVI_Reader plugin in functions.

After the reader plugin was made I realized that it is possible to make a plugin for encoding a stack into a video using same FFmpeg library. It currently allows a simple encoding scheme - it encodes a selected slice range into AVI container with video compressed as old standard MPEG-4, framerate and bitrate can be specified, and proportional frame rescaling can be done optionally.

Both plugins can be found at my update site:
FFmpeg_FrameReader is for video import (File>Import>Using FFmpeg…)
FFmpeg_FrameRecorder is for video export (File>Save As>Compressed AVI using FFmpeg…)

So, ok, long story. Now to the question. Additionally to their plugin-like functionality I’d like to have a possibility to use their code as API in other plugins. So it would be great to put them into imagej’s maven snapshots repository, and then use it as dependencies.
Both plugins are available at GitHub: this and this. I prepared things necessary for Travis, according to this page. And all this message is nothing but an attempt to “Contact an ImageJ admin in Gitter or the ImageJ forum and request that they file a PR which adds Travis support to your repository”.
It is also possible that these plugins could be useful for the community, I believe.

Thank you.

p.s. A warning for someone who will probably try these plugins.
current versions of the plugins require following dependencies in /jar folder:
ffmpeg-3.2.1-1.3.jar
ffmpeg-platform-3.2.1-1.3.jar
javacpp-1.3.2.jar
javacv-1.3.2.jar
javacv-platform-1.3.2.jar
/‘platform-short-name’/ffmpeg-3.2.1-1.3-‘platform-name’.jar

as for the “platform-name”, for example - /jar/win64/ffmpeg-3.2.1-1.3-windows-x86_64.jar
Some of ImageJ plugins use of older versions of javacv, javcpp and ffmpeg, so be careful to not break your installation


#2

Hi @anotherche and welcome to the forum!

Thanks for sharing your work!

I wonder how your plugins compare to the workflow currently documented on the wiki:

Did you try to import videos with the FFMPEG plugin served from the FFMPEG update site and compared this to your new plugins?

If yours provide some advantages, maybe they can be included into the main Fiji distribution in the future.


#3

Hi @imagejan!
Current “Out of the box” support of video is restricted by reading and writing of uncompressed video sources whereas most of videos available from elsewhere or recorded by devices are compressed (encoded with various video codecs). To use that kind of video with ImageJ one should first decompress it with some additional software. So, I wanted to make a plugin for direct opening compressed video as a virtual stack in ImageJ.
But you are right that I seem to “invent a bike”. It happened somehow that I didn’t note FFMPEG update site before. Now I have tried it, and more, I’ve noted that there is SCIFIO-JavaCV project which is supposedly based on the above FFMPEG_IO plugin. SCIFIO-JavaCV is supposed to offer out-of-the-box support for the compressed video formats. But now it is inactive and not finished so that there is no possibility to test it for a comparison.
Well, my plugins appeared to be similar to FFMPEG_IO in suggested functionality and they are based on similar methods - use of native ffmpeg libraries through jna layer. I could not find FFMPEG_IO sources, so I’m may be wrong, but it seems that FFMPEG_IO uses direct wrapping of ffmpeg cpp functions made with jna. In contrast, my plugins (and those by SCIFIO-JavaCV) use javacpp presets by Bytedeco (whose project JavaCV is actively evolving).
I noted that FFMPEG_IO is outperforming in most operations - video sources are opening faster, and navigation across the created stack is faster (especially in case of reverse direction which I still don’t know how to realize other than restarting reading from the beginning or by caching some number of previous frames, but which makes the stack not so virtual). I would like to know how the fast reading is realized in FFMPEG_IO plugin, but I could not find its sources.
On the other hand, FFMPEG_IO could not open some of my test files (which are opened successfully with my plugin) and it often crashes when encoding to a video file, with ImageJ unexpectedly closing (which was never seen with mine now).
A minor difference (not advantage or disadvantage) is that FFMPEG_IO uses mpeg1 encoder to write video files whereas I selected mpeg4 for my recording plugin. Basically, the selection of the video codec for the recording plugin can be customized to allow interactive configuration of the desired codec.
So, that’s it. As is seen, there is a need of the functionality to read/write compressed video sources (remember SCIFIO-JavaCV…). The suggested plugins provide the needed things, they depend on current version of javacv, and require some optimization. The fundamental question on the inclusion it in Fiji is probably that they require use of native libraries, so some internal support is required in Fiji updater and imagej-maven-plugin to automate the platform specific things.


#4

I found this post while googling for a solution to how to use ffmpeg to export movies from within ImageJ with better quality than the built-in AVI codec.

So far your solution is the best one of the four I have tried; and FFMPEG_IO is not working well for me. It usually (but not always) just crashes ImageJ / Fiji midway in the conversion. I noticed that the native libraries linked by FFMPEG_IO are from 2013 while you use a recent build; that could make a difference. I also tried to convert from the command line using the Zeranoe build of ffmpeg.exe and the libx264 codec. The latter approach gives slightly better results, but that is cumbersome as I then need to prepare a folder of single images as an intermediate step.

I installed your plugin manually in vanilla ImageJ 1.52m using your list of dependencies. It worked fine on the first attempt; I used the most recent build (3.4.1-1.4) of each file.

So keep up the good work!


#5

Thanks, @steinr !
I knew that there must be people on the planet who need it.
I have actually finished with this because I satisfied my need to export video from ImageJ with this plugin and I had no more time to improve this further.
But there was one additional improved version which I did not upload to the update site (because I have not done enough testing). In this new version a possibility is added to chose video format (avi, mov, mp4, mkv, or automatic by file extension) and encoder (mpeg4, h264, mjpeg, huffyuv, or custom, or automatic by video format) as well as to specify additional options for the encoder.
So, if you are interested in testing of these new capabilities I can upload it as is to the update site (I checked the build today against the latest JavaCV).


#6

Yes, I would be very interested in testing that!
My primary goal was to export videos with h264 codec. I can do that with ffmpeg on the command line from a folder of exported single images, and see that the result video quality is slightly better than with the mpeg4 codec for the same bitrate.