Plugins for reading and writing compressed video

maven
video
imagej
plugins
ffmpeg
#1

Hello everyone!
While developing a pair of specialized plugins for certain objects tracking in a stack I often missed a possibility to use various compressed video files as sources for such stacks.
AFAIK ImageJ can only uncompressed video. So I decided to write my own plugin which could open compressed video files as virtual stacks.

I used FFmpeg library for this task (trough javacv - a bundle of jna interfaced libraries by bytedeco) and I succeeded. Ok, there is some complicated story with dependencies because ImageJ updater cannot distinguish different parts of the library with platform specifications in their names (or may be this is me who cannot…). But finally I’ve managed with this somehow, more or less (although I still have to ignore some updater’s warnings from time to time).
The plugin allows loading video files supported by FFmpeg decoders into a virtual stack and frame by frame linear reading of the video. It is more or less similar to the AVI_Reader plugin in functions.

After the reader plugin was made I realized that it is possible to make a plugin for encoding a stack into a video using same FFmpeg library. It currently allows a simple encoding scheme - it encodes a selected slice range into AVI container with video compressed as old standard MPEG-4, framerate and bitrate can be specified, and proportional frame rescaling can be done optionally.

Both plugins can be found at my update site:
FFmpeg_FrameReader is for video import (File>Import>Using FFmpeg…)
FFmpeg_FrameRecorder is for video export (File>Save As>Compressed AVI using FFmpeg…)

So, ok, long story. Now to the question. Additionally to their plugin-like functionality I’d like to have a possibility to use their code as API in other plugins. So it would be great to put them into imagej’s maven snapshots repository, and then use it as dependencies.
Both plugins are available at GitHub: this and this. I prepared things necessary for Travis, according to this page. And all this message is nothing but an attempt to “Contact an ImageJ admin in Gitter or the ImageJ forum and request that they file a PR which adds Travis support to your repository”.
It is also possible that these plugins could be useful for the community, I believe.

Thank you.

p.s. A warning for someone who will probably try these plugins.
current versions of the plugins require following dependencies in /jar folder:
ffmpeg-3.2.1-1.3.jar
ffmpeg-platform-3.2.1-1.3.jar
javacpp-1.3.2.jar
javacv-1.3.2.jar
javacv-platform-1.3.2.jar
/‘platform-short-name’/ffmpeg-3.2.1-1.3-‘platform-name’.jar

as for the “platform-name”, for example - /jar/win64/ffmpeg-3.2.1-1.3-windows-x86_64.jar
Some of ImageJ plugins use of older versions of javacv, javcpp and ffmpeg, so be careful to not break your installation

1 Like

#2

Hi @anotherche and welcome to the forum!

Thanks for sharing your work!

I wonder how your plugins compare to the workflow currently documented on the wiki:

Did you try to import videos with the FFMPEG plugin served from the FFMPEG update site and compared this to your new plugins?

If yours provide some advantages, maybe they can be included into the main Fiji distribution in the future.

0 Likes

#3

Hi @imagejan!
Current “Out of the box” support of video is restricted by reading and writing of uncompressed video sources whereas most of videos available from elsewhere or recorded by devices are compressed (encoded with various video codecs). To use that kind of video with ImageJ one should first decompress it with some additional software. So, I wanted to make a plugin for direct opening compressed video as a virtual stack in ImageJ.
But you are right that I seem to “invent a bike”. It happened somehow that I didn’t note FFMPEG update site before. Now I have tried it, and more, I’ve noted that there is SCIFIO-JavaCV project which is supposedly based on the above FFMPEG_IO plugin. SCIFIO-JavaCV is supposed to offer out-of-the-box support for the compressed video formats. But now it is inactive and not finished so that there is no possibility to test it for a comparison.
Well, my plugins appeared to be similar to FFMPEG_IO in suggested functionality and they are based on similar methods - use of native ffmpeg libraries through jna layer. I could not find FFMPEG_IO sources, so I’m may be wrong, but it seems that FFMPEG_IO uses direct wrapping of ffmpeg cpp functions made with jna. In contrast, my plugins (and those by SCIFIO-JavaCV) use javacpp presets by Bytedeco (whose project JavaCV is actively evolving).
I noted that FFMPEG_IO is outperforming in most operations - video sources are opening faster, and navigation across the created stack is faster (especially in case of reverse direction which I still don’t know how to realize other than restarting reading from the beginning or by caching some number of previous frames, but which makes the stack not so virtual). I would like to know how the fast reading is realized in FFMPEG_IO plugin, but I could not find its sources.
On the other hand, FFMPEG_IO could not open some of my test files (which are opened successfully with my plugin) and it often crashes when encoding to a video file, with ImageJ unexpectedly closing (which was never seen with mine now).
A minor difference (not advantage or disadvantage) is that FFMPEG_IO uses mpeg1 encoder to write video files whereas I selected mpeg4 for my recording plugin. Basically, the selection of the video codec for the recording plugin can be customized to allow interactive configuration of the desired codec.
So, that’s it. As is seen, there is a need of the functionality to read/write compressed video sources (remember SCIFIO-JavaCV…). The suggested plugins provide the needed things, they depend on current version of javacv, and require some optimization. The fundamental question on the inclusion it in Fiji is probably that they require use of native libraries, so some internal support is required in Fiji updater and imagej-maven-plugin to automate the platform specific things.

1 Like

#4

I found this post while googling for a solution to how to use ffmpeg to export movies from within ImageJ with better quality than the built-in AVI codec.

So far your solution is the best one of the four I have tried; and FFMPEG_IO is not working well for me. It usually (but not always) just crashes ImageJ / Fiji midway in the conversion. I noticed that the native libraries linked by FFMPEG_IO are from 2013 while you use a recent build; that could make a difference. I also tried to convert from the command line using the Zeranoe build of ffmpeg.exe and the libx264 codec. The latter approach gives slightly better results, but that is cumbersome as I then need to prepare a folder of single images as an intermediate step.

I installed your plugin manually in vanilla ImageJ 1.52m using your list of dependencies. It worked fine on the first attempt; I used the most recent build (3.4.1-1.4) of each file.

So keep up the good work!

0 Likes

#5

Thanks, @steinr !
I knew that there must be people on the planet who need it.
I have actually finished with this because I satisfied my need to export video from ImageJ with this plugin and I had no more time to improve this further.
But there was one additional improved version which I did not upload to the update site (because I have not done enough testing). In this new version a possibility is added to chose video format (avi, mov, mp4, mkv, or automatic by file extension) and encoder (mpeg4, h264, mjpeg, huffyuv, or custom, or automatic by video format) as well as to specify additional options for the encoder.
So, if you are interested in testing of these new capabilities I can upload it as is to the update site (I checked the build today against the latest JavaCV).

0 Likes

#6

Yes, I would be very interested in testing that!
My primary goal was to export videos with h264 codec. I can do that with ffmpeg on the command line from a folder of exported single images, and see that the result video quality is slightly better than with the mpeg4 codec for the same bitrate.

0 Likes

#7

The new version is already there. There can be strange problems with the layout of the settings dialog if you run it on Linux (a button in the upper right corner is not visible but it fully functional). on Windows everything is ok.

0 Likes

#8

Thanks so much for making this. It’s the only thing that’s worked for me so far.

Is there a way to import higher than 8-bit bit depth clips and save an image sequence of tifs without loss of bit depth? I haven’t tried your “experimental” version yet.

0 Likes

#9

Could you share one of those for others to try? Which format and codec do your clips use?

0 Likes

#10

Sorry. I first wrote the following answer and then realized that the question was not about export but about import. Any way I will leave it here because it has some relation to the topic. As for the import, I am intrigued too - what can be the format of that clip.

Basically, ffmpeg can use 16 bit grey or 48 bit RGB images as input (check output of “ffmpeg -pix_fmts” if you have ffmpeg installed). But I am not aware of any video codec capable of encoding a movie with a high bit depth. I’ve never heard of any video player as well which could play such video. Moreover, most of video codecs are lossy compressors which convert images to lower effective bit depth (by reducing space resolution of chromatic component). So, if you want to encode without loss of bit depth you will probably want not to loss any information. Then it will be uncompressed video (or video with lossless compression which mostly reduce the file size ~twice). Then you will get ~1GB per 10 sec of movie with FHD resolution. That’s the standard problem with lossless/uncompressed video. Any way, if you still want it, higher bit depth material can be encoded as the sequence of 8-bit channels (but then, you will need some specialized decoder and player which renders it back to series of normal RGB or grey images, using methods similar to plugins managing RAW camera images, but making it on the fly).

P.S. After some googling I found that H.265 probably supports 10 or 12 bit grey format. At least “ffmpeg -h encoder=libx265” says that it can use gray10le and gray12le inputs. And there is information that ffmpeg can decode h.265 streams. But do not have such clips.

0 Likes

#11

I don’t currently have any high bit depth files myself but a friend of mine is capturing his old family 8mm films with some kind of camera setup that captures each frame at high bit depth. He’s still tinkering with it but he wants me to stabilize the film for him when the time comes so I thought I’d poke around and see what’s available in an attempt to keep the workflow as lossless as possible until the final encode. It could be that he just has to deal with current limitations.

0 Likes

#12

I checked few publicly available samples of 10 bit HDR video. The plugin opens it as usual 24 bit RGB (that is the images are imported in the standardl 8 bit depth). This is because imageMode parameter of the internal FFmpegFrameGrabber class is set default as 24 bit color. The two other possibilities for the imageMode are GREY (normal 8 bit image) and RAW (general byte buffer). It is possible to enable setting it to RAW but then we need information of the data format in this buffer and a support of ImageJ to represent the sequence of those buffers as the stack of images of the supported formats (floating point or 16 bit int with the adjustable depth scaling)

0 Likes