Working with .mov (and other movie) files in python / dask

Hi all

I’m looking for some tips on how to best work with .mov files in python (though I imagine what I’m looking for would also apply to other compressed movie files too).

One thing I’m trying to do is interactively visualize them in napari, but I might also be interested in doing some processing on them too, possibly with dask in a lazy fashion.

Right now I currently have .mov files that are ~5000 time points, ~1200x700 and rgb. The total size as a .mov file is ~60 MB which is very reasonable due to it’s compression - thanks @lambdaloop for this example movie

I can load these files into python and get a numpy array using imageio and ffmpeg as follows

import imageio
import numpy as np

vid = imageio.get_reader(path,  'ffmpeg')
movie = np.array([im for im in vid.iter_data()], dtype=np.uint8)

For the work I am doing though I find this a little slow (curious if there are better approaches), and the numpy array that is generated is quite large numpy with 12.6e9 elements. If I have longer movies this will become in-practical.

I don’t really want to work with the generator or the object called vid above, I’d really prefer something array-like, which makes trying to use dask seem like a good choice.

I think ideally I’d probably like to have a lazy loading scheme of the sort used in dask-image, see these blog posts https://blog.dask.org/2019/06/20/load-image-data and https://napari.org/tutorials/applications/dask for example, but dask-image imread doesn’t natively support reading the .mov file (and maybe I should be posting some of this question on the dask-image GitHub, but I’m trying here first), so I tried writing my own lazy style reader

import imageio
from dask import delayed
import dask.array as da

vid = imageio.get_reader(path,  'ffmpeg')
shape = vid.get_meta_data()['size'][::-1] + (3,)
lazy_imread = delayed(vid.get_data)
movie = da.stack([da.from_delayed(lazy_imread(i), shape=shape, dtype=np.uint8) for i in range(vid.count_frames())])

But I find this very slow too. It could be because I’ve done the dask delayed poorly, it could be for other reasons related to how ffmpeg works. Has anyone else tried this / succeeded here? Maybe @talley has ideas :slight_smile:

2 Likes

You could look at what it is done by deep learning frameworks. For example, pytorch has a dataloader based on pyav, so pyav itself would be an option. It also has its own ffmpeg bindings, that are supposed to be faster. Also opencv has ffmpeg bindings, and there are other simple libraries like lintel. And then there are more hardcore options, like nvidia DALI. None of these need to span new ffmpeg processes, but all of these are based on ffmpeg.

Just remember, the performance of reading video usually strongly depends on the order of frame reading - in many cases one needs to read the “keyframes” and frames that “sandwich” around a given frame. So full random access is not a great idea. Instead, many decoders are already parallelized and you should be able to leverage that if you really need maximum decoding speed.

2 Likes

Thanks for those tips @sdvillal I will poke around those libraries and do some benchmarking.

Note I’ve also decided to ask a more dask specific question over at https://github.com/dask/dask-image/issues/134

@sofroniewn In case you missed it this post sounds relevant!

thanks guys for a lot of information that i found here in regards with .mov files. i’m working on my project with such types of files quite a lot and i did found some info that i was looking for here.
may i ask you questions in case i would have any? thanks

Yes, ask away! I’m sure I, and others here would learn something from your questions too!!

1 Like

we load .mov files with openCV in deeplabcut guis/functions. This has been working well for us.

Also, to note at the hackathon this week we worked on this (in part) for integration of dlc into napari! Will make a proper post soon :slight_smile:

1 Like