Request for comment: road to scikit-image 1.0

rfc
scikit-image

#1

Hi everyone,

Back in July, I wrote a blog post proposing a roadmap for scikit-image [1]. Then I mostly ignored it for a few months. =P I’m hoping to come up with a final roadmap mid-november, incorporating as many comments as possible from the community. So I would like to give an opportunity to a wider community to comment, as currently most comments have come from scikit-image core developers.

We’ve got four mechanisms for comments:

  • comment on the relevant scikit-image GitHub issue [2]
  • comment on this forum topic
  • leave a comment on the blog post [1]
  • make an anonymous comment on pollev.com [3]

Thanks for listening!

…[1] https://ilovesymposia.com/2018/07/13/the-road-to-scikit-image-1-0/
…[2] https://github.com/scikit-image/scikit-image/issues/3263
…[3] https://pollev.com/juannunezigl611


#2

Hi @jni, sorry for the delay in reply! I had been intending to offer more detailed comments “when I have time” (ha). Since I have some minutes now, I’m just going to write what I can, and I hope it is helpful and/or thought-provoking somehow.

I really like the structure of the document, built around Vision, Mission and Values. It is laudable that you are investing the effort to take a big-picture view of scikit-image to direct it wisely and effectively.

insufficient transparency from our functions for new users to predict what will happen to their data. For example, we are quite liberal with how we deal with input data: it gets rescaled whenever we need to change the type, for example from unsigned 8-bit integers (uint8) to floating point.

We don’t do magic. … We prefer to educate our users over making decisions on their behalf (through quality documentation, particularly in docstrings).

This is a tough one. Certainly it is awesome to strive for software that tends toward educating users as they use it. But people want convenience, and convenience breeds “magic” and a corresponding lack of understanding of internals. The ImageJ approach is to structure API in layers, with the lower layers giving power and control, and the higher layers giving convenience but necessarily concealing what is happening deeper down. Users can dig into the source code of the convenience layers to see how the more powerful layers are being used to achieve it, but tools like IDEs are needed to do this effectively. I don’t have a better answer.

We use NumPy arrays instead of fancy façade objects that mask their complexity.

I myself used to fall heavily on the pro-magic side, but hard experience with this library has shown me the error of my ways.

I would love to hear some examples where you feel being too “magical” caused problems. It is of particular interest to me from an ImageJ perspective because “fancy façade objects” is arguably a closely related concept to “interface-driven”, and being interface-driven is one of SciJava’s design goals (associated core value: extensibility). So any pitfalls and lessons learned here would be really valuable to hear about from the skimage perspective.

we require excellent documentation , in the form of usage examples, docstrings documenting the API of each function, and comments explaining tricky parts of the code.

I think there is a deeper and more general core value here that is driving the desire for great docs, as well as the “don’t do magic” and “value elegant implementations” points. Educate and empower users? Make the library more accessible? More “scientific” and open? More contribution-friendly? (And for what it’s worth: SciJava shares those sorts of values).

This requirement has stalled a few PRs in the past

If the values are reframed along the above lines, then the fact that PRs are being stalled by documentation requirements is more clearly in opposition to the deeper values. Or to put another way: the core values should lead to a good balance between enabling community contribution in the short term (e.g. make it easier to get PRs merged) and community contribution in the long term (e.g. stronger docs across the whole project).

until now we have completely abdicated responsibility in them and simply ignore any metadata. I don’t think this is tenable for a scientific imaging library. We don’t have a good answer for how we will do it, but I consider this a must-solve before we can call ourselves 1.0.

Speaking from experience, image metadata is an incredibly complex problem. My advice to scikit-image would be to push for 1.0 without metadata support. I believe any metadata feature you add can be done without breaking existing APIs. There is a substantial set of use cases that do not require any fancy metadata beyond structural basics already present in a numpy array. Even if new API is added that accepts additional arguments relating to metadata, there can be sensible defaults such that the older metadata-free API still makes sense. For example, some algorithms care about spatial calibration (are samples isotropic?). But default spatial calibration of [1, 1, ..., 1] can be used unless the caller specifies otherwise.

At some point, it might be great to have a more extended discussion here on the forum about image metadata. I could describe the ImageJ project’s take on it, current plans and future directions, etc.

Typing support. I never want to move from simple arrays as our base data type, but I want a way to systematically distinguish between plain images, label images, coordinate lists, and other types, in a way that is accessible to automatic tools.

Coming from Java-land, I :+1: :100: :+1: support this. :smile: It will be vital for better ImageJ <-> scikit-image integration. We can put appropriate conversions into the imglyb and/or imagej modules as data types emerge on the Python side.

Thanks again @jni for posting this. I’m excited to continue integrating ImageJ into Python better so our projects can play together more. If you didn’t see already: the building blocks for the imagej python module are now all on PyPI and conda-forge: jgo, scyjava, imglyb. During the Jupyter notebook session at I2K I’ll show how to use ImageJ with not only the BeakerX Groovy kernel, but also the standard Python 3 kernel. Thanks to @hanslovsky and @hadim for pushing on this with me. :sunny:


#3

Hi @jni @ctrueden there is not much that I can add but here are my thoughts on

We are quite liberal with how we deal with input data: It gets rescaled whenever we need to change the type, for example from unsigned 8-bit integers (uint8) to floating point. Although we have good technical reasons for doing this, and rather extensive documentation about it, these conversions are the source of much user confusion. We are aiming to improve this in issue 3009. Likewise, we don’t handle image metadata at all.

As a developer I do prefer to have total control about what happens with the data. At the same time it is very useful to use reasonable defaults and meta data whenever available. From my experience in the imglib2 environment, I have really learned to appreciate to have a somewhat two-layer approach to image processing libraries:

  1. The low-level layer that knows only about pixels and is pure image processing:
  • Take input data as is, no modifications/transformations. It is callers responsibility to ensure the inputs are compatible with the algorithm.
  • Expose all meaningful parameters with reasonable defaults, e.g. if you care to respect image calibration for voxel resolution, add a parameter resolution = np.ones((n_dims,)) to your method.
  1. A higher-level layer that accepts higher-level objects that not only include pixel-values but also meta data.
  • Delegate to lower-level methods.
  • Ensure that inputs are in the right format, do some pre-processing if necessary.
  • Extract meta-data from inputs and pass them to lower-level methods appropriately.

We use NumPy arrays instead of fancy façade objects that mask their complexity.

Fancy façade objects can be extremely useful, especially when working with views (values are evaluated any time a specific voxel is requested, avoid copies). As Python does allow for duck-typing, this is probably more relevant for Java, anyways. Also, as @jni made me aware of not to long ago, the proposal of the numpy __array_function__ protocal (NEP 18) will make numpy arrays a little more like fancy façade objects. So hopefully, you will get the fancy façade objects for free without changing any of your interfaces.


#4

@ctrueden and @hanslovsky thanks for replying! Just in time as I’m definitely thinking of putting up the proposal for final adoption basically in the next week.

The ImageJ approach is to structure API in layers

Yes, certainly we will look at layers and nice defaults. However I’ll add that at scikit-image we do have the luxury of, in general, a more technical user base, since they have already picked up some Python by the time they get to us.

This is not what I meant by magic, though:

I would love to hear some examples where you feel being too “magical” caused problems.

This is not about good defaults in general, but about “smart” defaults, ie default values that vary with the data. The biggest thing is our automatic rescaling with img_as_float (converts anything to float, rescaling uint8s by dividing by 255, uint16s by dividing by 65535, etc.). A very large portion of our “bug reports” stems from this.

Another “smart” thing we used to do (and probably still do in some corners) is to guess whether the last dimension is channels or a spatial dimension. So if your image has shape (25, 25, 3), we assumed that last dim is RGB. Needless to say we heard from users with 3-layers-thick images, and this is increasingly common since people are e.g. looking at tensors with skimage functions.

We also have (currently!) gray2rgb be a no-op if the last dimension is length-3, but only for 3-dim arrays. This results in all manner of inconsistencies, e.g. when processing a collection of images instead of a single image.

For SLIC we convert to LAB colorspace automatically because that’s what the original paper did. However, for many scientific images, you are actually clustering not in perceptual space but in the space of your channels, so this is precisely the wrong thing to do by default for many of our users’ applications. (I was bitten by this one myself.)

… and so on. Those were just off the top of my head.

I think there is a deeper and more general core value here that is driving the desire for great docs

Yes! All those things. =)

My advice to scikit-image would be to push for 1.0 without metadata support. I believe any metadata feature you add can be done without breaking existing APIs.

I believe you are right here, and you are certainly not the first to give us that advice. Since we are Python3-only, we can really mess around with keyword-only arguments and this makes updating APIs much easier. Duly noted!

I also agree about having a discussion. For reference here are the discussions for imageio and scikit-image:


I’m excited to continue integrating ImageJ into Python better so our projects can play together more.

I agree 100% and we should organise a sprint for this next year. =)

I’ll show how to use ImageJ with not only the BeakerX Groovy kernel, but also the standard Python 3 kernel.

Awesome! Will I2K talks be available?

@hanslovsky:

From my experience in the imglib2 environment, I have really learned to appreciate to have a somewhat two-layer approach to image processing libraries

Absolutely, and scikit-image currently fills the first layer’s niche almost precisely as you describe. The question is whether scikit-image should aim to also fill in the second level. I think we won’t before 1.0.

As Python does allow for duck-typing, this is probably more relevant for Java, anyways.

I think so. There is actually a history of ndarray subclasses, including in scikit-image, which used to define an Image subclass. To be honest, I don’t understand deeply the issues that arose with this, but there were many (things like sub-indexing an image resulting in a single-element image, as opposed to a scalar with a NumPy array), and these things seemed to happen any time that NumPy array were subclassed — which is why the NumPy matrix subclass is slowly being deprecated, for example.

As you say, let’s hope that NEP18 can deal with most of the complexity here behind the scenes. Certainly, I very much hope that in the future, scikit-image will transparently allow dask arrays in place of NumPy arrays.

Thanks again both for chiming in! Will take your comments on board in the final proposal. Coming soon to a scikit-image homepage near you!


#5

:metal:

Certainly, the “ImageJ and Friends” talk by @tpietzsch and @dietzc and I will be public and linked from https://imagej.net/Presentations after it is ready. And same for my workshop materials.

@Christian_Tischer @fjug Are there plans in general to archive all the conference content?


#6

With @tischer we talked today about recording the workshops. That might be failing due to missing recording hardware… not sure.

Slides we can easily be collected and offered in an central place.

Does this answer your question @ctrueden?


#7

Yes, thank you! Please feel welcome to use https://github.com/imagej/conference for permanent storage of slides, etc., if it is useful.