Managing personal/group update sites... let us know what you think!

As part of the imagej.net migration away from MediaWiki to GitHub Pages, we will need to develop a new approach to managing people’s personal and group update sites on sites.imagej.net. We are committed to maintaining this infrastructure, but are looking for ideas from developers in the community with experience in this area.

  • How would you like the process to work in future to create update sites?
  • … to create users?
  • … to manage credentials? Etc.

In particular, if anyone knows of a good lightweight off-the-shelf tool for managing WebDAV, we’d love to hear about it!

@ctrueden and the entire ImageJ team… :slight_smile:

5 Likes

It would be nice to keep – or even enforce – the separation between user accounts and update sites, i.e. creation of a new user doesn’t necessarily mean creation of an update site with the same name; and write permission for (any number of) users can be given to (any number of) update sites.

Would OAuth be an option? So users can authenticate with their GitHub, Twitter, or Image.sc forum accounts. And the admins maintain a map of update sites and which accounts have permission to upload…

2 Likes

Just thinking loudly: Would it be possible to decentralize the update sites? What if I have a webserver and can put files on it. How could I make ImageJ understand that this is a proper (potentially read-only) update site? E.g. my update site URL could be something similar to: https://github.com/clij/clij2/releases :wink:

1 Like

@haesleinhuepf What you suggest is already how it works. (Step 0: It already works and you don’t have to do anything! :stuck_out_tongue_winking_eye: ) If you look at the List of update sites, you’ll see a few that are not on sites.imagej.net but rather some other server. People are free to set up whatever servers they want, and point ImageJ at them. If it’s desired to have such a site on the “Manage update sites” dialog automatically, they can file a PR against the list-of-update-sites repository adding it.

The question of this thread is: what sort of features would people like to see for managing the hosted update sites served from sites.imagej.net? And does anyone have any suggestions for tools and technologies that might improve the experience for the community?

Hmm, definitely worth exploring! Would be awesome if people don’t need yet another password. I was exploring OAuth2 awhile back anyway for the rewritten ImageJ issue reporter; perhaps we could reuse the same token for both purposes!

1 Like

Awesome! And how does it work? How can I make ImageJ write the db.xml into a local folder instead of a webdav-drive?

There are Uploader plugins making it extensible. The imagej-updater includes a FileUploader for “uploading” to a file system directly. ImageJ also comes bundled with two more uploader plugins: one for WebDAV and one for SSH. Based on the source code of FileUploader it looks like you just put file:///path/to/directory or similar into the Host and/or Directory on Host?

1 Like

Oh, it supports ssh - that sounds pretty much like another solution. Thanks! :smiley:

1 Like

If we are to make an overhaul of update sites in Fiji, what about taking inspiration from Icy (I know, I am biased)?

  • One download (update site for us) = one plugin.
  • Obligatory description, icon, …
  • One person named is responsible.
  • Searchable in-app by name and description.
  • Install on fly after search.

4 Likes

yes this would be very useful… it’s the thing that has stopped us to use update sites.

1 Like

I’m interested in this. I have been maintaining two plugins on a plugin site:

https://sites.imagej.net/Odinsbane

They’re not listed on the list of update sites.

We’re also working on setting up some github pages with improved instructions on usage. I would like to get our plugins a bit more integrated if possible.

Cheers
mbs

1 Like

As @tinevez says, Icy’s update site approach is cleaner and more usable.

I think @ctrueden has some qualms about @axtimwalde’s github-based maven as-is. Nevertheless, I suggest:

  • 1 update site == 1 git repository [edited to generalize beyond github]
  • 1 central fiji repository continues to host a list of update sites that would be curated by pull requests to add individual update sites
  • add some requirements to update sites so Fiji can deliver a pretty update site search interface like Icy. update site requirements can be enforced by CI
3 Likes

For reference, this is our development maven repository that we modeled after Eclipse’s SWT repository. You will see that many of our initially experimental artifacts have now migrated to https://maven.scijava.org. I think that is a good strategy.

I like the idea to host update sites in remote git repos but would prefer to not limit this to GitHub, although almost everybody will of course use it at this time.

Pretty is overrated, ugly is a good filter to keep the kids away :wink: (Half joking :stuck_out_tongue:).

1 Like

Thanks for all the feedback! Much appreciated! :sunny:

While I agree that we want to move toward an update mechanism that is plugin/extension-centric, like how Icy does it… that is not what we’re asking in this thread. We just want to discuss technical mechanisms for managing the credentials for sites.imagej.net as it already works today. An overhaul of the Updater to work in a better way is definitely highly desirable, but not something we can achieve in the next few weeks. In the meantime, we are going to need a way for people to: 1) create/delete update sites; 2) create/delete users; 3) manage user credentials (i.e. set/change password); 4) control which users have upload permission to which update sites. While we could roll our own on all of that, we thought we’d ask first if anyone knows of foundational technologies that would make it easier—or has other requirements beyond (1,2,3,4) above.

2 Likes

All of these reasons sound like votes for piggybacking on git. If github/gitlab/etc is used then it provides (1,2,3,4).

I did not mean to support plugin/extension-centric update sites actually from Icy. However, the other features seem desirable.

The hesitation is about whether swapping from WebDAV (who actually wants to keep using WebDAV???) to git will be particularly time/resource consuming. There are at least 2 ways of doing this:
1 - write it from scratch
2 - find the minimal modification that can replace webdav usage with git

Let’s ignore (1) because it sounds hard (at least relative to the immediate payoff).

(2):
db.xml.gz is just a file with a listing and a little bit of metadata about each file. This is easily supplied from a git repository (even by a raw file fetch of sth like https://raw.githubusercontent.com/kephale/fakeupdatesite/master/db.xml.gz).
The files supplied by the update site would be stored on master or some other branch relative to the Fiji.app root directory.

I’m kind of convinced that download would be easy. What about upload? After skimming this JGit tutorial, it looks like adding files and committing to a repository is pretty straightforward as well. FilesUploader from imagej-updater is on the more complex side, but a good amount of that is overhead from dealing with WebDAV for locks and providing timestamping mechanism (which we get for free in git). Then digging a bit deeper it looks like a lot of the additional complexity in imagej-updater is about dealing with streams and pushing bytes; however, jgit would be taking over that logic.

My remaining concern would be about the conflict resolution logic. However, I think that does not even necessarily have to change because the minimal effort implementation might just reuse the current db.xml.gz system.

Additional benefits:

  • Eventually we would probably find that db.xml.gz could be eliminated in favor of using the git history.
  • Although we can already revert update sites using the db.xml.gz, reverting changes does become quite straightforward with git.
3 Likes

@kephale Thanks for the analysis! I know that @frauzufall has done a lot of research and explorative coding around augmenting/rewriting/replacing the existing Updater logic with using Maven repositories, and as part of that work she also explored versioning the history with git locally using JGit. @frauzufall Do you have thoughts or ideas along these lines?

As much as I love this idea, I still think it would be a one-month-minimum project for a full-time developer to transform the Updater in this way, test it, and roll it out. No one on the LOCI team (and I’m guessing not the CSBD team neither) has the bandwidth to do that this summer—we need to finish the server migration, complete the pom-scijava 29 final release, roll out the major Fiji update, and get PyImageJ 1.0.0 finally stabilized first. But we could potentially work on the Updater in these directions next, during the fall months.

That said: if we are planning to improve the Updater in this way, such that GitHub becomes responsible for most of the technical concerns, perhaps we can get away with continuing to manage existing update sites manually via forum requests until that work is complete, rather than implementing some interim solution.

Still, I have reservations: there are a substantial fraction of hybrid plugin developers who don’t have the bandwidth to absorb the increasingly complex best practices of technologies like GitHub and Maven. These folks rely on being able to “just make a JAR” and then upload it to their site painlessly via the ImageJ Updater. So we’d have to ensure that for people with that level of comfort technically, they can still have (close to) the same workflow, making it as easy as possible to authenticate with GitHub one time and have ImageJ remember and just take care of everything else for them using the existing UI-based workflow.

9 Likes