Remote HPC cluster parallelization support in SciJava plugins

Hello everyone,

We are a small team based in Ostrava, Czech Republic and have recently joined this wonderful community within a project managed by @tomancak. In this forum post we would like to inform the community about our project milestones, which we have elaborated with guidance from @ctrueden. Any feedback from you is greatly appreciated!

The utmost goal is to enable end users to make use of parallelization on a remote infrastructure and develop a web interface so that all workflow tasks as well as data transfer to the remote infrastructure can be managed remotely, even from a mobile phone.

There are two strategic development directions which are to be worked on simultaneously, parallelization support and creating a Fiji web interface, aiming at combining them towards the end of the project (summer 2021).

Parallelization support

We have created a SciJava service, provisionally called ParallelService, representing an entry-point to the parallelization-providing functionality. This service provides upon request list of locally available parallelization paradigms, objects representing a unique parallelization approach and encapsulating both cluster access and task execution details. As the cluster-specific details are hidden in respective ParallelizationParadigm implementations, the plugins remain extensible and technology-agnostic. The general architecture is envisaged as follows:

Stakeholders

  1. A remote resource provider, say an HPC cluster center, supplies a specific implementation of the ParallelizationParadigm interface encapsulating cluster access and task execution details.
  2. FIJI plugin developers, wishing to leverage parallelization functionality in their plugins, retrieve a desired ParallelizationParadigm via ParallelService. Using their domain knowledge, they can delegate execution of plugin-specific tasks on particular computations units, such as HPC cluster nodes.
  3. Biologists, using a FIJI plugin utilizing the parallelization functionality, download a desired ParallelizationParadigm so that ParallelService can discover it. Provided that a biologist has access to the particular HPC infrastructure, the plugin will access it and executes its tasks there.

Milestones

  1. Gather desired use cases from the community and define ParallelService and ParallalizationParadigm functionality.
  2. Develop convenient data syncing between local and remote file system. For example, the user could be prompted to select local files/folders as a plugin input. In the next step, the data would be synced with the remote storage by calling parallelizationParadigm.syncData(myFiles).
  3. Create additional parallelization paradigms for specific clusters to validate the designed software architecture.
  4. Implement a method which would validate whether remote execution of a given task would actually pay off.
  5. Provide pre-defined parallelization-providing calls which users could utilize in their scripts and macros.
  6. Create a plugin enabling the end user to add/configure/choose parallelization paradigms.

An example of a cluster-specific ParallelizationParadigm implementation could be as follows:

More information on technologies used within this approach: 1, 2.

ImageJ Server and Fiji web interface

We are aiming to continue working on the ImageJ Server project. This RESTful server is intended to be a universal layer facilitating access to ImageJ functionalities from a non-Java environment. Specifically, ImageJ server can be very convenient for deployment on particular HPC cluster nodes, where it can wait to receive REST calls from a central node.

Moreover, the server can also be leveraged as a back-end for a client providing fundamental Fiji functionalities in a web browser. The client can comprise either a ported GUI (see projects Fun with JWt and Swing: WebGraphics2D, AjaxSwing, WebSwing) or a tailored web GUI. Particularly, we have started working on the latter approach and developed a basic Angular client. Although there might be some constrains, we firmly believe that we will enable users to enjoy the most popular Fiji plugins from a web browser and eventually to remotely manage parallelization of their workflows.

Milestones

  1. Gather desired use cases from the community.
  2. Implement direct and run-time access on cluster nodes from an end-user Fiji environment.
  3. Convert ImageJ Server from a wrapper into a pluggable ImageJ component.

Resources

ParallelService code snippets can be found https://github.com/PetrBainar/scijava-parallel (under development, caution needed!).

7 Likes