We are a small team based in Ostrava, Czech Republic and have recently joined this wonderful community within a project managed by @tomancak. In this forum post we would like to inform the community about our project milestones, which we have elaborated with guidance from @ctrueden. Any feedback from you is greatly appreciated!
The utmost goal is to enable end users to make use of parallelization on a remote infrastructure and develop a web interface so that all workflow tasks as well as data transfer to the remote infrastructure can be managed remotely, even from a mobile phone.
There are two strategic development directions which are to be worked on simultaneously, parallelization support and creating a Fiji web interface, aiming at combining them towards the end of the project (summer 2021).
We have created a SciJava service, provisionally called
ParallelService, representing an entry-point to the parallelization-providing functionality. This service provides upon request list of locally available parallelization paradigms, objects representing a unique parallelization approach and encapsulating both cluster access and task execution details. As the cluster-specific details are hidden in respective
ParallelizationParadigm implementations, the plugins remain extensible and technology-agnostic. The general architecture is envisaged as follows:
- A remote resource provider, say an HPC cluster center, supplies a specific implementation of the
ParallelizationParadigminterface encapsulating cluster access and task execution details.
- FIJI plugin developers, wishing to leverage parallelization functionality in their plugins, retrieve a desired
ParallelService. Using their domain knowledge, they can delegate execution of plugin-specific tasks on particular computations units, such as HPC cluster nodes.
- Biologists, using a FIJI plugin utilizing the parallelization functionality, download a desired
ParallelServicecan discover it. Provided that a biologist has access to the particular HPC infrastructure, the plugin will access it and executes its tasks there.
- Gather desired use cases from the community and define
- Develop convenient data syncing between local and remote file system. For example, the user could be prompted to select local files/folders as a plugin input. In the next step, the data would be synced with the remote storage by calling
- Create additional parallelization paradigms for specific clusters to validate the designed software architecture.
- Implement a method which would validate whether remote execution of a given task would actually pay off.
- Provide pre-defined parallelization-providing calls which users could utilize in their scripts and macros.
- Create a plugin enabling the end user to add/configure/choose parallelization paradigms.
An example of a cluster-specific
ParallelizationParadigm implementation could be as follows:
ImageJ Server and Fiji web interface
We are aiming to continue working on the ImageJ Server project. This RESTful server is intended to be a universal layer facilitating access to ImageJ functionalities from a non-Java environment. Specifically, ImageJ server can be very convenient for deployment on particular HPC cluster nodes, where it can wait to receive REST calls from a central node.
Moreover, the server can also be leveraged as a back-end for a client providing fundamental Fiji functionalities in a web browser. The client can comprise either a ported GUI (see projects Fun with JWt and Swing: WebGraphics2D, AjaxSwing, WebSwing) or a tailored web GUI. Particularly, we have started working on the latter approach and developed a basic Angular client. Although there might be some constrains, we firmly believe that we will enable users to enjoy the most popular Fiji plugins from a web browser and eventually to remotely manage parallelization of their workflows.
- Gather desired use cases from the community.
- Implement direct and run-time access on cluster nodes from an end-user Fiji environment.
- Convert ImageJ Server from a wrapper into a pluggable ImageJ component.
ParallelService code snippets can be found https://github.com/PetrBainar/scijava-parallel (under development, caution needed!).