I’d like to share my tool chunkflow here. It has been used to process petabyte-scale image volumes using Cloud Computing and Deep Learning.
I has a few features:
- Composable operators. The chunk operators could be composed in commandline for flexible usage.
- Hybrid Cloud Distributed computation in both local and cloud computers. The task scheduling frontend and computationally heavy backend are decoupled using AWS Simple Queue Service. The computational heavy backend could be any computer with internet connection and Amazon Web Services (AWS) authentication.
- Petabyte scale. We have used chunkflow to output over eighteen petabyte images and scaled up to 3600 nodes with NVIDIA GPUs across three regions in Google Cloud, and chunkflow is still reliable.
- Operators work with 3D image volumes.
- You can plugin your own code as an operator.