Running distributed cell profiler in non-default AWS regions

cellprofiler

#1

Hi all!,

I am currently setting up distributed cell profiler on an AWS account for the first time. I initiated all my necessary resources in US-East-2, including the project’s S3, CTRL node EC2 and ECS. I followed the docs in the distributed Cell-Profiler GitHub repo to set everything up. After submitting my first job, I went through the documentation and noticed Step 3 that the current Image-IDs and Snapshot-IDs available for fleet configuration are provided in East-1 and West-2 only.

When starting my fleet with an East-1 specific Snapshot-ID (and all other credentials being specific for my East-2 configuration) I receive - not too surprisingly - an error:

botocore.exceptions.ClientError: An error occurred (InvalidSpotFleetRequestConfig) when calling the RequestSpotFleet operation: Parameter: SpotFleetRequestConfig.IamFleetRole is invalid

I assume that the differences in AWS regions are the root cause for this error, as I do not have that IamFleetRole etc. configured for East-1.

Do you know any workarounds for this problem - for example - creating a US East-2 specific DCP Snapshot?


#2

My current solution:
I located the EBS-snapshots and AMIs and start transferring them to US-East-2.

FYI:

  • The current US-East-1 AMI can be found in US-East-2 under: ami-0d2984e1cc1a716d7
  • And I found a Snapshot in US-East-1 that was transferred before: snap-008391595528156c4, I’m going to try this one out.

EDIT: snap-008391595528156c4 did not work properly - we ended up starting the ami-0d… manually and using the snap-id of that instance for cluster-submission.