I think your images are likely just barely over the size limit. Arrays in Java are addressed with a Java int which has a maximum value of two billion. You have images that are about 1/2 billion pixels and I think we need four bytes per pixel to store RGB. That pushes you just barely over the limit. It’s an inherent limitation in Java that can’t be changed by increasing the heap size.
The upcoming release won’t alleviate this, but we are planning features for the release after that one. CellProfiler 2.2 will likely have a tiling feature that will work well with your images by analyzing them in tiles. We can use Bio-Formats to write the images in tiles as well - this will allow us to write color images with your format. The standard TIF file format can only support 2GB image sizes, but BioFormats can write the BigTIFF format and overcome this limit. Maximum JPEG image dimensions are 64K x 64K. The PNG specification supports up to 2G x 2G, theoretically.
It might be wisest to use a strategy that breaks your images into tiles and then processes each tile separately for now - I’m sure you will start running into more pitfalls with these large images and, even with enough memory to run without disk thrashing, I would bet that CP will run the tiled set faster than the single image.