Applying Mask with Image calculator creates artifacts


I am trying to apply a binary mask on an image with two channels using the AND operator in image calculator and this creates artifacts. My goal was to filter out everything outside the mask region. If there is a better way, please let me know. Thank you.

Here are two channels of my original image separated:

Here are the two channels of my AND result image separated:

Please check whether doing an AND operation between a 16 bit and and an 8 bit image is the right way to do this.