With linear unmixing, in a perfect world (hah), and assuming you had a sample with no dye, your unmixed (math above) calculation would result in an almost blank channel 2. Then, any further channel 2 signal would be only from the dye.
It sounds like, though, the problem is additional signal in channel 1… which is more problematic. You now have three variables and only two equations, which doesn’t seem solvable. If you could find a channel where you knew your protein signal was pure (even if not optimal), you might be able to use that. Otherwise… I dunno Sounds very math unlikely.
Pixel in channel 1 = protein+dye1
pixel in channel 2 = dye 2
If the dye1 to dye2 ratio (their relative emission strength in each channel) isn’t fixed, you cannot create an equation to solve for your variables, I think.
If you can get a protein only channel, then you can solve for at least dye1, with:
channel 3 = protein.
channel 1 = protein +( dye1=0) [in a sample with no dye]
Once you have that ratio, you can add the dye and extract the intensity of dye1… though at that point you may as well only take channel 2 and channel 3!
That said, I am basing this on the assumption that the emission change for your dye is in relative channel intensity, not just emission strength. IE the emission peak is shifting. If it just gets brighter or dimmer based on environment (but the ratio of a pixel in channel1 to channel 2 will always be the same), you should be able to eliminate the signal from the dye in Ch1, and ignore what I said above. Sorry for the somewhat meandering post.