I'm working on instance segmentation task for person class. My model works fine, it gives reasonable results and masks look fine, but when I color the image by the masks of every instance, overlapping areas are emerging. Here is what I mean.
You have multiple instances and multiple masks. Each mask contains predictions for each pixel, it means for some pixels you will have multiple predictions across different masks.
You just need to take the maximum prediction for every pixel, in other words, argmax. It means each pixel will be colored by an instance mask, which has the maximum prediction, the maximum probability for it.
You just colored the entire image mask by mask, without excluding other predictions for the same pixel.