A pixel’s color value is the integration over a set of light rays. These lights rays may vary in color if they originate from different objects or surfaces that vary in albedo. To achieve photorealism, image manipulations such as contrast and gamma adjustment, histogram equalization or color replacement should ideally be applied to each light ray instead of the aggregate of the rays. In this paper, we attempt to separate the set of colors contributing to a pixel. We assume the set can be approximated using a finite set of weighted colors. Experimental results shown that two colors are sufficient. By applying image manipulations to each of the two colors independently, improved results can be obtained. We demonstrate our approach on a variety of image manipulations and compare both quantitatively and qualitatively against ground truth and baseline methods, as well as via human studies.