Yingmei Lavin Alice Liu
In this project, we investigate the methods of automatic color balancing of an image taken by a digital camera made by a local startup company Silicon Vision. The camera's CCD array has 512 X 480 pixels, and its color mosaic consists of cyan, yellow, magenta, and green. Under normal operations, controls such as exposure, saturation, gamma value can be externally controlled, and color balancing is accomplished by choosing the lighting condition to be home, office, or natural. Since the objective of the project is to automatically color balance images, raw pixel values of the CCD were obtained. The R, G, and B values of each pixel were extracted from linear combinations of the cyan, yellow, and magenta values. This step was performed assuming a linear response of the raw pixel values with incident light intensity. After the reconstruction of the R, G, and B channels from the raw pixels, we checked for the proper exposure and filtered out noise. An algorithm was used to determine the proper scaling of the R, G, and B values for automatic color balancing. The balanced image undergone the above processes still appears to be not completely white balanced. The reason was found to be the very nonlinear response between the raw pixel values and the incident light intensity.
Task: Obtain raw pixel values directly from the CCD array.
Difficulties:
Solutions:
The color table below is the color filter layout in front of the CCD array:
figure 1.
In order to view the color image on the computer display, we need to recover the r, g, b matricies. The raw image from the camera represents the gray level for each pixel. So to get the red, green and blue values for each pixel, we need to use the cyan, yellow and magenta values from the neighboring pixels. The formulas for convertion from Cy, Ye and Mg to R, G and B are given below:
R = Ye + Mg - Cy
G = Ye + Cy - Mg
B = Cy + Mg - Ye
Problem:
This method will cause problems when the intensity signals from CCD array
are not really linear.
Each pixel in the raw image contains the calculated RGB values without gamma correction. Before any color balancing can be done, it is necessary to check if the image is underexposed or overexposed, and to reduce false peak values introduced by noise during the reconstruction of the raw RGB image. We make sure that the image contains less than 1% of saturated pixel values to prevent oversaturation, and that it contains less than 90% of the pixel values below 128 to prevent undersaturation. Due to the calculations of the raw RGB values, some noise peaks are introduced into the image. To reduce the effect of the noise peaks, we compare each pixel value with the mean value of its nearest 64 pixels. If the peak pixel value in the 64 pixel window is much greater than the mean, it is recognized as a noise pixel and the value is replaced with the mean value. Automatic color balancing is accomplished using an algorithm developed in US Patent #5504524, as shown in figure 2. We construct an 8X8 pixel window and convolve with the entire image, so that peak average values within the 64 pixel window and the window locations can be found. The peak average green value is denoted G(p), global peak average red value R(pw), and global peak average blue value B(pw). Since there is frequently a relatively high level of green component occurring in scenes under more common light sources such as natural, tungsten, and fluorescent lights, we choose to search within the peak average green window to find the peak red value R(p) and peak blue value B(p). We then compare the red and blue components with the green component. The objects inside the peak average green window can be either white, mostly green, mostly yellow, or mostly cyan. Inside the peak average green window, if the peak red value R(p) is higher than G(p), then it is decided that there is an excess of red, and that the image is imbalanced. The red component is then scaled down by a factor determined by the ratio of G(p)/R(p). If the peak red value R(p) is less than G(p), then the global peak average red value R(pw) is compared with the G(p) value, since the peak red value inside the peak average green window may not accurately represent the real peak value in the image. The lower red peak value inside the peak green window can be caused by either color imbalance or a shortage of red in the scene (for example, a cyan panel). Assuming the objects in the the entire image are diverse in terms of color reflection and emission, then if the global red peak R(pw) is also lower than G(p), then it is decided that the object within the peak green window is white, and the image is imbalanced. The red component is then scaled up according to a factor determined by the ratio of G(p)/R(wp). If the global red peak R(pw) is not lower than G(p), then it is decided that the object within the peak green window is mostly green or cyan, which have low red reflection. In this case, the image is judged to be balanced in red. The treatment of the blue component is exactly the same as that of the red component. In the case where the entire image is lacking substantial green, another more dominating color should be chosen, so that the green component and the third color component can be compared to this chosen color. After the image is properly balanced, gamma correction is added.
figure 2.
Cases of color failure:
The algorithm should work well under most situations with various lighting conditions,
but like all color balancing algorithms, it is not failure proof. In our case, color failure
occurs when the image consists of mostly objects that are monochromatic. For example,
an image consisting of only a completely white object under green lighting is mistaken to
be balanced, because the algorithm assumes that the object is green. This is a difficult
case to treat, because under similar circumstances even the human visual system can be
tricked. Another possibility for color failure is mixed lighting conditions in the same
image. This problem can be potentially avoided if color balancing is performed
separately in different subsections of the image. However, the time required for this
process may be longer.
Source codes of the image extraction, color matrix reconstruction and
color balancing are listed as following:
A list of c++ files:
gray pixel values
red image green image blue image reconstructed color image color balanced image
Discussions
As shown in the calculated R, G, and B images, there are some noise
pixels having abnormally high values. These noise pixels were
introduced by overflowing and underflowing problems during the
calculations of the RGB values. By convolving the R, G and B images
with a 8X8 filter first, then looking for the peak values in the
smoothed images, we eliminate the accidental high values caused by noises.
The color
balanced image shown above still has too much red in the backgound.
The color balancing algorithm has found the peak average green area to
be in the bright part of the white sweater with G(p) > R(p) and G(p) >
B(p). In addition, G(p) > R(pw) and G(p) < B(pw). According to the
algorithm, the following decisions were made: red is imbalanced, and
is increased according to the ratio of G(p)/R(pw); blue is unchanged.
However, since we know that the peak green average area is white, it
is not physically possible to have blue peak levels higher than the
green peak level! So we scaled the blue component to make a white
surface. The entire image was then scaled to bring up the brightness.
However, the resulting image still does not look right! Because of
the impossible behavior of the blue peak levels compared with the
green peak level, we suspected that our raw pixel values were not
linear with respect to the incident light. If this were true, then
our method of reconstructing the RGB values from the raw data would
not be valid. We placed a neutral density filter in front of the
camera and compared the values at the same pixel locations before and
after the insertion of the filter. Below is an example: a(100:104,100:104)
ans = c=b(100:104,100:104)./a(100:104,100:104)
A list of header files:
A list of matlab function files and scripts:
Results:
Discussion:
b(100:104,100:104)
ans =
33 33 37 37 33
33 37 37 29 29
33 33 33 29 29
37 33 33 29 29
37 37 33 33 37
25 26 33 33 29
29 25 29 36 35
26 26 34 37 35
29 29 33 33 32
30 34 38 34 33
c =
1.3200 1.2692 1.1212 1.1212 1.1379
1.1379 1.4800 1.2759 0.8056 0.8286
1.2692 1.2692 0.9706 0.7838 0.8286
1.2759 1.1379 1.0000 0.8788 0.9062
1.2333 1.0882 0.8684 0.9706 1.1212
where b is the image without the neutral density filter and a is the image with the neutral density filter. The ratio of b/a is displayed in c. The filter should cut off 3 times the light intensity, and the ratios of pixels in b and a should be around 3, but c clearly shows this not to be the case! Therefore, we have concluded that the camera response is nonlinear. Because of this, the calculated RGB raw values are no longer valid, and the subsequent color balancing cannot work properly to correct this problem.
Last modified: Thu Mar 12 17:00:17 PST 1998