High Dynamic Range Image Processing

Peter Catrysse and Robert Batchko

Final Project Psych 221

Stanford University, 3/13/98

Click here to start the Presentation

Motivation

Driven by the continuous advances in CMOS technology for processors and DRAM's, CMOS sensor arrays are becoming a more viable alternative to CCD sensor technology every day. Research on CMOS vision chip design with A/D conversion at the pixel level, presentated at the past SPIE Photonics West Conference by El Gamal et al. is leading us to unexplored terrain. The availability of such a device is one step along the way to bring us to the programmable digital camera-on-a-chip. But before this camera sees the daylight, a number of issues need to be researched. (Figure 1)

It is expected at this point that A/D conversion at the pixel level should give advantages when dealing with images taken from scenes with high dynamic range. Those scenes tend to be a problem for CCD based digital cameras (even for traditional film based photography) and lead to totally unacceptable images.In a first part of the project, we explore this particular topic from a device independent point of view focussing on algorithms and techniques. We started by writing our own decomposition tools and used to to try out a few ideas, but they should allow more rigorous work in the future.
Another issue that needs to be dealt with is the expected excess noise in CMOS sensors over CCD sensors. We made a first pass at evaluating the impact of CMOS noise on the image quality using the CIELAB L* metric. This part of the project should allow us to suggest A/D conversion schemes and the appropriate use of quantizers. (Figure 2)

Part I: High Dynamic Range Image Processing Tools.

Overview.

The goal of this portion of the project was to utilize spatial- and intensity-filtering techniques in order to enhance the viewability of information in a high dynamic range image. (Figure 3)

This work focused on implementing two techniques for signal processing of the high dynamic range image; bit plane decomposition and blurring. However, other techniques such as pyramid decomposition and spatial and spectral averaging were briefly explored and merit further study.
(Figure 4)

Step 1: Original source image.

An original (black and white) image was acquired. The image resolution was 256 x 256 pixel with 12 bit grey scale.
(Figure 5)

Step 2: Bit Plane Decomposition.

Using binary intesity decompositon, the original image is broken up into 12 bit planes. Each bit plane consists of the (single bit) information corresponding to a respective bit level from the 12 bits of intesity. Notice that in bit planes 7 and 8, image information is revealed which was not viewable in the original image.
(Figure 6)

Unweighted Reconstructed Image.

Without altering the values of (ie., weighting) the bit planes relative to eachother, a reconstructred image is built by recombining the bit planes. This unweighted reconstructed image is a duplicate of the original image.
(Figure 7)

Weighted Reconstructed Image.

By assigning an individual weighting coeficient to each bit plane, a weighted reconstructed image is created. This image reveals information not previously viewable in the original image. However, some "noise" has also been amplified thus degrading other parts of the original image.
(Figure 8)

Step 3: Blurring.

The third step of this work involved the use of blurring (ie., low-pass filtering) on the individual bit planes. Twelve separate Gaussian blurring kernels were constructed. Each kernel was convolved with a respective bit plane, thus performing a specific blurring function on each bit plane.
(Figure 9)

Bit Plane Blurring.

This slide shows the twelve (unweighted) bit planes with blurring. Notice that the lowest order bit planes (ie., bit planes 1-4) were given the greatest blurring while the hightest order order bit planes (ie., bit planes 9-12) were blurred the least.
(Figure 10)

Step 4: Resulting Image.

Last, the weighting factors are applied to each blurred bit plane. The resulting image now contains more viewable information than in the original image, with a small degradation of some image content due to amplified noise. Compare this resulting image (Figure 12) to the original image (Figure 11)

Conclusions for part I of the project.

Binary intensity decomposition combined with blurring and weighting of the bit planes is useful in displaying the full signal extent of high dynamic range images.
However, while these methods can improve certain areas of the image, other areas may be degraded. Spatial decomposition in combination with intensity decomposition has potential to overcome this limitation. (Figure 13)

Robert G. Batchko, March 1998

Part II: Noise Analysis and Reduction Techniques.

Overview.

The second part of the project consists of a noise analysis and modeling for CMOS sensors including shot noise, fixed pattern noise and quantization noise. It will in the future also include readout noise. We perform the following steps: first we create synthetic digital images; then we convert those images back to their analog representation using a model of the capturing device and the A/D converter used in the process; next we add noise to the image and finally we compare the re-digitized image with the original using the CIELAB metric (Figure 14). This processing pipeline can be reformulated in the shape of a project graph showing the different smaller parts of the task (Figure 15 and Figure 16).

Step 1: Synthetic Digital Images.

We developed a small set of synthetic images, because we soon realized that operating on real images of any decent size was going to be painful during the development phase of the MATLAB tools. The synthetic images also give us more flexibility in isolating particular image characteristics. At this point our library consists of a checkerboard pattern and a couple of grayscale ramps. This enabled us to experiment and try out the tools we wrote for this project and as need arises we will expand this library. (Figure 17)

Step 2: Digital-to-Analog Conversion.

In order to model the noise associated with the analog image as it is captured by the sensor array, we perform an digital-to-analog conversion based on two different quantizers: a linear or uniform one and a nonlinear (square root) one. Their characteristics are given in Figure 18.
The choice of quantizer is a definite choice that is available having the A/D conversion at the pixel level and we should definitely explore the options we have. The uniform quantizer was chosen as a reference and the nonlinear one was chosen because it allows us at least 8 bits of digital data for the given set of sensor parameters (Figure 19). Also it was chosen with the notion in the back of the head that shot noise has a square root type of behavior and this quantizer might be suitable to deal well with shot noise dominated images.

Step 3: Noise Modeling for the Analog Image.

We show the models for both the shot noise and the fixed pattern noise (Figure 20 and Figure 21). At this point we only investigate the impact of shot noise on image quality.

Step 4: Analog-to-Digital Conversion.

Of course we use the same quantizer here than the one used for digital-to-analog conversion, it wouyld not make sense to do otherwise

Step 5: CIELAB L* Vision Metric.

The L* value which is part of the CIELAB metric gives us the luminance that would be obtained from the rgb values of a pixel within an image. This is a visual metric for the way pixels would look similar to the eye. We use this value to compare our original synthetic digital image to the digitsal image that has been obtained after noise degradation.
We perform the comparison for both quantizers and notice that if the image is shot noise dominated that the nonlinear square root quantizer outperforms the linear quantizer. (Figure 22)

Step 6: Blurring of the Digital Images.

Blurring of both the original synthetic image and the noisy digital image allow us to see what the impact is of the limited resolution of the human eye as a function of the viewing distance but also allows us to explore if blurring improves image quality when an image has been degraded with noise.
Clearly, blurring does not improve the L* matching on a pixel to pixel basis between the both images as can be readily seen from Figure 23.

Conclusions for part I of the project.

Well Capacity determines maximum number of bits available after performing A/D conversion. If the only noise source is shot noise, then the nonlinear quantizer outperforms the linear quantizer. At the maximum allowable number of bit levels the nonlinear quantizer produces almost perfectly matched L* values. Blurring of both the original and the noisy digital image has a deteriorating effect on L* matching. (Figure 24)

Peter Catrysse, March 1998