Image Scaling

Abstract Introduction Scaling Up      Padding      Convolution      Box Filter       Bartlett Filter Scaling Down Scaling Color Images Discussion References Appendix

Abstract

Introduction

although pixel distances are not preserved, the expansion or contraction ratio is the same everywhere; and

if the scaling ratio is an integer, the pixel lattice of the target image is a subset or superset of the lattice of the original image.

For the case of scaling up (image enlargement), the methods used involve first resizing the image by padding with zeros, followed by convolution with a two-dimensional, spatially invariant linear filter. The filter performs a smoothing function, which dampens out high frequencies in the reconstructed image. Since we are dealing with matrices that represent discrete pixel values, we use the filter in the discrete domain. Hence each filter can be represented by the impulse function h, which is also called the mask of the filter. We consider two common filters, the Box filter and the Bartlett filter, in the enlargement of both greyscale and color images.

Scaling down (image reduction) increases the incidence of high frequencies and causes several pixels to collapse into one. Hence we need to apply a smoothing filter in order to minimize aliasing problems in the target image. Again we consider the effects of the Box and Bartlett filters in image reduction. More effective smoothing filters can also be used in order to obtain a final image that is better in certain aspects. While such filters minimize the introduction of high frequencies during image reconstruction, the tradeoff is a loss of definition in the final image.

Finally, we compare our results with the scaling algorithms used in software such as Adobe Photoshop and discuss issues such as efficiency of the scaling algorithms as well as the possibility of taking into account human visual perception when scaling images. A list of the Matlab code that we wrote can be found in the Appendix.

Scaling Up

Padding

zero-padding for a 2 by 2 expansion leads to

If the orginal image had n rows and m columns, and we want to scale it up by a factor of x in the horizontal direction and y in the vertical direction, then the zero-padded image will have x*n+1 rows and y*n+1 columns.
 

Convolution

Box Filter - Scaling by Pixel Replication

If the original image had n rows, this leads to an image with 2n rows, looking like this:

Hence the rows of zeros have been replaced with a copy of the pixel values directly above them, and the columns of zeros took on the values of the pixels to their immediate left. Note that this convolution kernel has a total mass of four, but since the image had been enlarged four times by zero-padding, the average intensity of the convolution result remains equal to that of the original.

In general, to scale an image x times in the horizontal direction and y times in the vertical direction, we need a mask of size y rows and x columns with all the elements taking a value of 1. We implemented this algorithm in Matlab code and scaled up the image of Einstein in figure 1. The result is shown in figure 2.

Figure 1: Original image of Einstein

Figure 2: Einstein image scaled by factor of 2x2 using the box filter

Note the slight jaggedness of diagonal lines (eg. Einstein's shirt collar) which is an artifact of using the box filter. This is due to the fact that dampening of high frequencies depends on the directions these frequencies occur on the image, resulting in clearly defined sharp edges between certain groups of pixels. With each increase in scaling factor, this reconstruction defect becomes more perceptible, and the enlarged image appears coarser. Nonetheless, scaling by replication is very popular due to its ease of implementation and computational efficiency. It is commonly found in hardware implementations.
 

Bartlett Filter - Scaling by Bilinear Interpolation

The above matrix shows a Bartlett filter used to scale an image up by a factor of 2 in both the horizontal and vertical directions. To construct a Bartlett filter, we first construct a row vector r with 2*scale_factor_x-1 elements and a column vector c with 2*scale_factor_y-1 elements. For example, in the case of scaling up by 2 in both the horizontal and vertical direction, r and c each has 3 elements as shown in the above matrix. Elements in r and c count up from 1, 2, ..., to scale_factor, and then count down toward 1again. As shown in the above matrix, the elements in r and c are 1, 2, 1. After filling in r and c, we can construct the Bartlett matrix by taking the dot product of r and c. Note that we also multiply the dot product of r and c by 1/(scale_factor_x * scale_factor_y). This is done because the sum of all the elements in the resulting dot product is always (scale_factor_x * scale_factor_y) times greater than the desired intensity. For example, in the case of scaling up 2x in both the horizontal and vertical directions, since each pixel is mapped into 4 pixels in the resulting image, we want the intensity to be 4 times greater than that of the original image in order to maintain the same average intensity level. Thus, we want the mass of the Bartlett filter to be 4. However, the mass of the dot product of r and c is 16. This is why we need to multiply it by 1/4.

The effect of convolving a padded image with a Bartlett filter is to insert between each two consecutive rows of the original image a new row that is the average of the two, and likewise for the columns. In other words, we perform linear interpolation between rows and columns independently. Because of this linear interpolation done by using the Bartlett filter, the image appears to be smoother than the image obtained using a box filter. Figure 3 shows the Einstein image scaled up 2x2 using the Bartlett filter. Note that the white collar area is smoother than the same image scaled up using the box filter as shown in figure 2. However the main drawback of a Bartlett filter is the fuzziness introduced to the image. In television, where spatial and color resolution are relatively low, zoom-ins using the Bartlett filter
yield very satisfactory results.

Figure 3: Einstein image scaled by factor of 2x2 using the Bartlett filter

Scaling Down

We then pick 1 out of every 4 pixels to constitute the new image. For the case of the box filter, each 2 x 2 block of the original image
 

results in the pixel value 1/4(f(i, j)+f(i+1, j)+f(i, j+1)+f(i+1, j+1)) being used in the reduced image. Figure 4 shows the scaled down image using the box filter, and figure 5 shows the image as a result of using the Bartlett filter.
 

Figure 4: Scaled down image using box filter

Figure 5: Scaled down image using Bartlett filter

From our observations, both the box and Bartlett filters give comparable results. However, for increasing scale factors, the Bartlett filter may give better results because of more smoothing. For comparison, we performed scaling down on the same image without the convolution step. Each pixel is just selected from the first of each block of 4 pixels in the original image. The result is shown in figure 6. Note that the quality of the reconstructed image is visibly poorer that those obtained using the filters. This indicates the importance of the convolution step in image scaling.
 

Figure 6: Scaled down image without smoothing

Scaling Color Images

Figure 7: Double-sized color image using Box Filter

Figure 8: Double-sized color image using Bartlett Filter

The box filter gives a coarser image, while the Bartlett filter results in a slightly blurry image due to more smoothing. Ideally, we would want the result to be the average of the two, so that the image is sharp and clear. In the next section, we will compare our results with the images obtained by scaling using Adobe Photoshop, and discuss what we have learnt in this project.

Discussion

Figure 9: Einstein image enlarged by 2x2 using Adobe Photoshop

Figure 10: Clown image enlarged by 2x2 using Adobe Photoshop

We can tell that the Photoshop images are visibly less coarse than the images using the box filter, but definitely not as blurry as the results using the Bartlett filter. This implies that algorithms like that used in Photoshop probably apply some sort of smoothing technique that is in between the box and Bartlett filters in terms of degree of smoothing. Commercial software must also take into account the computational efficiency of the smoothing algorithms. We found that the Bartlett images take up less space than the box images. This is probably due to the fact that the box filter retains more high frequency elements in the image, which, after JPEG compression, take up more space than the compressed Bartlett image. Also, there is the possibility of taking into account human visual perception when scaling images. Since human contrast-sensitivity changes with spatial frequency, a possible image scaling algorithm would take into account the variation of spatial frequencies in the image in determining how to scale it. Also, in scaling color images, a possible algorithm could scale the image based on human sensitivity to the colors in the image. These are ideas for future research in the topic of image scaling.

References

Appendix