Unsuccessful Methods

For a number of reasons, the first few methods implemented did not result in reasonable detectors. As the purpose of this site is to outline the work completed for the project, we nonetheless describe these attempts and their respective issues below.

Hough Transform

Though this concept is actually used in our final implementation, we experimented extensively with using this as the first stage in our detection pipeline. In particular, we first performed edge detection on the image, and then searched the image for rectangles of an aspect ratio that approximated a license plate. We used Canny, Sobel, and Prewitt edge detectors but still noted that in a number of our images, the license plate border was not well-defined. This was sometimes a result of glare, poor exposure, license plate holders, dirt, etc. Consequently, were we to set parameters such that a solely Hough Transform based method would detect a reasonable license plate region, the detector became so sensitive that a vast number of candidate regions were identified in the image and the method quickly became far too computationally intensive. Shown below is an example of a license plate border that is not well-defined due to the presence of a license plate holder. Note that Canny edge detection is the method used in this case.

Histograms of Oriented Gradients

This method has been shown to be very effective in identifying various objects, but it typically relies on building a model from training images. The basic idea is that an object can be described by its edge orientations; the procedure for building a descriptor consists of taking simple horizontal and vertical gradients and determining a gradient magnitude and direction for each pixel in the input image. The directions are then rounded to the nearest, for example, 45 degrees, and a histogram of these orientations is calculated for various image windows. We examined the oriented gradients and resulting histograms of plates in our test images, but after some experimentation decided that a model is, not surprisingly, necessary to easily implement a reliable detector using these descriptors.

Harris Corners

This method searches for "corner" image points by searching for certain eignevalue conditions of the sensitivity matrix, a 2x2 block calculated based on the local image gradients. Our idea was to identify Harris corners and subsequently search the set of corners for rectangles of the approximate aspect ratio of a license plate. The difficulty here turned out to be that in order to return Harris corners that actually corresponded to the corners of a plate in our test images, the detector had to be sensitive and as a result, returned a large number of corners, making the search intractable.

K-means Segmentation

K-means is a method of image segmentation with the goal of improving edge dectection and recognition. The process breaks up the images into k color spaces depending on the location and color of pixels within the image. For our project, the method was essentially an unsupervised nearest neighbors analysis in 5-space using X,Y pixel coordinates as well as all 3 RGB values. We were hoping that the license plate would be large and similar enough to warrant its own color cluster. Since the license plate color was a more important factor than its location, we weighted the X,Y coordinates at roughly 25% of R,G, and B values. The problem with k-means, and the reason we didn't use it in our project, is that the optimal number of centers, k, depends heavily on how far away the license plate is in the image. For distant license plates, at about 25 feet, using 10 centers might be optimal, but in closer images, this would usually result in the license plate being separated into at least two clusters.

Histogram Equalization

Histogram equalization (adaptive or otherwise) is a method of enhancing contast by balancing an image's histogram. In many images, including some of ours, the grayscale histogram of the image is skewed towards the light or dark pixels. This shift means that there less contrast within the image, making it harder to detect edges and recognize regions of interest. In order to combat this, equalization transforms the histogram to balance it evenly over the [0 255] brightness space. In adaptive equalization, blocks within the image are equalized, but not the whole image itself. Below is an illustration of equalization on an image that was initially too dark. The drawback of equalization is that it tends to enhance contrast both in regions of interest and in unexepected places. In many of our images for example, the contrast of the trees on the blue sky was greatly enchanced leading to many false postives above the horizon.

Related Work

As noted above, this paper concerns the license plate detection problem without the use of a classifier, so only previous works that do not include such a stage shall be discussed. A number of blind methods, either edge-based or color-based, have implemented edge-based and color-based methods for plate detection. Reference [4] outlines a solely edge-density-based method that, while reasonably effective, most certainly relied on a controlled imaging scenario.

Reference [6] approached the problem from an initially color-based standpoint. In particular, the authors first perform a mean-shift segmentation, and subsequently use edge-density and rectangularity criteria to choose license plate candidates from the original segmentation output regions. This is in sharp contrast to Abolghaesmi and Ahmadyfard’s work, in which a matched filtering technique is employed to extract candidate regions.

Our System

Preprocessing

To speed the implementation, the image is first downsampled by a factor of 2, and subsequent edge detection and filtering performed at this scale. In addition, several of the previous works discussed above rely on very specific functions for image enhancement, and perhaps with good reason. Here our only preprocessing stage is an optional adaptive histogram equalization step. In our implementation, the dimensions of the tiles for the adaptive equalization are chosen to match those of a typical plate in the target images.

Edge Detection, Filtering, and Thresholding

A key insight into the license plate detection problem is that the presence of a plate results in vertical edges in a rectangular region of a particular aspect ratio [5]. As a result, as is done in [5], the first stage in the detection pipeline consists of taking a simple centered horizontal image gradient, followed by a 21 by 50 pixel Gaussian weighting with standard deviation equal to 10 pixels. An example of a weighted edge map obtained from a car image is shown in the figure below

The next detection stage consists of choosing candidate regions from the edge map obtained above. Reference [5] accomplishes this with the filter shown below in Figure 2. Such a filter accomplishes the task reliably in the situation in which it was used: detecting Iranian license plates. Virtually none of these plates were surrounded by license plate holders, while a majority of our test images were California plates surrounded by license plate holders from dealerships, schools, etc. Edges detected in the wording on these license plate holders could be picked up by the extended negative lobes in the filter shown below:

For this reason, we use instead the filter shown here:

The output of the filtering stage is shown below. We threshold this result, using a value chosen experimentally to be any pixels that fall into the 99.5 percentile or higher. This choice often results in a large number of candidate regions, but is necessary largely due to the presence of multiple license plates and, at times, smaller plates with less well-defined edges.

After thresholding, region labeling is performed. We extract rectangular candidate regions from the returned set, centered at the mean of each cluster. The region dimensions are chosen proportional to the width of the cluster (a margin is provided for slightly distorted plates).

False Positive Rejection

As shown in in the figure above, simply returning all of the candidate regions output by the filtering stage would not be an acceptable detector. We thus require a method of rejecting the regions that likely do not contain license plates. We considered discarding small regions, or performing morphological erosion with a rectangular structuring element of the approximate aspect ratio of a license plate and then discarding small regions. Though these methods did drop many of the false positives, tuning the parameters to avoid also discarding license plate regions proved rather difficult across our set of images. Consequently, we employ two methods, described below.

Contextual Rejection

Although it is clearly possible for a license plate to be present in an image without a car, plates in the situations in which we are interested in performing license plate detection are usually attached to vehicles. Thus, we can reject false positive candidate windows by searching for characteristics of cars in the local region. However, even though various car detection methods have been successful (e.g. Histograms of Oriented Gradients), they typically use a training database and classifier.

We suggest the very simple approach of using the Hough transform to search above and below the candidate window for prominent horizontal lines, which, for an actual license plate region, often result from the edges of the plate itself, the license plate holder, or various portions of the bumper. This last distinction is important because the edges of the plate itself are often not particularly well defined in the edge map; dirt, glare, license plate holder and/or bumper color can all severely interfere with the detection of the plate border. This contextual rejection is most effective for image patches that contain, for example, tree branches or leaves against a sky or building background. In many cases, the branches result in a rectangular region of dense edge pixels that can easily be rejected via this method. In our implementation, the outer pixels of the original candidate rectangle output by the matched filtering stage, as well as a small margin around it, are searched for horizontal lines of at least half the length of the original window.

Color-based Rejection

Because the license plate region consists mostly of a solid color background, we’d like to segment the image region containing a plate candidate window in order to reject additional false positives. An algorithm such as k-means is not suitable because of the vast differences in output based on the choice of k, as well as the fact that at a minimum, the lettering on the plate will be segmented separately from the plate background.

The mean-shift algorithm, however, detects modes of a dataset, which, for images, can be done both in the spatial and range domains. In particular, this allows for a spatial- and color-based segmentation, which will achieve results much closer to the desired output. If it were perfect, however, we could have just used this algorithm as the basis for the detector, as in [6].

Instead, we utilize the algorithm to further reject windows that passed through the previous stage. To achieve a good segmentation result, we set the minimum region area parameter proportional to the region size selected by the filtering and thresholding stage. This often results in a favorable segmentation output that can be used for false positive rejection, or even region refinement. These two applications, however, were not fully explored as more development is needed in the analysis of the segmented regions. Currently, the image is downsampled by a factor of 8, mean-shift is segmentation is performed, followed by a morphological erosion is performed using a 2 by 5 (height x width) structuring element, and then regions that are sufficiently small or large are rejected. Additionally, if the rectangle enclosing all of the region’s pixels is sufficiently far from the aspect ratio of a plate, the region is rejected. The parameters were chosen experimentally. Note also that the regions needed to be upsampled to return a region in the units of the original image.

An example of reasonable plate separation by mean-shift segmentation is shown below. Note that without prior knowledge of candidate region size, the segmentation would not produce such a desirable result in many cases.

References

[1] D. Comanicu, P. Meer, "Mean shift: A robust approach toward feature space analysis," IEEE Trans. Pattern Anal. Machine Intell., May 2002.

[2] P. Meer, B. Georgescu, "Edge detection with embedded confidence," IEEE Trans. Pattern Anal. Machine Intell., 28, 2001.

[3] C. Christoudias, B. Georgescu, P. Meer, "Synergism in low level vision," 16th International Conference of Pattern Recognition, Track 1 - Computer Vision and Robotics, Quebec City, Canada, August 2001.

[4] D. Zheng, Y. Zhao, and J. Wang, “An efficient method of license plate location,” Pattern Recognition Letters, vol. 26, pp. 2431-2438, 2005.

[5] V. Abolghasemi and A. Ahmadyfard, “An edge-based color-aided method for license plate detection,” Image and Vision Computing, vol. 27, pp. 1134-1142, 2009.

[6] W. Jia, H. Zhang, and X. He, “Region-based license plate detection,” Journal of Network and Computer Applications, vol. 30, pp. 1324-1333, 2007.