Implementation
Build the Image Pyramid and Tiles
Given a high resolution image, we choose a Gaussian kernel, convolve with the image, subsample it to give a lower resolution image of a quarter size. Keep doing this for severl times will generate a Gaussian multiresolution pyramid.
We can use commands in the ISE pyrTools matlab toolbox to implement this process. Major commands include buildGlyr, pyrBand (buildLpyr for Laplacian pyramid). For simplicity, we just use the default Gaussian kernel "binom5".
bwpyr.m reads a grayscale image and build a Gaussian pyramid and a Laplacian pyramid. Notice that after each convolution and downsampling, the pixel value is enlarged by 2 times. So when saving the image, the pixel value needs to be scaled down accordingly.
For colored image, we have to build the pyramid separately for each of the red, green and blue planes, and then recombine the red, green and blue band at each resolution to generate a colored image at that resolution. colorgpyr_tif.m reads a tif colored image file and generates a Gaussian pyramid of multiple resolutions, saved in filename_level.tif.
Mablab Memory Problem
However this approach breaks down for a large high resolution image. For the image shown in the Java demo, whose resolution is 3500x2400, the tif image file is almost 25MB. Tif uses 3 bytes for each pixel, one for each color. When loaded in matlab, each pixel is represented by 3 doubles, each being 8 bytes. The image alone requires 25x8=200MB, before any processing can happen. Using pyrTools causes Matlab run out of memory quickly.
To deal with the memory problem, I tried a number of approaches. First I tried to download ISE pyrTools to a Sun workstation with 1G RAM and 1.5G swap space, hoping the matlab memory problem would go away. However, it didn't work, which implied that Matlab might have internal memory limitations, regardless of the physical RAM and swap space on the machine it's running on.
I finally decided to rewrite a number of commands in pyrTools toolbox, including buildGpyr and pyrBand. Basically, I had to break down a big task into multiple smaller tasks, which can be handled by matlab.within its memory limitation. I programmed extremely meticulously to clear each var whenever it has been used so that its memory space is released, and save it to a file if it will be used later and load back in when needed. Basically, at any given time, only keep the bare minimum amount variables in matlab to be able to proceed given the memory limitation.
In particular, in order to build a Gaussian pyramid for a color image, I first decompose the image into red, green and blue planes. For each plane, I do convolution with a Gaussian kernel and downsampling, then save the result immediately into a file, instead of building a pyramid on the fly as done in buildGpyr. Next for each band, I load the red, green and blue components and reconstruct a color image for that band.
The method described so far will still fail if the image is so big that convolution with just one color component is still not possible due to the memory limitation, then we have to break the image to blocks, and build a pyramid for each block separately and later on reconstruct them.
An alternative approach is to use C or C++, instead of using Matlab. However, image processing is not as convenient as in Matlab.
After building the pyramid, the image at each resolution is decomposed into tiles of 64x64 pixels. (For simplicity, in my implementation I use a tile size of 100x100 pixels.) Each image tile is saved in a separate image file.
A Java Applet is implemented as a FlashPix client. It uses a simplified version of Internet Image Protocol to request image tiles from the server.
The Java Applet uses a cache for the tiles for each resolution. Initially all tiles are empty.
When the user requests to draw a certain area of an image, the applet figures out which tiles are needed based on the requested resolution, the size of the applet window and the location of the area the user is interested in. It then checks the cache to see if the tiles needed are available. If a tile is availabe, it's displayed immediately. Otherwise, a request is sent to the server, and when the tile is received from the server, it is put in the cache and displayed in the window.
As described on the Java Demo page, The user can use several buttons to navigate around the image. When the user move to a certain direction, the resolution level doesn't change, but some new tiles are needed to be displayed. These tiles can either already cached which can be displayed immediately or need to be fetched from the server.
When the user does zooming, the resolution level is changed. Zoom in decrement the resolution level by 1.(Level 0 is the highest resolution). Zoom out increment the resolution level by 1. The applet then use the location of the user click point to compute which tiles are needed to zoom in around the click point as the center point, and they are either displayed from the cache right away or fetched from the server.
Advantage of FlashPix
(Repeated from the Java Demo page because of its importance.)
This applet demonstrate the power of FlashPix. You can load a low resolution image first, get a general idea of the image, then zoom in to the area you are interested in to get more details from a higher resolution image. Only limited amount of tiles are transfered for the higher resolution image. You can click on the redraw button to redraw the image, and then you can open the Java console from a Web browser and see what percentage of tiles are transmitted for each resolution. Typically, for the highest resolution, less than 10% is needed.
Future Optimizations
Currently, the applet generates one request for each tile. Multiple requests need multiple network commincations which is slow. A immediate optimization is to group several requests into one request packet and send the packet to the server. The server will bundle all the tiles requested and return it to the client in one packet..
Multiple windows can be used to display multiple areas or multiple resolutions of the image at the same time. The multiple windows share the same cache.
Multithreading can be used to further improve the performance.
The server is a simple simulation of the FlashPix file format using a file directory hierarchy. Each resolution is stored in a separate subdirectory. For each resolution, each tile is stored as a file, named as image_name_resolution_index_tile_index.jpg, for example medical_1_25_30.jpg is a tile at the 25th row and the 30th column for resolution 1 for the image medical.jpg. As you can see, each image is stored in the jpeg format to save disk space and reduce transmission time. For lossless storage, compressed tiff format can be used.
Performance is very good
This implementation of FlashPix is for evaluation and illustration purposes. However, it does have to potential to grow into a full blown implementation.
Compared to some real implementation of FlashPix on the web, this Java applet implemenation is extremely fast.While the major reason is that this is a very limited implementation of the FlashPix format, the good design of the Java applet does play a role.