OverView

Sergei Mikhailovich Prokudin-Gorskii (1863–1944) was a pioneering Russian photographer, best known for his early color photography,namely the Prokudin-Gorskii Photo Collection. In this photo collection, he captured three separate monochrome images of the same scene through red, green, and blue filters, which allowed a full-color image to be created when the three layers were combined.

In this project, we aimed to use digitized images of Prokudin-Gorskii glass plates and modern image processing techniques to automatically generate color images with as few visual artifacts as possible.

Approach

I assumed a simple (x,y) translation model is sufficient for proper alignment. I use the blue channel image as the reference image and try to move the red channel image and the green channel image to match the blue channel image. I start with a simple but inefficient method, namely search over the window of [-15, 15] x [-15, 15] of displacements and calculate the similarity between the two channel images. I use Normalized Cross Correlation(NCC) as the similarity and select the displacement with the best similarity as the best displacement. After that, I apply the corresponding displacements to the images of the two channels respectively to create a full-color image.

During the experiment, I discovered that image edges adversely affect the calculation of the optimal displacement. Therefore, I cropped 5% of both sides of the image and then calculated the similarity.

However, this algorithm is very inefficient when processing high-resolution images like .tif images. This is because for high-resolution images the algorithm needs to search a much larger window to find the best displacement. This can take anywhere from a few minutes to a few hours.

Thus, a pyramid search algorithm is used to process hi-res images. This method build a pyramid of progressively lower resolution images from the original images. It starts by searching through the best displacement at the coarsest level and then refine the search result by moving down the pyramid. Since you know the optimal displacement on the coarser image, you only need to search a limited area to get the optimal displacement on the current image. This refine procedure will continue until reaching the original resolution and acquire the optimal displacement.

In this project, I downscale images by some power of 2 until they are under 800*800 pixels. Then, I align the smallest image by searching over the window[-width*5%,width*5%] x [-height*5%,height*5%] and get the best displacement (dx,dy). After that, I scale up the image and search in a limited range, namely [2*dx-2,2*dx+2] X [2*dy-2,2*dy+2] to find the best displacement of current image. I continue this process until acquire the optimal displacement of the original image. After synthesis the full-color image, I cropped 7% of both sides of them to get better visual effects.

Bells & Whistles

In order to further improve the quality of the resulting full-color images, I tried to use the gradient image of the original image instead of the original image to calculate the optimal displacement.I use the sobel operator to process the original images and obtain their gradient images, which can better outline the boundaries of the shapes in the original images without being disturbed by brightness. This method gives better results on most .tifs.

This method has achieved remarkable results on the following images:

It can be noticed that using the image processed by the sobel operator for matching can get better output results.

Results

Displacement are listed as (dx,dy), dx represents horizontal displacement, dy represents vertical displacement.