Note that there are two aspects to the improvement. The first is the absence of the clumpy unreliable pixel groups. The second, although more subtle effect is the general cleanliness of the "regular" pixels in the dark frame.

Although the chip temperature was slightly different between the exposures, note that this would most likely have favored the "before" image, as it was taken at a lower temperature. Qualitative experience with darks taken under various conditions indicates that chip temperature was not a significant factor in the presence or severity of unreliable pixels.

Correcting for Unreliable Pixels in a CCD Array

by Russell Croman

Introduction

In CCD imaging, it is standard practice to calibrate individual object exposures using dark exposures. The theory is that each pixel's response in the object exposure is a sum of its response to light from the object being photographed, plus its own self-generated "dark current." By taking an exposure of equal length (and at the same chip temperature) with the camera's shutter closed, a record of each pixel's dark current is captured. This can then be subtracted from the object exposure to yield the true value due to photon flux experienced by each pixel.

In reality, this method is limited mainly by noise sources, and these are usually overcome by taking multiple dark frames and averaging them, similar to the practice of taking multiple exposures of the object and combining them to improve the signal-to-noise ratio.

While simple dark calibration as described above successfully calibrates most of the pixels in a CCD array, there are some pixels which behave in a way that defeats this method.

One way a pixel can fail to be calibrated correctly is if its dark current is very high. On long exposures, such a pixel may reach the saturation level of either its own full-well depth or the A/D converter that will eventually convert its charge level to a numerical value. If a pixel saturates in the dark exposure, it will certainly also saturate in the object exposure. When the dark exposure is subsequently subtracted from the object exposure, a "zero" pixel value will result.

Figure 1 shows a small portion of a raw exposure, taken with the SBIG STL-11000XM camera. The exposure time was five minutes, and the chip temperature was -20°C.) Note the numerous white pixels, caused by pixels with unusually high dark current.

Figure 2 shows a corresponding master dark frame, formed by combining 20 individual dark exposures.

Figure 3 shows how the object exposure of Figure 1 appears after dark calibration. Note the many black pixels that are a result of saturated pixels in the dark frame.

Figure 1: Raw object exposure

Figure 2: Master dark frame

Figure 3: Calibrated object exposure

In the past, a few methods have been used to handle the black pixels that result from saturated pixels in the dark frame. One is to apply a "dead pixel filter," in which each pixel is inspected in relation to its neighbors. If a pixel has a value significantly below its neighbors, it is replaced by some function combining the neighboring pixel values, usually the median value.

While this method works well for many detectors, it mostly fails on images taken with the KAI-11000M (the main detector in the STL-11000XM).

Here is a close-up of a portion of the calibrated image that illustrates why this is:

Figure 4: Close-up of calibrated exposure

As can be seen, with this detector most of the black pixels occur in vertical pairs. Therefore, when a black pixel is inspected by a dead pixel filter, it has a neighbor that is also black. Thus the filter does not apply a correction, as the black pixel is not significantly below its neighbors.

Unreliable Pixels

Closer inspection of Figure 4 reveals another problem. While many of the "bad" CCD pixels result in black pixels in the calibrated image, there are also many that are not completely black. Even if these did not occur in vertical pairs, a dead pixel filter might still miss them as they are not low enough below the neighboring pixel values to trigger the filter.

Lengthy inspection of the individual dark and object exposures was undertaken in an attempt to understand the mechanism behind these unreliable pixels, and a common denominator was found that yielded a way to detect these "unreliable" pixels. Note that the exact physics that produces unreliable pixels is beyond the scope of this work. Suffice it to say that the main goal was to come up with a method of detecting them and handling their consequences.

If a certain pixel location in the image is chosen, and the pixels values in this location in each dark exposure are inspected, one usually finds that the values vary according to statistics corresponding to the shot noise of the dark current and the readout noise of the A/D converter. What was found in the case of "unreliable" pixels was that the variation from exposure to exposure was much higher than the norm. Some "hot" pixels in the dark exposures did not appear in every exposure, or they changed their values drastically from exposure to exposure. Indeed, they seemed to almost turn on and off at random intervals. It was this behavior that suggested the term "unreliable pixel."

Figure 5 shows a close-up of the master dark frame, corresponding to the portion of the image shown in Figure 4.

Figure 6 shows a map of pixel "unreliability." The pixel values in this map are proportional to the relative deviation of the value of each pixel location across all of the individual dark frames. It is generated as an option during dark frame combination by the RC Console plug-in for MaxIm DL.

Figure 5: Close-up of master dark frame

Figure 6: Close-up of unreliable pixel map

Once the map of unreliable pixels is generated, it can be used to patch up the calibrated object image. This is done using a method similar to classic dead pixel filtering. If a pixel is flagged as unreliable by the map, it is replaced by an average of the neighboring pixel values, excluding any other unreliable pixels that happen to be neighbors.

Figure 7 shows the result of applying the unreliable-pixel correction. As a side note, about 3.3% of the pixels were deemed to be unreliable for this data set by the filter.

Figure 7: Calibrated exposure with unreliable pixels repaired

Figure 8 shows the larger portion, corresponding to Figures 1-3, with unreliable pixels repaired.

Figure 8: Calibrated exposure with unreliable pixels repaired

Note that all of the dark pixels have been repaired. However, there are a few hot pixels remaining. These are apparently due to hot pixels that were present (i.e., turned on) in the object exposures but not in the dark exposures. They are few enough that two iterations of hot pixel filtering in MaxIm DL removes them. (Two iterations are needed as these still occur in vertical pairs. One iteration handles one of each pair. The second iteration handles the remainder.)

Dithered guiding

As a side note, it is strongly recommended to use dithered guiding in combination with unreliable pixel repair. The method essentially involves the creation of artificial data to repair unreliable pixels, and dithered guiding will ensure that these pixels are not in the same location in each object exposure after alignment. Dithered guiding also has significant benefits for signal-to-noise ratio even in the absence of unreliable pixels, and should be done in any case.

One might ask why dithered guiding in conjunction with sigma-reject or median combination would not be sufficient to handle unreliable pixels.. Indeed, careful tests were conducted to evaluate this. The main reasons were found to be:

1. When dithered images are aligned prior to combination, interpolation causes unreliable pixels to corrupt (darken) neighboring pixels.
2. The dithering pattern must be fully non-duplicative, or the sigma-reject algorithm will be thrown off by unreliable pixels occurring in the same (or nearly the same) place more than once.

Here is a before-and-after example of sigma-reject combination of eight object exposures taken with dithered guiding, with and without unreliable pixel repair. The contrast has been increased to levels typical of image post-processing, and is the same for both images:

Figure 9: Sigma-reject combination of eight object exposures without unreliable pixel repair

Figure 10: Sigma-reject combination after unreliable pixel repair

Repairing unreliable pixels with RC Console

Unreliable pixel repair has been implemented in the latest version of the RC Console plug-in for MaxIm DL. The process consists of a few steps:

1. Generate the master dark frame and unreliable pixel map.
2. Calibrate the object exposures with the master dark.
3. Repair the unreliable pixels.

Each step is described below.

1. Generate the master dark and unreliable pixel map

An option check-box has been added to the sigma-reject combination dialog box to produce an unreliable pixel map in addition to the sigma-reject combined image.

Figure 11: New unreliable pixel map option in Sigma-Reject Combine

The unreliable pixel map will appear as a new MaxIm image with the name "UnreliablePixels." Save this and the sigma-reject combined image to disk for later use.

2. Calibrate the object exposures

This is done as usual, using master dark generated in step 1, and any flat-field image required.

3. Repair the unreliable pixels

This is done using the new repair function in RC Console. Start with all of the calibrated object exposures open from the previous step. Also open the unreliable pixel map image generated in step 1. Then invoke the RC Console plug-in. Select the object exposures you want to repair, but do not select the unreliable pixel map. Your screen should look something like this:

Figure 12: Setup for unreliable pixel repair

Next, click the "Fix Unreliable Pixels" button. You should get the following dialog box:

Figure 13: Unreliable pixel repair dialog box

The first box is the threshold for pixel repair, and refers to the pixel values in the unreliable pixel map. The best way to set this value is to inspect the unreliable pixel map and determine its background value. This corresponds to the relative deviation for "normal" pixels. Set the threshold to a value somewhat above this level to avoid repairing good pixels.

The next box is used to select which image open in MaxIm is the unreliable pixel map. If you had more than one image in the "available images" box in the RC Console main form, you may need to select the correct image.

The final box is the kernel size to be used to repair pixels. The filter will look in a box of pixels of this size around each unreliable pixel to determine what repair value to synthesize. The recommended value is 5x5 for well-sampled images, and 3x3 for under-sampled images, but may be varied to experiment.

When the values in the dialog box are set as desired, click "OK" to run the repair. A progress window will open and the repair will commence.

Repaired images can then be aligned and combined as usual. (Unreliable pixel repair must be performed prior to alignment.)

Unreliable pixels may move around over time, and they will most likely change with exposure time and chip temperature. It is therefore recommended that a new unreliable pixel map be generated for each dark data set.

Conclusion

A new method of detecting and repairing unreliable pixels in a CCD array has been presented. While this development was motivated by experience with the KAI-11000M detector, it should be noted that other detectors also have unreliable pixels, and images produced using these can also benefit from unreliable pixel repair.

To obtain the RC Console plug-in for MaxIm DL, go the following link:

http://www.rc-astro.com/resources/rc_console.html