| ccdman | Version-3.0 | ccdman |
V. CCD Reductions
1. Introduction
2. Getting the Data
3. Reducing the Data
3.1 Introduction
3.2 Reading the Data from Tape
3.3 Setting Up the Translation File
3.3.1 Setinstrument for CCD Used
3.3.2 Setting the Subsets Parameter and Preparations
3.4 Preparing Individual Calibration Frames
3.5 Processing the calibration frames
3.6 Preparing a Final Flat Field
3.7 Processing the Images
3.8 Creating the Bad Pixel File
3.9 Writing the Data to Tape
Lisa Wells
Mario Hamuy
September 30, 1992
CCDRED is a package in IRAF used for CCD data reduction. It is primarily used for CCD imaging although various aspects are also used for spectral reduction. This document is intended as a guideline to reducing direct CCD images using IRAF. If you are reducing spectra, see Section III or IV of this manual. If you do not have experience using IRAF we suggest that you start by reading "An Introduction to IRAF" which will give you a general idea of IRAF structure as well as a few fundamental tools. If you plan to use this package extensively, there is a demo on the computer as well which will run through the reduction process interactively. Once you login and enter IRAF type the following:
cl> noao
no> imred
im> ccdred
cc> ccdtest
cc>
The cc> prompt indicates the package has been loaded. ccdtest contains several tasks one of which is demo. Now type demo and follow the instructions. It will prompt you, from time to time, to continue to the next section.
The examples shown here are just that, examples. The user must decide upon the reduction procedure, naming, convention, etc..., appropriate for his/her own data and use the cookbook and examples as guidelines only. The examples are shown with prompts, for the package containing the tasks (do not type the prompts, of course). It is strongly recommended that you perform an lpar on every task immediately before using it, unless you are familiar with all of its parameters. This is not always shown in the examples, but is normal practice even among seasoned IRAF users.
IRAF uses two conventions that you should always keep in mind. First, images consist of lines and columns (not rows and columns). Keep in mind that the mountain Forth systems for the CCDs are zero-indexed and use rows and lines. IRAF uses one-indexed coordinates for images (see figure 1). Second, the "order" of a function is the number of independent coefficients for a polynomial, or the number of intervals for a spline curve. For example, a cubic (third-degree) polynomial is described as "order=4" (four coefficients).
If you require personal assistance in your reductions please contact either Mauricio Navarrete or Nelson Saavedra on Tololo (ex 422), or Mario Hamuy(ex 210) in La Serena.
Many observers often get confused about the number and kind of calibration frames that must be taken to properly reduce their data. During a whole run you should get the following:
1) Bias frames for each night are essential no matter what setup and chip you are using. 20-30 of these should be taken and combined.
2) Dome flats should be taken for each filter you are using. Preferably this should be done every day, but you can get by with just one per run, as long as you take a sufficient number of them, say 20-30.
3) Twilight sky flats are necessary if you want to do photometry to better than 2%-3%, if you are doing surface photometry of extended objects, or if sky subtraction is critical. We suggest that everyone take sky flats since it is a good check of your dome flat illumination. For some CCDs it is better to use a sky flat to flatten your objects and this may depend upon the filters being used. It was found at the 0.9-m and 1.5-m telescopes that sky flats do better in flattening your images in the U and B filters. It is therefore suggested that you concentrate on getting many U and B sky flats (10 or more) since you will probably process your data using them. These will be combined in the first steps of the reduction.
4) Darks are worth taking to check that things are working but dark correction is not necessary for any of the CCDs now used at CTIO. The dark current should be <10 e-/hour/pixel, if greater, then something is wrong and you should get it fixed.
5) Photometric standard stars should be taken when needed and as many as necessary (>20) to properly calibrate your objects.
You should refer to the instrument manual for more details. We suggest that you start taking your calibration frames early in the afternoon so that you have enough time left for supper. It is important to note on your calibration frames, the positions of bad pixels and then avoid observing your objects on these regions of the CCD, especially if you plan to do photometry. At the end of the reductions, you may wish to use a bad pixel map to correct the bad pixels. This will be discussed later (section 3.8) in more detail.
3. Reducing the Data
A full reduction of CCD data requires the following operations (see the flow diagram on the next page):
1) Combine the bias, flats and darks.
2) Fit and subtract a readout bias given by the overscan strip.
3) Trim the image of overscan strip and border rows and columns.
4) Subtract the dark, if appropriate.
5) Prepare a final flat.
6) Divide by the flat field.
7) Fix the bad pixels in all the images.
8) Fringing corrections may be done at the end.
The most general processing, described in this manual, consists of overscan subtracting, trimming, bias subtracting, dark subtracting, and flat fielding your data. Although dark subtraction is rarely used, it is included in this example for generality.
Individual bias, darks, dome and sky flats must be properly combined to give good signal to noise calibration frames and to remove cosmic rays. The algorithm used to combine the images must have 10 or more frames to do a good job of cleaning the cosmic rays. IRAF offers several algorithms for combining the individual frames. You should always carefully inspect all the individual frames, and the final image to check for potential problems.
Having obtained the combined calibration images you should flatten your sky flat using the dome flat and examine the result for any significant illumination variation. If these variations are significant they may be fit in order to correct your dome flat. Fringing corrections may be applied. This should be done separately. This is only needed for the RCA chips at the 4 meter PF/CCD and the TI chips in the I band. We do not currently support this but direction can be given as to what might possibly work.
At this level the images should be finished except for fixing any bad pixels that may not have been taken care of in the flat fielding. Once this is done you may do photometry. A photometry manual is available to do your analysis, see section VI of this manual, which describe the use of the aperture photometry routines in IRAF and the transformation from instrumental to standard magnitudes. There is also a manual which was written by the IRAF group in Tucson which is a good guide; "A User's Guide to Stellar CCD Photometry with IRAF".
==========================
= Set Instrument =
= and Translation File =
==========================
*****
***
*
==================================
= Combining Calibration Frames =
= and Removing Cosmic Rays =
==================================
*****
***
*
===============================
= Processing =
= Calibration Frames =
===============================
*****
***
*
================================
= Preparing a Flat Field =
================================
*****
***
*
==========================
= Processing the data =
==========================
*****
***
*
==========================
= Fixing bad pixels =
==========================
*****
***
*
==========================
= Fringe Corrections =
= (optional) =
==========================
Load the dataio package and allocate the appropriate tape drive:
cl> dataio
da> alloc mta
If you wish to read fits files from an exabyte, use mtx as the device designation. Now mount your tape on the tape drive and be sure you've removed the write ring. It is best to create a separate directory for each night. This is done using the command mkdir night1. Now change to this directory by typing cd night1. Read the data using the task rfits. You must specify the tape drive name, the list of files you wish to read and the "root" file name. If you transferred your data to the SUN using getpix then you may wish to use the naming convention given to your files, in this case just set the parameter oldirafname to yes. In choosing the root file name, it is usually a good idea to include a digit in the name to indicate the tape number (eg, "tape1" for the first tape; files would be called "tape10001, tape10002,.."); alternatively, an offset may be added (eg, offset=89 means the first files would be called "tape10090, tape10091,.." or 1+89, 2+89,..).
da> epar rfits (check the parameter list)
da> rfits mta 1-999 tape1
The file list "1-999" should more than cover the number of files on tape; the task will end gracefully at the end of the tape. When finished, rewind the tape and deallocate the drive,
da> rew mta
da> dealloc mta
and remove your tape from the drive. We assume that you kept the old IRAF name given to your files throughout the rest of this manual.
\fIrfits\fR
fits_file = "mta" FITS data source
file_list = "1-999" File list
iraf_file = "tape1" IRAF filename
(make_image = yes) Create an IRAF image?
(long_header = no) Print FITS header cards?
(short_header = yes) Print short header?
(datatype = "") IRAF data type
(blank = 0.) Blank value
(scale = yes) Scale the data?
(oldirafname = no) Use old IRAF name in place of iraf_file?
(offset = 0) Tape file offset
(mode = "ql")
3.3 Setting Up the Translation File
Start by loading the imred and ccdred packages.
cl> noao
no> imred
im> ccdred
cc>
Because the ccdred package was designed to work with data from many different observatories and instruments, an instrument translation file is required to define a mapping between the package parameters and the particular image header format. An example of a translation file is shown on the next page. You must define a translation file using the task setinstrument. Edit the parameters for the task setinstrument according to the list given below and run it. The choices for instrument can be found in Appendix A. If the CCD you used is not listed, use the generic "ccd" or "cfccd" in the instrument parameter in the task.
cc> epar setinstrument (check the parameter list)
cc> setinstrument
The task will run and prompt you for the instrument ID (a "?" will list the possible choices). Answer cfccd and the task will send you into the parameter editing mode for the task ccdred. It will automatically set the instrument (translation) parameter so do not change it. It is a good idea while processing your calibration frames to save a copy of the unprocessed images. This will be done by the package ccdred if you specify a prefix or subdirectory in the parameter backup. In our example, the files will be saved with the prefix B. When you type ctrl-z to exit you will then be sent to ccdproc to edit its parameters. We will edit ccdproc later so don't worry about it for now.
\fIsetinstrument parameters\fR
\fIinstrument\fR = "cfccd" Instrument ID (type ? for a list)
(\fIsite\fR = "ctio") Site ID
(\fIdirectory\fR = "ccddb$") Instrument directory
(review = yes) Review instrument parameters?
query = "" Instrument ID (type q to quit)
(mode = "ql")
\fIccdred parameters\fR
(pixeltype = "real real") Output and calculation pixel datatypes
(verbose = no) Print log information to the standard output?
(logfile = "ccdlog") Text log file
(plotfile = "ccdplot") Log metacode plot file
(\fIbackup\fR = "B") Backup directory or prefix
(\fIinstrument\fR = "ccddb$ctio/cfccd.dat") CCD instrument file
(\fIssfile\fR = "subsets") Subset translation file
(graphics = "stdgraph") Interactive graphics output device
(cursor = "") Graphics cursor input
(version = "2: October 1987")
(mode = "ql")
($nargs = 0)
An example of the instrument translation file follows.
exptime exptime
darktime darktime
imagetyp imagetyp
subset filters
biassec biassec
datasec datasec
trimsec trimsec
fixfile fixfile
fixpix bp-flag 0
overscan bt-flag 0
zerocor bi-flag 0
darkcor dk-flag 0
flatcor ff-flag 0
fringcor fr-flag 0
OBJECT object
DARK dark
"PROJECTOR FLAT" flat
"SKY FLAT" other
COMPARISON other
BIAS zero
"DOME FLAT" flat
MASK other
The instrument file consists of first, the IRAF parameter and its associated image header keyword from the objects. These header values may vary depending upon the instrument being used. The second section is the header flag section which is added to the object as it is being processed to mark that the operation has been done. Once an image is zero corrected, the keyword bi-flag parameter is added to the image header and set to 0 so that the process is not repeated later. The last section is the image-type specification. This is used when the imagetyp parameter is read. The header value OBJECT is translated to object, and a SKY FLAT becomes other. If you need to modify the translation file you may copy this file to your directory and change the appropriate parameters. If the name of the imagetype contains 2 words, then you must put quotes around the designation, for example, look at the SKY FLAT line in the above file. Comparison arcs are not used in direct imaging and are not needed.
Without the proper translation file, the task ccdlist will give the following:
cc> ccdlist *.imh
bias001.imh[400,420][short][unknown][]:median of 4 ave of 6 bias
flat002.imh[400,420][short][unknown][]:median of 4 ave of 6 flat
flat003.imh[400,420][short][unknown][]:median of 4 ave of 6 flat
obj004.imh[400,420][short][unknown][]:IC 481 V-filter ........
obj005.imh[400,420][short][unknown][]:IC 481 B-filter.........
: : : : : : : :
obj036.imh[400,420][short][unknown][]:Sky flat B-filter .......
obj037.imh[400,420][short][unknown][]:Sky flat V-filter .......
: : : : : : : :
The [unknown] in the listing means that the frame type is unknown, ie, if it is a dark, flat, bias,... etc. The blank brackets should contain the filter type and will be taken care of in the next section.
The ccdred package groups observations into subsets. The image header parameter used to identify the subsets is defined in the instrument translation file . For example, to select subsets by the header parameter filters the instrument translation file would contain the line
subset filters
Now do a ccdlist and you should see the correct image type specified for each frame:
cc> ccdlist *.imh bias001.imh[400,420][short][zero][]:median of 4 ave of 6 bias flat002.imh[400,420][short][flat][]:median of 4 ave of 6 flat flat003.imh[400,420][short][flat][]:median of 4 ave of 6 flat obj004.imh[400,420][short][object][]:IC 481 V-filter ...... obj005.imh[400,420][short][object][]:IC 481 B-filter....... : : : : : : : : : : : : : : obj036.imh[400,420][short][object][]:Sky flat B-filter ..... obj037.imh[400,420][short][object][]:Sky flat V-filter ..... : : : : : : :
Doing this will create a file in your directory called subsets. This is a file with a listing of the filter type read from the image header consisting of the combination of the upper and lower filter bolt positions. The flats were taken with a color balance filter so they have different values from the objects in the list. Therefore, you need to edit this file to set the filter types properly. The bias and dark were probably taken with a setting that corresponds to one of the filters, but it will be ignored. If however, the filter type set in the header corresponds to something other than a standard filter, set this to some arbitrary value, for example below it is assumed that the filter positions for the bias were 0 1, so this is set to 0.
cc> edit subsets
An example of the necessary changes is:
\fISubsets Before | Subsets After\fR
'1 0' 1 | '1 0' U
'2 0' 2 | '2 0' B
'3 0' 3 | '3 0' V
'4 0' 4 | '4 0' R
'5 0' 5 | '5 0' I
'0 1' 01 | '0 1' 0
'1 1' 11 | '1 1' U
'2 1' 21 | '2 1' B
'3 1' 31 | '3 1' V
'4 1' 41 | '4 1' R
'5 1' 51 | '5 1' I
If any of the parameters are not specified properly after doing ccdlist, then the translation file may be set up improperly. Consult Mauricio Navarrete, Nelson Saavedra, or Mario Hamuy.
You must also specify the overscan and trim region. The overscan is the region to the far right of the image when plotting a line of the image (see figure 2a). The overscan region should be specified by the beginning and ending column followed by the beginning and ending line. This region should begin at least 5-10 pixels from the edge of the active region of the CCD. The correct position can be found by plotting several lines using :l #1 #2 (#1 and #2 specify a range of lines to be averaged and plotted) in implot, and expanding the right side (see figure 2b). Notice the signal decays toward the right just the other side of the step with an e-folding distance of a few pixels. The overscan strip should begin in 3 to 4 e-folding distances from the break. You want to fit this region all along the image so the range of lines should include everything. Be sure to write down these values so you don't forget them.
The trim region is the section of the image to be kept after processing. Choose this area excluding the overscan region and any bad lines or columns at the edges of the image. Figures 2c and 2d show a plot of a line and column, respectively. They show the edges which should be trimmed from the image. Later (section 3.5) you will edit the parameters for ccdproc and enter the overscan and trim section.
The bad pixel file will be used after the images have been flattened. In many cases the flatfielding will correct the bad pixel regions also so it is best left until the end. If you specify the bad pixel file during the processing be aware that the corrections are done on the file first.
If you have taken many calibration frames (bias, dome flats, darks and sky flats) that you wish to combine in order to remove cosmic rays, we suggest you to perform a combination of images separately for each night. In some cases, when the instrument response remains stable during your run, it is worth combining data from many nights, especially the darks and sky flats, to improve the signal-to-noise ratio. If you have already combined the data on the mountain skip over this section and go on to the next one.
Most likely you used "getpix" to transfer your data from the LSI to the SUN so you have preserved the naming convention. This makes it very easy to combine your images.
There are tasks which combine each type of object together. For the bias do an epar on the zerocombine task, and fill the parameters according to the list given below. We suggest selecting the avsigclip rejection operation which applies a sigma clipping algorithm to each pixel to remove cosmic rays. This algorithm requires at least three input images (best with more than 10) to work effectively. According to your particular needs you may select other options to combine the data like minmax, ccdclip, etc. No scaling is performed on the biases. If your bias frames vary greatly, then there is something wrong with your setup. In this case, have someone check out the electronics.
cc> epar zerocombine (check the list given below)
Then combine the bias frames. Optionally you may add a & at the end of the command line to submit the task as a background job.
cc> zerocombine bias*.imh output=zero1 &
Now, proceed with the combination of the darks.
cc> epar darkcombine (check the list given below)
cc> darkcombine dark*.imh output=dark1 &
Before combining the flat frames, you must find a region free of bad pixels to be used to scale the individual flat frames. The box should be chosen where there is signal. Once you have determined the position of the box, you should first run imstat for all your images using this region.
cc> imstat flat*[200:250,300:330]
# IMAGE NPIX MEAN STDDEV MIN MAX
flat001.imh[200:250,300:330] 37901 5489. 1510. 678. 9020.
flat002.imh[200:250,300:330] 37901 5485. 1508. 694. 8484.
flat003.imh[200:250,300:330] 37901 5478. 1507. 691. 8472.
flat004.imh[200:250,300:330] 37901 5475. 1506. 671. 8397.
flat005.imh[200:250,300:330] 37901 5474. 1506. 659. 8540.
flat006.imh[200:250,300:330] 37901 5472. 1505. 663. 8422.
flat007.imh[200:250,300:330] 37901 5467. 1504. 655. 15513.
flat008.imh[200:250,300:330] 37901 5464. 1502. 673. 8471.
flat009.imh[200:250,300:330] 37901 5458. 1501. 684. 8503.
If the mean varies by a large amount, say greater than a few percent, then you should use the mode section for the scaling. In the example below the flats are combined with scaling. Enter this region in the statsec parameter in the following format,
[x1:x2,y1:y2]
where the values x1 and x2 are the limits of the box in the x-axis, y1 and y2 are the limits along the y-axis. This will only be used if median is specified in the scale parameter of the task flatcombine. In this example we have chosen to combine the individual flats using the ccdclip option, which rejects pixels using the CCD noise parameters, namely the read-out-noise and the gain. You must enter these values in the parameters rdnoise and gain in flatcombine. In this example we have entered a read-out-noise of 5 electrons, and a gain of 2.5 electrons/ADU.
cc> epar flatcombine (check the list given below)
Proceed with the combination of the flats.
cc> flatcombine flat.*imh output=flat1 &
If you have 3 or more sky flats, they may be combined in the same manner as the dome flats (change the ccdtype parameter to other).
cc> epar flatcombine (check the list given below)
cc> flatcombine sky.*imh output=sky1 &
The output at this point is a set of images clean of cosmic rays called zero1, dark1, flat1, and sky1, where the 1 stands for the first night and the flats and skys end in their respective filter type ie, flat1U, flat1B,... Using the parameter subsets, the combination occurred for all the flats and skys of the same filter type (subset). We suggest that you examine these images, either by displaying, or imploting them. For example;
cc> display zero1
cc> implot zero1
We suggest also to look at the logfile created by the package ccdred in order to check that the images were properly combined by the same filter type.
cc> page ccdlog
\fIzerocombine\fR
input = "bias*.imh" List of zero level images to combine
(output = "zero1") Output zero level name
(\fIcombine\fR = "average") Type of combine operation
(\fIreject\fR = "avsigclip") Type of rejection
(\fIccdtype\fR = "zero") CCD image type to combine
(process = no) Process images before combining?
(delete = no) Delete input images after combining?
(clobber = no) Clobber existing output image?
(\fIscale\fR = "none") Image scaling
(statsec = "") Image section for computing statistics
(nlow = 0) minmax: Number of low pixels to reject
(nhigh = 1) minmax: Number of high pixels to reject
(nkeep = 1) Minimum to keep (pos) or maximum to reject
(mclip = yes) Use median in sigma clipping algorithms?
(lsigma = 3.) Lower sigma clipping factor
(hsigma = 3.) Upper sigma clipping factor
(rdnoise = "0") ccdclip: CCD readout noise (electrons)
(gain = "1") ccdclip: CCD gain (electrons/DN)
(snoise = "0.") ccdclip: Sensitivity noise (fraction)
(pclip = -0.5) pclip: Percentile clipping parameter
(blank = 0.) Value if there are no pixels
(mode = "ql")
\fIdarkcombine\fR
input = "dark.*imh" List of dark images to combine
(output = "dark1") Output flat field root name
(\fIcombine\fR = "average") Type of combine operation
(\fIreject\fR = "avsigclip") Type of rejection
(\fIccdtype\fR = "dark") CCD image type to combine
(process = no) Process images before combining?
(delete = no) Delete input images after combining?
(clobber = no) Clobber existing output image?
(\fIscale\fR = "exposure") Image scaling
(statsec = "") Image section for computing statistics
(nlow = 1) minmax: Number of low pixels to reject
(nhigh = 1) minmax: Number of high pixels to reject
(nkeep = 1) Minimum to keep (pos) or maximum to reject
(mclip = yes) Use median in sigma clipping algorithms?
(lsigma = 3.) Lower sigma clipping factor
(hsigma = 3.) Upper sigma clipping factor
(rdnoise = "0.") ccdclip: CCD readout noise (electrons)
(gain = "1.") ccdclip: CCD gain (electrons/DN)
(snoise = "0.") ccdclip: Sensitivity noise (fraction)
(pclip = -0.5) pclip: Percentile clipping parameter
(blank = 0.) Value if there are no pixels
(mode = "ql")
\fIflatcombine for dome flats\fR
input = "flat.*imh" List of flat field images to combine
(output = "flat1") Output flat field root name
(combine = "median") Type of combine operation
(reject = "ccdclip") Type of rejection
(ccdtype = "flat") CCD image type to combine
(process = no) Process images before combining?
(subsets = yes) Combine images by subset parameter?
(delete = no) Delete input images after combining?
(clobber = no) Clobber existing output image?
(scale = "median") Image scaling
(statsec = "[m:n,p:q]") Image section for computing statistics
(nlow = 1) minmax: Number of low pixels to reject
(nhigh = 1) minmax: Number of high pixels to reject
(nkeep = 1) Minimum to keep (pos) or maximum to reject
(mclip = yes) Use median in sigma clipping algorithms?
(lsigma = 3.) Lower sigma clipping factor
(hsigma = 3.) Upper sigma clipping factor
(rdnoise = "5") ccdclip: CCD readout noise (electrons)
(gain = "2.5") ccdclip: CCD gain (electrons/DN)
(snoise = "0.") ccdclip: Sensitivity noise (fraction)
(pclip = -0.5) pclip: Percentile clipping parameter
(blank = 1.) Value if there are no pixels
(mode = "ql")
\fIflatcombine for sky flats\fR
input = "sky.*imh" List of flat field images to combine
(output = "sky1") Output flat field root name
(combine = "median") Type of combine operation
(reject = "ccdclip") Type of rejection
(ccdtype = "other") CCD image type to combine
(process = no) Process images before combining?
(subsets = yes) Combine images by subset parameter?
(delete = no) Delete input images after combining?
(clobber = no) Clobber existing output image?
(scale = "median") Image scaling
(statsec = "[m:n,p:q]") Image section for computing statistics
(nlow = 1) minmax: Number of low pixels to reject
(nhigh = 1) minmax: Number of high pixels to reject
(nkeep = 1) Minimum to keep (pos) or maximum to reject
(mclip = yes) Use median in sigma clipping algorithms?
(lsigma = 3.) Lower sigma clipping factor
(hsigma = 3.) Upper sigma clipping factor
(rdnoise = "5") ccdclip: CCD readout noise (electrons)
(gain = "2.5") ccdclip: CCD gain (electrons/DN)
(snoise = "0.") ccdclip: Sensitivity noise (fraction)
(pclip = -0.5) pclip: Percentile clipping parameter
(blank = 1.) Value if there are no pixels
(mode = "ql")
Although all the following steps may be done only in one step, our approach is to do them separately to allow you to start at any point in this section in case you have already started reducing the images on the mountain.
You must start by overscan subtracting and trimming zero1. Edit the parameters for the ccdproc task according to the list given below. There is a series of parameters that are set to yes and no. You must set only overscan and trim to yes and the rest to no. Also ccdtype must be set to zero. Be sure to properly specify the parameters biassec and trimsec which you determined while reading section 3.3.2. The range of columns comes first separated by a : and the range of lines is again separated by a :, i.e., [29:425,21:402].
Do not change these parameters until having processed all your images .
cc> epar ccdproc (check the list given below)
cc> ccdproc zero1
If the parameter interactive was set to yes you will be required to fit interactively the overscan region with a certain function. Once presented by the task with the plot of the overscan region (see figure 3) you may change the fitting function, for example, with :function chebyshev and its order with :order 4. To try a new fit type f. We suggest a spline3 of order 2. You may also select the sample to be fit using the s command twice. If you are happy with the fit type q to quit. The task will then process zero1 accordingly.
Continue by overscan subtracting, trimming and bias subtracting dark1. In this case you have to change in ccdproc the ccdtype parameter to dark and the zerocor parameter to yes.
cc> epar ccdproc (check the list given below)
cc> ccdproc dark1
The dark image must be examined before proceeding. For instance, if the dark level is low enough (<10 counts/hour/pixel) compared to the flats, you will probably disregard dark subtracting your images, to avoid introducing noise in your data. However, if you notice some structure in the dark image, it would be worth trying to dark correct the data. In the following examples, for generality's sake, we consider dark subtraction as part of the overall processing. If this is not your case, do not forget to set the darkcor parameter in ccdproc to no and leave it alone.
Then process the flats. Check that the ccdtype parameter is set to flat, that overscan, trim, zerocor and darkcor are all set to yes and execute the task.
cc> epar ccdproc (check the list given below)
cc> ccdproc flat1*imh
IRAF records all the reduction operations in the image headers. You may check from the headers of zero1, dark1, sky1 and flat1 that BT-FLAG, BI-FLAG, DK-FLAG are properly set. For instance,
cc> imheader flat1R l+
It is also possible to check this by using the ccdlist task which will check the header automatically and list the operations which have been performed on the image.
cc> ccdlist flat1R
You should get the following,
flat1R[496,501][real][flat][R][OTZ]:dflat R
ccdlist should tell you the image name (flat1R), the image size [496,501]
left after trimming, the pixel type [real] of the image (the processed
images should normally have pixels in real format, as opposed to
the raw images which generally have pixels in short format),
the image type [flat], the filter used in the observation
[R], the level of processing of the image [OTZ], and the image title (dflat R).
At this level you should have the flats processed up through [OTZ] which means that the image has been [O]verscan subtracted, [T]rimmed, and [Z]ero subtracted. Additional codes for other operations in ccdproc are [F]lat correction, [B]ad pixel correction, [I]llumination correction.
\fIccdproc for zero1\fR
images = "zero1" List of CCD images to correct
(\fIccdtype\fR = "zero") CCD image type to correct
(max_cache = 0) Maximum image caching memory (in Mbytes)
(noproc = no) List processing steps only?
(fixpix = no) Fix bad CCD lines and columns?
(\fIoverscan\fR = yes) Apply overscan strip correction?
(\fItrim\fR = yes) Trim the image?
(\fIzerocor\fR = no) Apply zero level correction?
(\fIdarkcor\fR = no) Apply dark count correction?
(\fIflatcor\fR = no) Apply flat field correction?
(illumcor = no) Apply illumination correction?
(fringecor = no) Apply fringe correction?
(readcor = no) Convert zero level image to readout cor?
(scancor = no) Convert flat field image to scan correction?
(readaxis = "line") Read out axis (column|line)
(fixfile = "") File describing the bad lines and columns
(\fIbiassec\fR = "[m:n,*]") Overscan strip image section
(\fItrimsec\fR = "[r:s,t:u]") Trim data section
(\fIzero\fR = "zero1") Zero level calibration image
(\fIdark\fR = "dark1") Dark count calibration image
(\fIflat\fR = "") Flat field images
(illum = "") Illumination correction images
(fringe = "") Fringe correction images
(minreplace = 1.) Minimum flat field value
(scantype = "shortscan") Scan type (shortscan|longscan)
(nscan = 1) Number of short scan lines\n
(interactive = yes) Fit overscan interactively?
(function = "spline3") Fitting function
(order = 2) Number of polynomial terms or spline pieces
(sample = "*") Sample points to fit
(naverage = 1) Number of sample points to combine
(niterate = 1) Number of rejection iterations
(low_reject = 3.) Low sigma rejection factor
(high_reject = 3.) High sigma rejection factor
(grow = 0.) Rejection growing radius
(mode = "ql")
\fIccdproc for dark1\fR
images = "dark1" List of CCD images to correct
(\fIccdtype\fR = "dark") CCD image type to correct
(max_cache = 0) Maximum image caching memory (in Mbytes)
(noproc = no) List processing steps only?
(fixpix = no) Fix bad CCD lines and columns?
(\fIoverscan\fR = yes) Apply overscan strip correction?
(\fItrim\fR = yes) Trim the image?
(\fIzerocor\fR = yes) Apply zero level correction?
(\fIdarkcor\fR = no) Apply dark count correction?
(\fIflatcor\fR = no) Apply flat field correction?
(illumcor = no) Apply illumination correction?
(fringecor = no) Apply fringe correction?
(readcor = no) Convert zero level image to readout corr?
(scancor = no) Convert flat field image to scan correction?
(readaxis = "line") Read out axis (column|line)
(fixfile = "") File describing the bad lines and columns
(\fIbiassec\fR = "[m:n,*]") Overscan strip image section
(\fItrimsec\fR = "[r:s,t:u]") Trim data section
(\fIzero\fR = "zero1") Zero level calibration image
(\fIdark\fR = "dark1") Dark count calibration image
(\fIflat\fR = "") Flat field images
(illum = "") Illumination correction images
(fringe = "") Fringe correction images
(minreplace = 1.) Minimum flat field value
(scantype = "shortscan") Scan type (shortscan|longscan)
(nscan = 1) Number of short scan lines\n
(interactive = yes) Fit overscan interactively?
(function = "spline3") Fitting function
(order = 2) Number of polynomial terms or spline pieces
(sample = "*") Sample points to fit
(naverage = 1) Number of sample points to combine
(niterate = 1) Number of rejection iterations
(low_reject = 3.) Low sigma rejection factor
(high_reject = 3.) High sigma rejection factor
(grow = 0.) Rejection growing radius
(mode = "ql")
\fIccdproc for flat1\fR
images = "flat1*imh" List of CCD images to correct
(\fIccdtype\fR = "flat") CCD image type to correct
(max_cache = 0) Maximum image caching memory (in Mbytes)
(noproc = no) List processing steps only?
(fixpix = no) Fix bad CCD lines and columns?
(\fIoverscan\fR = yes) Apply overscan strip correction?
(\fItrim\fR = yes) Trim the image?
(\fIzerocor\fR = yes) Apply zero level correction?
(\fIdarkcor\fR = yes) Apply dark count correction?
(\fIflatcor\fR = no) Apply flat field correction?
(illumcor = no) Apply illumination correction?
(fringecor = no) Apply fringe correction?
(readcor = no) Convert zero level image to readout corr?
(scancor = no) Convert flat field image to scan correction?
(readaxis = "line") Read out axis (column|line)
(fixfile = "") File describing the bad lines and columns
(\fIbiassec\fR = "[m:n,*]") Overscan strip image section
(\fItrimsec\fR = "[r:s,t:u]") Trim data section
(\fIzero\fR = "zero1") Zero level calibration image
(\fIdark\fR = "dark1") Dark count calibration image
(\fIflat\fR = "") Flat field images
(illum = "") Illumination correction images
(fringe = "") Fringe correction images
(minreplace = 1.) Minimum flat field value
(scantype = "shortscan") Scan type (shortscan|longscan)
(nscan = 1) Number of short scan lines\n
(interactive = yes) Fit overscan interactively?
(function = "spline3") Fitting function
(order = 2) Number of polynomial terms or spline pieces
(sample = "*") Sample points to fit
(naverage = 1) Number of sample points to combine
(niterate = 1) Number of rejection iterations
(low_reject = 3.) Low sigma rejection factor
(high_reject = 3.) High sigma rejection factor
(grow = 0.) Rejection growing radius
(mode = "ql")
Next we want to check the illumination of the flat by applying it to the sky flat. Load the imred and ccdred packages if they are not already loaded. There is no tried and true method for testing the illumination of the sky flats except for applying them on a long exposure taken in the same filter and implotting it or doing image statistics.
We must use ccdproc to overscan subtract, trim, bias subtract, dark subtract (if needed), and flat field the sky images. Again, we assume here that the dark is necessary for the reductions. Be sure that the images you are using for the flat field division have the proper filter identification. First edit the parameters for ccdproc, and then run this task to process your skys to see if the illumination of the dome flat is uniform;
cl> noao
im> imred
im> ccdred
cc> epar ccdproc (check the parameter list)
cc> ccdproc sky1R
Now we must check the sky to see if it really is flat. This is done using implot and plotting some lines or a group of lines,
cc> implot sky1R
:l 150 200
:c 200 250
q
Ideally if the illumination was uniform, sky1R should not show any variation. In plotting several lines together, you may get a small slope (see figure 4). If you see any variations, including a slope even near the edges of the image, then you must run the task mkskyflat. Select a smoothing box size which preserves the large scale features while adequately suppressing the noise. The scale of the largest features expected shouldn't be smaller than say 1/10 to 1/12 the image size.
cc> epar mkskyflat (check the parameter list)
cc> mkskyflat sky1R finalflat1R
The output image is the corrected flat image so call it "finalflat1R". We need to recopy the sky flat to process it again. Do an imcopy of the sky1R from the backup image after deleting the processed one.
cc> imdel sky1R
cc> imcopy Bsky1R sky1R
cc> epar ccdproc (change the parameter 'flat' from
flat1R to finalflat1R)
cc> ccdproc sky1R
Again, check to see if the sky is really flat. If it is flat (see figure 5), then you are done and must process the flats for your other filters. If it is not flat, go back to the beginning and ask for help. It may be that you need to play with the smoothing parameters. Repeat these steps until you are satisfied. Now you can process your images.
\fIccdproc for sky flats\fR
images = "sky1*imh" List of CCD images to correct
(\fIccdtype\fR = "other") CCD image type to correct
(max_cache = 0) Maximum image caching memory (in Mbytes)
(noproc = no) List processing steps only?
(fixpix = no) Fix bad CCD lines and columns?
(\fIoverscan\fR = yes) Apply overscan strip correction?
(\fItrim\fR = yes) Trim the image?
(\fIzerocor\fR = yes) Apply zero level correction?
(\fIdarkcor\fR = yes) Apply dark count correction?
(\fIflatcor\fR = yes) Apply flat field correction?
(illumcor = no) Apply illumination correction?
(fringecor = no) Apply fringe correction?
(readcor = no) Convert zero level image to readout corr?
(scancor = no) Convert flat field image to scan correction?
(readaxis = "line") Read out axis (column|line)
(fixfile = "") File describing the bad lines and columns
(\fIbiassec\fR = "[m:n,*]") Overscan strip image section
(\fItrimsec\fR = "[r:s,t:u]") Trim data section
(\fIzero\fR = "zero1") Zero level calibration image
(\fIdark\fR = "dark1") Dark count calibration image
(\fIflat\fR = "flat1*imh") Flat field images
(illum = "") Illumination correction images
(fringe = "") Fringe correction images
(minreplace = 1.) Minimum flat field value
(scantype = "shortscan") Scan type (shortscan|longscan)
(nscan = 1) Number of short scan lines\n
(interactive = yes) Fit overscan interactively?
(function = "spline3") Fitting function
(order = 2) Number of polynomial terms or spline pieces
(sample = "*") Sample points to fit
(naverage = 1) Number of sample points to combine
(niterate = 1) Number of rejection iterations
(low_reject = 3.) Low sigma rejection factor
(high_reject = 3.) High sigma rejection factor
(grow = 0.) Rejection growing radius
(mode = "ql")
\fImkskyflat\fR
input = "sky1R" Input CCD images
output = "finalflat1R" Output images (same as input if none)
(ccdtype = "") CCD image type to select
(\fIxboxmin\fR = 0.1) Minimum smoothing box size in x at edges
(\fIxboxmax\fR = 0.1) Maximum smoothing box size in x
(\fIyboxmin\fR = 0.1) Minimum smoothing box size in y at edges
(\fIyboxmax\fR = 0.1) Maximum smoothing box size in y
(clip = yes) Clip input pixels?
(lowsigma = 2.5) Low clipping sigma
(highsigma = 2.5) High clipping sigma
(ccdproc = "") CCD processing parameters
(mode = "ql")
This can be setup to be done all at once. The final flat has been made so now we divide it into your objects. If the calibration frames themselves have not been processed it will do this as well. The program takes into account the differing filters. Be sure that you have deleted all the individual calibration frames that were combined. Edit the parameters for ccdproc and run it. For tasks that take a long time, it is a good idea to set them going as a background job and send the output to a file which you can watch to keep track of the progress. You may wish to make a list of the objects to be processed to avoid reprocessing your sky flats.
cc> epar ccdproc (check the parameter list)
cc> ccdproc obj*.imh
To check the progress of the task, just type
cc> tail ccdlog
This will print out the last lines of the file named ccdlog which is the latest operation done by the task ccdproc. Once this is done, check your images to see that they have been properly flat fielded. If you are satisfied, then you may now check your images for bad pixel regions, the structure of which may not have been processed out of your images.
We suggest also to do a ccdlist on the objects in order to check whether the frames have been duly processed, i.e.,
cc> ccdlist obj*.imh
You should get a listing of your images in the format that follows,
obj1010.imh[496,501][real][object][R][OTZF]:rubin 152 R
obj1011.imh[496,501][real][object][I][OTZF]:rubin 152 I
obj1012.imh[496,501][real][object][B][OTZF]:rubin 149 B
obj1013.imh[496,501][real][object][V][OTZF]:rubin 149 V
obj1014.imh[496,501][real][object][R][OTZF]:rubin 149 R
obj1015.imh[496,501][real][object][I][OTZF]:rubin 149 I
obj1016.imh[496,501][real][object][B][OTZF]:rubin 149 B
obj1017.imh[496,501][real][object][V][OTZF]:rubin 149 V
obj1018.imh[496,501][real][object][R][OTZF]:rubin 149 R
obj1019.imh[496,501][real][object][I][OTZF]:rubin 149 I
where the [OTZF] code reveals the degree of processing of the images.
\fIccdproc for objects\fR
images = "obj*imh)" List of CCD images to correct
(\fIccdtype\fR = "object") CCD image type to correct
(max_cache = 0) Maximum image caching memory (in Mbytes)
(noproc = no) List processing steps only?
(fixpix = no) Fix bad CCD lines and columns?
(\fIoverscan\fR = yes) Apply overscan strip correction?
(\fItrim\fR = yes) Trim the image?
(\fIzerocor\fR = yes) Apply zero level correction?
(\fIdarkcor\fR = yes) Apply dark count correction?
(\fIflatcor\fR = yes) Apply flat field correction?
(illumcor = no) Apply illumination correction?
(fringecor = no) Apply fringe correction?
(readcor = no) Convert zero level image to readout corr?
(scancor = no) Convert flat field image to scan correction?
(readaxis = "line") Read out axis (column|line)
(fixfile = "") File describing the bad lines and columns
(\fIbiassec\fR = "[m:n,*]") Overscan strip image section
(\fItrimsec\fR = "[r:s,t:u]") Trim data section
(\fIzero\fR = "zero1") Zero level calibration image
(\fIdark\fR = "dark1") Dark count calibration image
(\fIflat\fR = "finalflat1*imh") Flat field images
(illum = "") Illumination correction images
(fringe = "") Fringe correction images
(minreplace = 1.) Minimum flat field value
(scantype = "shortscan") Scan type (shortscan|longscan)
(nscan = 1) Number of short scan lines\n
(interactive = yes) Fit overscan interactively?
(function = "spline3") Fitting function
(order = 2) Number of polynomial terms or spline pieces
(sample = "*") Sample points to fit
(naverage = 1) Number of sample points to combine
(niterate = 1) Number of rejection iterations
(low_reject = 3.) Low sigma rejection factor
(high_reject = 3.) High sigma rejection factor
(grow = 0.) Rejection growing radius
(mode = "ql")
Now it is time to check your images to see if structure remains in the bad pixel regions. If you were careful to observe your objects far away from any obvious bad regions then you may only need to fix pixels for cosmetic reasons. We suggest you to examine these files using the display window, imtool (this can only be done at one of the SUN consoles), and then plotting them using implot. Start by loading the noao, images, tv, imred, and ccdred packages. Display a processed image and plot it using implot,
cl> noao
no> images
im> tv
tv> imred
im> ccdred
cc> display obj002 1
cc> implot obj002
Do not exit from implot, but move the cursor to the imtool window and press F6. The cursor coordinates are now visible at the bottom of the screen. Find the position of a bad pixel using this, return to the graph and plot it using implot with the :l #(line) and :c #(column) commands. You can define them more precisely using implot by overplotting lines and columns around the center of the bad regions. Do this by typing o followed by :c # and :l #. Once you have these regions written down, check some of your images to see if the bad pixels are a problem, ie >1% in signal. If it is a problem then create a file called badpix with the listing of the pixels you wish to fix;
cc> edit badpix (according to the following format)
The following example is to illustrate the format of a bad pixel file.
84 87 1 450
87 90 105 109
87 89 110 111
124 126 112 116
206 206 80 80
Each line stands for a rectangular region of the image to be fixed. The regions are specified by four numbers giving the starting and ending columns followed by the starting and ending lines. The starting and ending points may be the same to specify a single column, line or pixel. Note that each region is "fixed" by interpolating across its shortest dimension and that regions are processed in the order that they appear in the bad pixel list. For bad regions of complex shape, some care may be required in specifying the regions in order to achieve an optimal result. It is strongly recommended that you test your bad pixel file by using it to correct a copy of one of your images before going into production . This bad pixel file will be specified in the task ccdproc in the parameter fixfile. So now edit and run the task:
cc> epar ccdproc
cc> ccdproc obj*.imh
Remember that ccdproc already knows that the images have been processed so those operations will not be performed again. Now you have final images that are ready to be used. If you plan to do photometry in IRAF see section VI of this manual.
\fIccdproc\fR
images = "obj*imh)" List of CCD images to correct
(\fIccdtype\fR = "object") CCD image type to correct
(max_cache = 0) Maximum image caching memory (in Mbytes)
(noproc = no) List processing steps only?
(\fIfixpix\fR = yes) Fix bad CCD lines and columns?
(\fIoverscan\fR = yes) Apply overscan strip correction?
(\fItrim\fR = yes) Trim the image?
(\fIzerocor\fR = yes) Apply zero level correction?
(\fIdarkcor\fR = yes) Apply dark count correction?
(\fIflatcor\fR = yes) Apply flat field correction?
(illumcor = no) Apply illumination correction?
(fringecor = no) Apply fringe correction?
(readcor = no) Convert zero level image to readout corr?
(scancor = no) Convert flat field image to scan correction?
(readaxis = "line") Read out axis (column|line)
(\fIfixfile\fR = "badpix") File describing the bad lines and columns
(\fIbiassec\fR = "[m:n,*]") Overscan strip image section
(\fItrimsec\fR = "[r:s,t:u]") Trim data section
(\fIzero\fR = "zero1") Zero level calibration image
(\fIdark\fR = "dark1") Dark count calibration image
(\fIflat\fR = "finalflat1*imh") Flat field images
(illum = "") Illumination correction images
(fringe = "") Fringe correction images
(minreplace = 1.) Minimum flat field value
(scantype = "shortscan") Scan type (shortscan|longscan)
(nscan = 1) Number of short scan lines\n
(interactive = yes) Fit overscan interactively?
(function = "spline3") Fitting function
(order = 2) Number of polynomial terms or spline pieces
(sample = "*") Sample points to fit
(naverage = 1) Number of sample points to combine
(niterate = 1) Number of rejection iterations
(low_reject = 3.) Low sigma rejection factor
(high_reject = 3.) High sigma rejection factor
(grow = 0.) Rejection growing radius
(mode = "ql")
Allocate the device, for the SUNs it is "mta". Then mount your tape, don't forget to put in a write ring. If you wish to use the exabyte drive, use "mtx" for the device name. Load the package dataio and do an epar on wfits.
cl> dataio
da> epar wfits
da> wfits *.imh mta.6250 new+
The parameter newtape should be "yes" if you are starting on a new tape. Otherwise you will not want to overwrite data which has already been placed on a tape, so use "no".
After writing all the desired files to tape, deallocate the drive and retrieve your tape.
da> dealloc mta