Version 1.2 -- March 2008
This user’s guide is focused on how to run the software and obtain improved results. For more detailed discussion, consult Laidler et al., “TFIT: A Photometry Package Using Prior Information for Mixed-Resolution Datasets,” PASP 119:1325.
Given a high resolution image (HRI) and a low resolution image (LRI) of the same galaxy, we can use the information from the HRI to constrain our interpretation of the LRI.
More importantly, since galaxies don't occur in isolation, we can use the information from HRI of several galaxies, close enough on the sky to overlap in the LRI, to obtain more accurate photometry of these galaxies in the LRI by simultaneously fitting the several HRIs to the LRI.
We begin with a library of FITS files, each containing a single object, and a corresponding catalog of xy positions and isophotal fluxes. These images have been cut out from a high resolution image containing many objects which has been cataloged by a tool such as SExtractor (Bertin & Arnouts, 1996) that produces a segmentation map or some other means by which to exclude pixels that don't belong to the object in question from the cutout. The local sky background has been subtracted from each cutout, and for convenience, each cutout has also been normalized to have a total flux of 1. Each FITS file contains a WCS that accurately locates the image on the sky.
We convolve each cutout with a convolution kernel selected to transform the HRI to the LRI. If the HRI is sufficiently high resolution, it can be treated as ``perfect'', and the point spread function (PSF) of the LRI can be used as the convolution kernel. Since the LRI is generally at a coarser pixel resolution than the HRI, we rebin the convolved images to the LRI pixelation. The resulting library of FITS files, convolved to the spatial resolution and rebinned to the pixel resolution of the LRI, are referred to as templates.
We now consider an area of the LRI that is large enough to contain all the light produced by an object close to its center, but no larger. This area is typically twice the FWZI (full width, zero intensity) size of the PSF, and is likely to contain several objects. Such a region is referred to as a cell.
Using the aforementioned catalog of the high resolution image to guide us, we select the templates for all objects which occur in this cell. We use their WCS information to place them correctly with respect to the LRI. This allows us to create a matrix equation
LRI = A0 + A1T1 + A2T2 + ... + AnTn
where the Ti are the templates for the n objects that occur in the region. (Actually, Ti = Ti/Ei, where Ei is the RMS error value for the relevant pixels in the LRI.) We solve this equation to determine the best fit flux for each object in the LRI, plus a background term for the cell.
The flux that is produced in this way is something like an isophotal flux, but there is a subtlety if the bandpasses of the LRI and HRI differ, because the isophotes are defined by the templates, and are thus defined in the band of the HRI.
In addition to an output catalog, these fluxes are used to prepare a model image, or collage, using the templates as a diagnostic product. The collage is then subtracted from the LRI to produce a residual image. This residual image is the primary diagnostic product. It can be inspected for accurate background subtraction, PSF matching, and registration. Objects which appear in the LRI but do not have corresponding detections in the HRI will also appear in this residual image.
If, as is frequently the case, the LRI has astrometric properties (such as distortion) not captured by the WCS CD matrix, this will be evident from the residual image. In a final diagnostic step, the collage is cross-correlated against the LRI in rectangular zones. New transfer kernels are then created which incorporate this shift, and the convolution and fitting stages can be repeated to obtain improved results.
TFIT requires some careful preparation of the data sets before you attempt to run. Specifically these requirements are:
· Both lores and hires images must have a valid WCS specified in RA and Dec
· The CRVALs of both images must be identical values.
· The CRPIXs of both images must be half-integer values.
· The images must not be rotated with respect to each other.
· The pixel scale of the lores image must be an integer multiple of the pixel scale of the hires image.
· The lores image should be background-subtracted as accurately as possible.
You will also need to prepare at least one transfer kernel suitable for convolving the hires image cutouts to produce templates that are psf-matched to the lores image. Accuracy of this kernel will significantly affect the accuracy of the final results. If the images have very different resolutions, one may use the PSF of the lores image; otherwise, the PSF of the hires image should be deconvolved from the lores PSF. The resulting convolution kernel must be:
· a FITS image with a pixel scale identical to the pixel scale of the hires image
· centered in its array
If your lores image has significant spatial variation in its PSF, you may prepare multiple kernels, each in its own FITS file, along with a lookup table defining the rectangular zones on the lores image that correspond to each kernel.
To run TFIT, you need to have:
A sample task parameter file will be installed in the param/ directory. (It can also be found in the lib/
directory of the source tree.) The parameters are divided into sections, and
each section is annotated with explanations of the parameters. You should copy
this file into your work area and edit it appropriately.
An annotated version of the task parameter file is presented in an appendix to this document.
Many of the parameters are filenames, and many of these
files function as the output from one stage and the input to another stage. This must be well understood in order to
successfully re-run parts of the pipeline. The information is included in the annotations and also in the table below.
The TFIT pipeline is designed to be run from the shell command line. It produces a great deal of output that is useful mainly for debugging when something goes wrong, so the output should be directed to a file. If installed according to the directions in this manual, the command usually given is:
tfit taskparfile.param >& logfile.log &
TFIT produces the following files:
In addition to the TFIT pipeline itself, the TFIT software package provides several utility scripts that can be run to provide additional diagnostic information.
This script can be used to apply the results of a registration dance to a new convolution kernel. It generates a new set of shifted kernels and a new lookup table that can be used by a pass2 (multikernel) run. This may be desirable if significant PSF improvements are made after the first pass through the pipeline has already been run.
applydance.py <name of new kernel file> <ddiags filename>
This script generates a DS9 style region file that can be used to visualize the boundaries of the overlapping cells.
cell2region.py <name of tfitcell file>
This script generates individual FITS files containing the covariance matrix for each cell from the covariance diagnostic file produced by the fitter. These files can be examined with any FITS viewer, and are used by the add_covar_info.py script.
covar_proc.py <tfitcovar file> <dirname for covar FITS files>
script can be used to generate a collage (named collage.fits) from a
set of templates and a _best file, without rerunning the diags stage
and re-creating the _best file. It can be useful if the diags stage
crashed after creating the _best file but before successfully writing
It can also be used without applying the fitting results contained in the _best file, to generate a collage of normalized templates. This is rarely but occasionally useful.
If the _best file is not supplied, the collage will be made from all templates in the directory; if it is supplied, the _best file will be used to select the objects.
make_collage.py <template directory> <lores filename> [<tfitcat_best file>]
script can be used to generate two matching collages named
hrpsf_collage.fits and lrpsf_collage.fits. Both collages have the
fitted scale factor supplied; both are at the pixel resolution of the
hires image. The hrpsf_collage.fits file is created from the
unconvolved cutouts; the lrpsf_collage.fits file is created from the
convolved, but high-resolution, templates which may be optionally
created in the convolve stage.
The resulting pair of collages may be useful in PSF investigations.
psf_collage.py <cutout directory> <hr_template directory> <hires filename> <tfitcat_best file>
This script can be used to add the covariance index (in a column named maxcvratio) to the "best" file. The object id of the most influential object is also added (maxcvid), as is the signal to noise (sn). This information can be used when working with the catalog to identify objects that are effectively isolated or whose photometry is significantly affected by their neighbors. You must first run the cover_proc.py script to generate individual FITS files for the covariance matrices.
set_covar_info.py <tfitcat_best file> <dirname for covar FITS files>
Note that objects that were very bright in the hires image are unlikely to subtract well, if the template is dominated by instrumental effects rather than intrinsic object morphology.
The table below summarizes the actions performed within each
of the TFIT processing stages, and identifies their input and output products
by the associated keyword in the task parameter file. The filenames of some
output products are constructed from others; this is indicated by providing the
prefix or suffix in <angle brackets>. Square brackets indicate optional products that may be produced depending on the task parameters.
Note that the psflookup file is an output from the dance stage, and an input for the convolve stage only when the multikernels option is set to true. The posframe parameter determined whether the pixel coordinates in the psflookup file are in hires or lores coordinates. When using a file produced by the dance stage, posframe should be set to lores.
Cut out objects from HRI
Create template library:
Transform catalog to LR frame
Cull template library
Generate grid of overlapping
Select a single best fit
value for each object
Cross-correlate zones of
model image and LRI
Plot shifts computed by dance
The TFIT pipeline software permits the user to specify any or all of the processing stages in any order (although the user is responsible for sensible ordering; the software doesn't check for this). Thus, it's easy to customize the processing for a given situation. This section discusses some common use modes.
The cutout stage assumes that each of the hires and lores images are contained in a
single file. For very high resolution data of large fields, however, this may
not be the case. In this case, each hires image should be processed separately,
through the cutout stage only, using a single input catalog so as to
avoid object duplication or name collisions. Then all cutouts can be combined
into a single directory, and subsequent processing can begin from the transform
If no hires image exists that precisely corresponds to the lores image (for example, because the high resolution data is too big to fit into a single mosaic image, so that it was processed in sections), the parameter file should be modified so that the hiresfile parameter points to a text file containing the relevant WCS information, i.e.
Similarly, one may wish to use one set of high resolution data with several sets of low resolution data. There's no need to regenerate the cutouts each time; just start with transform for each set of processing after the first.
problems such as distortion are present, the registration dance will detect nonnegligible shifts, and create a set of compensating
transfer kernels to be used. A second pass through the pipeline can then be
run, starting with the convolve stage in multikernel
mode, placing the templates into the same directory to avoid reconvolving any objects that don't need to have a shift
applied. The transform stage can be skipped since the catalog does not
change, and processing can then proceed to the cull, fit, and diags stages.
Because the dance stage generates a new set of transfer kernels, it should not be run as part of a second pass.
Note that a separate task parameter file should be prepared for second pass processing, with careful attention to any filename changes necessary to avoid overwriting first-pass products.
Sometimes due to errors, it will be necessary to restart the pipeline from a specific stage. In order to do this,
You may also want to modify any filenames for partially-completed stages. At present it is necessary to manually clean up the results of a partially completed stage.
"best" file: The
final TFIT catalog file, containing only one measurement per object. The best
measurement is determined to be the measurement from a cell in which the object
was located closest to the cell center.
cell: a rectangular zone on the lores image over with the fit stage performs a simultaneous fit for all templates contained therein
collage: a FITS file containing a model image produced by adding templates (usually scaled by the best fit fluxes) in the correct locations in an empty array. This diagnostic product provides a visualization of what TFIT determined was the best fit for all objects. Also known as model image.
convolution kernel: a FITS file containing an image which, when convolved with the hires image, would produce the lores image. Also known as transfer kernel.
convolve: a TFIT processing stage that generates templates from cutouts.
max(abs(covar[ij]/covar[jj])), where covar is the covariance matrix for
the cell. The covariance index provides an estimate, for a given
object, of how strongly affected its photometric measurement was by its
neighbors, compared to the intrinsic uncertainty in the measurement. It
depends on nearest neighbor distance but also on relative brightness:
faint objects are influenced by bright neighbors more than bright
objects are influenced by faint objects.
cull: a TFIT processing stage that compares the bounds of each template to the bounds of the lores image array. Templates that fall entirely outside the lores image are culled: removed from the catalog and placed in a separate subdirectory. Templates that fall partially outside the lores image are clipped.
cutout (noun): a FITS file containing a single object cut out from the hires image, with the segmentation map applied to exclude contributions for other objects.
cutout (verb): a TFIT processing stage that produces cutouts from the hires image.
dance: a TFIT processing stage that uses IRAF tasks to measure subpixel misregistration between the collage and the lores image, and produce a set of compensating convolution kernels. Also known as registration dance.
diags: a TFIT processing stage that produces the "best" catalog, the collage, and the residual image.
first pass: the normal mode of TFIT pipeline processing, from cutout through dance (or plotdance) stages. So named because the dance stage frequently indicates that a second pass is necessary to compensate for misregistration.
fit: a TFIT processing stage that fits the templates to the lores image in overlapping cells.
hires image or HRI: the "truth" image that will be used for a priori information in measuring photometry on a lower resolution image
hires template or hr template: a FITS file produced by convolving a cutout with the convolution kernel, and saved at the pixel resolution of the HRI. Usually normalized.
lores image or LRI: the target image on which
you are attempting to measure photometry
plotdance: a TFIT processing stage that generates a diagnostic plot from the results of the dance stage
residual image: a FITS file containing an image produced by subtracting the collage from the lores image.
second pass: TFIT processing through the convolve, cull, fit, and diags stages, using a set of convolution kernels to compensate for misregistration.
stage: a processing step of the TFIT pipeline
template: a FITS file produced by convolving a cutout with the convolution kernel, and then binning down if necessary (in a flux-conserving manner) to match the pixel resolution of the LRI. Usually normalized.
transfer kernel: see convolution kernel
transform: a TFIT processing stage that transforms the hires catalog to the reference frame of the lores image.
a script that is not part of the TFIT pipeline, but can be run
independently to perform some useful operation on the TFIT data
An annotated version of the
task parameter file is included here for convenience:
#Params relating to the hi-res image set
# These are input parameters for the cutout stage. The
# hiresfile is the image from which the cutouts will be made,
# using the catalog and segmentation information in the other
# two files.
#hicatcolnames NUMBER, X_IMAGE, Y_IMAGE, XMIN_IMAGE, YMIN_IMAGE, XMAX_IMAGE, YMAX_IMAGE, BACKGROUND, FLUX_ISO
#This is the default set of column names from the input catalog,
#used if the value of "hicatcolnames" is None.
#If you need to specify a different set of names, be sure that
#the order corresponds to the default set. All names must be
#specified on the same line. Alternatively, you can provide
#a filename as the value here, with the actual column names
#in the file, all on the same line.
#Params relating the two images
#This is the relative pixel scale between the two images.
#For example if the hires images has a pixel scale of 0.1
#arcsec/pixel, and the lores image has a pixel scale of 0.5
#arcsec/pixel, the value of relscale should be 5.
#Params relating to the lo-res image set
#TFIT can subtract a global background constant from the
#lores image before fitting. We do not recommend this; instead
#it is recommended that you background-correct the lores
#image as perfectly as possible before fitting.
#TFIT is designed to work with an RMS map as the error map,
#but it will also accept a weight map, or a single constant
#value of the RMS from which an RMS map is generated. The
#errtype specifies which kind of error image is provided.
#For best results, use a source-weighted RMS map, to prevent
#the bright objects from dominating the fit.
#The flag map is used at several points in the pipeline to
#avoid constructing templates, and fitting pixels, that contain
#bad data. At present this is implemented as a maximum value:
#templates containing at least one pixel with a flag map value
#greater than maxflag will not be created. Eventually we may
#change this to work as a mask.
#Params relating to the transfer kernel
#The transfer kernel must be a FITS image on the same
#pixel scale as the high res image. It should contain
#a centered, normalized image T such that
# hires PSF convolved with T = lores PSF.
#TFIT supports the use of multiple kernels to accommodate
#a spatially varying PSF. To use this option, set the
#multikernels value to true, and provide a psflookup file
#that divides the lores image into rectangular zones,
#specified as pixel ranges, and provides a transfer kernel
#filename for each zone. Any objects which fall in a zone
#not included in the lookup file will use the transfer kernel
#specified as psffile.
#The posframe parameter determines whether the pixel ranges
#specified in the psflookup files are in lores or hires pixels,
#and, accordingly, which catalog file will be used as an imput
#to this stage:
# - if posframe = lores, the lrfcat will be used. This is
# the normal mode for pass 2 processing.
# - if posframe = hires, the cutoutcat will be used. This is the
# mode used when using multiple kernels in the first pass run.
#Params relating to the cutout stage
#cutoutcat should be the filename containing only those columns
#from the hirescat used by TFIT.
#cutoutdir is the directory containing the cutouts.
#These are output parameters if you start from the cutout stage;
#they are input parameters if you start from the convolve stage.
#The normalize parameter determines whether the cutouts will be
#normalized or not; it is normally set to true, so that the
#final output catalog will contain fluxes, rather than colors.
#Params relating to the convolve stage
#templatedir is the directory containing the templates.
#It is an output parameter for the convolve stage,
#and an input parameter for all subsequent stages.
#The next two parameters control the optional production of
#hr templates: ie, templates that have been convolved with the
#kernel, but not binned down to the LR pixel resolution.
#If these images are created, they may (soon) be used by the
#compl_convolve stage, to apply a shift computed by the dance
#rather than re-convolving. They may also be used to construct
#an "hr collage" as a diagnostic product for use in constructing
#a better kernel.
#Params relating to the transform stage
#This is the "cutoutcat" file, transformed into the
#pixel reference frame of the lores image. It is an output
#parameter for the transform stage, and an input parameter
#for the cull stage.
#Params relating to the cull stage
#This is the "lrfcat" file, modified to exclude or modify objects
#that were culled because their templates fell entirely or
#partially outside the bounds of the lores image. It is an
#output parameter for the cull stage, and an input parameter
#for all subsequent stages.
#Params relating to the fitting stage
#The fitting stage has the most complex set of parameters.
#These are all output parameters. The "tfitpars" file
#specifies the name of the special parameter file for the fitting
#stage that will be generated from the parameters in this file.
#The others are filenames for the output catalog, cell, and
#covariance files, respectively.
#...Then control parameters
#These are the most important control parameters.
#cell_xdim and cell_ydim specify the original size of the cell,
#in lores pixels. It should be large enough to contain 2-3 lores
#PSFs. The cell will grow to avoid chopping off partial templates,
#so this is a minimum size.
#The cell_overlap is also specified in lores pixels.
#..including many that don't change too often
#These are control parameters.
# dithercell=true dithers the pattern of overlapping cells over
# the lores image, to avoid corner problems.
# cellmask=true uses a mask to exclude pixels from the fit which
# do not contain a value of at least "maskfloor" in
# at least one template. This is necessary because
# of the cell growth.
# maskfloor=1e9 should be set to a value at which it's okay to clip
# the templates.
# fitbackground=false does not fit the background in each cell. This
# is necessary because of the cell growth.
# writecovar=true writes the covariance information out to the covar
#Params pertaining to diagnostics
#This is an output parameter. It will contain the collage made by
#multiplying each template by its best flux and dropping it into the
#right place. An additional diagnostic file will be created: it will
#contain the difference image (loresimage - modelfile). Its filename
#will be created by prepending "resid_" to the modelfile.
#Params pertaining to the dance
#The first two are control parameters.
# dzonesize specifies the size of the rectangular zones over which a
# shift will be calculated. It should be comparable to the size over
# which the misregistration is roughly constant; but, it must be
# large enough to contain enough objects to provide a good signal
# to the crosscorrelation.
# maxshift specifies the maximum size of the x/y shift, in lores
# pixels, which is considered to be valid. Any shift larger than this
# is considered spurious and dropped from the final results, and
# replaced by an interpolated value from the surrounding zones.
# ddiagfile is an output parameter for the dance stage, and an input
# parameter for the plotdance stage.
# dlogfile is an output parameter; it simply contains the output from
# the crosscorrelation process.
#Param defining the pipeline
#Typical firstpass run
order cutout, convolve, transform, cull, fit, diags, dance, plotdance
#The typical firstpass run contains all the valid TFIT stages.
#There are several possible paths through the TFIT pipeline, depending
#on the state of the data. The pipeline software does not enforce
#any choices about the makeup or ordering of these stages; the user
#must be familiar enough with the pipeline to choose a path that makes
#Other possible paths include:
#order convolve, cull, fit, diags
#This is a typical secondpass run, using the multikernel option to
#regenerate templates using spatially varying transfer kernels
#generated by the dance, and omitting the transform stage because it's
#already been performed.
#order convolve, transform, cull, fit, diags, dance, plotdance
#This is a typical GOODS ACS-to-lores firstpass run, using previously
Symptom: The TFIT log file shows the following:
Uhoh, TFIT failed: exit status 32512
Probable cause: The libcfitsio.so file cannot be located
Examine the environment variable LD_LIBRARY_PATH (or equivalent on other platforms) and determine whether it contains a path to the cfitsio library.
Update the LD_LIBRARY_PATH appropriately and restart from the fit stage.
Symptom: The TFIT log file shows the following:
error map ... Reading
Probable cause: The specified error map file is missing.
Examine the parameter file for the file name & path associated with keyword loreserr. Determine whether this file exists.
Place the correct file in the directory; or correct the filename in the task parameter file.
Symptom: The following error messages are generated during the convolve stage:
Original cutout crpix1/2: -5988.5 -6409.5
Modified newwcs crpix1/2: -576.5 -619.5
Illegal value None
omitting h[cunit1] for this object
Illegal value None
omitting h[cunit2] for this object
card is too long, comment is truncated.
Cause: cunit1 and cunit2 keywords are missing from FITS headers for the cutouts.
Remedy: None needed - this is harmless, because we don't use those keywords. They are read in for completeness.
Symptom: The TFIT program runs to completion, but the output catalog is empty. Log file indicates that all objects were discarded from every cell. The diagnostic stage ends with an error like this one:
ValueError: No catalog entries were found in isaac9ks_tfit.cat_pass1. Task aborting.
Probable cause: The templatecat specified in the parameter file is incorrect.
Diagnostic procedure: Check to make sure that the templatecat file exists. Plot the xy positions of the objects and make sure they are in the lores image's pixel reference frame.
Remedy: Correct the templatecat, and rerun starting from the fit stage.
Symptom: Residual map shows areas with significantly-too-faint fits.
Possible cause: Multiple occurrences of some objects in the _culled catalog fed to TFIT.
Identify an object id in the affected region by measuring its (x,y) in DS9, then looking for an object with the same (x,y) in the tfitcat_best file.
Search for rows starting with that object id in the templatecat file; there should be only one row per object id. (IE, object ids must be unique.)
Sort the templatecat file with the unique option to ensure only one entry per object id, and restart from the fit stage.
In principle, the TFIT package should run on any flavor of Unix that also supports IRAF. However, it has only been tested only on SunOS 5.8 and Redhat Linux (RHEL 3 and 4).
Python - version 2.4 or greater
pyFITS - version 1.1 or greater
numpy - version 1.0 or greater
convolve - version 2.0 or greater
matplotlib - version 0.84 or greater (recommended but not required)
PyRAF - version 1.1 or greater
STSDAS - version 3.2 or greater
some version of make
some c++ compiler (gcc recommended)
The stsci_python package (version 2.5 or greater) includes PyRAF and PyFITS, and is bundled with numpy and convolve for convenience.
(Matplotlib is presently used only for the diagnostic step PLOTDANCE. You may equally well use your favorite plot package to plot the calculated shifts. However, it's a fairly spiffy plotting package and any additional plots incorporated into TFIT will make use of it.)
Although the TFIT package itself does not incorporate SExtractor, it relies on the object catalog and segmentation map generated by SExtractor. SExtractor can be obtained at
TFIT now uses the Python distutils package to simplify software installations.
1. Download the gzipped tarfile and unpack it in some convenient location. This is
the source directory, and is not the same as the installation directory.
2. Make sure that the LD_LIBRARY_PATH includes the location of the libcfitsio.so library; and the LHEASOFT environment variable is set to the parent directory. This will happen automatically if you have the LHEASOFT or HEASOFT6_64 packages installed on your system.
3. Check to make sure there exists a src/Macrodefs.<OSname> file for your OS,
and that its definitions are appropriate for your system. Create and/or modify
if necessary. (If you need to create a new Macrodefs
file, please send it to us so we can include it in future distributions.)
4. python setup.py install [--local=/some/local/place/]
This command builds the C++ program, pre-compiles the python
code, and places all scripts and executables either in the standard places, or
(if the --local option is used) in the specified directory tree.
(Note: C++ and its libraries are notoriously finicky. You may need to edit the Macrodefs or Makefile for your particular setup. At present, even a clean build generates a number of warnings.)
5. If you use the --local option, you must ensure the following variables are set in order to correctly run TFIT:
PATH to include /some/local/place/bin/
PYTHONPATH to include /some/local/place
6. To verify that the package has been built and installed correctly, you may download and unpack the test set provided, and follow the instructions for testing.