STScI User's Guide to the IRAF FITS kernel

Changes from previous versions

The FITS kernel as released as part of IRAF v2.11 has a couple of significant changes to its behavior from those previously used at STScI. This version of the FITS User Guide has been revised to include those changes and specifically point out the change from the previous versions. In particular, the FITS kernel now requires explicit identification of input extensions if the file is not a simple FITS file; the defaulting mechanism has been eliminated. Extensions are no longer appended to output files by default.

What is the big deal about the "FITS kernel" in IRAF? Why do I need to know anything about it? You may not, but if you will be getting data from STIS or NICMOS, and you want to inspect or analyze it using IRAF/STSDAS/TABLES, you will need to learn something about it.

In a few sentences: The FITS kernel allows IRAF to access FITS files directly without having to go through the muss or fuss of converting them into '.imh' or '.hhh' files. Most IRAF/STSDAS/TABLES tasks that access images are able to access images in FITS files just as easily through the FITS kernel as other IRAF image formats (but see FAQ question 6 if trying to access FITS files created from WFPC, WFPC2, FOS, GHRS, or FOC data).

OK, great. The FITS kernel lets me access images in FITS files easily. But why do I need to learn anything about it? Isn't this all transparent to the user? We wish it was, but the simple fact is that there is a lot more to it than that, especially when it comes to using HST data. Unless you understand much of what follows, you will be destined for headache and confusion. If you know what is good for you, READ THIS BEFORE YOU USE FITS FILES IN IRAF. One exception: if the only FITS files that you will be using do not have any extensions and you will not be writing new FITS files out, you may go. But virtually all HST data in FITS format does not fit in this exclusion. Remember, you were warned.

One last introductory comment. You will see that the FITS kernel on occasion requires complicated syntax as part of running an application, and yet there is no mention of the syntax in the help for the application. This syntax, like other system level syntax, e.g., image sections, is associated with the image kernel, not the application.

There are two basic reasons for the complexity of dealing with FITS files in IRAF. Briefly, the first has to do with the fact that FITS files can contain multiple images whereas IRAF's image kernel only deals with single images. Some mechanism is necessary to deal with the selection and manipulation of individual images within a FITS file that contains more than one. The second complication deals with keyword inheritance issues which will be discussed later.

FITS files can contain any number of 'extensions' after its first header/data unit. Each extension itself consists of header/data units essentially of the same form of the first. The first is called the "Primary Header/Data Unit" or PHDU for short. Likewise, those for extensions are called EHDUs (for...you figure it out). These extensions may be of different types, the most likely ones you will encounter are IMAGE or BINTABLE. The FITS kernel is only concerned with IMAGE extensions, though it must properly recognize the existence of all extensions and skip over those that are not of IMAGE type.

The use of IMAGE extensions allows one to package several images into one file. Unlike the disk format for previous STScI data products (GEIS, i.e., .hhh), the different IMAGE extensions contained within a FITS files are not required to be the same size or type, nor do the headers need to contain the same keywords. Because several images can be combined into one file, there needs to be a means of specifying which image IRAF is to get. If this was all there was to the issue, the solution would be straightforward; use the same means as is used to specify group number in GEIS images, i.e., by using the extension number in brackets (e.g., infile.fits[3]). And indeed that will work for selecting image extensions in the IRAF FITS kernel.

But there is more to it than that. Both STIS and NICMOS generate sets of image arrays for each exposure or readout. Typically, there is a science image, data quality image, and error image in every set. (NICMOS has two more, a samples image and integration time image). Each of these is saved as a separate extension in a FITS file. Such an association of FITS extensions will hereafter be referred to as an image set or imset for short. Furthermore, in the case of both instruments, particularly NICMOS, there may be several of these imsets within one FILE. Specifying the 5th science image becomes more awkward if only a numbering system is used to specify extensions. What's more, such a scheme is heavily dependent on the order of extensions remaining unchanged, something that cannot be counted on.

For this reason a means of selecting extensions by name also was chosen and implemented in the FITS kernel. How it works will be described a little later. Just keep in mind that in many respects a FITS file is now somewhat analogous to a directory so many of the same operations needed for manipulating files within a directory are now needed to manipulate extensions within a FITS file. Consider the issues of what should be done when copying an extension to another FITS file. If one already exists, should the file be replaced, or should the extension be added? What if there is already an extension with the same name in the output file? How does one insert or overwrite extensions in the output file?

These questions will be answered, but the main point is that unless you think of these FITS files this way, you may find how the kernel works hard to understand. What you think is the obvious outcome is not necessarily so. One user may expect that the output file should be overwritten, while another one expects that it should be appended to. With the packaging aspects come inevitable ambiguity that needs resolution and hence more complicated syntax. This is the price of a more flexible file format.

The second cause of complexity has to do with 'keyword inheritance.' Since HST data sets contain large numbers of header keywords, it was desired that most of these keywords not be repeated in every extension header but somehow be treated as though they were. This inheritance scheme places 'global' keywords in the Primary header and when the image kernel opens an extension image, it will automatically include all the global header keywords (with some reasonable exceptions) in the extension header if the inheritance feature is enabled. Adding this feature has introduced complexity and will result in some unexpected behavior if the user is unaware of how it works.

The simplest way to select an extension is by number. The PHDU is referred to by 0, the first extension by 1, and so on. When a FITS file does not contain any extensions and it has data in the PDU, then it is not necessary to specify an extension as part of the file name. But if that is not true, it is necessary to specify the extension (note: this is contrary to earlier versions of the FITS kernel in use at STScI!)

The name of an extension is specified by the combination of the value of two extension header keywords, EXTNAME and EXTVER, which take a string and integer value respectively. If only EXTNAME is specified, the first extension that matches the EXTNAME is opened, otherwise the FITS kernel selects the first extension that matches both. Use of EXTVER was chosen so that could be used to refer to the nth imset, and EXTNAME used to name components of that set (e.g., SCI for science data). For all intents and purposes, EXTVER represents imset for HST data; they are one and the same.

For selecting an input extension the syntax is of the form

fitsfile.fits[<extension number>][<keyword section>][<image section spec.>]

All of the bracketed sections are optional. The last bracket is for the usual image section specification. The <keyword section> bracket may contain a number of items though it is primarily used to specify the extension name. That is the only use that will be shown in this section. When specifying an input extension, specifying both the extension number and extension name would be redundant. Note that the FITS kernel saves EXTNAME with the exact case that a user specified to create the name, but it ignores case when trying to match names. In other words, if EXTNAME="Sci", use of "SCI", "sci", or "Sci" would be sufficient to match that extension.

fitsfile.fits				# This only works if the FITS file
					# does not have any extensions,
                                        # otherwise an error will result.

fitsfile.fits[3]			# get the third extension, if it is
					# not of type IMAGE an error will
					# result.

fitsfile.fits[SCI]			# get the first image extension that
					# has EXTNAME=SCI (EXTVER is ignored)

fitsfile.fits[ERR,3]			# get the first image extension that
					# has EXTNAME=ERR, EXTVER=3
					# Note that the value of EXTVER
					# does not imply order within the
					# file.

The next two examples show equivalent forms of the last two examples

fitsfile.fits[EXTNAME\=SCI]	        # You need the backslash only if you
			   		# are in the IRAF cl.
fitsfile.fits[EXTNAME\=ERR,EXTVER\=3]

"fitsfile.fits[EXTNAME=ERR,EXTVER=3]"   # a way of avoiding backslashes before
                                        # the '=' while in the CL is by
                                        # placing double quotes around the
                                        # filename specification.

Specifying output extensions:

By default, if an output file is specified and does not exist, a new file will be created. If the file does exist, it will either result in an error or cause the file to be overwritten depending on the value of 'imclobber' (yes: overwrite, no: error, default: no). Note: this is contrary to the behavior of previous versions of the FITS kernel at STScI where the default action was to append to existing files. If the user desires that the extension selected be appended to an existing file, they must specify the 'append' option as part of the output file specification (see examples). For the most part, it is not necessary to specify the output extension by number when appending. If an output extension name is specified, that is the new name of the extension. An example:

imcopy fitsin.fits[SCI,3] fitsout.fits[FLAT,2,append]

This copies the first extension that matches EXTNAME=SCI and EXTVER=3 from file fitsin.fits to the end of the existing file fitsout.fits; this new extension will have EXTNAME=FLAT and EXTVER=2. If the file fitsout.fits does not exist then a null PDU (i.e. NAXIS=0) will be created plus an extension containing the input extension; again, with the same extname and extver as above.

If instead:

			
imcopy fitsin.fits[SCI,3] fitsout.fits[append+]

Please use the catfits utility to list the extensions of a FITS file. This will give you one line per extension. If you want to see the Extension number and the EXTVER in this listing then you will need to edit the parameter file for catfits as follows (this is unnecessary for STSDAS/TABLES v2.0 since this is the default format):

 
  cl> tables
  tb> fitsio
  fi> cd    			   # to return to home$
  fi> copy fitsio$format.mip .     # to make your own copy.
  fi> edit format.mip
      insert before the 1st line: EXT#  -5.5
      insert after FILENAME line: EXTVER -5.5
      delete the INSTRUMENT line.
      done
   fi> catfits.format_file="home$format.mip"     # to update your catfits 
						 # parameter file.

   fi> catfits file.fits     # Now you can see the new EXT# and EXTVER
			     # The EXTNAME is the FILENAME column.

Normally the kernel will prevent you from creating extensions that duplicate existing extensions in the filename so that uniqueness is preserved.

One can use various keywords in the keyword section to specify optional behavior. Their use is shown primarily in the following examples.

First, details on keyword syntax:

For boolean options (inherit, noinherit, overwrite) the following forms are usually acceptable:

option			equivalent to option=yes
option=yes
option=no
option+			equivalent to option=yes
option-			equivalent to option=no

1. By default, inheritance of the primary header keywords can only occur when the primary data unit is null.
2. An extension inherits the primary header by default only when the extension header contains a keyword,value combination of INHERIT=T.
3. FITS-required keywords are not inherited.
4. Keywords in the primary header that appear as well in the extension header will take their value from the extension header always.
5. The Commentary keywords COMMENT and HISTORY will never be inherited.
6. Changes to an inherited keyword only appear in the output extension header, not the primary header. This can cause confusion (e.g., see FAQ question 1). One example is that hedit normally requires add=yes to add new keywords to a header; however, applying hedit to a primary header keyword results in the updated keyword appearing in the extension header if inheritance applies, even without add=yes.
7. After the inheritance operation is performed, the default action is that INHERIT=F is set in the writing of an inherited image header to a new file. (i.e., inheritance turns itself off on propagation). This is not the case if an operation is 'in-place,' e.g., hedit has been used to edit a header

Examples

imheader stis_data.fits[dq,inherit\=no] long=yes

We assumed that the extension header with EXTNAME=dq has INHERIT=T; then 'imheader' will display only the extension header. Without inherit=no, the display will have a merged global (i.e., primary header) and extension header if INHERIT=T in the extension header and the PDU is null.

imcopy in.fits[24][noinherit][100:400,*] out.fits[6][overwrite,inherit]

The input image subsection from extension not containing the PHU keywords (noinherit) will overwrite the existing extension number 6 in the output file, and the output extension header will inherit the output file's PHU. Notice that the old output extension 6 need not be of the same size, or even an IMAGE extension.

imsum in.fits[im1,1],in.fits[im1,2],in.fits[im3] sum.fits[sum123,append]

1) I edited the keyword in an image, but the program used the old value. What happened?

If the science data is in the first extension and the keyword being edited is in the primary header, the edited version will appear in the extension header. Some programs, however, explicitly look at the value of the primary header keyword which has not changed (e.g., CALSTIS or CALNIC). To change the primary header keyword it is necessary to do: hedit file.fits[0]

2) I used imcopy to copy my science image to another file. When running an analysis program on it, it complains that it can't find the data quality file (or some other file).

Some programs expect to see a set of extensions corresponding to the science extension (i.e., the imset). The task imcopy (like the vast majority of IRAF tasks) deals with only one image at a time unless an explicitly named list is given. Your use of imcopy once only copied one of the elements of the imset and not the rest.

3) I just copied this image to an output FITS file but when I display the FITS file, I see a different image. What gives?

Most likely, you copied your image to an existing FITS file. Instead of warning you about an existing file, as IRAF will do with other image formats, it simply appends it. If you display the file, it will use the old image by default. Get into the habit of running catfits on these files if problems arise to see what they really contain.

4) Editing a header or overwriting an extension takes a really long time. Why?

It is probably because you have changed the size of one of the data units or headers in the file. When that happens the whole file must be rewritten. This can take a while if it is very large.

5) When I try to display a NICMOS TIME extension, I get an error message. Why?

The STIS and NICMOS pipelines have adopted a convention to save disk space when an image array contained in an image extension has a constant value. In that case the extension may have no data at all and the extension header indicates the size of the array and the constant value contained in it. Although the pipeline routines and some other tasks that were developed to use these data are aware of this convention and automatically generate the constant array when the extension is read in, the FITS kernel does not. Thus the task complains when it sees that there is no data array.

6) When I try accessing WFPC2 FITS files from the archive I get error messages of various sorts (extension not found, array not two-dimensional, etc.). I thought this was able to access FITS files?

There are FITS files and then there are FITS files. The GEIS files produced for the old instruments were saved as FITS files, but pretty strange FITS files. For example, a WFPC2 data file was converted into a 800x800x4 three-dimensional array, and the group parameters appended as a table. This retains all the previous information, but it is not organized in a way that makes accessing the separate chips the same as if they were in GEIS (.hhh) files. The simplest thing to do is to convert the FITS file back into GEIS format.

7) Not all features of the FITS kernel work under earlier versions of the FITS kernel. In particular, a number of things (like overwrite) do not work on the version being used in OPUS.

8) I get an error now when I don't specify the input extension. This used to work before; why not now?

Yes, it used to work, but NOAO decided that it was dangerous to allow such defaulting on files with extensions since users may inadvertently copy only a part of the data file without realizing there were more data in the file. Now a user must explicitly specify which data entity of a FITS file they wish to access if it is not a simple FITS file (i.e., no extensions).

9) Previously, the FITS kernel appended extensions to the specified output file, now it gives me an error message (or worse, overwrites the file).

NOAO changed the default behavior. Now a user must explicitly state that they want to append to the output file by use of the append option. Otherwise, the kernel will behave in the same way it does for other file formats.

10) I have FITS files with extensions different from .fits; how do I access them?

IRAF v2.11 uses a new IRAF environmental variable imextn to define which file extensions map to which file types. The following example shows how this works.

imextn = "oif:imh fxf:fits,fit plf:pl qpf:qp stf:hhh,??h"

This is the default value. What it says is that any file that has a .imh extension should be considered an OIF (Old IRAF Format) file, any file with an extension of .fits or .fit should be considered a FITS file (identified by the name fxf), any file with an extension of .pl is a pixel list ( plf), any file with a .qp extension is a QPOE format file (qpf), and any file with an extension of .hhh or any three character extension ending in h is a GEIS (or ST) format file. Note that these are listed in search order. Obviously .imh matches the requirements for GEIS files also, but since it appears first, it is associated with OIF. Users wishing to add other extensions for FITS files can edit the above example to suit their needs.

11) The way that IRAF creates filename extensions seems to have changed. How does it work now?

(To be filled in, it depends on which version of the kernel is installed.)