*************************
TABLES APPLICATIONS CHANGES
FITSIO:
- General Changes
Added .dta (otfc delta files) to the list of ascii extensions.
Fixed problem with using the TNULL parameter in fits table extensions.
Fixed problem in converting NaN to INDEF.
TTOOLS:
- General Changes
The following tasks were modified to set the input to STDIN or the output
to STDOUT, without getting the cl parameter, if the I/O is redirected:
imtab, tabim, tcopy, thistogram, tinfo, tintegrate, tlcol, tlinear,
tprint, tproject, tquery, trebin, tselect, tstat, ttranspose.
The 'columns' parameter in ttools tasks now accept either ! or ~ as the
negation character.
For ttools tasks that create an output table based at least partly on an
input table, if the input and output are in FITS files, the primary header
of the input will be copied to output, under some conditions. The primary
header will be copied if the output file doesn't exist yet, and if the input
primary data unit is empty (i.e. if the PHDU is just a header).
- TCHCOL
Warning messages about changing column name and/or units for text tables
were removed, since this is now possible.
- TCOPY
The line that is printed by tcopy in verbose mode, giving the names of the
input and output table names, is now preceded by "# ". This is to prevent
problems when piping text tables through tcopy.
- TCREATE
Column attributes are now possible for text tables. See the description
in the TABLE I/O section below for more details.
In the data file for tcreate, the data values corresponding to an integer
column are now allowed to be expressed as floating point numbers, as long as
the fractional part is zero. For example, 7.00 and 300.e-2 would be allowed.
Another change to tcreate is that it can be used to create a text table,
in particular, a text table with explicit column definitions.
- TEDIT
After getting the name of the original table, any expressions in brackets
appended to the file name are now stripped off before calling the function
to get the name of the directory.
Fixed bugs in the cut/paste buffer when using fits files and in the
set command.
- TEXPAND
Added code to handle short integer columns and array columns. Also
changed to ignore rules that do not apply to a table instead of
flagging them as errors.
- TINFO
A new task parameter, subtype, has been added to tinfo, and the values for
the table type, tbltype, have been modified. The table types "row" and
"column" have been changed to "stsdas", with subtypes "row ordered" and
"column ordered" respectively. For text tables, the subtypes are "simple"
(plain ascii table) and "explicit column definitions" (if the file has
column descriptor lines). Tables in FITS files have subtypes "ascii",
"binary", and "primary header" (which can be opened as a table just to
access the keywords). If the subtype differs from the default (simple,
row ordered, or binary), it is now printed on the same line with the
table type.
- TJOIN
Added support for outer joins, joins on multiple columns, and joins on
row numbers.
- TLCOL.PAR
The default value for the 'nlist' parameter was changed from 1 to 4.
- TMATCH, TPRODUCT
Changed so that primary header of FITS files is copied.
- TMERGE
The algorithm for handling column names for text tables was modified.
This was made necessary by the introduction of explicit column definitions
for text tables.
- TPROJECT
Modified so that row order is not changed when unique option is
specified.
- TPRINT
A new "html" option has been added to tprint. The output can be redirected
to a file with extension ".html", which can then be loaded directly into
a browser. For option = "plain" or "html" (but not "latex" or "tex" yet),
if any column that is to be printed contains arrays, all array elements
will now be printed; previously, only the first element was printed.
Boolean values in FITS tables can be true, false or undefined. Tprint
now prints INDEF for undefined boolean values.
- TQUERY, TSELECT
Changed to reduce memory usage.
- TREBIN
A bug was fixed in trebin for the case that the input data are in decreasing
order. The test on whether the input value (at which the data should be
interpolated) was outside the range of the input data was incorrect for this
case.
- TUNITS (new task)
A new task that converts a table column from one set of units to
another.
- TUPAR
In the tupar task, when the value of an existing keyword is changed (with
the "p" instruction), the data type will not be changed unless it was
explicitly given. There's now more flexibility for specifying/printing
Boolean parameters.
TABLES LIBRARY CHANGES
TABLES I/O routines:
CFITSIO is now used instead of SPP FITSIO. CFITSIO is being better
maintained than the SPP version of FITSIO and is also about twice
as fast as FITSIO. CFITSIO does not support IRAF networking, however,
so if that is needed, the library should be built using SPP FITSIO.
The mkpkg file now has an ifdef on SPPFITSIO. If this is defined, then the
SPP version of FITSIO will be compiled and included in the libtbtables.a
library; otherwise, the SPP to C interface tbfxff.c will be compiled, so
that calls to FITSIO routines (beginning with fs) will link with CFITSIO.
The value for sppfitsio is arbitrary; mkpkg only checks whether it is
defined. The sppfitsio flag is used in the following way:
mkpkg -p tables sppfitsio=yes
For the CFITSIO version, use:
mkpkg -p tables
Two new features were added for text tables: (1) Explicit column definitions
can be given in the file. (2) Header keywords can be specified. A table
"subtype" was introduced, and macros were added to tables$lib/tbset.h to
access this information. Text tables are either TBL_SUBTYPE_SIMPLE (the
old kind, except that they can have keywords) or TBL_SUBTYPE_EXPLICIT
(if they have explicit column definitions). The subtype can be gotten
using tbpsta with macro TBL_SUBTYPE, and the subtype can be set to
TBL_SUBTYPE_EXPLICIT.
The table subtype applies to FITS tables as well. The subtypes are
TBL_SUBTYPE_ASCII, TBL_SUBTYPE_BINTABLE or TBL_SUBTYPE_IMAGE (primary
header). The subtype can be gotten with tbpsta, but it cannot be set,
as ASCII tables are supported only for input.
The maximum line length for a text table has been increased from 1023
to 4095. The last line of a text table is no longer required to end with
a newline.
The name of a file containing FITS tables is no longer required to have
extension ".fits", ".fit" or ".??f". (The filename extension still must
be one of those when creating a new FITS table, however.) The fact that
an existing file is in FITS format is now determined by reading the beginning
of the file, rather than relying on the filename extension.
- further information about changes to tables:
1. Several of the ttools tasks have been modified so that if the input
and/or output are redirected, the cl parameter is automatically set to
STDIN or STDOUT respectively, rather than prompting for a value. This
makes it easier to pipe text tables through the tasks, e.g.
tselect a.fits expr="x > 0" | tstat column=x
Note: This caused a problem with a user's cl script. He was calling
tcopy with output redirected to dev$null to prevent messages being sent
to the screen: tcopy ("a.tab", "b.tab", verbose-, >& "dev$null")
This redirection of output resulted in the output table being set to
"STDOUT" and then sent to dev$null. Setting verbose to no was all that
he needed to prevent messages from being printed.
2. Two new features have been added to the table I/O routines for text
tables, explicit column definitions and header keywords. These are both
optional; simple text tables are still supported just as before. Column
definitions means that real column names can be used, not just "c1", "c2",
etc., units may be specified, and you can set the print format. See
tables$doc/text_tables.doc for further information.
Note: This can cause a problem if you convert from an stsdas or FITS
table to an ascii table and then try to access the columns in the ascii
table as c1, c2, etc. In the past that would have worked, but now their
names will have been inherited from the input file and will likely be
different from c1, etc. If a.fits has columns x, y, z, for example,
you might have used something like the following in the past:
tt> tselect a.fits STDOUT "x > 0" > a.txt
tt> tstat a.txt c3
But now you would need to use:
tt> tstat a.txt z
Header keywords have the following syntax:
#k keyword = value comment
The "#k " must be the first three characters of the line, and the space
following "k" is required. The "k" is not case sensitive. Header keywords
can be added to any text table, and they can appear anywhere in the file.
Column definitions have the following syntax:
#c column_name data_type print_format units
The "#c " must be the first three characters of the line, and the space
following "c" is required. The "c" is not case sensitive. Aside from the
"#c ", the syntax is the same as the output from tlcol or the input cdfile
for tcreate. Only the column name is required, although in most cases you
will also need to give the data type (the default is d, double precision).
If any column is defined this way, all columns in the file must be defined,
and all column definitions must precede the table data.
3. The following describes low-level changes.
lib/tbtables/tblfits.h
/fitssppb/fitsio.h
/fitsio_spp.h
The macros SZ_FTTYPE, SZ_FTFORM, and SZ_FTUNIT were increased to 70.
All three files contain macros with the same names, used by different
sets of code (tbtables, tbfxff.c, and FITSIO).
tbtables.h
A new member of the tbtables structure was added, COL_NELEM. A TB_FILE2
element was added just after TB_FILE, to support eight-byte pointers for
CFITSIO on certain platforms. TB_OS_FILENAME was added. This is used by
CFITSIO, since that package uses C file I/O rather than IRAF file I/O.
Changes were made having to do with table subtypes, for FITS tables and
for text tables.
tbltext.h
This is a new file. It contains the SZ_TEXTBUF macro, which was previously
in a source file, and it contains macros used for text tables with explicit
column definitions and/or header keywords.
tbfrcd.x was rewritten to read all header keywords that are used for column
definitions in one call, instead of reading specific keywords one column
at a time. Tbuopn was modified because it calls tbfrcd.
tbept.x
The TB_MODIFIED flag was not being set to true. This bug could result
in a text table not actually being changed after calling one of these
routines.
tbfag.x
A bug was fixed in this file. The routines were trying to get all nelem
elements, regardless of the starting element or the actual number of
elements in the array. These have been modified so that nret is set to
the minimum of nelem and the number of elements from the first requested
to the array length.
tbfap.x
The tbfapt routine in tbfap.x was modified to fix a bug. In the section
for column data type short, the index j was not being incremented, so when
an array of values was being written, the values were all being written to
the first element.
tbfnew.x
When creating a new FITS file, the keyword NEXTEND is now added to the
primary header, with a value of one. When adding an extension to an
existing FITS file, NEXTEND is updated (or added, if it doesn't exist)
to be the number of the extension that is being added. However, NEXTEND
is neither modified nor added under either of two conditions: if the
primary header/data unit contains a non-null data portion, or overwrite=yes
was specified in order to replace an existing table extension.
tbrcmp.x, tbrcsc.x
Boolean elements are gotten using a short integer buffer, in order to
preserve indef values, since FITS tables support indef boolean.
For entries that contain arrays, tbrcmp now compares all array elements,
or as many as needed to decide which of the two entries is larger.
lib/tbtables/selector/tcsopen.x
The pattern matching code used to match column names containg
wildcards was incorrect and was revised. Also, the negation character
was changed so that either a tilde or a bang could be used to agree
with the row selector syntax.
TTOOLS
pltpar.par
The 'errtype' parameter was deleted, since it was not used by the code.
ttools/mkpkg
was added as a dependency for several files.
GFLIB
Library was rewritten to replace it with a version that can portably
access all image formats. The old version of the gf library, which
could only handle GEIS files, was renamed to gilib. It continues to
be used under gflib to access GEIS files and is also used by some
other software which only needs to access GEIS files, such as strfits.
STXTOOLS
- Several bug fixes were made to support template expansion using fits
files.
- The expression compiler was changed to store constants and function
names in the pseudo-code, just as is done with synphot. This enables
the compiler to set a strict upper bound on the length of the compiled
code, eliminating "expression too complex" errors.