STSDAS v3.3 Full Release Notes CHANGES TO APPLICATIONS ANALYSIS.DITHER The versions of PyDrizzle and MultiDrizzle released with STSDAS V3.2 have been deprecated, and renamed as 'opydriz' and 'omdriz' respectively. They have been replaced by new, redesigned versions of PyDrizzle and MultiDrizzle, versions which have been installed for use in the HST data processing pipeline and archive. These obsolete versions have been updated from what was released with STSDAS V3.2, although those updates do not contain all the fixes and revisions included in the redesigned versions. Furthermore, 'opydriz' and 'omdriz' will be removed from all future versions of STSDAS. Python tasks have also been written as part of the redesign of MultiDrizzle, with only the Python code necessary for setting up the IRAF EPAR interfaces being distributed with the Dither package itself. The Python code for these tasks, including PyDrizzle and MultiDrizzle, now gets distributed and installed from the STSCI_PYTHON package. - DRIZZLE Drizzle was updated from Version 2.92 to Version 3.4. This new version includes: - double-precision computations for WCS keywords - separate I/O routines for IRAF and C-based interfaces - F2C generated versions of all non-IRAF based code and Python C-API module used for building Python interface for 'drizzle' and 'blot' tasks without requiring IRAF - changes to the code to allow it to compile under Mac OS X and Windows, as well as Solaris and Linux - added 'wcs2dr', 'wtranback', 'wtraxy', and 'wdrizzle' tasks to the Dither package, tasks which rely on WCS of an image to specify the output frame. - updated IRAF parameter files for 'traxy' and other coordinate transformation tasks built as part of 'drizzle' - added support for distortion correction images, if specified in headers (see below for description of new parameters) This reorganization now supports both the full IRAF distribution of tasks, as well as the building of a Python-callable version of 'drizzle' and 'blot' which does NOT require the use of IRAF or PyRAF in any way. This new callable interface can be built under Solaris, Linux, Mac OS X, and even Windows. ************************** IMPORTANT INTERFACE CHANGE: This version of drizzle introduces two new parameters: xgeoim, ygeoim These parameters allow the user to specify separate additional distortion correction images that provide non-linear corrections to any distortion model specified in the 'coeffs' file. Although the default value for these will be blank and these are strictly optional, it is recommended that you 'unlearn drizzle' and check any script you may have to include the new parameters. - MULTIDRIZZLE The Multidrizzle software automatically creates cosmic-ray cleaned, geometric distortion corrected, mosaics from multiple input images. Multidrizzle was originally released as part of STSDAS v3.2 as a large, complex, Python/IRAF script. Since that time, Multidrizzle has been completely re-implemented as an object-oriented application written in Python and C. Since Multidrizzle is still undergoing significant development and enhancements, please check http://stsdas.stsci.edu/multidrizzle for the latest information regarding its use, known bugs, and any updates. The Multidrizzle process is as follows: - Recompute the World Coordinate System (WCS) information for each input image. - Populate a mask file for each input image using available data quality information. - Create a "static" bad pixel mask from all of the input data. All pixels that deviate by more than a user defined sigma below the image median are flagged as "bad". This static mask is applied to the individual image's bad pixel masks. - Compute the sky background for each exposure and subtract it. - Drizzle each input image onto the geometrically corrected output image grid (the one that will be used for the final combined image). Note that at this stage each input image is drizzled into its own separate output image that shares the same WCS as all the the other drizzled input images, i.e., they are not drizzled into the same output image. Only the pixels flagged during the "static mask" creation step are rejected during this step. - Combine these individually drizzled images to created a cosmetically clean, cosmic ray free, mosaic. This combined image is created using either a clipped 'average', median, or 'minmed' (median, or minimum if only two valid pixels at a pixel location) algorithm. This combination process is simple (pixel-by-pixel) since all the combined drizzled images share the same WCS - Blot (distort) the mosaic image back into the reference frame of each of the input images to produce a cosmic-ray-free image to compare with.. - Compare the input images with the corresponding blotted images and identify significant differences as "bad" pixels. The gradient of the input image is also used in deciding whether a pixel is to be labeled as bad; the criterion for labeling a pixel as bad is loosened in regions where the gradient is large otherwise most star-like objects would be mislabeled as cosmic rays. - Drizzle each image input onto a single, geometrically corrected, output image grid to produce the final combined image. This step uses masks which are the union of the static, data quality, and cosmic-ray masks to reject bad pixels. Although this is the same general procedure followed by the previous version of Multidrizzle, there are a number of substantial differences that should be noted. These differences include: - All IRAF dependencies have been removed from Multidrizzle. - Internally to Multidrizzle, all computations occur in units of electrons. If the "workinplace" parameter is set to yes, the 'sci' extensions of the input files will also be in units of electrons upon the completion of Multidrizzle. - For all steps in the Multidrizzle process, every attempt was made to eliminate the creation of intermediate files typically seen in IRAF tasks. This allows the user to save space on disk and speeds up the processing by approximately 20%. - The sky subtraction algorithm now updates a primary header keyword, MDRIZSKY', with the value computed for the input image. The units of 'MDRIZSKY' are the native units of the input image. Upon successful completion of Multidrizzle processing, the computed sky value will NOT have been subtracted from the 'sci' extensions of the input images. The sky value can be subtracted from the input images by setting the boolean parameter "workinplace" to yes. For input images with multiple 'sci' extensions, the sky subtraction step will now compute a sky level for each chip, but only subtract the lowest value from each of the 'sci' extensions. This has the effect of creating a more uniform sky across an image when one of the chips is dominated by an extended object. In the case of WFPC2, plate scale differences are taken into account. The 'MDRIZSKY' header value for WFPC2 is taken to be on the WF3 plate scale. Since this version of Multidrizzle assumes only one exposure per file, for STIS data it is necessary to split repeated exposures into separate files (this won't be necessary in future versions of Multidrizzle) - During the single drizzle step, only the mask generated during the "static mask" creation step is used to flag bad pixels. No data quality information is used. - There are now only 4 valid combination types allowed in the median creation step. These 4 types are average, median, sum, and minmed. - The DRIZ_CR step now updates the data quality arrays of all input images with the pixels it identifies as "bad". The bit value used to update the DQ array is instrument specific. The bit value used for each instrument is as follows: - ACS : 4096 - WFPC2 : 4096 - STIS : 8192 - NICMOS: 4096 The input image's data quality array will not be updated if the "crbit" parameter in the user interface is set to "0". Note: all STIS associations have to be split using stsdas.mssplit prior to running MultiDrizzle in order to get any meaningful cosmic-ray detection by this step. - The ability to apply a weighting image during the final drizzle step has now been implemented. This new implementation of the Multidrizzle software has been included as part of the ACS calibration pipeline as of 22 September 2004. Because of this, all imaging modes of ACS are supported by Multidrizzle. At this time, only the most common types of WFPC2 observing modes have been tested and are supported. Support for STIS and NICMOS imaging is only preliminary at this time. Full support for WFPC2, STIS, and NICMOS is under development. Until the PA_V3 keyword is automatically included in the header of STIS and NICMOS science exposures, it should be read from the primary header of the spt files and added to the primary header of the science exposures manually before running Multidrizzle on such data. Although Multidrizzle's basic algorithm's remain in place, the re-reimplementation provides a more robust, efficient, and user friendly software platform for the user. Although every effort was made to test the different modes of Multidrizzle prior to release, for a software project of this complexity we anticipate that some problems do exist. We strongly encourage users to submit bug reports to help@stsci.edu. - PYDRIZZLE This version takes a major departure from previous releases of PyDrizzle for one primary reason: it no longer relies on any IRAF functionality. Instead, it relies on new callable interfaces to 'drizzle' and 'blot', uses PyFITS and readgeis solely for image input, and replaces IRAF variable name expansion with a Python-based function. An IRAF interface for PyDrizzle has been retained for this version and the previous version of PyDrizzle still remains part of the Dither package in STSDAS, albeit renamed as 'opydriz'. The IRAF interface gets distributed as part of the STSDAS source tree for the Dither package, while the source code for this new version of PyDrizzle gets distributed as a Python package on its own alongside other Python packages such as MultiDrizzle. - Source Code Packaging This separation allows PyDrizzle to be installed as a Python task without the need for STSDAS, or alongside STSDAS, giving the user a choice of interfaces to use for running the task; specifically, Python command-line, IRAF command-line or IRAF graphical EPAR. This same source code for the callable version of 'drizzle' and related tasks gets distributed with this version of PyDrizzle, and can be used to update the version of the 'drizzle' code in the STSDAS tree as well. This allows the same source code to be used for building both a Python callable version and the IRAF executables. - Code Upgrades and Additions Algorithms developed by Richard Hook for computing shifts from WCS keywords and converting shifts from undistorted frames back to the original input frames significantly improved the image registration and shift computations in PyDrizzle as well. This allows PyDrizzle rely completely on WCS information, and any user-supplied corrections, for registering the images and defining the output frames, regardless of whether the inputs are full-frame or subarray readouts. IDCTAB distortion models can now also use an optional OFFTAB reference table which provides time-dependent chip-to-chip offsets, used initially for WFPC2 images. PyDrizzle, through the use of an updated 'drizzle' task, can also now apply sub-pixel distortion residuals stored in DGEOFILE reference images. These new reference files are derived from the wavelength dependent distortion models stored in the IDCTAB reference table, and are completely optional for use with PyDrizzle. The WCSObject class in the 'wcsutil' module has been updated as well, especially to support computation of new WCS keywords by 'makewcs' as used by MultiDrizzle. This class has new methods for writing, saving a copy, and restoring a WCS, in addition to the coordinate transformation methods (xytord and rdtoxy). PyDrizzle has also been updated to support GEIS input, along with multi-extension FITS input, for WFPC2 data. Multi-extension FITS copies will be made for GEIS input when necessary. Naming conventions for output products have also changed. The HST pipeline populates association tables with all upper case rootnames, however, it uses all lower case filenames. This convention had been enforced by PyDrizzle by forcing all output names to be generated as all lower case. This, however, was changed such that the output names will now be case sensitive based on whatever the association table or input filename rootnames specify. Output files, though, for drizzling to separate output files (single=yes option in 'run()') have been reduced by eliminating the output of the context images for that step. Finally, many file utilities have been reorganized into the 'fileutil' module and a help method has been added to that module to highlight the available functionality. Many of these utilities can be used for general file operations, regardless of the file format (FITS or GEIS). - Revision History: The whole list of version revisions can be found in the 'History' file delivered with the source code for PyDrizzle. ANALYSIS.ISOPHOTE The cl scripts for ISOIMAP and BMODEL were modified to explicitly not save and restore parameters because this was causing an unnecessary prompt from TCALC, and because one parameter (ttout for TINFO) was not restored. - BMODEL A bug in task 'bmodel' was fixed. The task crashed under PyRAF due to ambiguous spelling of an internal CL parameter (the IRAF CL did not catch the bug). - N2GAUSSFIT The help file for task 'n2gaussfit' was updated in order to call the users attention to the fact that the task *will* read parameter initial guesses from the 'tgausspars' pset, regardless of the actual parameter input source. Values set to INDEF in the pset *will* be used to control the task execution, even thhough they are not used in the computations themselves. GRAPHICS.STPLOT - IGI: Added ellipticity as an option to the ELLIPSE command of IGI. - SKYMAP Fixed a bug of the task SKYMAP to prevent arithmetic error when drawing circles on certain platforms. HST_CALIB.ACS: - CALACS CALACS has been updated to Version 4.5.1 based on the following modifications: - Implemented support for applying CFLTFILE and SPOTTAB (spot flats and their shifts) in the FLATCORR step. This replaces the hand-processing described on the ACS web pages for coronographic flat-fielding. This processing also gets reported in the trailer files. - Removed code from calacs.c that deletes input FLT images for the SFL product image. This insures that all generated FLT images will be available for use with MultiDrizzle in the pipeline for REPEAT-OBS ASNs. - Fixed an error caused a 32 image REPEAT-OBS ASN to fail in the pipeline. A MsgText string variable used to build the list of inputs to ACSSUM was revised to be a dynamically allocated variable to avoid array out-of-bounds memory problems with long lists of inputs. - RUNCALDRIZ (New task) A new task, runcaldriz, has also been added to allow operation of both CALACS and the new MultiDrizzle as they are run in the pipeline, including creation of the trailer files and updating of header keywords. HST_CALIB.ACS.AXE aXe was upgraded to v1.4. This version differs in various aspects from previous aXe releases. A new set of High Level Tasks simplifies the user interaction and reduce the number of commands that need to be issued in order to perform a basic data reduction. Another important addition is the aXedrizzle technique which combines for each object the 2D grism images extracted from the individual images to a deep, 2D drizzled grism image before deep 1D spectrum are extracted. Details are available from the aXe web site, http://www.stecf.org/software/aXe/ HST_CALIB.CTOOLS - MSREADNOISE, MSBADPIX.(Removed) Retired the tasks MSREADNOISE and MSBADPIX. - CHCALPAR The psets for CHCALPAR for STIS now include 'tdstab'.(Time dependent sensitivity correction) HST_CALIB.GHRS CALHRS occasionally produced incorrect results, because some vignetting data was corrupted on input. The problem occurs when reading two groups in a GEIS file and the second group is located before the first group. The data in the second group is corrupted. The code has been rewritten to avoid this problem by reading sequentially the entire file. HST_CALIB.NICMOS Cleaned up tasks in the NICMOS package. Deleted tasks (MSBADPIX, MSREADNOISE, and MSSTREAKFLAT) and psets (NSTREAKPAR, WSTREAKPAR, and WFDQPAR) from the NICMOS package because they are duplicated in the CTOOLS package. Removed task definitions of NDARK and NLINCORR that do not exist anymore. Removed the pset DQPAR in NICMOS package since it is identical to another pset NICDQPAR. Renamed the parameter DQPAR to NICDQPAR in tasks (BIASEQ, PEDSKY, and PEDSUB) using this parameter. HST_CALIB.STIS - CALSTIS An 'angle' parameter has been added to the wavecal task and to the hidden task _cs4; it may be necessary to unlearn both of these tasks (or "unlearn stis"). This parameter should normally be left at its default value of INDEF. For echelle data taken with a long slit (e.g. 6X0.2), the angle of the slit images on the detector can be as large as about six degrees, and specifying an appropriate value for 'angle' will significantly improve the wavecal processing. CALSTIS can now use the new EPC tables (engineering products for calibration) to get the temperatures for correcting the CCD sensitivity for its dependence on temperature for side B data Some STIS CCD data have a column with very high counts that extends down into the virtual overscan region. The current algorithm for identifying outliers in this region is run for each column, and it was not finding these hot columns because all the values in the column were comparable. An additional clipping step has been added to CALSTIS to look for outliers within the entire virtual overscan region. There were two variables in CALSTIS that under some circumstances would not be initialized, which could result in a segmentation violation. These are now initialized correctly. CALSTIS now writes HISTORY records for TDSCORR and CTECORR. In the 1-D spectral extraction step of CALSTIS (X1D), background smoothing is done for most data, but the calculation of the ERROR column was still including a significant contribution from the background regions. CALSTIS now does not include any contribution from the background if background smoothing has been done. The X1D task didn't always set calibration switches to COMPLETE. The steps to be done are based on task parameters rather than the switches (DISPCORR, etc) in the primary header. When updating the header switches after processing, however, X1D would only set a keyword to COMPLETE if the keyword was originally PERFORM (and if the step was actually done, of course). - SC2DREF (Removed) The SC2DREF task has been deleted, as this is no longer needed. - SSHIFT (New task) SSHIFT is a new PyRAF post-processing task for aligning dithered spectral images. The dithering must be along the slit and use integer pixel shifts. The output images can then be combined with ocrreject, which will reject hot pixels like it does with cosmic rays. The POSTARG2 and BINAXIS2 keywords are used to determine the number of rows to be shifted. The shifts will be toward the center of the CCD. - NORMSPFLAT, MKFRINGEFLAT, DEFRINGE The fringe-flat tasks, NORMSPFLAT, MKFRINGEFLAT, and DEFRINGE, have been modified to handle subarrays and binned images. They have also been modified to run under PyRAF by removing the GOTO statements. The flatfield file is obtained from the header of the input file. As a result, the PREPSPEC task is now deprecated. - TRXYEQ, TREQXY, UCRPIX (New tasks) These tasks have been added to the stis package. These are for converting between pixel coordinates and right ascension and declination in STIS images, taking into account the optical distortion. HST_CALIB.SYNPHOT In June 2004, a new spectrum for Vega (Alpha Lyrae) was made available in CDBS. This composite flux standard consists of IUE data from 1255-1675A, STIS CCD fluxes from 1650-4200A (Bohlin & Gilliland 2004), and a specially tailored Kurucz model longward of 4200A and from 900-1255A (Kurucz 2003). The help files for a number of synphot tasks which explicitly referred to this file were modified to reflect the new filename, crcalspec$alpha_lyr_stis_002.fits The new standard was adopted for use by synphot internals in converting to and from units of vegamag. A synphot-local copy of this spectrum was placed in stsdas$lib/synphot/vega.dat. The previous spectrum, which has been in use since 1998, was retained for historical purposes in the file vega_98_to_04.dat. - THERMBACK The help file for thermback was removed from the thermback directory and correctly placed into the doc directory. HST_CALIB.SYNPHOT.SIMULATORS The simulators subpackage has a separate copy of the refdata parameter set, which contains default filename patterns for the graph and component tables. These values were updated to use the current default patterns for the FITS files, mtab$*_tmg.fits and mtab$*_tmc.fits. The simcatp parameter set used by the simulators package was also updated to reflect the new filename for the spectrum of Alpha Lyrae. TOOLBOX.IMGTOOLS.MSTOOLS Tasks 'msarith' and 'msstatistics' were modified to add support for ACS files. One should note that this support is provisional in nature, especially for the 'msarith' task. At this point both tasks handle ACS files in the exact manner they handle STIS files (see the help page for details). Users should be aware that subtle differences may exist between STIS and ACS files; these differences must be taken into account when using these tasks on ACS files and interpreting their results.