THE_NON-LINEAR_INTERACTIVE_FITTING_PACKAGE · EXAMPLES · SEE_ALSO
fitparams -- solve for the parameters of the transformation equations
fitparams observations catalogs config parameters
- The list of files containing the observational data. Observations files are multi-column text files whose columns are delimited by whitespace, and whose first column is usually reserved for the object id. All observations files in the list must have the same format. The format of legal observations files is described in further detail in the description section.
- The list of files containing the catalog data. Catalog files are multi-column text files whose columns are delimited by whitespace, and whose first column is always reserved for the object id. If more than one entry with the same id exists in the list of catalogs, only the first entry is used. All catalog files in the list must have the same format. The format of legal catalog files is described in further detail in the description section.
- The name of the configuration file. The configuration file is a text file specifying the format of the input catalog and observations files, and the form of the transformation equations. A brief description of the syntax and grammar of this file is given in the configuration file section.
- The name of the output database file. Parameters is a text database file to which the error anaysis of the fit and the parameter values and errors for each transformation equation are written. The output of fitparams is appended to parameters as a set of new records, one for each of the transformation equations. If more than one record exists with the same record name, the last record written takes precedence.
- weighting = "uniform"
- The following weighting schemes are supported.
- The data points are all assigned a weight of one.
- The total error squared for each data point is set to the total error in the catalog variables squared plus the total error in the observations variables squared and the weight for each data point is set to 1.0 / error ** 2. This option assumes that all the sources of error are in the photometric indices (magnitudes and colors), that error columns (see the description of the configuration file below) have been declared for at least one photometric index, and that the contribution of each catalog or observations variable to the total error is weighted by the number of times it occurs in the transformation equation. If addscatter is "yes" then an additional "scatter" term is fit and added to the weights.
- The weight equation (see the description of the configuration file below) is evaluated for each point and the weight for that point is set to that value. If there is no weight equation the weights are all set to one. If addscatter is "yes" then an additional "scatter" term is fit and added to the weights.
- addscatter = yes
- Add an additional scatter term to the weights if the average error in the fit is much greater than the average error in the measurements? Addscatter has no effect if weighting is "uniform". Addscatter is recommended if weighting is "photometric" as the intrinsic error in the transformations is often much greater than the formal errors of measurement and the scatter term stabilizes the fit. Users of the weighting equals "equations" option may wish to turn off addscatter .
- tolerance = 3.0e-5
- The convergence tolerance for the non-linear least squares fit. The fit will stop iterating when the fractional change in the reduced chi-square of the residuals from iteration to iteration is less than tolerance .
- maxiter = 15
- The maximum number of iterations for the non-linear least squares fit. When this number is reached the fitting process will terminate even if the fit has not converged.
- nreject = 0
- The maximum number of bad data rejection iterations. If nreject is greater than zero the initial fit is used to detect and reject deviant points before performing the final fit. No rejection is performed if nreject is less than or equal to zero.
- low_reject = 3.0, high_reject = 3.0
- The lower and upper rejection limits in units of the rms of the fit. Points deviating from the initial fit by more than this amount are rejected before performing the final fit. No rejection is done if both limits are zero.
- grow = 0.0
- The default rejection growing radius. Points within a distance given by this parameter of any rejected point are also rejected.
- interactive = yes
- Fit equations interactively ? When this parameter is yes , the user will be presented with plots of the data and can interact with the fitting process.
- logfile = "STDOUT"
- The name of the output text file to which selected detailed results of the fitting process are written. By default logfile is the standard output. If logfile is "", logging is turned off altogether. Otherwise new output is appended to logfile which can therefor become quite large.
- log_unmatched = yes
- Write the list of observations with no corresponding catalog entries to logfile? This option is useful for checking for errors in the observed object id names and for users who like to run fitparams in non-interactive mode.
- log_fit = no
- Write the error analysis of the final fit in logfile? This option is useful for users who like to run fitparams in non-interactive mode.
- log_results = no
- Write the results of the current fit to logfile? This option is useful for users who like to run fitparams in non-interactive mode.
- catdir = ")_.catdir"
- The directory containing the supported standard star catalogs. The default parameter value redirects catdir to a package parameter of the same name. A list of standard catalogs may be obtained by printing the file "photcal$catalogs/README". Alternatively the user may create their own standard star catalogs and standard star catalog directory.
- graphics = "stdgraph"
- The default graphics device. This parameter is used only if interactive=yes .
- cursor = ""
- Graphics cursor input. When null the standard graphics cursor is used. Otherwise the specified cursor command file is used. This parameter is used only if interactive=yes .
FITPARAMS parses the configuration file config checking for grammar and syntax errors. FITPARAMS attempts to recover from any errors and to finish parsing the configuration file, but it will not process the input data if errors are present. The configuration file is described briefly in the configuration file section and in detail in the help page for the configuration file.
Once the configuration file is successfully parsed, FITPARAMS reads the list of catalog files and loads the values of the catalog variables declared in config into memory. If no catalog section is declared in config , if the catalog section is empty, or if catalogs is "", no catalog data is read and all the required input data is assumed to be in observations . After the catalog data is read, FITPARAMS reads the observations files observations , matches the object ids of the observations with the corresponding catalog object ids, and loads all the observations variables declared in config into memory. Id matching is disabled if no catalog data is read, otherwise only those observations which have a matching catalog entry will be used in the fit. If a catalog section declaration was made in config , even an empty one, FITPARAMS assumes that the object ids are in column 1 of observations .
Legal catalog and observations files are multi-column text files whose columns are delimited by whitespace. The first column of a catalog file is always reserved for an object id. The first column of an observations file is usually reserved for an object id which can be used to match the observational data with the corresponding catalog data. All other columns may contain any quantity which can be expressed as an integer or real number. Sexagesimal format numbers (hh:mmm:ss) are interpreted internally as real numbers. The constant INDEF can be used to represent data that is missing or undefined. Double precision and complex data are not supported. Lines beginning with "#" are treated as comment lines.
FITPARAMS solves the fit for each equation in the configuration file either interactively or non-interactively depending on the value of interactive , and writes the solution in the output file parameters for later use by the evaluation routines EVALFIT or INVERTFIT. Selected results can also be written to logfile if any of the switchs log_unmatched , log_fit , or log_results are enabled. In interactive mode the user can use all the interactive capabilities of the interactive non-linear least squares package INLFIT. INLFIT is described more fully below.
The configuration file is a text file which specifies how the data is organized in the input files and how the transformation equations are to be fit.
The input data are assumed to come from two different sources that may be either in the same input file or in different input files. These sources are known as the catalog and the observations respectively.
The catalog contains values indexed by a name called the matching name. This name must be in the first column of the catalog and is also assumed to be unique, i.e, each catalog entry is assumed to be unique.
The observations are values that may be either indexed by a matching name if a catalog section is specified in the configuration file, or a stream of input values in an ordinary text file. If a catalog section is specified and non-empty, each observation is matched against the catalog entries, and only observations whose matching names are found in the catalog are used to compute the transformation equations. Otherwise all values are used.
The configuration file is divided in three sections: the catalog section which describes the format of the catalog files, the observations section which describes the format of the observation files, and the transformation section which defines the transformation equations in that order.
The catalog and observations sections permit the user to assign names to the input file columns. These columns can later be referenced by name in the configuration file by using these assigned names as if they were variables in a programming language.
The transformation section is used to define the equations to solve, and assign initial values to the fitting parameters. The user may also optionally define equations for the derivatives of the transformation equations with respect to the parameters, the weights to be used in the fit, the errors of the fit and the default equations to be plotted in the interactive fitting process. It is possible to specify any number of transformation equations in this section.
SAMPLE CONFIGURATION FILES
Example 1. Configuration file for reducing UBV photoelectric photometry.
# Configuration file for reducing UBV photoelectric photometry. catalog V 2 # V magnitude BV 3 # B - V color UB 4 # U - B color observation v 2 # v instrumental magnitude b 3 # b instrumental magnitude u 4 # u instrumental magnitude ev 5 # error in v instrumental magnitude eb 6 # error in b instrumental magnitude eu 7 # error in u instrumental magnitude X 8 # airmass transformation fit v1 = 0.0, v2=0.16, v3=-0.043 VFIT: V = v1 + v - v2 * X + v3 * (b - v) weight(VFIT) = 1.0 / ev ** 2 plot(VFIT) = V, V - (v1 + v - v2 * X + v3 * (b - v)) fit b1 = 0.0, b2=0.09, b3=1.21 BVFIT: BV = b1 - b2 * X + b3 * (b - v) weight (BVFIT) = 1.0 / (eb ** 2 + ev ** 2) plot(BVFIT) = BV, BV - (b1 - b2 * X + b3 * (b - v)) fit u1 = 0.0, u2=0.300, u3=0.861 UBFIT: UB = u1 - u2 * X + u3 * (u - b) weight (UBFIT) = 1.0 / (eu ** 2 + eb ** 2) plot(UBFIT) = UB, UB - (u1 - u2 * X + u3 * (u - b))
Example 2. Configuration file for reducing UBV CCD photometry.
catalog V 2 # V magnitude BV 3 # B - V color UB 4 # U - B color error(V) 5 # error in V magnitude error(BV) 6 # error in B-V color error(UB) 7 # error in U-B color observation m1 2 # filter 1 instrumental magnitude error(m1) 3 # error in filter 1 instrumental magnitude Xm1 4 # airmass of filter 1 observation m2 6 # filter 2 instrumental magnitude error(m2) 7 # error in filter 2 instrumental magnitude Xm2 8 # airmass of filter 2 observation m3 10 # filter 3 instrumental magnitude error(m3) 11 # error in filter 3 instrumental magnitude Xm3 12 # airmass of filter 3 observation transformation fit u1 = 27.0, u2=0.68, u3=0.05 UFIT: m3 = u1 + V + BV + UB + u2 * Xm3 + u3 * UB fit b1 = 26.0, b2=0.30, b3=0.18 BFIT: m2 = b1 + V + BV + b2 * Xm2 + b3 * BV fit v1 = 25.0, v2=0.17, v3=-0.02 VFIT: m1 = v1 + V + v2 * Xm1 + v3 * BV
INLFIT fits an n-dimensional function to a set data points, iterating until the reduced chi-squared changes by less than tolerance percent between sucessive iterations, or machine precision is reached and the fit converges, or until the maximum number of iterations maxiter is reached. If the maximum number of iterations is reached before convergence a status flag is set.
After computing an initial fit, INLFIT presents the user with a plot of the fit and activates the graphics cursor. At this point the user may examine and/or interact with the fit by, for example, reprogramming the default graph keys, editing the default convergence or bad data rejection parameters, deleting and undeleting points, altering which parameters in the fitting function are actually to be fit and which are to be held constant, and refitting the data.
If nreject is greater than zero the RMS of the residuals is computed and points whose residuals are less than low_reject * RMS or high_reject * RMS value are excluded from the fit. Points within a distance grow of a rejected point are also excluded from the fit. The function is then refit without the rejected points. The rejection algorithm is executed until the number of rejection iterations reaches nreject or no more points are rejected.
INLFIT uses the standard Levenberg-Marquardt non-linear least squares algorithm to fit the data. Detailed descriptions of the algorithm can be found in the following two references.
1. Bevington, P.R., 1969, Data Reduction and Error Analysis for the Physical Sciences, Chapter 11, page 235. 2. Press, W.H. et al., 1986, Numerical Recipes: The Art of Scientific Computing, Chapter 14, page 523.
The following interactive cursor keystroke commands are available from with the INLFIT paclage.
- The terminal is cleared and a menu of cursor keystroke and colon commands is printed.
- The id, coordinates of the data point nearest the cursor, along with the function value, the fitted value and the residual, are printed on the status line.
- The data point nearest the cursor and not previously deleted is marked with an X. It will not be used in further fits until it is undeleted.
- The function is fit to the data and the fit is graphed using the default plot type.
- Redefine the graph keys "h-l" from their defaults. A prompt is issued for the
graph key to be redefined. Another prompt is issued for the data to be
plotted at which point the user must enter the x and y axis data to plot,
delimited by a comma. The data types are the following (they can be
abreviated to up to three characters).
function Dependent variable or function fit Fitted value residuals Residuals (function - fit) ratio Ratio (function / fit) nonlinear Nonlinear component identifier Independent variable named "identifier" (if defined) var n Independent variable number "n" user n User defined plot equation "n" (if defined)The application program can define independent variable names and user plot functions, aside from the standard options provided. If variable names are supplied, the user can reference them by their names. Otherwise they can be always referenced by "var n", where "n" is the variable number (the user has to know the variable order in this case). The ":variables" command will list the currently defined variables by name and number. The application program may define any number of plot equations aside from the defaults provided. In this case the user may refence them by "user n", where "n" is the plot function number (the user must know the equation order in this case).
- h, i, j, k, l
- By default each key produces a different graph. The graphs are described by
the data which is graphed along each axis as defined above. The default graph
which may be redefined by the application program or interactively by using
the g key, are the following.
h function, fit i function, residuals j function, ratio k var 1, function l user 1, user 2 (default)The initial graph key, if not redefined by the application program is h.
- Overplot the next fit provided the graph format has not changed.
- Exit from the interactive curve fitting package.
- Redraw the current graph.
- Toggle fit overplotting on and off. If this option is on the data and fitted values are overplotted. Otherwise only data points are plotted. The fitted values are marked using boxes.
- Undelete the data point nearest the cursor which has been previously deleted. This option does not work over points marked as deleted by the application program before calling inlfit.
- w [key]
- Set the graph window or data range along each axis to be graphed.. This is a gtools option which prints the prompt "window:". The available cursor keystroke commands are printed with ? and on-line help is available by typing "help gtools".
- Interrupt the task immediately without saving the current fit.
Colon commands are used to show or set the values of parameters. The application program calling inlfit can add more commands. Parameter names can be abbreviated. The following commands are supported.
- :show [file]
- Show the current values of the fitting parameters high_reject, low_reject, niterate, grow, tol, itmax. The default output device is the terminal (STDOUT) and the screen is cleared before the information is output. If a file is specified then the information is appended to the named file.
- :variables [file]
- List the currently loaded variables. The number, id, minimum value and maximum value of each variable is printed. The default output device is the terminal (STDOUT) and the screen is cleared before the information is output. If a file is specified then the information is appended to the named file.
- :data [file]
- List the raw data. The value of each standard catalog and observations catalog variable for each data point is printed. The default output device is the terminal (STDOUT) and the screen is cleared before the information is output. If a file is specified then the information is appended to the named file.
- :errors [file]
- Show the error analysis of the current fit. The number of iterations, total number of points, the number of rejected and deleted points, the standard deviation, the reduced chi, average error (always = 1.0 if weight = 1.0, otherwise = 1.0 / <weight>), average scatter (always = 0.0 if no weights scatter term is fit) and the rms value are printed on the screen. The fitted parameters and their errors are also printed. The default output is the terminal (STDOUT) and the screen is cleared before the information is output. If a file is specified then the information is appended to the named file.
- :results [file]
- List the results of the current fit. The function value, the fitted value, the residual, and the weight are printed for each data point. The default output device is the terminal (STDOUT) and the screen is cleared before the information is output. If a file is specified then the information is appended to the named file.
- :vshow [file]
- A verbose version of ":show" which is equivalent to a ":show" plus a ":errors" plus a ":results". The default output device is the terminal (STDOUT) and the screen is cleared before the information is output. If a file is specified then the information is appended to the named file.
- :page file
- Page through the named file.
- :tolerance [value]
- Show or set the value of the fitting tolerance. Tolerance is the maximum fraction by which the reduced chi-squared can change from one iteration to the next for the fit to meet the convergence criteria.
- :maxiter [value]
- Show or set the maximum number of fitting iterations.
- :nreject [value]
- Show or set the maximum number of rejection iterations. A value of zero means that automatic bad data rejection is turned off.
- :low_reject [value], :high_reject [value]
- Show or set the values of the bad data rejection limits. If both low_reject and high_reject are zero then automatic bad data rejection is turned off. If either of the high or low rejection rejection limits are greater than zero, and nreject is greater than zero, the rms of the initial fit is computed. Points with residuals more than low_reject * rms below zero and high_reject * rms above zero are removed before the final fit. Rejected points are marked on the graphs with diamonds.
- :grow [value]
- Show or set the value of the rejection growing radius. Any points within this distance of a rejected point are also rejected.
- :fit [parameter] [value]
- Set the starting guess value for the named coefficient and allow the parameter value to change (converge) during the fit. If the value is not specified inlfit will use the last starting guess.
- :const [parameter] [value]
- Set the named parameter to be a constant with the specified value, i.e, its value won't change during the fit. If the value is not specified inlfit will use its last starting value.
- Print help for the graph formatting options (the w key).
- Print help for the general IRAF graphics options.
1. Fit a set of UBV standard star data non-interactively using the automatic bad data rejection algorithm and the configuration file shown in example 2 under the configuration file section.
ph> fitparams m92.obs m92.cat m92.config m92.fit nreject=10 inter- ... compute valued for the parameters in all the transformation equations ph> page m92.fit ... check that the fitted parameter values are reasonable ph> invertfit m92.obs m92.cat m92.config m92.fit m92.out ... evaluate the transformation equations for all the standard stars
2. Fit the same set of data interactively but deleting bad points by eye instead of using the automatic rejection algorithm.
ph> fitparams m92.obs m92.cat m92.config m92.fit ... a default plot of the UFIT equation comes up on the screen (the fit or right-hand side of the equation is plotted versus the function or left-hand side of the equation) ... type '?' to show the available commands ... type 'i' to plot the residuals versus the function (LHS of the equation) ... delete bad points with the 'd' key and refit using the 'f' key ... check for any dependencies of the residuals on the the color term by reprogramming the graph key 'l' using the 'g' key (type 'g' to enter the reprogramming menu, 'l' after the prompt to reprogram the 'l' key, and "UB, residuals" in response to the question of which axes to plot ... list the plot windowing menu by typing 'w' followed by '?' after the "window:" prompt ... type 'w' followed by 'z' after the ":window" prompt to zoom up on an interesting area in the plot, a 'w' followed by 'a' will return to normal scaling ... type 'q' to quit the fit for this equation ... answer "yes" to the question about saving the fit ... proceed to the next fit by typing "next" in response to the prompt