EXES Redux Developer’s Manual

Introduction

Document Purpose

This document is intended to provide all the information necessary to maintain the EXES Redux pipeline, used to produce Level 2 and 3 reduced products for EXES data, in either manual or automatic mode. Level 2 is defined as data that has been processed to correct for instrumental effects; Level 3 is defined as flux-calibrated data. A more general introduction to the data reduction procedure and the scientific justification of the algorithms is available in the EXES Redux User’s Manual.

This manual applies to EXES Redux version 3.0.0.

Redux Revision History

The EXES pipeline was originally developed as three separate packages: the EXES, which provided the image processing algorithms; FSpextool, which provided the spectral extraction algorithms and some supporting libraries; and Redux, which provided the interactive GUI, the automatic pipeline wrapper, and the supporting structure to call EXES and FSpextool algorithms.

EXES was a software package developed in IDL for the reduction of EXES data. Dr. John Lacy initially developed the algorithms in FORTRAN, for the reduction of data from TEXES (EXES’s predecessor instrument). The FORTRAN algorithms were translated into IDL and adapted for EXES by Melanie Clarke for the SOFIA DPS team in 2013 and 2014. Version 1.0.0 was released for use at SOFIA in 2015. The EXES IDL package was never used as a standalone data reduction package, so it did not provide interface scripts for its algorithms; it was designed specifically for incorporation into Redux.

FSpextool was built on top of a pre-release version of Spextool 4, an IDL-based package developed by Dr. Michael Cushing and Dr. William Vacca for the reduction of data from the SpeX instrument on the NASA Infrared Telescope Facility (IRTF). Spextool was originally released in October 2000, and has undergone a number of major and minor revisions since then. The last stable public release was v4.1, released January 2016. As Spextool does not natively support automatic command-line processing, FSpextool for SOFIA adapted the Spextool library to the SOFIA architecture and instruments; version 1.0.0 was originally released for use at SOFIA in July 2013.

Redux was developed to be a general-purpose interface to IDL data reduction algorithms. It provided an interactive GUI and an object-oriented structure for calling data reduction processes, but it does not provide its own data reduction algorithms. It was developed by Melanie Clarke for the SOFIA DPS team, to provide a consistent front-end to the data reduction pipelines for multiple instruments and modes, including EXES. It was first released in December 2013.

Between 2015 and 2021, the EXES Redux package was maintained by the EXES PI team, under Dr. Matthew Richter at UC Davis. The principal developers were Dr. Curtis DeWitt and Dr. Edward Montiel. In 2021, the SOFIA DPS team rebased the pipeline from the PI team version for release as EXES Redux v2.0.0, in support of EXES’s transition from PI class to facility class instrument.

In 2022, the EXES pipeline was entirely reimplemented in Python, as a set of software modules in the SOFIA Redux Python package:

  • sofia_redux.instruments.exes: processing algorithms specific to the EXES instrument

  • sofia_redux.spectroscopy: spectral extraction algorithms

  • sofia_redux.pipeline: interactive and batch mode interface tools for managing data reduction processes

  • sofia_redux.toolkit: numerical algorithms and supporting utilities

  • sofia_redux.visualization: data analysis and visualization tools

The exes module reimplements the algorithms in the EXES IDL package, pipeline reimplements the Redux tools, and spectroscopy reimplements FSpextool algorithms.

The SOFIA Redux package was developed as a unified Python package to support data reduction for all facility class instruments for SOFIA, replacing all legacy pipelines with an integrated, shared code base. The package was developed by the SOFIA DPS team, starting in 2018. The principal developers for SOFIA Redux prior to the EXES 3.0.0 release were Daniel Perera, Dr. Rachel Vander Vliet, and Melanie Clarke, for the SOFIA DPS team, with additional contributions from Dr. Karishma Bansal.

Overview of Software Structure

The sofia_redux package has several sub-modules organized by functionality:

sofia_redux
├── calibration
├── instruments
│   ├── exes
│   ├── fifi_ls
│   ├── flitecam
│   ├── forcast
│   └── hawc
├── pipeline
├── scan
├── spectroscopy
├── toolkit
└── visualization

The modules used in the EXES pipeline are described below.

sofia_redux.instruments.exes

The sofia_redux.instruments.exes package is written in Python using standard scientific tools and libraries.

The data reduction algorithms used by the pipeline are straight-forward functions that generally take a data array, corresponding to a single image file, as an argument and return the processed image array as a result. They generally also take as secondary input a variance array to process alongside the image, a header structure to track metadata, and keyword parameters to specify non-default settings.

The exes module also stores any reference data needed by the EXES pipeline. This includes bad pixel masks, nonlinearity coefficients, dark files, and default header values and distortion correction parameters. The default files may vary by date; these defaults are managed by the readhdr function in the exes module. New date configurations may be added to the caldefault.txt file in exes/data/caldefault.txt.

sofia_redux.toolkit

sofia_redux.toolkit is a repository for classes and functions of general usefulness, intended to support multiple SOFIA pipelines. It contains several submodules, for interpolation, image manipulation, multiprocessing support, numerical calculations, and FITS handling. The utilities used by EXES are generally simple functions that take input as arguments and return output values.

sofia_redux.spectroscopy

The sofia_redux.spectroscopy package contains a library of general-purpose spectroscopic functions. The EXES pipeline uses these algorithms for spectroscopic image rectification, aperture identification, and spectral extraction. Most of these algorithms are simple functions that take spectroscopic data as input and return processed data as output. However, the input and output values may be more complex than the image processing algorithms in the exes package. The Redux interface in the pipeline package manages the input and output requirements for EXES data and calls each function individually. See the sofia_redux.spectroscopy API documentation for more information.

sofia_redux.visualization

The sofia_redux.visualization package contains plotting and display routines, relating to visualizing SOFIA data. For the EXES pipeline, this package currently provides a module that supports generating quick-look preview images.

sofia_redux.pipeline

Design

Redux is designed to be a light-weight interface to data reduction pipelines. It contains the definitions of how reduction algorithms should be called for any given instrument, mode, or pipeline, in either a command-line interface (CLI) or graphical user interface (GUI) mode, but it does not contain the reduction algorithms themselves.

Redux is organized around the principle that any data reduction procedure can be accomplished by running a linear sequence of data reduction steps. It relies on a Reduction class that defines what these steps are and in which order they should be run (the reduction “recipe”). Reductions have an associated Parameter class that defines what parameters the steps may accept. Because reduction classes share common interaction methods, they can be instantiated and called from a completely generic front-end GUI, which provides the capability to load in raw data files, and then:

  1. set the parameters for a reduction step,

  2. run the step on all input data,

  3. display the results of the processing,

and repeat this process for every step in sequence to complete the reduction on the loaded data. In order to choose the correct reduction object for a given data set, the interface uses a Chooser class, which reads header information from loaded input files and uses it to decide which reduction object to instantiate and return.

The GUI is a PyQt application, based around the Application class. Because the GUI operations are completely separate from the reduction operations, the automatic pipeline script is simply a wrapper around a reduction object: the Pipe class uses the Chooser to instantiate the Reduction, then calls its reduce method, which calls each reduction step in order and reports any output files generated. Both the Application and Pipe classes inherit from a common Interface class that holds reduction objects and defines the methods for interacting with them. The Application class additionally may start and update custom data viewers associated with the data reduction; these should inherit from the Redux Viewer class.

All reduction classes inherit from the generic Reduction class, which defines the common interface for all reductions: how parameters are initialized and modified, how each step is called. Each specific reduction class must then define each data reduction step as a method that calls the appropriate algorithm.

The reduction methods may contain any code necessary to accomplish the data reduction step. Typically, a reduction method will contain code to fetch the parameters for the method from the object’s associated Parameters class, then will call an external data reduction algorithm with appropriate parameter values, and store the results in the ‘input’ attribute to be available for the next processing step. If processing results in data that can be displayed, it should be placed in the ‘display_data’ attribute, in a format that can be recognized by the associated Viewers. The Redux GUI checks this attribute at the end of each data reduction step and displays the contents via the Viewer’s ‘display’ method.

Parameters for data reduction are stored as a list of ParameterSet objects, one for each reduction step. Parameter sets contain the key, value, data type, and widget type information for every parameter. A Parameters class may generate these parameter sets by defining a default dictionary that associates step names with parameter lists that define these values. This dictionary may be defined directly in the Parameters class, or may be read in from an external configuration file or software package, as appropriate for the reduction.

EXES Redux

To interface to the EXES pipeline algorithms, Redux defines the EXESReduction and EXESParameters classes. See Fig. 112 for a sketch of the Redux classes used by the EXES pipeline. The EXESReduction class calls the sofia_redux.instruments.exes reduction functions, with support from the sofia_redux.toolkit and sofia_redux.spectroscopy packages. The EXESParameters class defines default parameter values for all reduction steps.

The EXESReduction holds definitions for all reduction steps for the EXES pipeline:

The recipe attribute for the reduction class specifies the above steps, in that order.

If an intermediate file is loaded, its product type is identified from the PRODTYPE keyword in its header, and the prodtype_map attribute is used to identify the next step in the recipe. This allows reductions to be picked up at any point, from a saved intermediate file. For more information on the scientific goals and methods used in each step, see the EXES Redux User’s Manual.

The EXESReduction class also contains several helper functions, that assist in reading and writing files on disk, and identifying which data to display in the interactive GUI. Display is performed via the QADViewer class provided by the Redux package. Spatial profiles and aperture locations are additionally displayed by the MatplotlibViewer class. One-dimensional spectra are displayed by the EyeViewer class.

EXES Redux Classes

Fig. 112 EXES Redux class diagram.

Detailed Algorithm Information

The following sections list detailed information on the functions and procedures most likely to be of interest to the developer.

sofia_redux.instruments.exes

sofia_redux.instruments.exes.calibrate Module

Functions

calibrate(data, header, flat, variance[, ...])

Calibrate spectral image to physical units.

sofia_redux.instruments.exes.cirrus Module

Functions

cirrus(data, header, abeams, bbeams, flat)

Correct nod-off-slit data for residual sky noise.

sofia_redux.instruments.exes.clean Module

Functions

clean(data, header, std[, mask, radius, ...])

Correct bad pixels.

sofia_redux.instruments.exes.coadd Module

Functions

coadd(data, header, flat, variance[, illum, ...])

Combine individual frames to increase signal-to-noise.

sofia_redux.instruments.exes.correct_row_gains Module

Functions

correct_row_gains(data)

Correct odd/even row gain offsets.

sofia_redux.instruments.exes.debounce Module

Functions

debounce(data, header, abeams, bbeams, flat)

Correct for optics shifts between nods (bounces).

sofia_redux.instruments.exes.derasterize Module

Functions

derasterize(data, header[, dark_data, ...])

Read and recombine a rasterized flat file.

sofia_redux.instruments.exes.derive_tort Module

Functions

derive_tort(data, header[, maxiter, fixed, ...])

Derive distortion parameters and identify orders and illuminated regions.

sofia_redux.instruments.exes.despike Module

Functions

despike(data, header[, variance, abeams, ...])

Correct outlier pixels due to temporal spikes.

sofia_redux.instruments.exes.diff_arr Module

Subtract sky frames or nod pairs.

Functions

diff_arr(data, header, abeams, bbeams, variance)

Subtract sky frames, nod pairs, or a dark frame.

sofia_redux.instruments.exes.get_atran Module

Functions

clear_atran_cache()

Clear all data from the atran cache.

get_atran_from_cache(atranfile, resolution)

Retrieves atmospheric transmission data from the atran cache.

store_atran_in_cache(atranfile, resolution, ...)

Store atmospheric transmission data in the atran cache.

get_atran(header, resolution[, filename, ...])

Retrieve reference atmospheric transmission data.

sofia_redux.instruments.exes.get_badpix Module

Functions

get_badpix(header[, clip_reference, ...])

Get a bad pixel mask from a reference file.

sofia_redux.instruments.exes.get_resolution Module

Functions

get_resolution(header)

Retrieve expected spectral resolving power.

sofia_redux.instruments.exes.lincor Module

Functions

lincor(data, header)

Correct raw readout frames for detector nonlinearity.

sofia_redux.instruments.exes.make_template Module

Functions

make_template(data, header, weighting_frame)

Make a spectral template for weighting and shifting spectra.

sofia_redux.instruments.exes.makeflat Module

Functions

makeflat(cards, header, variance[, robust, ...])

Generate calibrated flat frame; set distortion parameters.

blackbody_pnu(wavenumber, temperature)

Black-body photon function.

bnu(wavenumber, temperature)

Black-body intensity function.

bb_cal_factor(wavenumber, bb_temp, ...)

Calibration factor for EXES blackbody source + flat mirror.

sofia_redux.instruments.exes.mergehdr Module

Functions

mergehdr(headers[, reference_header])

Merge input headers.

sofia_redux.instruments.exes.readhdr Module

Functions

readhdr(header[, check_header, config_file, ...])

Read and update an EXES FITS header.

sofia_redux.instruments.exes.readraw Module

Functions

readraw(data, header[, do_lincor, ...])

Correct for nonlinearity combine individual readouts.

sofia_redux.instruments.exes.spatial_shift Module

Functions

spatial_shift(data, header, flat, variance)

Shift spectra for spatial alignment.

sofia_redux.instruments.exes.submean Module

Functions

submean(data, header, flat, illum, order_mask)

Subtract residual sky background from nod-on-slit data by removing the mean value at each wavelength.

sofia_redux.instruments.exes.tort Module

Functions

tort(data, header[, variance, skew, order, ...])

Correct image for optical distortion.

sofia_redux.instruments.exes.tortcoord Module

Functions

tortcoord(header[, skew])

Calculate undistorted image coordinates.

sofia_redux.instruments.exes.utils Module

Functions

get_detsec(header_or_str)

Parse the DETSEC keyword in a header or from a string.

check_data_dimensions(**kwargs)

Check the data dimensions and return number of frames.

check_variance_dimensions(variance, nx, ny, nz)

Check variance dimensions for expected shape.

get_reset_dark(header)

Get a reset dark image from a file on disk.

set_elapsed_time(header)

Set the TOTTIME key in the header to the total elapsed time.

parse_central_wavenumber(header)

Parse the central wavenumber from the input header.

sofia_redux.instruments.exes.wavecal Module

Functions

wavecal(header[, order])

Generate a wavelength calibration map from the grating equation.

sofia_redux.toolkit

sofia_redux.toolkit.convolve.base Module

Classes

ConvolveBase(*args[, error, mask, stats, ...])

Convolution class allowing error propagation.
Class Inheritance Diagram
digraph inheritance555cb6bce0 { bgcolor=transparent; rankdir=LR; size=""; "ConvolveBase" [URL="../../../api/sofia_redux.toolkit.convolve.base.ConvolveBase.html#sofia_redux.toolkit.convolve.base.ConvolveBase",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,margin=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Convolution class allowing error propagation."]; "Model" -> "ConvolveBase" [arrowsize=1.2,arrowtail=empty,dir=back,style="setlinewidth(0.5)"]; "Model" [URL="../../../api/sofia_redux.toolkit.utilities.base.Model.html#sofia_redux.toolkit.utilities.base.Model",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,margin=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base model Class for fitting N-dimensional data"]; }

sofia_redux.toolkit.convolve.kernel Module

Functions

apply_ndkernel(data, kernel[, axes, ...])

Apply a kernel over multiple features

convolve(*args[, error, mask, stats, ...])

Convolve an N-dimensional array with a user defined kernel or fixed box.

savitzky_golay(*args[, order, error, mask, ...])

Apply a least-squares (Savitzky-Golay) polynomial filter
Classes

KernelConvolve(*args[, error, mask, stats, ...])

Generic convolution with a kernel

BoxConvolve(*args[, error, mask, stats, ...])

Convolution with a box kernel (mean)

SavgolConvolve(*args[, error, mask, stats, ...])

Convolve using Savitzky-Golay filter
Class Inheritance Diagram
digraph inheritance7459f88665 { bgcolor=transparent; rankdir=LR; size=""; "BoxConvolve" [URL="../../../api/sofia_redux.toolkit.convolve.kernel.BoxConvolve.html#sofia_redux.toolkit.convolve.kernel.BoxConvolve",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,margin=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Convolution with a box kernel (mean)"]; "KernelConvolve" -> "BoxConvolve" [arrowsize=1.2,arrowtail=empty,dir=back,style="setlinewidth(0.5)"]; "ConvolveBase" [URL="../../../api/sofia_redux.toolkit.convolve.base.ConvolveBase.html#sofia_redux.toolkit.convolve.base.ConvolveBase",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,margin=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Convolution class allowing error propagation."]; "Model" -> "ConvolveBase" [arrowsize=1.2,arrowtail=empty,dir=back,style="setlinewidth(0.5)"]; "KernelConvolve" [URL="../../../api/sofia_redux.toolkit.convolve.kernel.KernelConvolve.html#sofia_redux.toolkit.convolve.kernel.KernelConvolve",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,margin=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Generic convolution with a kernel"]; "ConvolveBase" -> "KernelConvolve" [arrowsize=1.2,arrowtail=empty,dir=back,style="setlinewidth(0.5)"]; "Model" [URL="../../../api/sofia_redux.toolkit.utilities.base.Model.html#sofia_redux.toolkit.utilities.base.Model",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,margin=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base model Class for fitting N-dimensional data"]; "SavgolConvolve" [URL="../../../api/sofia_redux.toolkit.convolve.kernel.SavgolConvolve.html#sofia_redux.toolkit.convolve.kernel.SavgolConvolve",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,margin=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Convolve using Savitzky-Golay filter"]; "ConvolveBase" -> "SavgolConvolve" [arrowsize=1.2,arrowtail=empty,dir=back,style="setlinewidth(0.5)"]; }

sofia_redux.toolkit.convolve.filter Module

Functions

savgol(data, window[, order, axes, check, ...])

Apply Savitzky-Golay filter to an array of arbitrary features

savgol_windows(order, window, *samples[, scale])

Creates the correct windows for given order and samples

sobel(input_array[, kderiv, kperp, pnorm, ...])

Edge enhancement Sobel filter for n-dimensional images.

sofia_redux.toolkit.fitting.fitpeaks1d Module

Functions

parse_width_arg(model_class, width_arg)

Simple convenience lookup to get width parameter name for various models.

get_fitter(fitter_class[, robust, ...])

Creates the object fitting a model to data

dofit(fitter, model, x, y, **kwargs)

A simple wrapper to fit model to the data

box_convolve(model, box_class[, box_width, ...])

Convolve a model with a box (or another model)

get_search_model(peak_model[, box_class, ...])

Create the initial_search peak fitting model

initial_search(fitter, model, x, y[, ...])

Perform an initial search for peaks in the data

get_background_fit(fitter, peak_model, ...)

Return a background model with initialized parameters

get_final_model(peak_class, pinit[, ...])

Refine the initial fit and return a set of models

fitpeaks1d(x, y[, npeaks, xrange, ...])

Fit peaks (and optionally background) to a 1D set of data.

medabs_baseline(_, y)

Default data preparation for initial_search and baseline function.

guess_xy_mad(x, y)

Default peak guess function for initial_search.

sofia_redux.toolkit.fitting.polynomial Module

Functions

polyexp(order[, ndim, indexing])

Returns exponents for given polynomial orders in arbitrary dimensions.

polysys(samples, order[, exponents, error, ...])

Create a system of linear equations to solve n-D polynomials

linear_equation(design_matrix, values[, ...])

Create a system of linear equations

gaussj(alpha, beta[, invert, preserve])

Linear equation solution by Gauss-Jordan elimination and matrix inversion

poly1d(x, coeffs[, covar])

Evalulate polynomial coefficients at x

polynd(v, coefficients[, exponents, ...])

Evaluate a polynomial in multiple features

zero_order_fit(data[, error])

Calculate the zeroth order polynomial coefficients and covariance

linear_polyfit(samples, order[, exponents, ...])

Fit a polynomial to data samples using linear least-squares.

gaussj_polyfit(samples, order[, exponents, ...])

Fit a polynomial to data samples using Gauss-Jordan elimination.

nonlinear_polyfit(samples, order[, ...])

Solve for polynomial coefficients using non-linear least squares fit

polyfitnd(*args[, error, mask, covar, ...])

Fits polynomial coefficients to N-dimensional data.

linear_vector_lstsq(alpha, beta, matrix_out)

nonlinear_coefficients(matrix, data[, ...])

nonlinear_evaluate(matrix_in, data, matrix_out)

polyfit2d(x, y, z[, kx, ky, full])

Least squares polynomial fit to a surface

poly2d(x, y, coeffs)

Evaluate 2D polynomial coefficients

polyinterp2d(x, y, z, xout, yout[, kx, ky, ...])

Interpolate 2D data using polynomial regression (global)
Classes

Polyfit(*args[, error, mask, covar, stats, ...])

Fits and evaluates polynomials in N-dimensions.
Class Inheritance Diagram
digraph inheritance812590e9a1 { bgcolor=transparent; rankdir=LR; size=""; "Model" [URL="../../../api/sofia_redux.toolkit.utilities.base.Model.html#sofia_redux.toolkit.utilities.base.Model",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,margin=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base model Class for fitting N-dimensional data"]; "Polyfit" [URL="../../../api/sofia_redux.toolkit.fitting.polynomial.Polyfit.html#sofia_redux.toolkit.fitting.polynomial.Polyfit",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,margin=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Fits and evaluates polynomials in N-dimensions."]; "Model" -> "Polyfit" [arrowsize=1.2,arrowtail=empty,dir=back,style="setlinewidth(0.5)"]; }

sofia_redux.toolkit.image.adjust Module

Functions

shift(data, offset[, order, missing, ...])

Shift an image by the specified amount.

rotate(data, angle[, order, missing, ...])

Rotate an image.

frebin(data, shape[, total, order, ...])

Rebins an array to new shape

image_shift(data, shifts[, order, missing])

Shifts an image by x and y offsets

rotate90(image, direction)

Replicates IDL rotate function

unrotate90(image, direction)

Un-rotates an image using IDL style rotation types

register_image(image, reference[, upsample, ...])

Return the pixel offset between an image and a reference

upsampled_dft(data, upsampled_region_size[, ...])

Upsampled DFT by matrix multiplication.

sofia_redux.toolkit.image.coadd Module

Functions

coadd(hdr_list, data_list, var_list, exp_list)

Coadd total intensity or spectral images.

sofia_redux.toolkit.image.combine Module

Functions

combine_images(data[, variance, method, ...])

Combine input image arrays.

sofia_redux.toolkit.image.fill Module

Functions

clough_tocher_2dfunc(d, cin, cout, **kwargs)

spline_interp_2dfunc(d, cin, cout, **kwargs)

maskinterp(image[, func, mask, apstep, ...])

Interpolates over image using a mask.

image_naninterp(data)

Fills in NaN values in an image

polyclip(i, j, pol_x, pol_y[, area])

Clip a polygon to a square unit pixel

polyfillaa(px, py[, xrange, yrange, ...])

Finds all pixels at least partially inside a specified polygon

polygon_area(ppath)

Uses the shoelace method to calculate area of a polygon

polygon_weights(polygon[, xrange, yrange, ...])

Get pixel weights - depreciated by polyfillaa

sofia_redux.toolkit.image.resize Module

Functions

resize(image, output_shape[, order, mode, ...])

Replacement for skimage.resize.

sofia_redux.toolkit.image.smooth Module

Functions

quadfit(image)

Quick and simple cubic polynomial fit to surface - no checks

bicubic_coefficients(z, dx, dy, dxy, nx, ny)

Returns the coefficients necessary for bicubic interpolation.

bicubic_evaluate(z, dx, dy, dxy, xrange, ...)

Parameters:

fitplane(points)

Fit a plane to distribution of points.

fiterpolate(image, nx, ny)

Fits a smooth surface to data using J.

sofia_redux.toolkit.image.utilities Module

Functions

to_ndimage_mode(mode)

Convert from numpy.pad mode name to the corresponding ndimage mode.

fix_ndimage_mode(mode)

Allow translation of modes for scipy versions >= 1.6.0

clip_output(original, warped, mode, cval, clip)

Clip the array to the range of original values.

map_coordinates(data, coordinates[, order, ...])

A drop in replacement for ndimage.map_coordinates().

sofia_redux.toolkit.image.warp Module

Functions

warp_image(data, xin, yin, xout, yout[, ...])

Warp data using transformation defined by two sets of coordinates

polywarp(xi, yi, xo, yo[, order])

Performs polynomial spatial warping

polywarp_image(image, x0, y0, x1, y1[, ...])

Warp an image by mapping 2 coordinate sets with a polynomial transform.

is_homography_transform(transform, n_dimensions)

Check if a transform is homographic.

full_transform(coordinates, transform)

Apply a metric transform to the supplied coordinates.

warp_terms(terms, coefficients)

Apply coefficients to polynomial terms.

estimate_polynomial_transform(source, ...[, ...])

Estimate the polynomial transform for (x, y) coordinates.

warp_coordinates(coordinates, source, ...[, ...])

Apply the warping between two sets of coordinates to another.

warp_array_elements(source, destination, shape)

Warp the indices of an array with a given shape using a polynomial.

warp(data, source, destination[, order, ...])

Warp an n-dimensional image according to a given coordinate transform.
Classes

PolynomialTransform([source, destination, order])

Initialize a polynomial transform.
Class Inheritance Diagram
digraph inheritance2c28666683 { bgcolor=transparent; rankdir=LR; size=""; "ABC" [URL="https://docs.python.org/3/library/abc.html#abc.ABC",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,margin=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Helper class that provides a standard way to create an ABC using"]; "PolynomialTransform" [URL="../../../api/sofia_redux.toolkit.image.warp.PolynomialTransform.html#sofia_redux.toolkit.image.warp.PolynomialTransform",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,margin=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "ABC" -> "PolynomialTransform" [arrowsize=1.2,arrowtail=empty,dir=back,style="setlinewidth(0.5)"]; }

sofia_redux.toolkit.interpolate.interpolate Module

Functions

line_shift(y, offset[, order, missing])

Shift an equally spaced array of data values by an offset

interpolate_nans(x, y, xout[, missing, ...])

Interpolate values containing NaNs

spline(x, y, xout[, sigma])

Perform cubic spline (tensioned) interpolation

sincinterp(x, y, xout[, dampfac, ksize, ...])

Perform a sinc interpolation on a data set

interp_1d_point(x, y, xout)

Perform linear interpolation at a single point.

interp_1d_point_with_error(x, y, error, xout)

Perform linear interpolation at a single point with error propagation.

interp_error_1d(x, error, xout[, cval])

Perform linear interpolation of errors

interp_error_nd(points, error, xi[, cval])

Propagate errors using Delaunay triangulation in N-dimensions

interp_error(points, error, xi[, cval])

Propagate errors using linear interpolation in N-dimensions

tabinv(array, xvals[, missing, fast])

Find the effective index of a function value in an ordered vector with NaN handling.

findidx(ix, ox[, left, right])

Finds the effective index of a function value in an ordered array.
Classes

Interpolate(*args[, method, cval, cubic, ...])

Fast interpolation on a regular grid
Class Inheritance Diagram
digraph inheritanceb8ee21f950 { bgcolor=transparent; rankdir=LR; size=""; "Interpolate" [URL="../../../api/sofia_redux.toolkit.interpolate.interpolate.Interpolate.html#sofia_redux.toolkit.interpolate.interpolate.Interpolate",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,margin=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Fast interpolation on a regular grid"]; }

sofia_redux.toolkit.stats.stats Module

Functions

find_outliers(data[, threshold, keepnans, axis])

Determines the outliers in a distribution of data

meancomb(data[, variance, mask, rms, axis, ...])

(Robustly) averages arrays along arbitrary axes.

medcomb(data[, variance, mask, mad, axis, ...])

Combines a data set using median

moments(data[, threshold, mask, axis, get_mask])

Computes statistics on a data set avoiding deviant points if requested

robust_mask(data, threshold[, mask, axis, ...])

Computes a mask derived from data Median Absolute Deviation (MAD).

sofia_redux.toolkit.utilities.base Module

Classes

Model(*args[, error, mask, covar, stats, ...])

Base model Class for fitting N-dimensional data
Class Inheritance Diagram
digraph inheritanced18b721993 { bgcolor=transparent; rankdir=LR; size=""; "Model" [URL="../../../api/sofia_redux.toolkit.utilities.base.Model.html#sofia_redux.toolkit.utilities.base.Model",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,margin=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base model Class for fitting N-dimensional data"]; }

sofia_redux.toolkit.utilities.fits Module

Functions

hdinsert(header, key, value[, comment, ...])

Insert or replace a keyword and value in the header

add_history(header, msg[, prefix, refkey])

Add HISTORY message to a FITS header before the pipeline.

add_history_wrap(prefix)

Make a function to add HISTORY messages to a header, prefixed with a string.

robust_read(filename[, data_hdu, ...])

Retrieve the data and header from a FITS file

getheader(filename[, hdu, verbose])

Returns the header of a FITS file

getdata(filename[, hdu, verbose])

Returns the data from a FITS file

header_to_chararray(header)

Convert a FITS header to an array of strings

chararray_to_header(chararray)

Convert an array of strings to a FITS header

gethdul(filename[, verbose])

Returns the HDUList from a FITS file

write_hdul(hdul[, outdir, overwrite])

Write a HDULists to disk.

get_key_value(header, key[, default])

Get a key value from a header.

set_log_level(level)

Context manager to temporarily set the log level.

order_headers(headers)

Order headers based on contents.

merge_headers(headers, keyword_configuration)

Merge input headers.

sofia_redux.toolkit.utilities.func Module

Functions

robust_bool(value)

Check for 'truthy' values.

valid_num(value)

Check for valid numbers.

natural_sort(string_list[, reverse])

Returns list sorted in a human friendly manner

goodfile(filename[, read, write, execute, ...])

Check if a file exists, and optionally if it has the correct permissions.

date2seconds(datestring[, dformat])

Convert a header datestring to seconds

str_to_value(text)

Convert a string to an int or float.

slicer(array, axis, index[, ind])

Returns a slice of an array in arbitrary dimension.

setnumber(value[, minval, maxval, default, ...])

Sets a value to a valid number type

gaussian_model(x, x0, amplitude, fwhm, y0)

Gaussian model for curve_fit

to_array_shape(value, shape[, dtype])

Broadcast an array to the desired shape.

recursive_dict_update(original, new)

Recursively update a dictionary

stack(*samples[, copy])

faststack(*samples)

taylor(order, n)

Taylor expansion generator for Polynomial exponents

bytes_string(size_bytes)

Convert a number of bytes to a string with correct suffix

remove_sample_nans(samples, error[, mask])

Remove any samples containing NaNs from sample points

bitset(arr, bits[, skip_checks])

Return a byte array the same size as the input array.

julia_fractal(sy, sx[, c0, c1, iterations, ...])

Generate a 2-D Julia fractal image

nantrim(xin[, flag, trim, bounds])

Derive a mask to trim NaNs from an array

nansum(a[, axis, dtype, out, keepdims, missing])

Emulates the behaviour of np.nansum for NumPy versions <= 1.9.0.

sofia_redux.toolkit.utilities.multiprocessing Module

Functions

get_core_number([cores])

Returns the maximum number of CPU cores available

relative_cores(jobs)

Return the actual number of cores to use for a given number of jobs.

valid_relative_jobs(jobs)

Return a valid number of jobs in the range 1 <= jobs <= max_cores.

multitask(func, iterable, args, kwargs[, ...])

Process a series of tasks in serial, or in parallel using joblib.

pickle_object(obj, filename)

Pickle a object and save to the given filename.

unpickle_file(filename)

Unpickle a string argument if it is a file, and return the result.

pickle_list(object_list[, prefix, ...])

Pickle a list of objects to a temporary directory.

unpickle_list(pickle_files[, delete])

Restore pickle files to objects in-place.

in_main_thread()

Return whether the process is running in the main thread.

log_with_multi_handler(logger)

Context manager to temporarily log messages for unique processes/threads

log_for_multitask(logger)

Context manager to output log messages during multiprocessing.

purge_multitask_logs(log_directory, ...[, ...])

Remove all temporary logging files/directories and handle log records.

wrapped_with_logger(func, ...)

Return the results of the function in multitask and save log records.

log_records_to_pickle_file(logger, pickle_file)

Store the log records in a pickle file rather than emitting.

wrap_function(func, args[, kwargs, logger, ...])

Wrap a function for use with multitask().
Classes

MultitaskHandler()

A log handler for multitask.
Class Inheritance Diagram
digraph inheritance7a3b669de2 { bgcolor=transparent; rankdir=LR; size=""; "Filterer" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,margin=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A base class for loggers and handlers which allows them to share"]; "Handler" [URL="https://docs.python.org/3/library/logging.html#logging.Handler",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,margin=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Handler instances dispatch logging events to specific destinations."]; "Filterer" -> "Handler" [arrowsize=1.2,arrowtail=empty,dir=back,style="setlinewidth(0.5)"]; "MultitaskHandler" [URL="../../../api/sofia_redux.toolkit.utilities.multiprocessing.MultitaskHandler.html#sofia_redux.toolkit.utilities.multiprocessing.MultitaskHandler",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,margin=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A log handler for multitask."]; "Handler" -> "MultitaskHandler" [arrowsize=1.2,arrowtail=empty,dir=back,style="setlinewidth(0.5)"]; }

sofia_redux.spectroscopy

sofia_redux.spectroscopy.binspec Module

Functions

binspec(x, y, delta[, xout, lmin, lmax, average])

Bin a spectrum between lmin and lmax with bins delta wide

sofia_redux.spectroscopy.earthvelocity Module

Functions

cartesian_lsr([definition])

Find the radial LSR velocity towards sky coordinates.

cartesian_helio(time[, center, location])

Calculate the Cartesian velocity of the Sun.

earthvelocity(ra, dec, time[, equinox, ...])

Provide velocities of the Earth towards a celestial position.

sofia_redux.spectroscopy.extspec Module

Functions

col_subbg(col_arc, col_image, col_var, ...)

Fit background to a single column.

extspec(rectimg[, profile, spatial_map, ...])

Extracts spectra from a rectified spectral image.

sofia_redux.spectroscopy.findapertures Module

Functions

find_apertures(profiles[, npeaks, orders, ...])

Determine the position of the aperture(s) in a spatial profile.

sofia_redux.spectroscopy.fluxcal Module

Functions

get_wave_shift(flux, correction, ...[, ...])

Get pixel shift between flux and correction curve.

fluxcal(spectra, atran[, response, ...])

Calibrate and telluric correct spectral flux.

sofia_redux.spectroscopy.getapertures Module

Functions

get_apertures(profiles, apertures[, ...])

Determine aperture radii for extraction.

sofia_redux.spectroscopy.getspecscale Module

Functions

getspecscale(stack[, refidx])

Determines the scale factors for a _stack of spectra

sofia_redux.spectroscopy.mergespec Module

Functions

mergespec(spec1, spec2[, info, sum_flux, ...])

Combine two spectra into a single spectrum

sofia_redux.spectroscopy.mkapmask Module

Functions

mkapmask(slit, wave, apertures[, background])

Constructs a 2D aperture mask.

sofia_redux.spectroscopy.mkspatprof Module

Functions

mkspatprof(rectimg[, atran, atmosthresh, ...])

Construct average spatial profiles.

sofia_redux.spectroscopy.radvel Module

Functions

radvel(header[, equinox])

Calculate the expected extrinsic radial velocity wavelength shift.

sofia_redux.spectroscopy.readflat Module

Functions

readflat(filename)

Reads a Spextool flat field FITS image

sofia_redux.spectroscopy.readwavecal Module

Functions

readwavecal(filename[, rotate, info])

Read a Spextool wavecal file

sofia_redux.spectroscopy.rectify Module

Functions

rectify(image, ordermask, wavecal, spatcal)

Construct average spatial profiles over multiple orders

sofia_redux.spectroscopy.rectifyorder Module

Functions

get_rect_xy(xarray, yarray, xvals, yvals[, ...])

Given arrays of x and y coordinates, interpolate to defined grids

trim_xy(xarray, yarray, xgrid, ygrid[, ...])

Trim rows and columns from the edges of the coordinate arrays.

rectifyorder(image, ordermask, wavecal, ...)

Construct average spatial profiles for a single order

update_wcs(result, spatcal)

Update a FITS header with spectral WCS information.

rectifyorder(image, ordermask, wavecal, ...)

Construct average spatial profiles for a single order

sofia_redux.spectroscopy.smoothres Module

Functions

smoothres(x, y, resolution[, siglim])

Smooth a data to a constant resolution

sofia_redux.spectroscopy.tracespec Module

Functions

tracespec(rectimg, positions[, orders, ...])

Trace spectral continua in a spatially/spectrally rectified image.

sofia_redux.visualization

sofia_redux.visualization.quicklook Module

Functions

make_image(filename[, extension, colormap, ...])

Generate a map image from a FITS file.

make_spectral_plot(axis, wavelength, ...[, ...])

Generate a plot of spectral data.

sofia_redux.visualization.redux_viewer Module

Classes

EyeViewer()

Redux Viewer interface to the Eye of SOFIA.
Class Inheritance Diagram
digraph inheritance28e1a7d94e { bgcolor=transparent; rankdir=LR; size=""; "EyeViewer" [URL="../../../api/sofia_redux.visualization.redux_viewer.EyeViewer.html#sofia_redux.visualization.redux_viewer.EyeViewer",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,margin=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Redux Viewer interface to the Eye of SOFIA."]; "Viewer" -> "EyeViewer" [arrowsize=1.2,arrowtail=empty,dir=back,style="setlinewidth(0.5)"]; "Viewer" [URL="../../../api/sofia_redux.pipeline.viewer.Viewer.html#sofia_redux.pipeline.viewer.Viewer",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,margin=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Parent class for Redux data viewers."]; }

sofia_redux.visualization.controller Module

Standalone front-end for Eye of SOFIA display tool.

Functions

main()

The Eye of SOFIA spectral viewer.

parse_args(args)

Parse command line arguments.

check_args(args)

Check arguments for validity.

sofia_redux.visualization.eye Module

Classes

Eye([args, view_])

Run the Eye of SOFIA.
Class Inheritance Diagram
digraph inheritancebb54a4d2b2 { bgcolor=transparent; rankdir=LR; size=""; "Eye" [URL="../../../api/sofia_redux.visualization.eye.Eye.html#sofia_redux.visualization.eye.Eye",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,margin=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Run the Eye of SOFIA."]; }

sofia_redux.pipeline

The Redux application programming interface (API), including the EXES interface classes, are documented in the sofia_redux.pipeline package.

Appendix A: Pipeline Recipe

This JSON document is the black-box interface specification for the EXES Redux pipeline, as defined in the Pipetools-Pipeline ICD.

{
    "inputmanifest" : "infiles.txt",
    "outputmanifest" : "outfiles.txt",
    "env" : {
        "DPS_PYTHON": "$DPS_SHARE/share/anaconda3/envs/exes/bin"
    },
    "knobs" : {
        "REDUX_CONFIG" : {
            "desc" : "Redux parameter file containing custom configuration.",
            "type" : "string",
            "default": "None"
        }
    },
    "command" : "$DPS_PYTHON/redux_pipe infiles.txt -c $DPS_REDUX_CONFIG"
}