SOFIA Redux

Introduction

SOFIA Redux (sofia_redux) contains data processing pipelines and algorithms for instruments on the Stratospheric Observatory for Infrared Astronomy (SOFIA).

SOFIA raw and processed data can be accessed from the SOFIA archive. Archived data may not match the results of data processed with this pipeline software.

SOFIA pipelines are developed internally by the USRA/SOFIA data processing software team, then are published publicly at the SOFIA Redux GitHub project. Contributions and feedback are welcome via the GitHub project, but merge requests cannot be directly accepted. They will be internally reviewed, and pushed to the public site as needed.

Getting Started

Installation

Stable release

The sofia_redux package is available via anaconda or pip:

conda install -c sofia-usra -c conda-forge sofia_redux

or:

pip install sofia_redux

From source

Obtain the source code for this package from the SOFIA Redux GitHub project, then install via one of the two methods below.

Via Anaconda

We recommend Anaconda for managing your Python environment. A conda environment specification is included with this package, as environment.yml.

To install a sofia_redux environment with Anaconda from the package directory:

conda env create -f environment.yml

Activate the environment:

conda activate sofia_redux

Install an editable copy of the sofia_redux package:

pip install -e .

Deactivate the environment when done:

conda deactivate

Remove the environment if necessary:

conda remove --name sofia_redux --all
Via Pip

Alternately, prerequisites for the package can be installed with:

pip install -r requirements.txt

and the package can then be installed as usual:

pip install -e .

Optional Requirements

PyQt5

If sofia_redux is installed via pip, the PyQt5 package, required for the pipeline GUI interface, is not automatically installed as a dependency. To use the GUI tools, install PyQt5 via pip:

pip install PyQt5

or conda:

conda install pyqt

Please note that there may be some incompatibilities between some versions of PyQt5, some versions of the package dependencies, and some host OS versions. If you run into difficulty, we recommend using the Anaconda installation method.

DS9

Some optional visualization tools in the SOFIA Redux interface also use the pyds9 and regions packages to interface with the external SAOImage DS9 tool. To use these tools, install DS9, then install pyds9 and regions directly via pip:

pip install pyds9 regions

or using the provided optional requirements file:

pip install -r optional_requirements.txt

Please note that pyds9 requires gcc to compile, and is not available on the Windows platform. On MacOS, you will need to make a ds9 executable available in your PATH environment variable; see the DS9 FAQs for more information.

Reference data

Some pipeline modes require additional reference data for optimal data quality. These files are too large to distribute with the pipeline code, so they are provided separately.

Atmospheric models

For optimal telluric correction, FORCAST, FLITECAM, and FIFI-LS spectroscopic reductions require a library of FITS files, containing model atmospheric transmission spectra, derived from the ATRAN model.

Two versions of the model libraries are available for each instrument, except FLITECAM. ATRAN files parameterized by water vapor are not available for FLITECAM.

The EXES pipeline does not use ATRAN models for telluric correction, but it does attach a reference atmospheric model at a matching altitude and zenith angle to output spectral products, if available. The models used are derived from the Planetary Spectrum Generator (PSG).

  • FORCAST:

    • Approximate models, not accounting for water vapor variation

    • More accurate models, enabling programmatic optimization of the telluric correction

  • FIFI-LS:

    • Approximate models, not accounting for water vapor variation

    • More accurate models, enabling use of water vapor values recorded in the FITS headers for more accurate telluric correction

  • FLITECAM:

    • Approximate models, not accounting for water vapor variation

  • EXES:

    • Approximate models, not accounting for water vapor variation

After downloading and unpacking the library, its location can be provided to the pipeline as an optional parameter in the Calibrate Flux step for FORCAST or FLITECAM, the Telluric Correct step for FIFI-LS, or the Extract Spectra step for EXES.

Standard flux models

In addition to the ATRAN models, a library of standard flux models is required to reduce FORCAST or FLITECAM standard spectra to instrumental response curves. This should be rarely needed for standard scientific reductions, since reference response curves are provided for most data. If needed for re-deriving spectral flux calibrations, the standard model spectra are provided in the source distribution of this package, at sofia_redux/instruments/forcast/data/grism/standard_models or sofia_redux/instruments/flitecam/data/grism/standard_models.

FLITECAM and EXES auxiliary data

The default auxiliary calibration and reference data for FLITECAM and EXES reductions are too large to be included in the software packages provided via PyPI or Anaconda.

These files are provided in full in the source distribution of this package. Since they are required for most data reductions for these instruments, they may also be automatically downloaded as needed for non-source installations (i.e. via pip or conda). Downloaded calibration files are cached for later use in a ‘.sofia_redux’ directory in the user’s home directory. For offline pipeline reductions, the source installation is recommended.

For FLITECAM, the data provided in this manner includes nonlinearity correction coefficients, spectroscopic order masks, and wavelength calibration files. For EXES, the large data files are bad pixel masks, reset dark files, and nonlinearity correction coefficients.

Troubleshooting

Please note that direct support for this project will end in September 2023.

Prior to this time, please submit a ticket on the GitHub issues page for installation assistance.

After this time, the source distribution of this package will remain available, but it will not be maintained for the latest versions of all dependencies. It is recommended that users fork their own copies of this package for continued maintenance and development.

The last working set of installed versions of all dependencies is recorded in the freeze_requirements.txt file in this package. If errors are encountered in the other listed installation methods, it may be useful to install the frozen versions directly. For example, to install from source using conda to create a new Python environment from the sofia_redux package directory:

conda create --name sofia_redux python=3.10
pip install -r freeze_requirements.txt
pip install -e .

Running SOFIA pipelines

Out of the box, Redux provides two primary command-line scripts:

  • redux (sofia_redux.pipeline.sofia.redux_app): An interactive graphical interface (GUI) for the SOFIA pipelines.

  • redux_pipe (sofia_redux.pipeline.sofia.redux_pipe): A command-line interface to the SOFIA pipelines.

Currently, the EXES, FIFI-LS, FLITECAM, FORCAST, and HAWC+ instruments are supported by these pipelines. See the user’s and developer’s manuals for each instrument for full descriptions of all supported observing modes and scientific algorithms.

In general, Redux works by reading in input data, deciding which reduction to run, then running a pre-defined set of reduction steps.

The GUI allows interactive parameter editing, and intermediate product display for each step. To begin a reduction, start the GUI by typing redux, then load in a set of data with the File->Open New Reduction menu.

The command-line interface allows fully automatic pipeline reductions. To begin a reduction with the automatic pipeline, type redux_pipe and provide the file names of the data to reduce on the command line. Non-default parameters can also be provided by specifying a configuration file in INI format. Type redux_pipe -h for a brief help message.

Tutorials

Tutorials are available for data reduction procedures, via the SOFIA website:

Manuals

User’s Manuals:

Developer’s Manuals:

Submodules