Integration

class sofia_redux.scan.integration.integration.Integration(scan=None)

Bases: ABC

Initialize a scan integration.

An integration is used to represent a chunk of timestream data from a given scan (frames). A valid scan must contain one or more integrations. The integration class provides methods to operate on the frame data for various reduction tasks, and also provides a link between the timestream data, and the instrument channels. Each frame/channel combination is referred to as a sample.

Parameters:

scansofia_scan.scan.scan.scan.Scan

The scan to which the integration belongs.

Attributes Summary

channel_flagspace	Return the channel flagspace.
configuration	Return the integration configuration (from channel configuration).
flagspace	Return the frame flagspace.
info	Return the Info object for the integration.
instrument_name	Return the instrument name.
motion_flagspace	Return the motion flagspace.
n_channels	Return the number of channels in the integration.
reference_attributes	Return attributes that should be referenced rather than copied.
scan_astrometry	Return the scan astrometry.
size	Return the number of frames in the integration.

Methods Summary

acceleration_clip([max_acceleration])	Invalidate frames with excessive acceleration.
acceleration_cut(max_acceleration)	Invalidate frames with excessive acceleration.
add_dependents(dependents)	Add dependents to the integration.
add_details([table])	Add integration details to an astropy table.
add_signal(signal)	Add a signal to the integration signals.
apply_configuration()	Apply configuration options to an integration.
apply_notch_filter(frequencies, width)	Apply notch filtering to frame data for given frequencies and width.
bootstrap_weights()	Calculates channel weights from frame data if required.
calculate_scan_speed_stats()	Calculates and reports statistics on the scanning speed.
calculate_source_nefd()	Set the source Noise-Equivalent-Flux-Density (NEFD).
check_consistency(channels, frame_dependents)	Check consistency of frame dependents and channels.
check_range()	Checks the range of frame data values, flagging as necessary.
clone()	Return a copy of the integration without frame type data.
condense_covariance(covariance)	Strips a covariance matrix of all zero and NaN column/row pairs.
copy()	Return a new copy of the integration.
decorrelate(modality_name[, robust])	Decorrelate a modality.
decorrelate_signals(modality_name[, robust])	Decorrelate a modality signal.
dejump_frames()	Remove jumps in the frame data.
despike([level])	Despike frame data.
despike_absolute(level)	Identifies and flags spikes in frame data using absolute method.
despike_gradual(level[, depth])	Identifies and flags spikes in frame data using gradual method.
despike_multi_resolution(level)	Identifies and flag spikes in frame data using multi-resolution.
despike_neighbouring(level, delta)	Identifies and flags spikes in frame data using neighbours method.
detector_stage()	Divide the frame data by the channel hardware gains.
downsample([factor])	Downsample frame data by a given, or derived factor.
downsampling_window(factor)	Create a normalized Hann window and the new indices.
fill_gaps()	Add invalid frames in cases where there are timestream gaps.
filter_frames_for([spec, default_time])	Return the number of filtering frames for a given specification.
flag_spiky_blocks()	Flag all frames in a block if one or more frames is flagged as spiky.
flag_spiky_channels([flag_fraction, flag_count])	Flag channels with excessive spiky frames as SPIKY.
flag_spiky_frames([frame_spikes])	Flag frames with excessive spiky channels as FLAG_SPIKY.
flag_weights()	Check if channel weights should be flagged and update source NEFD.
frames_for([time])	Return the number of frames for a given time interval.
get_acceleration_signal(direction[, mode])	Return an acceleration signal.
get_accelerations()	Return the telescope accelerations.
get_ascii_header()	Return a string header for output text files.
get_channel_weights([method])	Derive and set channel weights.
get_correlated_signal_class()	Return the correlated signal class.
get_coupling_gains(signal)	Return the coupling gains for a given channel.
get_covariance()	Return the channel covariance.
get_crossing_time([source_size])	Return the crossing time for a given source size.
get_default_scaling_factor()	Return the default scaling factor from the configuration.
get_dependents(name)	Return or create and return dependents of the given name.
get_differential_channel_weights()	Derive and set differentially derived weights for live channels.
get_display_id()	Get the display integration ID.
get_downsample_factor()	Return the factor by which to downsample frames.
get_duration()	Return the total integration duration.
get_exposure_time()	Return the total exposure time.
get_file_id()	Get the file integration ID.
get_filter(filter_name)	Return a filter of the given name.
get_first_frame([reference])	Return the first valid frame.
get_first_frame_index([reference])	Return the first valid frame index of the integration.
get_fits_data()	Return integration data as an astropy table.
get_floats()	Return a float array suitable for FFT.
get_frame_count([keep_flag, discard_flag, ...])	Return the number of frames in an integration.
get_frame_gaps()	Return the gaps between frames.
get_full_covariance(covariance)	Return the full covariance array for all initial channels.
get_full_id([separator])	Return the full integration ID.
get_group_covariance(division, covariance)	Return the covariance for all groups in a channel division.
get_id()	Return the simple integration ID.
get_integration_class(name)	Return an Integration instance given an instrument name.
get_last_frame([reference])	Return the first valid frame.
get_last_frame_index([reference])	Return the last valid frame index of the integration.
get_mean_level([channels, start, end, robust])	Return the median data values and associated weights for channels.
get_mjd()	Return the midpoint MJD of the integration.
get_modulation_frequency(signal_flag)	Return the modulation frequency.
get_pa()	Return the position angle of the integration.
get_point_crossing_time()	Return the time required to cross the point size of the scan.
get_position_signal(motion_flag, direction)	Return a position signal.
get_positions(motion_flag)	Return positions based on a type of motion.
get_rms_channel_weights()	Derive and set channel weights for live channels based on data values.
get_robust_channel_weights()	Derive and set robustly derived channel weights for live channels.
get_scanning_velocities([return_position])	Return the scanning velocity (including chopping motion).
get_signal(mode)	Return the signal for a given mode.
get_smooth_positions(motion_flag)	Return the positions, smoothed according to the configuration.
get_spectra([window_function, window_size])	Return the power spectrum of the data.
get_speed_clip_range()	Determine the range of permissible scanning speeds.
get_standard_id([separator])	Get the standard integration ID.
get_table_entry(name)	Return a parameter value for the given name.
get_tau(spec[, value])	Get the tau value based on configuration settings.
get_tau_coefficients(spec)	Return the tau coefficients for the tau specification.
get_thread_count()	Return the number of parallel tasks for the integration.
get_time_stream([channels, weighted])	Return the time stream data.
get_time_weights([channels, block_size, flag])	Derive and set frame weights.
get_typical_scanning_speed()	Get the mean (robust) scanning speed and weight (inverse variance).
get_velocities(motion_flag[, return_position])	Return the scanning velocity (including chopping motion).
has_gaps([tolerance])	Check if there are gaps in the time stream data.
has_option(option)	Return whether the configuration option is configured.
jackknife()	Randomly invert integration and channel gains.
level([channel_means, channel_weights, ...])	Remove the mean frame data from each channel and update dependents.
local_level(parms[, start, end, robust])	Level frame data between in a certain frame range.
merge(integration)	Merge frames from another integration onto this integration.
next_iteration()	Perform certain steps prior to an iteration.
next_weight_transit(level[, start, above])	Find the next frame index with relative weight above or below a level.
notch_filter()	Apply notch filtering based on configuration options.
offset(value)	Add an offset value to frame data for all unflagged channels.
perform(task)	Perform a reduction task on the integration.
pointing_at(offset[, indices])	Applies pointing correction to coordinates via subtraction.
power2_frames_for([time])	Return the number of frames for a given time interval using pow2ceil.
process_notch_filter_block(start, end, ...)
readout_stage()	Multiply the frame data by the channel hardware gains.
reindex()	Reset the frame fixed indices to range sequentially from 0 -> size.
reindex_channels()	Reindex the channels such that all data are self-consistent.
reindex_frames()	Reindex the time dependent data.
remove_channel_drifts(channel_group, parms, ...)	Remove drifts from channels.
remove_dc_offsets()	Remove the DC offsets, either if explicitly requested or to allow bootstrapping pixel weights when pixeldata is not defined.
remove_drifts([target_frame_resolution, robust])	Remove drifts in frame data given a target frame resolution.
remove_offsets([robust])	Remove the DC offsets from the frame data.
scale(factor)	Scale all data (in data) by factor.
search_corners(perimeter_pixels, projection)	Find the corners of the mapping range of the integration.
select_frames()	Delete frames not inside the frame range defined in the configuration.
set_iteration(iteration[, rounds])	Set the configuration for a given iteration
set_scaling([factor])	Set the gain scaling for the integration.
set_scan(scan)	Set the parent scan of the integration.
set_tau([spec, value])	Set the tau values for the integration.
set_tau_value(value)	Set the zenith tau if ground based, or the transmission.
set_temp_despike_levels(channel_data, level)	Set a temporary despiking level reference in the channel data.
set_thread_count(threads)	Set the number of parallel tasks for the integration.
set_weights_from_var_stats(channel_group, ...)	Set channel weights from the calculated variance.
set_zenith_tau(tau_value)	Set the zenith tau value.
setup_filters()	Set up the FFT frame data filters.
shift_frames(time_or_frames)	Shift the frame data.
slim()	Slim frame/channel type data to those channels still present.
trim([start, end])	Remove invalid frames from the start and end of the integration.
update_gains(modality_name[, robust])	Update gains in a modality following decorrelation.
update_inconsistencies(channels, ...)	Check consistency of frame dependents and channels.
validate()	Validate an integration following the initial read.
velocity_clip([speed_range, strict, sigma_clip])	Clip frames that are outside of the permissible scanning speed range.
write_ascii_time_stream([filename])	Write the frame data to a text file.
write_coupling_gains(signal_names)	Write the coupling gains to file.
write_covariance_to_file(filename, covariance)	Write a covariance matrix to file.
write_covariances()	Write the covariances to file.
write_products()	Write the integration products to an output file.
write_scan_pattern([filename])	Write the scan pattern to file.
write_spectra([window_name, window_size])	Write the spectra to file.

Attributes Documentation

channel_flagspace

Return the channel flagspace.

Returns:

ChannelFlagsclass

configuration

Return the integration configuration (from channel configuration).

Returns:

Configuration

flagspace

Return the frame flagspace.

Returns:

FrameFlagsclass

info

Return the Info object for the integration.

Returns:

Info

instrument_name

Return the instrument name.

Returns:

str

motion_flagspace

Return the motion flagspace.

Returns:

MotionFlagsclass

n_channels

Return the number of channels in the integration.

Returns:

int

reference_attributes

Return attributes that should be referenced rather than copied.

Returns:

set

scan_astrometry

Return the scan astrometry.

Returns:

AstrometryInfo

size

Return the number of frames in the integration.

Returns:

int

Methods Documentation

acceleration_clip(max_acceleration=None)

Invalidate frames with excessive acceleration.

Parameters:

max_accelerationastropy.units.Quantity

The maximum allowable acceleration. Should be in size/time/time units. If not supplied, is read from the ‘aclip’ configuration option in units of arcseconds/second/second. If neither value is available, no acceleration clipping will occur.

Returns:

None

acceleration_cut(max_acceleration)

Invalidate frames with excessive acceleration.

Parameters:

max_accelerationastropy.units.Quantity

The maximum allowable acceleration. Should be in size/time/time units.

Returns:

None

add_dependents(dependents)

Add dependents to the integration.

Parameters:

dependentsDependents

The dependents to add.

Returns:

None

add_details(table=None)

Add integration details to an astropy table.

Parameters:

tableastropy.table.Table, optional

If supplied, new columns will be added. Otherwise, a new table will be created.

Returns:

Table

add_signal(signal)

Add a signal to the integration signals.

The integration signals is a dictionary of the form {mode: signal).

Parameters:

signalSignal

Returns:

None

apply_configuration()

Apply configuration options to an integration.

Returns:

None

apply_notch_filter(frequencies, width)

Apply notch filtering to frame data for given frequencies and width.

Parameters:

frequenciesnumpy.ndarray (float)

The frequencies to filter.

widthfloat

The width of the filter.

Returns:

None

bootstrap_weights()

Calculates channel weights from frame data if required.

If weighting is required but were not extracted from the pixeldata configuration option, or set to ‘uniform’, will derive values using the get_differential_channel_weights method.

Returns:

None

calculate_scan_speed_stats()

Calculates and reports statistics on the scanning speed.

Returns:

None

Notes

This should be run before and frame clipping/invalidation, since smoothing positions (which is used prior to velocity calculations) results in identical position values for frames near the edge of a block of invalid frames.

calculate_source_nefd()

Set the source Noise-Equivalent-Flux-Density (NEFD).

Returns:

None

check_consistency(channels, frame_dependents, start_frame=None, stop_frame=None)

Check consistency of frame dependents and channels.

This is intended to be overridden for specific instruments as needed. The generic implementation just returns True.

Parameters:

frame_dependentsnumpy.ndarray (float)

channelsChannelGroup

start_frameint, optional

The starting frame (inclusive). Defaults to the first (0) frame.

stop_frameint, optional

The end frame (exclusive). Defaults to the last (self.size) frame.

Returns:

consistentnumpy.ndarray (bool)

An array of size self.size where True indicates a consistent frame.

check_range()

Checks the range of frame data values, flagging as necessary.

In addition to flagging frames, channels may also be flagged as having bad DAC ranges and dead if a certain fraction of frames for a given channel is above a threshold.

The configuration must contain a “range” branch for range checking to occur. If a range value is assigned to “range”, it is explicitly used to set the range of valid DAC values. If not, only NaN values will be flagged as out-of-range. The “range.flagfraction” configuration option is used to specify what fraction of frames flagged as out-of-range for a channel is required to flag that channel as bad.

Returns:

None

clone()

Return a copy of the integration without frame type data.

Returns:

Integration

static condense_covariance(covariance)

Strips a covariance matrix of all zero and NaN column/row pairs.

Parameters:

covariancenumpy.ndarray (float)

The covariance matrix of shape (n_channels, n_channels).

Returns:

condensed_covariancenumpy.ndarray (float)

The condensed covariance matrix without zero or NaN values of shape (n_valid, n_valid).

copy()

Return a new copy of the integration.

Returns:

decorrelate(modality_name, robust=False)

Decorrelate a modality.

Parameters:

modality_namestr

The name of the modality to decorrelate.

robustbool, optional

If True, use the robust (median) method to derive means. Otherwise, use a standard mean.

Returns:

decorrelatedbool

True if the modality was decorrelated and gains were updated, False otherwise.

decorrelate_signals(modality_name, robust=False)

Decorrelate a modality signal.

Parameters:

modality_namestr

The name of the modality to decorrelate.

robustbool, optional

If True, use the robust (median) method to derive means. Otherwise, use a standard mean.

Returns:

decorrelatedbool

True if the modality was decorrelated. False otherwise.

dejump_frames()

Remove jumps in the frame data.

This will also calculate frame relative weights if necessary.

“Jumps” are determined from the noise (weights) of the frame data. At every transition between high and low weights (determined by the ‘dejump.level’ configuration option), the average channel level data is subtracted, and dependencies are updated. Integration signals will also be updated.

Returns:

None

despike(level=None)

Despike frame data.

Parameters:

levelfloat, optional

The despiking level. If not supplied, will be extracted from the configuration ‘despike.level’ setting.

Returns:

None

despike_absolute(level)

Identifies and flags spikes in frame data using absolute method.

Identify spikes in the data using the relation:

|x| > threshold

where:

threshold = level * channel_noise / sqrt(w)

level is supplied by the user, x is the frame data, and w is the relative weight of a given frame.

Samples that obey the above inequality are flagged with the SAMPLE_SPIKE flag while all others will be unflagged. Note that this is only applicable to the “live channels”, and those frames that are not flagged with the SAMPLE_SKIP or SAMPLE_SOURCE_BLANK frame flags. Invalid frames are also ignored.

Parameters:

levelfloat

The sigma level used to identify spikes.

Returns:

None

despike_gradual(level, depth=0.1)

Identifies and flags spikes in frame data using gradual method.

Identify spikes in the data using the relation:

|x| > threshold

where:

threshold = max(level * channel_noise / sqrt(w),
                gain * depth * max(x/gain <for all channels>)).

level and depth are supplied by the user, x is the frame data, and w is the relative weight of a given frame.

Samples that obey the above inequality are flagged with the SAMPLE_SPIKE flag while all others will be unflagged. Note that this is only applicable to the “live channels”, and those frames that are not flagged with the SAMPLE_SKIP or SAMPLE_SOURCE_BLANK frame flags. Invalid frames are also ignored.

Parameters:

levelfloat

The sigma level used to identify spikes.

depthfloat, optional

The maximum allowable data value as a factor of the maximum channel data value.

Returns:

None

despike_multi_resolution(level)

Identifies and flag spikes in frame data using multi-resolution.

Flags are identified in the frame data at a series of decreasing resolutions (power of 2). The first resolution (1) determines the significance as:

s = |d[i] - d[i-1]| * sqrt(w_sum)

where:

w_sum = (w[i] * w[i-1]) / (w[i] + w[i-1])

and flagged frame i as a spike if s > level where d is the timestream data (frame_data * frame_relative_weight) and w is the frame_relative_weight.

Timestream data is then set to half the resolution by setting

d[i] -> d[i] + d[i-1] w[i] = w_sum

The process is repeated at decreasing resolution up to a maximum block size defined as half the number of frames in the filter time scale.

Parameters:

levelfloat

The significance above which data are flagged as spikes.

Returns:

None

despike_neighbouring(level, delta)

Identifies and flags spikes in frame data using neighbours method.

Identify spikes in the data using the relation:

|x[i + delta] - x[i]| > threshold

where:

threshold = level * channel_noise * sqrt(1/w[i] + 1/w[i + delta])

level and delta are supplied by the user, x is the frame data, and w is the relative weight of a given frame.

Samples that obey the above inequality are flagged with the SAMPLE_SPIKE flag while all others will be unflagged. Note that this is only applicable to the “live channels”, and those frames that are not flagged with the SAMPLE_SKIP or SAMPLE_SOURCE_BLANK frame flags. Invalid frames are also ignored.

Parameters:

levelfloat

The sigma level used to identify spikes.

deltaint

The frame difference between two “neighbours” used for the spike comparison.

Returns:

None

detector_stage()

Divide the frame data by the channel hardware gains.

Will not be performed if this operation has already been completed. i.e. the detector is already staged, and the readout is not staged.

Returns:

None

downsample(factor=None)

Downsample frame data by a given, or derived factor.

Parameters:

factorint, optional

The factor by which to downsample frames. If not supplied, will be determined using the get_downsample_factor method.

Returns:

static downsampling_window(factor)

Create a normalized Hann window and the new indices.

Parameters:

factorint

The downsampling factor.

Returns:

window, indicesnumpy.ndarray (float), numpy.ndarray (int)

The Hann kernel, and the indices over which the center of the kernel should be placed to generate the downsampled data.

fill_gaps()

Add invalid frames in cases where there are timestream gaps.

The MJD of the invalid frames are set to NaN.

Returns:

None

filter_frames_for(spec=None, default_time=None)

Return the number of filtering frames for a given specification.

The available specifications are ‘auto’, ‘max’, or a float value in seconds. ‘auto’ derives the time from the crossing time, max returns the maximum number of frames in the integration.

Note that the return value will be rounded up to the nearest power of 2.

Parameters:

specfloat or str, optional

The specification. May take values of ‘auto’, ‘max’, or a float value (in seconds).

default_timeastropy.units.Quantity

If a specification is not supplied or cannot be parsed, use this default time instead.

Returns:

framesint

Raises:

ValueError

If no spec or default value is provided.

flag_spiky_blocks()

Flag all frames in a block if one or more frames is flagged as spiky.

The number of frames in a “block” is defined as the number of frames within the filter time scale.

Returns:

None

flag_spiky_channels(flag_fraction=1.0, flag_count=inf)

Flag channels with excessive spiky frames as SPIKY.

The number of frames required to flag a channel as spiky is given as max(flag_fraction * n_frames, flag_count). This procedure also set the channel data spikes attribute and performs a census of all channels.

Parameters:

flag_fractionfloat, optional

The minimum fraction of frames flagged as spiky required to flag a channel as spiky.

flag_countint or float, optional

The number of frames flagged as spiky required to flag a channel as spiky.

Returns:

None

flag_spiky_frames(frame_spikes=None)

Flag frames with excessive spiky channels as FLAG_SPIKY.

Parameters:

frame_spikesint, optional

Frames with a number of spiky channels above this limit will be flagged. The default is all channels.

Returns:

None

flag_weights()

Check if channel weights should be flagged and update source NEFD.

Returns:

None

frames_for(time=None)

Return the number of frames for a given time interval.

Parameters:

timeastropy.units.Quantity, optional

The time interval. If not supplied, defaults to the sampling interval.

Returns:

n_framesint

The number of frames in the time interval.

get_acceleration_signal(direction, mode=None)

Return an acceleration signal.

Parameters:

directionstr or MotionFlag

The direction of motion.

modeMode, optional

Returns:

Signal

get_accelerations()

Return the telescope accelerations.

Returns:

astropy.units.Quantity (numpy.ndarray)

An array of accelerations of shape (N, 2) containing the (az, alt) telescope accelerations in the default size units/second/second.

get_ascii_header()

Return a string header for output text files.

Returns:

headerstr

get_channel_weights(method=None)

Derive and set channel weights.

Parameters:

methodstr

The method to derive channel weights. Must be one of {robust, differential, rms}. If not supplied, will be determined from the configuration. Defaults to ‘rms’ if no valid method can be found.

Returns:

None

static get_correlated_signal_class()

Return the correlated signal class.

Returns:

CorrelatedSignal

get_coupling_gains(signal)

Return the coupling gains for a given channel.

Parameters:

signalSignal

The signal for which to extract channel gains.

Returns:

coupling_gainsnumpy.ndarray (float)

The coupling gains as an array of shape (self.channels.size). Values for channels not included in the signal will be set to zero.

get_covariance()

Return the channel covariance.

Returns:

covariancenumpy.ndarray (float)

The covariance matrix of shape (n_channels, n_channels).

get_crossing_time(source_size=None)

Return the crossing time for a given source size.

Parameters:

source_sizeastropy.units.Quantity, optional

The size of the source. If not supplied, defaults to (in order of priority) the source size in the scan model, or the instrument source size.

Returns:

timeastropy.units.Quantity

The crossing time in time units.

get_default_scaling_factor()

Return the default scaling factor from the configuration.

Returns:

scalefloat

get_dependents(name)

Return or create and return dependents of the given name.

Parameters:

namestr

The name of the dependents.

Returns:

Dependents

get_differential_channel_weights()

Derive and set differentially derived weights for live channels.

The variance and variance weight for a channel i is given as:

var = sum(w * (data[:, i] - data[:, i+delta])^2)
weight = sum(w)

taken over all valid frames for the channel i and i+delta, and zero flagged samples for the channel i and i+delta. These values are then passed to set_weights_from_var_stats to set the channel degrees of freedom, weights, and variances for channels.

In this case delta is defined as 10 times the number of frames required to cross the point response of the instrument (based on scanning speed).

Returns:

None

get_display_id()

Get the display integration ID.

Returns:

str

get_downsample_factor()

Return the factor by which to downsample frames.

The downsampling factor is retrieved from the configuration ‘downsample’ option which may either be valued as ‘auto’, or an integer value. Any value derived at this stage may be corrected by the ‘downsample.autofactor’ configuration setting valued as an integer or float. i.e, final factor = downsample * autofactor.

Returns:

factorint

A factor greater than 1 indicates downsampling should occur.

get_duration()

Return the total integration duration.

Returns:

timeastropy.units.Quantity

The integration duration in time units.

get_exposure_time()

Return the total exposure time.

Returns:

timeastropy.units.Quantity

The exposure time in time units.

get_file_id()

Get the file integration ID.

Returns:

str

get_filter(filter_name)

Return a filter of the given name.

Parameters:

filter_namestr

The name of the filter.

Returns:

Filter

get_first_frame(reference=0)

Return the first valid frame.

Parameters:

referenceint, optional

The first actual frame index after which to return the first valid frame. The default is the first (0).

Returns:

Frames

get_first_frame_index(reference=0)

Return the first valid frame index of the integration.

Parameters:

referenceint, optional

If supplied, finds the first frame from reference, rather than the first index (0). May take negative values to indicate an index relative to the last.

Returns:

first_frameint

get_fits_data()

Return integration data as an astropy table.

Returns:

astropy.table.Table

get_floats()

Return a float array suitable for FFT.

Returns:

numpy.ndarray (float)

get_frame_count(keep_flag=None, discard_flag=None, match_flag=None)

Return the number of frames in an integration.

A number of flags may also be supplied to return the number of a certain type of frame.

Parameters:

keep_flagint or ChannelFlagTypes, optional

Flag values to keep in the calculation.

discard_flagint or ChannelFlagTypes, optional

Flag values to discard_flag from the calculation.

match_flagint or ChannelFlagTypes, optional

Only matching flag values will be used in the calculation.

Returns:

n_framesint

The number of matching frames in the integration.

get_frame_gaps()

Return the gaps between frames.

Returns:

frame_gaps, time_gapsnumpy.ndarray (int), units.Quantity

get_full_covariance(covariance)

Return the full covariance array for all initial channels.

Parameters:

covariancenumpy.ndarray (float)

The covariance array for all channels (not including the initial channels removed).

Returns:

full_covariancenumpy.ndarray (float)

The full covariance array of shape (store_channels, store_channels).

get_full_id(separator='|')

Return the full integration ID.

Parameters:

separatorstr, optional

The separator character/phase between the scan and integration ID.

Returns:

str

static get_group_covariance(division, covariance)

Return the covariance for all groups in a channel division.

Parameters:

divisionChannelDivision

The channel division for which to extract covariances.

covariancenumpy.ndarray (float)

The covariance matrix of shape (all_channels, all_channels) for all available channels.

Returns:

group_covariancenumpy.ndarray (float)

The covariance for channels in the division.

get_id()

Return the simple integration ID.

Returns:

str

classmethod get_integration_class(name)

Return an Integration instance given an instrument name.

Parameters:

namestr

The name of the instrument.

Returns:

Integrationclass

get_last_frame(reference=None)

Return the first valid frame.

Parameters:

referenceint, optional

The last actual frame index before which to return the last valid frame. The default is the last.

Returns:

Frames

get_last_frame_index(reference=None)

Return the last valid frame index of the integration.

Parameters:

referenceint, optional

If supplied, finds the last frame before reference, rather than the last index (self.size). May take negative values to indicate an index relative to the last index.

Returns:

last_frameint

get_mean_level(channels=None, start=None, end=None, robust=False)

Return the median data values and associated weights for channels.

Parameters:

channelsChannels or ChannelData, optional

The channels for which to calculate median levels.

startint, optional

Calculate the median after (and including) this starting frame.

endint, optional

Calculate the median from before (not including) this end frame.

robustbool, optional

If True, use smart_median() to determine the mean values. Otherwise, use a simple mean.

Returns:

values, weightsnumpy.ndarray (float), numpy.ndarray (float)

The median channel values and associated weight of the calculation, both if shape (channels.size,).

get_mjd()

Return the midpoint MJD of the integration.

The midpoint MJD is defined as the average of the first and last valid frame MJD values.

Returns:

mjdfloat

get_modulation_frequency(signal_flag)

Return the modulation frequency.

The modulation frequency is taken from the chopper frequency if available, or set to 0 Hz otherwise.

Parameters:

signal_flagFrameFlagTypes or str or int

The signal flag (not relevant for this implementation).

Returns:

frequencyastropy.units.Quantity.

The modulation frequency in Hz.

get_pa()

Return the position angle of the integration.

The position angle is derived from the average position angles of the first and last valid frames.

Returns:

position_angleastropy.units.Quantity

get_point_crossing_time()

Return the time required to cross the point size of the scan.

Returns:

timeastropy.units.Quantity

The point crossing time in time units.

get_position_signal(motion_flag, direction, mode=None)

Return a position signal.

Parameters:

motion_flagMotionFlagTypes or str or int

The motion type for which to extract positions.

directionstr or MotionFlag

The direction of motion.

modeMode, optional

Returns:

Signal

get_positions(motion_flag)

Return positions based on a type of motion.

Available motion types are TELESCOPE, SCANNING, or CHOPPING. Combinations of TELESCOPE|CHOPPING|PROJECT_GLS and SCANNING|CHOPPING are also permitted. i.e., TELESCOPE and CHOPPING should be supplied with or without CHOPPING, and PROJECT_GLS is available for the TELESCOPE flag. If these flags are not present, an array containing zero values will be returned.

Parameters:

motion_flagMotionFlagTypes or str or int

The motion type for which to extract positions.

Returns:

positionastropy.units.Quantity (numpy.ndarray)

An array of shape (N, 2) containing the (x, y) positions.

get_rms_channel_weights()

Derive and set channel weights for live channels based on data values.

The variance and variance weight for a channel i is given as:

var = sum(w * data^2) weight = sum(w)

taken over all valid frames for the channel i, and zero flagged samples for the channel i. These values are then passed to set_weights_from_var_stats to set the channel degrees of freedom, weights, and variances for channels.

Returns:

None

get_robust_channel_weights()

Derive and set robustly derived channel weights for live channels.

The variance and variance weight for a channel i is given as:

var = median(relative_weight * frame_data[:, i] ** 2)
weight = sum(relative_weight)

taken over all valid frames for the channel i and zero flagged samples for the channel i. These values are then passed to set_weights_from_var_stats to set the channel degrees of freedom, weights, and variances for channels.

Returns:

None

get_scanning_velocities(return_position=False)

Return the scanning velocity (including chopping motion).

Parameters:

return_positionbool, optional

If True, return the position in addition to velocity.

Returns:

velocity, [position]Coordinate2D, [Quantity]

The position will be returned in addition to velocity if return_position is True. Both contain (x, y) coordinates.

get_signal(mode)

Return the signal for a given mode.

If the signal exists in the current integration signals dictionary, it will be returned as is. Otherwise, if a response mode is supplied, a new signal will be created, added to the dictionary, and returned. If the signal does not exist, and the mode is not a response mode, no signal will be returned.

Parameters:

modeMode

Returns:

signalSignal or None

get_smooth_positions(motion_flag)

Return the positions, smoothed according to the configuration.

The frames are smoothed using a box kernel with width specified in time (seconds), and should be set using “positions.smooth”.

Parameters:

motion_flagMotionFlagTypes or str or int

The motion type for which to extract positions. See get_positions for details on motion flag types.

Returns:

positionastropy.units.Quantity (numpy.ndarray)

An array of shape (N, 2) containing the (x, y) positions.

get_spectra(window_function='hamming', window_size=None)

Return the power spectrum of the data.

The returned power spectrum is computed using Welch’s method of dividing the data into periodograms and averaging the result.

Parameters:

window_functionstr or function, optional

The window filter name. The default is “hamming”. Must be an available function in scipy.signal.windows.

window_sizeint, optional

The size of the filter in frames. The default is twice the number of frames in the filtering time scale.

Returns:

frequency, spectrumQuantity, Quantity

The frequency spectrum is of shape (nf2,) in units of Hertz. The spectrum is of shape (nf2, n_channels) in units of Janskys.

get_speed_clip_range()

Determine the range of permissible scanning speeds.

The permissible scanning speeds are determined from the configuration ‘vclip’ (velocity clipping) branch. If vclip = ‘auto’, the minimum speed is 5 FWHMs over the stability timescale, and the maximum speed is 1/2.5 beams per sample to avoid smearing. Otherwise, the ‘vclip’ configuration value should be a range in units of arcseconds per second.

If the ‘chopped’ configuration option is set, the minimum speed is set to zero.

Returns:

speed_rangeRange

The minimum and maximum speeds are astropy.unit.Quantity values in units of the default size unit per second.

get_standard_id(separator='|')

Get the standard integration ID.

Parameters:

separatorstr, optional

The separator character/phase between the scan and integration ID.

Returns:

str

get_table_entry(name)

Return a parameter value for the given name.

Parameters:

namestr

The name of the parameter to retrieve.

Returns:

value

get_tau(spec, value=None)

Get the tau value based on configuration settings.

The returned value is given as:

t_b + t_a * (value - band_b) / band_a

where band_a/b are retrieved from the configuration as tau.<spec>.a/b, and t_a/b are retrieved from the configuration as tau.<instrument>.a/b.

Parameters:

specstr

The name of the tau specification.

valuefloat, optional

If not provided, defaults to the zenith tau.

Returns:

float

get_tau_coefficients(spec)

Return the tau coefficients for the tau specification.

The tau a and b coefficients are retrieved from the configuration as tau.<spec>.a and tau.<spec>.b.

Parameters:

specstr

Returns:

t_a, t_bfloat, float

The tau a and b coefficients.

get_thread_count()

Return the number of parallel tasks for the integration.

Returns:

threadsint

get_time_stream(channels=None, weighted=False)

Return the time stream data.

Returns the frame time stream data for a given set of channels (default is the integration channels). Samples flagged as MODELING flags, have nonzero sample flags, or marked as invalid are returned as NaN if unweighted or zero otherwise.

Parameters:

channelsChannels or ChannelData

The channels for which to extract time streams. If not supplied, defaults to the integration channels.

weightedbool, optional

If True return the frame data multiplied by relative weight. In addition, return the weights.

Returns:

data, [weight]numpy.ndarray (float), [numpy.ndarray (float)]

An array of shape (n_frames, n_channels). Invalid values are returned as NaN values if unweighted, or zero otherwise.

get_time_weights(channels=None, block_size=None, flag=True)

Derive and set frame weights.

Relative weight values are derived for chunks of frames in turn. The block_size parameter determines how many frames are to be included in each chunk. If not supplied it is either retrieved from the ‘weighting.frames.resolution` option (in seconds), or defaults to the number of frames in 10 seconds.

This process sets the both the frame relative weights, the frame degrees of freedom, flags frames that do not have sufficient degrees of freedom, and optionally flags frames that do not have weights (noise values) within the acceptable range (determined by the ‘weighting.frames.noiserange` configuration option).

Parameters:

channelsChannels or ChannelData or ChannelGroup, optional

The channels from which to derive weights.

block_sizeint, optional

Processing occurs serially in blocks, and values are derived for each block independently. The block size determines the number of frames in each block.

flagbool, optional

If True, flag frames that are outside of the weight range determined by the ‘weighting.frames.noiserange’ configuration option. Such flags will be marked with the FLAG_WEIGHT flag.

Returns:

None

get_typical_scanning_speed()

Get the mean (robust) scanning speed and weight (inverse variance).

Returns:

speed, weightQuantity, Quantity

The speed in distance/time units, and weight in time^2/distance^2 units.

get_velocities(motion_flag, return_position=False)

Return the scanning velocity (including chopping motion).

Parameters:

motion_flagMotionFlagTypes or str or int

The motion type for which to extract positions. See get_positions for details on motion flag types.

return_positionbool, optional

If True, return the position in addition to velocity.

Returns:

velocity, [position]Coordinate2D, [Coordinate2D]

The position will be returned in addition to velocity if return_position is True. Both have (x, y) coordinates.

has_gaps(tolerance=1)

Check if there are gaps in the time stream data.

This assumes a uniform sampling interval between frame measurements and calculates gaps based on deviation from the expected MJD of a frame vs. it’s recorded value.

Parameters:

toleranceint or float

The maximum number of frame missing in a gap.

Returns:

bool

has_option(option)

Return whether the configuration option is configured.

In order to be considered “configured”, the option must exist and also have a value.

Parameters:

optionstr

Returns:

configuredbool

jackknife()

Randomly invert integration and channel gains.

The integration gain will be inverted if the configuration contains a ‘jackknife’ option. The frame signs will be inverted if the configuration contains a ‘jackknife.frames’ options.

Returns:

None

level(channel_means=None, channel_weights=None, channels=None, start=0, end=None, frame_dependents=None, robust=True)

Remove the mean frame data from each channel and update dependents.

Parameters:

channel_meansnumpy.ndarray (float), optional

The mean for each channel. Will be calculated if necessary. An array of shape (n_channels,).

channel_weightsnumpy.ndarray (float), optional

The weight derived for each channel during the mean operation of shape (n_channels,)

channelsChannels, optional

The channels on which to perform the levelling.

startint, optional

The starting frame from which to derive means and consistencies.

endint, optional

The ending frame from which to derive means and consistencies.

frame_dependentsnumpy.ndarray (float), optional

The frame dependents to update of shape (n_frames,). If not supplied, defaults to an array of zeros.

robustbool, optional

If True, use the robust method (median) to determine the channel means.

Returns:

consistentnumpy.ndarray (bool)

An array of shape (n_frames,) where True indicates a consistent frame.

local_level(parms, start=0, end=None, robust=True)

Level frame data between in a certain frame range.

Parameters:

parmsDependents

startint, optional

The starting frame (inclusive). The default is the first (0) frame.

endint, optional

The end frame (exclusive). The default is the last frame (self.size).

robustbool, optional

Level using a robust mean determination (median).

Returns:

None

merge(integration)

Merge frames from another integration onto this integration.

Parameters:

integrationIntegration

Returns:

None

next_iteration()

Perform certain steps prior to an iteration.

Returns:

None

next_weight_transit(level, start=0, above=True)

Find the next frame index with relative weight above or below a level.

Parameters:

levelfloat

The weight threshold reference value.

startint, optional

The starting frame. The default is the first (0).

abovebool, optional

If True, return the first frame above level. Otherwise, return the first frame below level.

Returns:

frame_indexint

The first frame index above or below the given level threshold. If no frame is found, returns -1.

notch_filter()

Apply notch filtering based on configuration options.

The frequencies to filter are taken from the configuration ‘frequencies’ option, in the notch section. Other options of interest are ‘width’ (width of the filter), ‘harmonics’ (number of harmonics to include), and bands. If “bands” is available, extra frequencies ranging from min(band):max(band) are added with separation of ‘width’.

Returns:

None

offset(value)

Add an offset value to frame data for all unflagged channels.

Parameters:

valuefloat

Returns:

None

perform(task)

Perform a reduction task on the integration.

Parameters:

taskstr

The name of the task to perform.

Returns:

performedbool

Indicates whether the task was performed.

pointing_at(offset, indices=None)

Applies pointing correction to coordinates via subtraction.

Parameters:

offsetastropy.units.Quantity (numpy.ndarray)

An array of

indicesnumpy.ndarray (int), optional

The frame indices that apply. The default is all indices.

Returns:

None

power2_frames_for(time=None)

Return the number of frames for a given time interval using pow2ceil.

pow2ceil raises the number of frames to the nearest power of 2. E.g., 5 -> 8, 12 -> 16 etc.

Parameters:

timeastropy.units.Quantity, optional

The time interval. If not supplied, defaults to the sampling interval.

Returns:

n_framesint

The number of frames in the time interval.

process_notch_filter_block(start, end, window_size, bins, zero_phase)

readout_stage()

Multiply the frame data by the channel hardware gains.

Will not be performed the detector is not staged (i.e. the readout is staged, and the detector is not staged)

Returns:

None

reindex()

Reset the frame fixed indices to range sequentially from 0 -> size.

Channels are also reset to ensure everything is self-consistent.

Returns:

None

reindex_channels()

Reindex the channels such that all data are self-consistent.

Returns:

None

reindex_frames()

Reindex the time dependent data.

Returns:

None

remove_channel_drifts(channel_group, parms, drift_n, robust=False)

Remove drifts from channels.

Removes the average values from frames in blocks of drift_n frames. Dependents are updated as necessary. If there is more than one block, then channel filtering time scales and source filtering are updated to account for the new length of the drift.

Parameters:

channel_groupChannelGroup

The channels from which to remove drifts.

parmsDependents

The dependents to update.

drift_nint

The number of frames in a “drift” (resolution).

robustbool, optional

If True, use the robust (median) method to calculate means. Otherwise, use a simple mean.

Returns:

None

remove_dc_offsets()

Remove the DC offsets, either if explicitly requested or to allow bootstrapping pixel weights when pixeldata is not defined. This must be done before direct tau estimates

Returns:

None

remove_drifts(target_frame_resolution=None, robust=None)

Remove drifts in frame data given a target frame resolution.

Will also set the filter time scale based on the target frame resolution.

Parameters:

target_frame_resolutionint, optional

The number of frames for the target resolution. If not supplied, extracted from the configuration ‘drifts’ option in seconds and then converted to frames.

robustbool, optional

If True use the robust (median) method to determine means. If not supplied determined from the ‘estimator’ configuration option (robust=median).

Returns:

None

remove_offsets(robust=None)

Remove the DC offsets from the frame data.

Parameters:

robustbool

If True, use the robust (median) method to determine means.

Returns:

None

scale(factor)

Scale all data (in data) by factor.

Parameters:

factorint or float, optional

Returns:

None

search_corners(perimeter_pixels, projection)

Find the corners of the mapping range of the integration.

Parameters:

perimeter_pixelsChannelGroup

A channel group ideally containing pixels near the edge of the array (Although this is not strictly necessary - just speeds things up).

projectionProjection2D

A grid projection used to convert pixel positions onto a map grid.

Returns:

map_rangeCoordinate2D

The minimum (x, y) and maximum (x, y) map offsets.

select_frames()

Delete frames not inside the frame range defined in the configuration.

The “frames” configuration setting is responsible for defining a valid frame range. All fixed frame indices outside of this range will be removed from the integration. Note that the “fixed” index of the frames will be used - the index as initially read from the data file.

Returns:

None

set_iteration(iteration, rounds=None)

Set the configuration for a given iteration

Parameters:

iterationfloat or int or str

The iteration to set. A positive integer defines the exact iteration number. A negative integer defines the iteration relative to the last (e.g. -1 is the last iteration). A float represents a fraction (0->1) of the number of rounds, and a string may be parsed in many ways such as last, first, a float, integer, or percentage (if suffixed with a %).

roundsint, optional

The maximum number of iterations.

Returns:

None

set_scaling(factor=None)

Set the gain scaling for the integration.

The factor is applied relatively.

Parameters:

factorfloat, optional

The factor by which to scale the integration gain. Retrieved from the configuration ‘scale’ option if not supplied.

Returns:

None

set_scan(scan)

Set the parent scan of the integration.

The info is copied from the scan, and the configuration is unlinked (separate copy from the scan configuration).

Parameters:

scanScan

Returns:

None

set_tau(spec=None, value=None)

Set the tau values for the integration.

If a value is explicitly provided without a specification, will be used to set the zenith tau if ground based, or transmission. If a specification and value is provided, will set the zenith tau as:

((band_a / t_a) * (value - t_b)) + band_b

where band_a/b are retrieved from the configuration as tau.<spec>.a/b, and t_a/b are retrieved from the configuration as tau.<instrument>.a/b.

Parameters:

specstr, optional

The tau specification to read from the configuration. If not supplied, will be read from the configuration ‘tau’ setting.

valuefloat, optional

The tau value to set. If not supplied, will be retrieved from the configuration as tau.<spec>.

Returns:

None

set_tau_value(value)

Set the zenith tau if ground based, or the transmission.

Parameters:

valuefloat

The tau value.

Returns:

None

static set_temp_despike_levels(channel_data, level)

Set a temporary despiking level reference in the channel data.

The despiking reference level is set to sqrt(variance) * level, and stored in the temp attribute of the channel data.

Parameters:

channel_dataChannelData

The channel data for which to set the despike reference level.

levelfloat

The relative reference level.

Returns:

None

set_thread_count(threads)

Set the number of parallel tasks for the integration.

Parameters:

threadsint

The number of parallel tasks to perform.

Returns:

None

static set_weights_from_var_stats(channel_group, var_sum, var_weight)

Set channel weights from the calculated variance.

In addition to channel weights, the degrees of freedom (DOF) and channel variances are also set for a given channel group where:

DOF = 1 - (dependents / var_weight) channel_variance = variance weight = DOF / variance

Parameters:

channel_groupChannelGroup or ChannelData

The channel group or data for which to set weights. Should be of size n_channels.

var_sum: numpy.ndarray (float)

The sum of weight * variance over all valid frames (n_channels,).

var_weightnumpy.ndarray (float)

The variance weights of shape (n_channels,).

Returns:

None

set_zenith_tau(tau_value)

Set the zenith tau value.

Parameters:

tau_valuefloat

Returns:

None

setup_filters()

Set up the FFT frame data filters.

Returns:

None

shift_frames(time_or_frames)

Shift the frame data.

Parameters:

time_or_framesastropy.units.Quantity or int

The time or number of frames by which to shift frame data.

Returns:

None

slim()

Slim frame/channel type data to those channels still present.

Returns:

slimmedbool

True if any channels were removed, False otherwise.

trim(start=True, end=True)

Remove invalid frames from the start and end of the integration.

Parameters:

startbool, optional

If True (default), delete all frames before the first valid frame in the integration.

endbool, optional

If True (default), delete all frames after the last valid frame.

Returns:

None

update_gains(modality_name, robust=False)

Update gains in a modality following decorrelation.

Parameters:

modality_namestr

The name of the modality to update.

robustbool, optional

If True, use the robust (median) method to derive means. Otherwise, use a standard mean.

Returns:

updatedbool

True if the modality gains were updated. False otherwise.

update_inconsistencies(channels, frame_dependents, drift_size)

Check consistency of frame dependents and channels.

Looks for inconsistencies in the channel and frame data post levelling and updates the inconsistencies attribute of the channel data.

Parameters:

frame_dependentsnumpy.ndarray (float)

channelsChannelGroup

drift_sizeint

The size of the drift removal block size in frames.

Returns:

None

validate()

Validate an integration following the initial read.

Returns:

None

velocity_clip(speed_range=None, strict=None, sigma_clip=None)

Clip frames that are outside of the permissible scanning speed range.

Parameters:

speed_rangeRange, optional

The minimum and maximum speeds are astropy.unit.Quantity values in units of the default size unit per second. If not supplied, will be determined from the configuration via the get_speed_clip_range method, and the ‘vclip’ configuration value.

strictbool, optional

Strict (True) velocity clipping completely invalidates all rejected frames. The alternative flags rejected frames with the SKIP_SOURCE_MODELING flag. The default is taken from the ‘vclip.strict’ configuration setting.

sigma_clipfloat, optional

Invalidates frame speeds using an iterative sigma clipping to sequentially remove speeds that are sigma_clip times the standard deviation away from the median speed value.

Returns:

None

write_ascii_time_stream(filename=None)

Write the frame data to a text file.

Parameters:

filenamestr, optional

The name of the file to output the time stream results.

Returns:

None

write_coupling_gains(signal_names)

Write the coupling gains to file.

Parameters:

signal_nameslist (str)

The signals to write.

Returns:

None

write_covariance_to_file(filename, covariance)

Write a covariance matrix to file.

Parameters:

filenamestr

The path of the file to write to.

covariancenumpy.ndarray (float)

The covariance matrix.

Returns:

None

write_covariances()

Write the covariances to file.

Returns:

None

write_products()

Write the integration products to an output file.

Returns:

None

write_scan_pattern(filename=None)

Write the scan pattern to file.

Parameters:

filenamestr, optional

The path to the output file.

Returns:

None

write_spectra(window_name=None, window_size=None)

Write the spectra to file.

Parameters:

window_namestr, optional

The name of the filtering window. If not supplied, will be extracted from the configuration option “write.spectrum” or default to ‘Hamming’ if not found.

window_sizeint, optional

The size of the filtering window. Will be extracted from the configuration option “write.spectrum.size” if not supplied. If not found in the configuration, the default will be 2 times the number of frames in the filtering time scale.

Returns:

None