Tags: view all tags

This page gathers the enhancements and current limitations on the JMMC OITools library focused on the OIFITS data handling.

Its main features are:

read / write OIFITS files (V1 / V2 + IMAGE_OI extensions = OImaging)
read / write FITS image(s) and cubes (primary HDU, extra Image HDU)
data model description: meta-data for keywords, columns, tables
in-memory data structures to deal with tables (columns, keywords) and derived columns (hard-coded & expression support)
validation (command-line interface + web interface)
serialization in TSV / XML formats (used by oidb)

Enhancements

The rules for OIFits Files

General rules

Rule_OI_1: OIFITS files must comply with the FITS standard (Hanisch et al. 2001), where shall be found acronyms and symbols not defined in this document.
Rule_OI_2: OIFITS files use binary table extensions following the primary header-data unit. Data types for each column are defined by the value of the TFORMn keywords in each extension header. This version introduces the use of the data type J (32 bit integers).
Rule_OI_3: A valid OI exchange-format FITS file must contain one (and only one) OI_TARGET table, one or more OI_ARRAY tables, one or more OI_WAVELENGTH tables, plus any number of the data tables: OI_VIS, OI_VIS2, or OI_T3. Each data table must refer to an OI_WAVELENGTH and OI_ARRAY table that are present in the file. The requirement for the presence of an OI_ARRAY table was not part of version 1 of the format. All other new tables are optional.
Rule_OI_4: It may be useful to incorporate other tables in an OIFITS file. For example, there might be one that contains instrument specific information, such as the backend configuration. Another optional table could contain information relevant to astrometry. The only requirement imposed by the format is that EXTNAMEs of additional tables must not begin with "OI_";.
Rule_OI_5: It would facilitate the addition of new keywords and columns in future releases of the standard if the non-standard keywords and column names were given a particular prefix, such as "NS_" to avoid conflicts.

Rule_OI_6: Cross-referencing between FITS tables is not a generic feature of the FITS format, but is necessary for correct description of the interferometric data. When writing OIFITS tables, particular care must be taken to maintain the cross-references. Specifically, each data table must refer:
- to a particular OI_WAVELENGTH table describing the wavelength channels for the measurements, via the INSNAME keyword; and
- to an OI_ARRAY table, via the ARRNAME keyword; and
- optionally, to the OI_INSPOL table containing the same INSNAME; and
- optionally, to an OI_CORR table via the CORRNAME keyword.

Notes:

Rule_OI_1: FITS support is provided by the nom.tam.fits library: TODO: upgrade + add any other specific validation rules for the FITS format ?
Rule_OI_2: implemented (data type J)
Rule_OI_3: implemented
Rule_OI_4: support of any other tables (not OIFITS): TODO + check "OI_" prefix (warning)
Rule_OI_5: TODO validator: warning if non-standard keyword/column found ? check for "NS_" prefix ?
Rule_OI_6: Question = "to the OI_INSPOL table containing the same INSNAME" seems to indicate that an OIFITS file can only contain (0 or 1) OI_INSPOL table : confirm ? + TODO validator

OIFITS V2 new features (table, columns, keywords)

Header keywords:

In addition to the mandatory FITS binary extension keywords defined in Hanisch et al. (2001), the keywords must be present in the table headers. We recommend the use of DATASUM and CHECKSUM in the main header and all table headers3, TUNITn values are mandatory for all columns except those representing unitless values (such as STA_INDEX). Any other keyword is expected to be ignored by OIFITS readers. The FITS format defines a null value for each supported data type. The OIFITS format uses these type-dependent null values to identify individual data as being unmeasured, invalid, etc. and this document refers to such values as NULL.

- - TTYPEn1 yes name of the column T3PHIERR
  - TUNITn1 yes unit of datum deg
  - TDIMn1 no dimensionality of the data (3, 4)
  - DATASUM no
  - CHECKSUM no

For OIFITS writers needing to add such ancillary information in the main header, we suggest using the additional keywords listed for required keywords:

- - SIMPLE T yes Standard FITS format
  - BITPIX %d yes Number of bits per data pixel
  - NAXIS %d yes Number of data axes1 0
  - EXTEND T yes Extensions may be present
  - ORIGIN %s yes Institution responsible for file creation ESO
  - DATE %s yes Date the HDU was written 2015-01-13T16:12:56.57
  - DATE-OBS %s yes Start date of observation 2014-10-15T18:33:26.06
  - CONTENT %s yes Must contain only the string “OIFITS2”2 OIFITS2

* Keywords set to “MULTI” for heterogenous content of the OIFITS (see text)

- - TELESCOP %s yes A generic identification of the ARRAY3 CHARA
  - INSTRUME %s yes A generic identification of the instrument3 VEGA
  - OBSERVER %s yes Who acquired the data3 S.M.Eisenstein
  - OBJECT %s yes Object Identifier3 HD123456
  - INSMODE %s yes Instrument mode3 Low_JHK

OI_TARGET, The only differences from revision 1 are the optional additional column giving the category assigned to a target for the purpose of calibrating the data, and the recommendation to use EQUINOX 2000 for coordinates.
OI_ARRAY: Therefore, we have added two columns to the OI_ARRAY table, FOV, which gives the radius of PFOV and FOVTYPE, which specifies whether the PFOV radial profile is Gaussian or rectangular.
- Each OI_ARRAY table in a file must have a unique value for ARRNAME.
OI_WAVELENGTH: In this revision, we introduce the possibility of storing purely monochromatic values (EFF_BAND=0) in OI_WAVELENGTH, since such a table can be referenced, via the INSNAME keyword, by a new table, OI_FLUX, which may contain monochromatic spectra
- Each OI_WAVELENGTH table in a file must have a unique value for INSNAME. Particular attention to this point is recommended when aggregating OIFITS files which may have similar INSNAME values but different channel wavebands.

Data tables:

- In OIFITS version2 only MJD and DATE-OBS must be used to express time. The TIME column is retained only for backwards compatibility and must contain zeros. The value in the MJD column shall be the mean UTC time of the measurement expressed as a modified Julian Day.

OI_VIS2: new keyword/column: CORRNAME (optional) Identifies corresponding OI_CORR table, CORRINDX_VIS2DATA (optional) Index into correlation matrix for 1st VIS2DATA element
OI_VIS: There are four new keywords in the header, AMPTYP, PHITYP, AMPORDER, and PHIORDER. (And CORRNAME too), and a lot of column:

- AMPTYP defines the type of data stored in VISAMP: absolute, differential or correlated flux. In the last case the usual TUNITn FITS keyword must be used to specify the flux unit.
- PHITYP is one of absolute or differential. The default for AMPTYP and PHITYP, if not present, is absolute.

- If AMPTYP or PHITYP is differential, the newly-defined data column VISREFMAP must be present since this column contains the channel number(s) whose combination defines the reference channel used in computing the VISAMP and VISPHI values.
- CORRINDX_VISAMP J (1) (optional) Index into correlation matrix for 1st VISAMP element
- CORRINDX_VISPHI J (1) (optional) Index into correlation matrix for 1st VISPHI element
- VISREFMAP L (NWAVE,NWAVE) (optional†) Matrix of indexes for establishing the reference channels (see text).
- RVIS D (NWAVE) (optional) Complex coherent flux (Real) in units of TUNITn
- RVISERR D (NWAVE) (optional) Error on RVIS
- CORRINDX_RVIS J (1) (optional) Index into correlation matrix for 1st RVIS element
- IVIS D (NWAVE) (optional) Complex coherent flux (Imaginary) in units of TUNITn
- IVISERR D (NWAVE) (optional) Error on IVIS
- CORRINDX_IVIS J (1) (optional) Index into correlation matrix for 1st IVIS element
- if some values of RVIS or IVIS or RVISERR or IVISERR are meaningless, NULL values are to be used instead.

OI_T3: FLAG marks both the corresponding bispectrum amplitude (T3AMP) and phase (T3PHI) as invalid. To mark only the amplitude or phase as invalid, the relevant data values must be replaced by NULL (with FLAG remaining as false). If the dataset does not provide triple product amplitudes at all, T3AMP and T3AMPERR must contain only NULL values.
New keywords/Column: CORRNAME (optional) Identifies corresponding OI_CORR table, CORRINDX_T3AMP (optional) Index into correlation matrix for 1st T3AMP element, CORRINDX_T3PHI (optional) Index into correlation matrix for 1st T3PHI element

New Tables:

OI_FLUX:
- OIFITS version 2 introduces this optional table intended as a container for raw or calibrated flux measurements. Corresponding wavelengths are referenced with the INSNAME keyword.
- If CALSTAT is U (Uncalibrated), ARRNAME and the STA_INDEX column must be present, and the flux stored in FLUXDATA is the flux measured on the telescope of array ARRNAME indicated by STA_INDEX. In this case, the table shall not contain FOV and FOVTYPE. (The field of view is given in the OI_WAVELENGTH table referenced by INSNAME.)
- The keywords FOV and FOVTYPE can be completely different and therefore independent of the column FOV and FOVTYPE from OI_ARRAY

OI_CORR:
- There is a need to describe correlations between any kinds of OIFITS observables, over a limited timespan (hours at most) determined by the calibration procedures.
- An OIFITS file may contain multiple such sets, to accommodate merging of multiple data sets.
- Each of the data tables in a correlated data set shall have a keyword CORRNAME that is used to look up the corresponding OI_CORR table. The correspondence between the position of a value in a data table and its index into the correlation matrix is given by the following new columns: CORRINDX_VISAMP, CORRINDX_VISPHI, CORRINDX_RVIS and CORRINDX_IVIS in OI_VIS, CORRINDX_VIS2DATA in OI_VIS2, CORRINDX_T3AMP and CORRINDX_T3PHI in OI_T3 and CORRINDX_FLUXDATA in OI_FLUX .

OI_INSPOL:
- However, we feel compelled to provide a way to note in the OIFITS format that the measurements were subject to instrumental polarisation, and, in the case of simultaneous or related measurements in various states of polarisation, provide an internal link between the classical observables of OIFITS and the instrumental polarisation state in which they were obtained. This new table is purely instrumental and named OI_INSPOL.
- The new OI_INSPOL table is the central hub that links all the INSNAMEs (all the polarized measurements).
- The OI_INSPOL table stores a Jones matrix for each wavelength, time interval, and polarisation state as cross-referenced via the INSNAME column.
- OI_INSPOL is an optional table: if OI_INSPOL is present, it must encompass at least all the time entries and baselines present in all the other tables referenced by all the values stored in its INSNAME column. OI_INSPOL is also experimental: OIFITS readers or writers should make provision for the possible augmentation of this table in future releases.

FeedBack OIFits2 Standard:

We have now type 'J' is integer 32 bits, so is INT. But before we have 'I' integer 16bits(SHORT) and we have use 'I' like a 'J' before V2. We need clarification on keyword types, all keywords stay INT ?
We have different synthax for keyword %s, %f, %d for header keyword.
The new tables sometimes have relationships difficult to understand, indeed OI_INSPOL have column that must refer to a keyword from another table (INSNAME). We assume we have only one OI_inspol table? What would mean that in all the OI_WAVE on the file have the same number of lines?
INSNAME column has no dimension in the OI_INSPOL
Need more explain for NPOL.
There is also the difficulty of building the OI_CORR matrix, validation et index I and J: how to calculate the dimension of the matrix ? And how does the index begin (0 or 1)?

Tests

TODO: describe the JUnit tests + file structure
TODO: Headercards type is string for Int/float/double too
TODO: Test update ? (History order) / (Hierarch)
- see strict compare Nom.tam.fits
TODO: CreateOIFitsV2
TODO: imageHDU should be returned even if no image to hold keywords -> could rely on the firstOnly flag ?
Improve the OIFitsValidator test to compare the validation results

Junit Tests

Copy files

Objective:

Load & write reference FITS (images) & OIFits (V1, V2, with IMAGE_OI extensions) files
Compare the OIFitsFile structures (all tables, all keywords, all columns, header cards): OK ?
Compare the nom.tam.Fits (raw HDUs):

TODO: FIX the lenient comparison mode (strict is too exhaustive) to ensure the copied files have not lost or modified original keyword / data

See README in the src/test/resource folder describing the reference files

TODO

Improve the OIFITS validator (V1 / V2)

add specific rules for V2 (inter-dependencies)
improve the validation results:
- extract the severity, origin (table/keyword or column) from the validation message
- sort & filter validation results

Problem 1: We have multi min/max methodes

In several file we have methode for calculate min/max value
for exemple : oifitsFile.getMinWavelengthBound() and oiWavelength.getEffWaveMin()
We need just 1 method to compute all kind of bound on any kind of mesurements (see also xmlOutputVisitor.appendRecord() )

Problem 2: We need to change Tam Fits Test for scrict comparison

Currently we ignore some error for the strict test or we need to fix that
We need to chose the best chose fix or ignore error, depending on severity.

See: oitools/src/test/resources/ref/WriteOIFitsTest.log

Ignore case:

ARRAYXYZ: 123.0 vs 123.
NAXIS1 = row size different (taille des String)
TFORM = Data format
- different value of header card[TFORM2] '6A' <> '16A'
- different value of header card[TFORM3] 'I' <> '1I'

Extra Tables

Binary Tables that are not in the OIFits standards (V1 or V2) are totally skipped (OIFitsLoader? / OIFitsWriter? )

Consequence: Copying a FITS file may loose 'unknown' tables (load then write) !
Solution: When loading such a table, it seems possible to parse its header and create keyword / column descriptors on the fly then load it as usual (GenericTable extends Table)

Extra Columns in table

Try to detect the displaced columns ... => avoid false positives

file: PRODUCT_V838_Mon_1-copy.90-2.53micron_2013-04-15T01_49_24.8034--------0.00000------------inf--------1--------2--------3

keywords:
- OI_VIS Missing header card TTYPE15 was = VISCOVRI
- OI_VIS Missing header card TFORM15 was = 16D
Columns:
- different number of columns 17 <> 14 in VISCOVRI

file: GRAVITY-copy.2016-01-09T05-37-06_singlesci_calibrated

different number of columns 16 <> 14 in ASTROMETRIC_DELAY
different number of columns 3 <> 2 in POLAR

OI_VIS Missing header card TTYPE15 was = ASTROMETRIC_DELAY
OI_VIS Missing header card TTYPE15 was = VISCOVRI

OI_T3 different number of header card 50 <> 52

Dimensions (TDIM)

file: AMBER_070409-copy

OI_ARRAY Missing header card TDIM5 was = (3)
OI_VIS Missing header card TDIM7 was = (20)
different values for column[VISDATA]
different values for column[VISERR]

See OIFits V2 standard that mentions how to use the TDIM keyword

Problem 6: Setters API

To easily fill tables (complex keywords, columns), a simple setter API will be implemented:

/**
* Define the column value for the given row (index)
*/
(OITable).setColumn(int row, (Type) value)

/**
* Define all column values for the given row (index)
*/
(OITable).setRow(int row, values...)

Limitations

Cube Image support: Fixed

Last version can load FITS image cubes from one ImageHDU but also write back to a FITS or OIFits file into one ImageHDU (with the proper wavelength keywords - AXIS3)

This use case is not yet required but next ones certainly will require code enhancement:

support of polychromatic data in OImaging

Feedback on the OIFits V2 standard

INSNAME column has no dimension in the OI_INSPOL

External resources

https://github.com/jsy1001/oifitslib provides some V2 test files

Topic revision: r17 - 2017-09-21 - CharleenKemps