OITools V2

This page gathers the enhancements on the JMMC OITools library focused on the OIFITS data handling.

Its main features are:

• read / write OIFITS files (V1 + extensions OImaging)
• read / write FITS image(s) (primary HDU, extra Image HDU)
• data model description: meta-data for keywords, columns, tables
• in-memory data structures to deal with tables (columns, keywords) and derived columns (hard-coded & expression support)
• validation (command-line interface + web interface)
• serialization in TSV / XML formats (used by oidb)

Enhancements

Tests

TODO: describe the JUnit tests + file structure

• Copy files
  • Load & write reference OIFits files
  • README on the file OIFits for explain the chose
  • Creating file for a fits file V1
  • Compare the OIFitsFile structures (all tables, all keywords, all columns, headercards)
  • Compare the nom.tam.Fits (raw HDUs): TODO: fix comparisons

TODO

For Monday:

• Look how table are load and save, try to add a new table (nondescript)
• In primary header on v2 we have new keywords to describe tables but we have keywords for OIImage too on the same spot (need new classe for manage 2 keywords type ?)
• Do JUnit test on validator
• Do JUnit test on Fits image
• Read the V2 doc for find the rule and write it

Problem 1: We need to manage Fits Image in the OIFits Files.

• Currently we can't manage image and we can't read and write it.
• Propose adapt methods for reading images and writing.

Problem 2: We need to test the validator

• Currently we have checker and validator for file, we need to test it.
• Maybe need to prepare for the V2

Problem 3: We have multi min/max methodes

• In several file we have methode for calculate min/max value
• for exemple : oifitsFile.getMinWavelengthBound() and oiWavelength.getEffWaveMin()
• We need just 1 method to compute all kind of bound on any kind of mesurements (see also xmlOutputVisitor.appendRecord() )

Problem 4: We need extend Column for all types

• Currently, for exemple we have getColumnDerived for type 1D, 2D, 1I, 2I, 1A, 1F.
• Maybe need to do all in a getColumnAsDouble : do Derv Column and not for Double.
• We need getter for prepare the new type can comming for V2.

Problem 5: We need to change Tam Fits Test for scrict comparison

• Currently we ignore some error for the strict test or we need to fix that
• We need to chose the best chose fix or ignore error, depending on severity.

See: oitools/src/test/resources/ref/WriteOIFitsTest.log

Ignore case:

• ARRAYXYZ: 123.0 vs 123.
• NAXIS1 = row size different (taille des String)
• TFORM = Data format
  • different value of header card[TFORM2] '6A' <> '16A'
  • different value of header card[TFORM3] 'I' <> '1I'

Primary HDU comment

• Images (Primary HDU)
• file: GRAVITY..
• ESO keywords dans Primary HDU (600)

Extra Tables

• Tables that are not in the OIFits V1 standard are totally skipped (OIFitsLoader / OIFitsWriter)
• Consequences:
  • OIFits V2 tables are ignored, can not be copied (load then write) so information is lost !
  • any unknown table is lost

Related: how to deal with the various data models ?

• OIFits V1 STRICT
• OIFits V1 + OI_VIS extensions (VISDATA / VISERR as complex values)
• OIFits V1 + OI_IMAGE (OImaging extensions)
• OIFits V2
• OIFits V1 + V2 (mixed ?)
• OIFits V2 + any of those extensions

Extra Columns in table

• Try to detect the displaced columns ... => avoid false positives

1. file: PRODUCT_V838_Mon_1-copy.90-2.53micron_2013-04-15T01_49_24.8034--------0.00000------------inf--------1--------2--------3

• keywords:
  • OI_VIS Missing header card TTYPE15 was = VISCOVRI
  • OI_VIS Missing header card TFORM15 was = 16D
• Columns:
  • different number of columns 17 <> 14 in VISCOVRI

1. file: GRAVITY-copy.2016-01-09T05-37-06_singlesci_calibrated

• different number of columns 16 <> 14 in ASTROMETRIC_DELAY
• different number of columns 3 <> 2 in POLAR

• OI_VIS Missing header card TTYPE15 was = ASTROMETRIC_DELAY
• OI_VIS Missing header card TTYPE15 was = VISCOVRI

• OI_T3 different number of header card 50 <> 52

Dimensions (TDIM)

1. file: AMBER_070409-copy

• OI_ARRAY Missing header card TDIM5 was = (3)
• OI_VIS Missing header card TDIM7 was = (20)
• different values for column[VISDATA]
• different values for column[VISERR]

See OIFits V2 standard that mentions how to use the TDIM keyword

Problem 6: Setters API

To easily fill tables (complex keywords, columns), a simple setter API will be implemented:

/**
* Define the column value for the given row (index)
*/
(OITable).setColumn(int row, (Type) value)

/**
* Define all column values for the given row (index)
*/
(OITable).setRow(int row, values...)

-- LaurentBourges - 11 Jul 2017