JmmcOiTools < Jmmc/Software

This page gathers the enhancements and current limitations on the JMMC OITools library focused on the OIFITS data handling. 

Its main features are:
   * read / write OIFITS files (V1 / V2 + IMAGE_OI extensions = OImaging)
   * read / write FITS image(s) and cubes (primary HDU, extra Image HDU)
   * data model description: meta-data for keywords, columns, tables
   * in-memory data structures to deal with tables (columns, keywords) and derived columns (hard-coded & expression support)
   * validation (command-line interface + web interface)
   * serialization in TSV / XML formats (used by oidb)

%TOC%


---+ Enhancements

---++ The rules for OIFits Files

   * The table for all rules (and apply to) has been created.
   * There are also tables for DataModel version 1 and version 2

---++ FeedBack OIFits2 Standard:

   * We have different synthax for keyword %s, %f, %d for header keyword.


---++ Tests

   * TODO: describe the JUnit tests + file structure
   * TODO: Headercards type is string for Int/float/double too
   * TODO: Test update ? (History order) / (Hierarch)
      *    see strict compare Nom.tam.fits
   * TODO: imageHDU should be returned even if no image to hold keywords -> could rely on the firstOnly flag ?
   * Improve the OIFitsValidator test to compare the validation results


---+++ Junit Tests

---++++ Copy files
Objective: 
   * Load & write reference FITS (images) & OIFits (V1, V2, with IMAGE_OI extensions) files
   * Compare the !OIFitsFile structures (all tables, all keywords, all columns, header cards): OK ?
   * Compare the nom.tam.Fits (raw HDUs): 
TODO: FIX the lenient comparison mode (strict is too exhaustive) to ensure the copied files have not lost or modified original keyword / data

See README in the src/test/resource folder describing the reference files


---++ TODO

---+++ Improve the OIFITS validator (V1 / V2)

   * improve the validation results: 
      * extract the severity, origin (table/keyword or column) from the validation message
      * sort & filter validation results


---+++ Problem 1: We have multi min/max methodes
   * In several file we have methode for calculate min/max value
   * for exemple : oifitsFile.getMinWavelengthBound() and oiWavelength.getEffWaveMin()
   * We need just 1 method to compute all kind of bound on any kind of mesurements (see also xmlOutputVisitor.appendRecord() )


---+++ Problem 2: We need to change Tam Fits Test for scrict comparison
   * Currently we ignore some error for the strict test or we need to fix that
   * We need to chose the best chose fix or ignore error, depending on severity.

See: oitools/src/test/resources/ref/WriteOIFitsTest.log

---++++ Ignore case:
   * ARRAYXYZ: 123.0 vs 123.
   * NAXIS1 = row size different	(taille des String)
   * TFORM = Data format
      * different value   of header card[TFORM2] '6A' <> '16A'
      * different value   of header card[TFORM3] 'I' <> '1I'


---++++ Extra Tables

    Binary Tables that are not in the OIFits standards (V1 or V2) are totally skipped (OIFitsLoader? / OIFitsWriter? )

   * Consequence:
        Copying a FITS file may loose 'unknown' tables (load then write) !
   * Solution:
        When loading such a table, it seems possible to parse its header and create keyword / column descriptors on the fly then load it as usual (GenericTable extends Table)


---++++ Extra Columns in table
   * Try to detect the displaced columns ... => avoid false positives

   1 file: PRODUCT_V838_Mon_1-copy.90-2.53micron_2013-04-15T01_49_24.8034--------0.00000------------inf--------1--------2--------3
   * keywords:
      * OI_VIS Missing header card TTYPE15 was = VISCOVRI
      * OI_VIS Missing header card TFORM15 was = 16D
   * Columns:
      * different number of columns 17 <> 14 in VISCOVRI

   1 file: GRAVITY-copy.2016-01-09T05-37-06_singlesci_calibrated
	
   * different number of columns 16 <> 14 in ASTROMETRIC_DELAY
   * different number of columns 3 <> 2 in POLAR

   * OI_VIS Missing header card TTYPE15 was = ASTROMETRIC_DELAY
   * OI_VIS Missing header card TTYPE15 was = VISCOVRI

   * OI_T3 different number of header card 50 <> 52


---++++ Dimensions (TDIM)

   1 file: AMBER_070409-copy

   * OI_ARRAY Missing header card TDIM5 was = (3)
   * OI_VIS Missing header card TDIM7 was = (20)
   * different values for column[VISDATA]
   * different values for column[VISERR]

See OIFits V2 standard that mentions how to use the TDIM keyword


---+++ Problem 6: Setters API

To easily fill tables (complex keywords, columns), a simple setter API will be implemented:
<pre>
/**
* Define the column value for the given row (index)
*/
(OITable).setColumn(int row, (Type) value)

/**
* Define all column values for the given row (index)
*/
(OITable).setRow(int row, values...)
</pre>
 

---++ Limitations

---+++ Cube Image support: Fixed
Last version can load FITS image cubes from one !ImageHDU but also write back to a FITS or OIFits file into one !ImageHDU (with the proper wavelength keywords - AXIS3)

This use case is not yet required but next ones certainly will require code enhancement:
   * support of polychromatic data in OImaging

---+ External resources
   * https://github.com/jsy1001/oifitslib provides some V2 test files
This topic: Jmmc/Software > WebHome > JmmcOiFits > JmmcOiTools
Topic revision: r18 - 2017-10-05 - CharleenKemps