(r4) JmmcOiTools < Jmmc/Software

Tags: view all tags
---+ OITools V2

This page gathers the enhancements on the JMMC OITools library focused on the OIFITS data handling. 

Its main features are:
   * read / write OIFITS files (V1 + extensions OImaging)
   * read / write FITS image(s) (primary HDU, extra Image HDU)
   * data model description: meta-data for keywords, columns, tables
   * in-memory data structures to deal with tables (columns, keywords) and derived columns (hard-coded & expression support)
   * validation (command-line interface + web interface)
   * serialization in TSV / XML formats (used by oidb)


---+ Enhancements

---++ Tests

TODO: describe the JUnit tests + file structure


   * Copy files
      * Load & write reference OIFits files
      * README on the file OIFits for explain the chose
      * Creating file for a fits file V1
      * Compare the OIFitsFile structures (all tables, all keywords, all columns, headercards)
      * Compare the nom.tam.Fits (raw HDUs): TODO: fix comparisons


---++ TODO

---+++ Problem 1: We need to manage Fits Image in the OIFits Files.
   * Currently we can't manage image and we can't read and write it.
   * Propose adapt methods for reading images and writing.


---+++ Problem 2: We need to test the validator
   * Currently we have checker and validator for file, we need to test it.
   * Maybe need to prepare for the V2


---+++ Problem 3: We have multi min/max methodes
   * In several file we have methode for calculate min/max value
   * for exemple : getMinWavelengthBound in OIFits File
   * We need just 1 method for calcul all


---+++ Problem 4: We need extend Column for all types
   * Currently, for exemple we have getColumnDerived for type 1D, 2D, 1I, 2I, 1A, 1F.
   * Maybe need to do all in a getColumnAsDouble : do Derv Column and not for Double.
   * We need getter for prepare the new type can comming for V2.

---+++ Problem 5: We need to change Tam Fits Test for scrict comparison
   * Currently we ignore some error for the strict test or we need to fix that
   * We need to chose the best chose fix or ignore error, depending on severity.

See: oitools/src/test/resources/ref/WriteOIFitsTest.log

---++++ Ignore case:
   * ARRAYXYZ: 123.0 vs 123.
   * NAXIS1 = row size different	(taille des String)
   * TFORM = Data format
      * different value   of header card[TFORM2] '6A' <> '16A'
      * different value   of header card[TFORM3] 'I' <> '1I'


---++++ Primary HDU comment
   * Images (Primary HDU)
   * file: GRAVITY..
   * ESO keywords dans Primary HDU (600)


---++++ Extra Tables
   * Tables that are not in the OIFits V1 standard are totally skipped (OIFitsLoader / OIFitsWriter)
   * Consequences: 
      * OIFits V2 tables are ignored, can not be copied (load then write) so information is lost !
      * any unknown table is lost

Related: how to deal with the various data models ?
   * OIFits V1 STRICT
   * OIFits V1 + OI_VIS extensions (VISDATA / VISERR as complex values)
   * OIFits V1 + OI_IMAGE (OImaging extensions)
   * OIFits V2
   * OIFits V1 + V2 (mixed ?)
   * OIFits V2 + any of those extensions


---++++ Extra Columns in table
   * Try to detect the displaced columns ... => avoid false positives

   1 file: PRODUCT_V838_Mon_1-copy.90-2.53micron_2013-04-15T01_49_24.8034--------0.00000------------inf--------1--------2--------3
   * keywords:
      * OI_VIS Missing header card TTYPE15 was = VISCOVRI
      * OI_VIS Missing header card TFORM15 was = 16D
   * Columns:
      * different number of columns 17 <> 14 in VISCOVRI

   1 file: GRAVITY-copy.2016-01-09T05-37-06_singlesci_calibrated
	
   * different number of columns 16 <> 14 in ASTROMETRIC_DELAY
   * different number of columns 3 <> 2 in POLAR

   * OI_VIS Missing header card TTYPE15 was = ASTROMETRIC_DELAY
   * OI_VIS Missing header card TTYPE15 was = VISCOVRI

   * OI_T3 different number of header card 50 <> 52


---++++ Dimensions (TDIM)

   1 file: AMBER_070409-copy

   * OI_ARRAY Missing header card TDIM5 was = (3)
   * OI_VIS Missing header card TDIM7 was = (20)
   * different values for column[VISDATA]
   * different values for column[VISERR]

See OIFits V2 standard that mentions how to use the TDIM keyword


---+++ Problem 6: Setters API

To easily fill tables (complex keywords, columns), a simple setter API will be implemented:
<pre>
/**
* Define the column value for the given row (index)
*/
(OITable).setColumn(int row, (Type) value)

/**
* Define all column values for the given row (index)
*/
(OITable).setRow(int row, values...)
</pre>
 


-- Main.LaurentBourges - 11 Jul 2017
Topic revision: r4 - 2017-07-13 - LaurentBourges