(r22) AmberDrsReleaseNotes < Jmmc/Software

Tags: view all tags
-- Main.GillesDuvert - 01 Jul 2009
---+ <span class="red">V3.0</span>
<br />The V3.0 release consists in:
   1 The C library, amdlib, which is autonomous since it provides command-line tools to  reduce the data
   1 The Yorick wrapper calling the amdlib.i yorick library, that provides added pipe-line processing and visualization tools.
    <br /> %ICON{stop}% Starting from this version, users are requested, if they use this software for a scientific publication, to acknowledge the use of JMMC tools *and cite the following paper*:
     <br />Chelli, A., Hernandez Utrera, O., Duvert, G., "Optimised Data Reduction for the VLTI/AMBER instrument", A&A 502, 705-709 (2009) 
   
---++ The amdlib Library and its command-line tools
---+++ The usual command-line tools are :
   * [[JmmcAmberDrsAmdlibCommandLineHelp][amdlibComputeP2vm]]: is the first thing to do. It computes a P2VM fits file wich contains all the calibration aspects. At the moment it is recommended to always use the options of amdlibComputeP2vm to modify manually the "spectral shifts" and the wavelength values. 
   * [[JmmcAmberDrsAmdlibCommandLineHelp][amdlibComputeOiData]]: reduce a science or calibrator observation and produce an OI-FITS file. An observation is one raw data file. There are typically 5 raw datat files on an object (science or calibrator) plus 1 dark and 1 sky in each Observing Block.
   * [[JmmcAmberDrsAmdlibCommandLineHelp][amdlibAppendOiData]]: Adds several OI-FITS files together.
   * [[JmmcAmberDrsAmdlibCommandLineHelp][amdlibPerformFrameSelection]]: from various selection criteria, produces a file with an index of frames to be retained to compute a final OI-FITS.
<br /> See [[JmmcAmberDrsAmdlibCommandLineHelp][the help]] for details about the options.
---+++ Changes in the library wrt. version 2
---++++ General issues
   * This is a completely new release involving a rewriting of many parts of the amdlib C library.
   * The code implements most of the algorithms published by [[http://www.aanda.org/index.php?option=article&access=doi&doi=10.1051/0004-6361/200811039][Chelli et al. 2009]] as well as workarounds of some of the problems audited by the ATF team (see [[http://arxiv.org/abs/0808.1315][ATF report]]).
   * We have tried to maintain, at least this time, the interfaces of the amdlib C library 'almost' compatible with the previous ones, to avoid rewriting of the Yorick interface.However, a number of internal changes were necessary, as for example the fact that all computations are now in double precision. Another important change is that _amdlib OI-FITS files have FLAGS and BLANKING VALUES_ . User programs reading these files should be aware that, depending on filters implicit or explicit in amdlibV3, some values may be flagged or blanked.
---++++ In Particular
   * detector noise is computed per pixel from the dark file (darks thus preferably contain more than 500 frames for good statistics)
   * on-the-fly bad pixel detection can be done
   * since the dark in Amber is usually a hot dark, dark noise is taken into account AND the additional use of SKY files is FORBIDDEN
   * However, one can use a SKY file in place of a DARK, which should be better in case of, e.g., sky glow.
   * Starting June 2009, day cold darks are taken by ESO. These should be used preferentially to any other type of dark or sky. 
   * Flats are defintely unnecessary, except for spectrophotometric purposes wich are not the scope of amdlib --- this version has not been tested yet using an AMBER Flat, only this [[http://www-laog.obs.ujf-grenoble.fr/twiki/pub/Laog/GRIL/Informatique/JmmcAmberDrsReleaseNotes/VraimentFlat.fits.gz][really flat flat]].
   * additionnally, the software
   1 implements the linearization factor of Chelli et al 2008. Information is displayed as:
         <br /> =Average Linearization Factor: 0.953354=, meaning that the data is good. if this factor is markedly different from 1.0, (say, below 0.8 or above 1.2 ), you could wonder if the <nop>P2VM is good enough or if the bad pixel map is up to date.
   1 uses a refined algorithm to compute pistons, and an heuristic scheme to evaluate this piston 'goodness of fit'. This is mostly useful in Low Resolution witout FINITO where the atmosphere piston can cause a complete loss of coherence. The present version does not use the improved estimate of piston to correct from the theoretical extenuation. However, it has options ( =-J= and =-C= ) to tag bad piston values and use piston closure for piston evaluation improvement. Use the =-c= and =-r= options to filter out pistons larger than some fraction of the coherence length (say, 20 microns in LRK).
   1 uses a goodness of fit test to tag individual visibilities which are not well fitted by the carrying waves of the interferogram. A message is associated, like: =Correlated Flux fit statistics: mean = 1.015236, rms = 0.276510, median = 0.988546=. The fact that this chi2 is very close to 1 means that all the variances have been well estimated. =amdlibComputeOiData= rejects by default all fits with a chi2 &gt; 5 times the rms. Can be overriden with the =-X= option
   1 drops all visibilities where one of the photometries is &lt; 0. The option =-P= permits to raise this value above zero.
   1 tags all bad values in the OI-FITS file using the FLAG columns (Tables OI_VIS, OI_VIS2 and OI_T3) and  put BLANK values for the bad values in the AMBER_DATA table.  
   1 All of the above is known to permit to avoid Frame Selection in MR or HR mode. In LR mode w/o FINITO, a selection on piston seems necessary. All FINITO-aided modes are abolutely not guaranteed yet, due to the absence of FINITO residuals in the AMBER raw data files.     
   1    Bug fix for the stations indexes management
----++ The Yorick Interface
   * %ICON{wip}%In construction.
   * The GUI Browser gets one  "remove" button 
   * Bug correction: put back wlen table to meters  in OI fits file
   * A suffix is added to amdlibGetProductDir
   * Bug correction: spaces are properly handled into keywords !
   * Bug correction: amdlibReadLog supports some empty log file
   * Bug correction for hardcopy file of graphs
   * Bug corrections: amdlibSetSpec, is now able to write spectra !
   * amdlibP2VM.i, amdlibRawData.i: Add bandwidth to amdlibRawDataXXX  functions and correct a long-lasting bug about P2VM and science
   data wavelength matching. Hopefully it will fix other side-effect bugs also

<hr/><hr/>

---++ V2.1 -> V2.2
   *  Correction of bugs happening when appending OI-FITS files amd memory leaks in the C code part.
   * Correction of bugs in the yorick scripts:
      * corrected amdlibDivideOiData crashes (due to zeroed differential visibilities)
      * corrected bugs relative to NaNs and zero values
      * corrected UV coordinates bugs
      * now avoid to write temporary files
      * amdlibAppendOiData and amdlibAppendAllOiData have been added to the list of amdlib functions
      * update of amdlibShowOiData behavior
   * updated documentation

---++ V2.0 -> V2.1
   * This version (2.1) corrects a problem wrong estimate of spectral resolution elements invalidated by the spectral displacement between the photometric and interferometric beams. This caused a of loss of valid spectral channels during the P2VM computation, only noticeable in Low resolution JHK mode, when the spectral displacement was large, which, to our knowledge, occured only between September 2007 and January 2008. However, this bug correction may be beneficial for all observations, providing a few (1 to 2) more wavelengths available in the reduced datasets, in the long-wavelength end of LR and HR modes, and short-wavelength end in MR modes. 
   * zeroed values of VISAMP and VISAMPERR until computation of Differential Visibility Amplitude has been accepted.
   * Fix yorick plugin compilation on Linux-x86_64

---++ Beta 1 -> Beta 2
   * Set default frame selection criterion to 'fringe SNR'
   * Change message when selecting directory to perform all frame selection
   * Update for gcc 4.x 
   * Fix bug in amdlibPerformFrameSelection (yorick plugin only) which invoked flux selection when calling snr selection. 
   * Fix bug in amdlibLoadSelection which crashed, now called amdlibLoadFrameSelection
   * Implement amdlibSaveFrameSelection to create your own selection file
   * Automatic check of the log integrity when available and recompute the missing parts if necessary
   * Change as much as possible parameter names in yorick-plugin according to document of G.Zins (second proposition) (jblebou).
   * Remove as much as possible 'verbose=' parameter in yorick-plugin (jblebou)
   * Add PRO CATG keyword in frame selection file header
   * Add procedures to handle logged messages ; i.e. message formatting, verbosity control, ...
   * Implement these Logging procedure in amdlibOiData.i, amdlibP2vm.i, amdlibDataQuality.i, amdlibGraphics.i  (jblebou)
   * Fix bug in amdlibShowData when OI file contains 'NaN' values
   * Change format of OI fits files in the AMBER_DATA table for ESO compliance. amdlib-2.0-Beta-1-generated OI files should normally still work but it is highly recommended to recompute them.
   * Corrected a graphical bug in amdlibShowP2vm about Vk plots : now plotted by telescopes and not by baseline.
   * Change parameter name 'skyFileName=' into 'inputSkyFile=', to be consistent with new parameter names.
   * Fix bug in amdlibAppendOiData

---++ Known bugs
   * amdlibLoadSelection does not work (fixed in Beta 2)
   * amdlibPerformFrameSelection (amdlib library) seems to not select frames for J-band raw OI files
   * amdlibComputeOiData generates 'NaN' values when data quality is very low

   * [[%ATTACHURL%/VraimentFlat.fits.gz][VraimentFlat.fits.gz]]: A really Flat Flat!
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
gz	VraimentFlat.fits.gz	r1	manage	1.4 K	2008-10-17 - 12:23	GillesDuvert	A really Flat Flat!
Topic revision: r22 - 2010-01-06 - GuillaumeMella