-- GillesDuvert - 17 Oct 2008

V3 Beta 1

This is a completely new release involving a rewriting of many parts of the amdlib C library.
The code implements most of the algorithms published by Chelli et al. (2008, Astron Astrophys, submitted) as well as workarounds of some of the problems audited by the ATF team (see ATF report).
We have tried to maintain, at least this time, the interfaces of the amdlib C library 'almost' compatible with the previous ones, to avoid rewriting of the Yorick interface.
However, a number of internal changes were necessary, as for example the fact that all computations are now in double precision. Another important change is that amdlib OI-FITS files have FLAGS and BLANKING VALUES . User programs reading these files should be aware that, depending on filters implicit or explicit in amdlibV3, some values may be flagged or blanked.
In this v3-Beta version, the Yorick interface does not take yet into account blanked values associated with Pistons and Photometries, which are not plotted correctly (principally in the amdlibShowOiData command).
For beta testers, it is recommended to use the shell interface commands amdlibComputeP2vm and amdlibComputeOiData in case of doubt. These have more options now than the yorick interface.
This version cannot be used as it is for ESO pipeline or inside AMBER observation software, the compatibility of version 2.x has been lost in the rewriting of code.
Starting from this version (amdlibV3Beta1), users are requested, if they use this software for a scientific publication, to acknowledge the use of JMMC tools and cite the following paper:
Chelli, A., Hernandez Utrera, O., Duvert, G., "Optimised Data Reduction for the VLTI/AMBER instrument", A&A (submitted) (to be replaced by the exact citation when published, keep in touch).

Summary of changes:

• detector noise is computed per pixel from the dark file.
• since the dark in Amber is a hot dark, dark noise is taken into account AND the additional use of SKY files is FORBIDDEN
• However, one can use a SKY file in place of a DARK, which should be better in case of, e.g., sky glow.
• Flats are defintely unnecessary, except for spectrophotometric purposes wich are not the scope of amdlib --- this version has not been tested yet using an AMBER Flat, only this really flat flat.
  
• shell commands:
  • amdlibComputeP2vm should be called with the option -s o1,o2,o3 where o1,etc.. are the constant offset values found by the ATF team.
  • amdlibComputeP2vm can have a 4th argument to the -s option, to displace by a fractional number of pixels the wavelength table.
  • amdlibComputeOiData
    1. implements the linearization factor of Chelli et al 2008. Information is displayed as:
       Average Linearization Factor: 0.953354, meaning that the data is good. if this factor is markedly different from 1.0, (say, below 0.8 or above 1.2), you could wonder if the P2VM is good enough or if the bad pixel map is up to date.
    2. uses a refined algorithm to compute pistons, and an heuristic scheme to evaluate this piston 'goodness of fit'. This is mostly useful in Low Resolution witout FINITO where the atmosphere piston can cause a complete loss of coherence. The present version does not use the improved estiamte of piston to correct from the theoretical extenuation. However, it has options ( -J and -C ) to tag bad piston values and use piston closure for piston evaluation improvement. Use the -c and -r options to filter out pistons larger than some fraction of the coherence length (70 microns in K band).
    3. uses a goodness of fit test to tag individual visibilities which are not well fitted by the carrying waves of the interferogram. A message is associated, like: Correlated Flux fit statistics: mean = 1.015236, rms = 0.276510, median = 0.988546. The fact that this chi2 is very close to 1 means that all the variances have been well estimated. amdlibComputeOiData rejects by default all fits with a chi2 > 1.8. Can be overriden with the -X option
    4. drops all visibilities where one of the photometries is < 0. The option -P permits to raise this value above zero.
    5. tags all bad values in the OI-FITS file using the FLAG columns (Tables OI_VIS, OI_VIS2 and OI_T3) and put BLANK values for the bad values in the AMBER_DATA table.
    6. All of the above is known to permit to avoid Frame Selection in MR or HR mode. In LR mode w/o FINITO, a selection on piston seems necessary. All FINITO-aided modes are abolutely not guaranteed yet, due to the absence of FINITO residuals in the AMBER raw data files.

---

V2.1 -> V2.2

• Correction of bugs happening when appending OI-FITS files amd memory leaks in the C code part.
• Correction of bugs in the yorick scripts:
  • corrected amdlibDivideOiData crashes (due to zeroed differential visibilities)
  • corrected bugs relative to NaNs and zero values
  • corrected UV coordinates bugs
  • now avoid to write temporary files
  • amdlibAppendOiData and amdlibAppendAllOiData have been added to the list of amdlib functions
  • update of amdlibShowOiData behavior
• updated documentation

V2.0 -> V2.1

• This version (2.1) corrects a problem wrong estimate of spectral resolution elements invalidated by the spectral displacement between the photometric and interferometric beams. This caused a of loss of valid spectral channels during the P2VM computation, only noticeable in Low resolution JHK mode, when the spectral displacement was large, which, to our knowledge, occured only between September 2007 and January 2008. However, this bug correction may be beneficial for all observations, providing a few (1 to 2) more wavelengths available in the reduced datasets, in the long-wavelength end of LR and HR modes, and short-wavelength end in MR modes.
• zeroed values of VISAMP and VISAMPERR until computation of Differential Visibility Amplitude has been accepted.
• Fix yorick plugin compilation on Linux-x86_64

Beta 1 -> Beta 2

• Set default frame selection criterion to 'fringe SNR'
• Change message when selecting directory to perform all frame selection
• Update for gcc 4.x
• Fix bug in amdlibPerformFrameSelection (yorick plugin only) which invoked flux selection when calling snr selection.
• Fix bug in amdlibLoadSelection which crashed, now called amdlibLoadFrameSelection
• Implement amdlibSaveFrameSelection to create your own selection file
• Automatic check of the log integrity when available and recompute the missing parts if necessary
• Change as much as possible parameter names in yorick-plugin according to document of G.Zins (second proposition) (jblebou).
• Remove as much as possible 'verbose=' parameter in yorick-plugin (jblebou)
• Add PRO CATG keyword in frame selection file header
• Add procedures to handle logged messages ; i.e. message formatting, verbosity control, ...
• Implement these Logging procedure in amdlibOiData.i, amdlibP2vm.i, amdlibDataQuality.i, amdlibGraphics.i (jblebou)
• Fix bug in amdlibShowData when OI file contains 'NaN' values
• Change format of OI fits files in the AMBER_DATA table for ESO compliance. amdlib-2.0-Beta-1-generated OI files should normally still work but it is highly recommended to recompute them.
• Corrected a graphical bug in amdlibShowP2vm about Vk plots : now plotted by telescopes and not by baseline.
• Change parameter name 'skyFileName=' into 'inputSkyFile=', to be consistent with new parameter names.
• Fix bug in amdlibAppendOiData

Known bugs

• amdlibLoadSelection does not work (fixed in Beta 2)
• amdlibPerformFrameSelection (amdlib library) seems to not select frames for J-band raw OI files
• amdlibComputeOiData generates 'NaN' values when data quality is very low

• VraimentFlat.fits.gz: A really Flat Flat!