V3.0

• amdlibComputeOiData : reduce a science or calibrator observation and produce an OI-FITS file. An observation is one raw data file. There are typically 5 raw datat files on an object (science or calibrator) plus 1 dark and 1 sky in each Observing Block.

Show usage  Hide usage 

  Usage: amdlibComputeOiData [-h] [-f] [-s] [-v level] [-b <num>] [-p PHASOR] [-e STATISTIC|THEORIC] [-c FLUX|FRG|OPD] [-r <ratio>] [-B J|H|K] <badPixelMap> <flatFieldMap> <p2vm> <dark> <input> <output>
  Options:
    -h help
    -v verbosity level [0..4]
    -f force the use of another P2VM
    -s split OI-FITS; one file per band
    -b <num> of frames to average (bin) over <num> frames
    -p PHASOR (default) for piston algorithm
    -e THEORIC (default) or STATISTIC for error bars estimation
    -c FRG, FLUX or OPD criterion for frame selection
    -r <ratio> for frame selection
       if 0<ratio<=1, its a ratio, else it is a threshold.
       flux are in (e), OPD in microns.
    -B J, H or K to select a spectral band

Options (new Options are in bold face):

• -J to tag Pistons with Jitter/Error gt. value (in microns)
• -X to tag visibilities with chi2 gt. value. This removes P2VM fits of complex coherent fluxes that did not behave well, that is, above some chi2. The default is -X0 , that is: tag data whose chi2 is above the mean chi2 by 3 times the chi2 rms. -X -1 or any negative value tages nothing. Normally AMBER data gives a mean chi2 between 0.9 and 1.8. Thus, -X 1.8 would select "normal" frames.
• -C to enhance piston precision using piston closure info. To be used with caution: The idea is to compute the piston on the baseliens with the smallest S/N as minus the sum of the two others. useful sometimes in Low Res.
• -P to retain photometries gt. value (in e). Default is 0, and default should not normally cause problems.
• -F to avoid using the continuum correction factor. This linearisation factor is part of the new way to reduce AMEBR data. Wizard only.
• -D to drop the first frames. Absolutely MANDATORY until the bias effects on the camera have been mastered. Drop at least the first 20 or 30 frames if possible. For FINITO-assisted data when the number of frames is too small, just pray .
• -n to avoid applying global bias estimate and correction. This should be used along with the -D option. If you drop frames, you do not need the global bias estimate.
• -A to avoid updating the badPixel info with bad pixels found in the dark file (Such On-The-Fly Bad Pixels are computed only if the number of frames is >500 anyway).
• -S to normalize star's spectrum by the spectrum of the P2VM (not exactly a transmission, the optical path being different: mostly useful in LowRes.). Wizard only.*

The default command should be something like that:

amdlibComputeOiData -D30 -n -b 99999 -p PHASOR -e THEORIC -s <badPixelMap> <flatFieldMap> <p2vm> <dark> <input> <output>

Note that it is recommendend to avoid selection (option -b 9999) unless you are in Low Res, where selection on PISTON ( -c OPD -r 15 ) would be a good idea if you are interested in differential visibilities.

• amdlibAppendOiData : Adds several OI-FITS files together.

Show usage  Hide usage 

 Usage: amdlibAppendOiData [-h] [-v level] <inOiFitsFile1> <inOiFitsFile2> [... <inOiFitsFileN>] <outOiFitsFile>
  Options:
    -h help
    -v verbosity level [0..4]
 

• amdlibPerformFrameSelection : from various selection criteria, produces a file with an index of frames to be retained to compute a final OI-FITS.

Show usage  Hide usage 

Usage: amdlibPerformFrameSelection [-h] [-v level] [-c FLUX_THR|FLUX_PCG|FRG_THR|FRG_PCG|OPD_THR|OPD_PCG] [-r <ratio>] [-i <inSelFile>] [-o <outSelFile>] <inOiFitsFile> [<outOiFitsFile>]
  Options:    -h help
    -v verbosity level [0..4]
    -c <criterion> for frame selection. Possible values are:
        FLUX_THR - threshold on flux (electron count)
        FLUX_PCG - percentage of frames kept based on flux selection
        FRG_THR  - threshold on fringe SNR (number)
        FRG_PCG  - percentage of frames kept based on fringe SNR selection
        OPD_THR  - threshold on piston (microns)
        OPD_PCG  - percentage of frames kept based on piston selection
    -r <value> frame selection value
    -i <inSelFile> input selection file
    -o <outSelFile> output selection file
 

Changes in the library wrt. version 2

General issues

• This is a completely new release involving a rewriting of many parts of the amdlib C library.
• The code implements most of the algorithms published by Chelli et al. 2009 as well as workarounds of some of the problems audited by the ATF team (see ATF report).
• We have tried to maintain, at least this time, the interfaces of the amdlib C library 'almost' compatible with the previous ones, to avoid rewriting of the Yorick interface.However, a number of internal changes were necessary, as for example the fact that all computations are now in double precision. Another important change is that amdlib OI-FITS files have FLAGS and BLANKING VALUES . User programs reading these files should be aware that, depending on filters implicit or explicit in amdlibV3, some values may be flagged or blanked.

In Particular

• detector noise is computed per pixel from the dark file (darks thus preferably contain more than 500 frames for good statistics)
• on-the-fly bad pixel detection can be done
• since the dark in Amber is usually a hot dark, dark noise is taken into account AND the additional use of SKY files is FORBIDDEN
• However, one can use a SKY file in place of a DARK, which should be better in case of, e.g., sky glow.
• Starting June 2009, day cold darks are taken by ESO. These should be used preferentially to any other type of dark or sky.
• Flats are defintely unnecessary, except for spectrophotometric purposes wich are not the scope of amdlib --- this version has not been tested yet using an AMBER Flat, only this really flat flat.
• additionnally, the software

1. implements the linearization factor of Chelli et al 2008. Information is displayed as:
   Average Linearization Factor: 0.953354, meaning that the data is good. if this factor is markedly different from 1.0, (say, below 0.8 or above 1.2 ), you could wonder if the P2VM is good enough or if the bad pixel map is up to date.
2. uses a refined algorithm to compute pistons, and an heuristic scheme to evaluate this piston 'goodness of fit'. This is mostly useful in Low Resolution witout FINITO where the atmosphere piston can cause a complete loss of coherence. The present version does not use the improved estimate of piston to correct from the theoretical extenuation. However, it has options ( -J and -C ) to tag bad piston values and use piston closure for piston evaluation improvement. Use the -c and -r options to filter out pistons larger than some fraction of the coherence length (say, 20 microns in LRK).
3. uses a goodness of fit test to tag individual visibilities which are not well fitted by the carrying waves of the interferogram. A message is associated, like: Correlated Flux fit statistics: mean = 1.015236, rms = 0.276510, median = 0.988546. The fact that this chi2 is very close to 1 means that all the variances have been well estimated. amdlibComputeOiData rejects by default all fits with a chi2 > 5 times the rms. Can be overriden with the -X option
4. drops all visibilities where one of the photometries is < 0. The option -P permits to raise this value above zero.
5. tags all bad values in the OI-FITS file using the FLAG columns (Tables OI_VIS, OI_VIS2 and OI_T3) and put BLANK values for the bad values in the AMBER_DATA table.
6. All of the above is known to permit to avoid Frame Selection in MR or HR mode. In LR mode w/o FINITO, a selection on piston seems necessary. All FINITO-aided modes are abolutely not guaranteed yet, due to the absence of FINITO residuals in the AMBER raw data files.
7. Bug fix for the stations indexes management

The Yorick Interface

• In construction.
• The GUI Browser gets one "remove" button
• Bug correction: put back wlen table to meters in OI fits file
• A suffix is added to amdlibGetProductDir
• Bug correction: spaces are properly handled into keywords !
• Bug correction: amdlibReadLog supports some empty log file
• Bug correction for hardcopy file of graphs
• Bug corrections: amdlibSetSpec, is now able to write spectra !
• amdlibP2VM.i, amdlibRawData.i: Add bandwidth to amdlibRawDataXXX functions and correct a long-lasting bug about P2VM and science data wavelength matching. Hopefully it will fix other side-effect bugs also

V2.1 -> V2.2

• Correction of bugs happening when appending OI-FITS files amd memory leaks in the C code part.
• Correction of bugs in the yorick scripts:
  • corrected amdlibDivideOiData crashes (due to zeroed differential visibilities)
  • corrected bugs relative to NaNs and zero values
  • corrected UV coordinates bugs
  • now avoid to write temporary files
  • amdlibAppendOiData and amdlibAppendAllOiData have been added to the list of amdlib functions
  • update of amdlibShowOiData behavior
• updated documentation

V2.0 -> V2.1

• This version (2.1) corrects a problem wrong estimate of spectral resolution elements invalidated by the spectral displacement between the photometric and interferometric beams. This caused a of loss of valid spectral channels during the P2VM computation, only noticeable in Low resolution JHK mode, when the spectral displacement was large, which, to our knowledge, occured only between September 2007 and January 2008. However, this bug correction may be beneficial for all observations, providing a few (1 to 2) more wavelengths available in the reduced datasets, in the long-wavelength end of LR and HR modes, and short-wavelength end in MR modes.
• zeroed values of VISAMP and VISAMPERR until computation of Differential Visibility Amplitude has been accepted.
• Fix yorick plugin compilation on Linux-x86_64

Beta 1 -> Beta 2

• Set default frame selection criterion to 'fringe SNR'
• Change message when selecting directory to perform all frame selection
• Update for gcc 4.x
• Fix bug in amdlibPerformFrameSelection (yorick plugin only) which invoked flux selection when calling snr selection.
• Fix bug in amdlibLoadSelection which crashed, now called amdlibLoadFrameSelection
• Implement amdlibSaveFrameSelection to create your own selection file
• Automatic check of the log integrity when available and recompute the missing parts if necessary
• Change as much as possible parameter names in yorick-plugin according to document of G.Zins (second proposition) (jblebou).
• Remove as much as possible 'verbose=' parameter in yorick-plugin (jblebou)
• Add PRO CATG keyword in frame selection file header
• Add procedures to handle logged messages ; i.e. message formatting, verbosity control, ...
• Implement these Logging procedure in amdlibOiData.i, amdlibP2vm.i, amdlibDataQuality.i, amdlibGraphics.i (jblebou)
• Fix bug in amdlibShowData when OI file contains 'NaN' values
• Change format of OI fits files in the AMBER_DATA table for ESO compliance. amdlib-2.0-Beta-1-generated OI files should normally still work but it is highly recommended to recompute them.
• Corrected a graphical bug in amdlibShowP2vm about Vk plots : now plotted by telescopes and not by baseline.
• Change parameter name 'skyFileName=' into 'inputSkyFile=', to be consistent with new parameter names.
• Fix bug in amdlibAppendOiData

Known bugs

• amdlibLoadSelection does not work (fixed in Beta 2)
• amdlibPerformFrameSelection (amdlib library) seems to not select frames for J-band raw OI files
• amdlibComputeOiData generates 'NaN' values when data quality is very low

• VraimentFlat.fits.gz: A really Flat Flat!