Standard Components in an NDF

An NDF comprises a main data array plus a collection of objects drawn from a set of standard items and extensions (see SUN/33). Only the main data array must be present; all the other items are optional.

example.sdf is an NDF which contains all the standard NDF components, except a WCS component and a FITS extension; it also has a FIGARO extension. The structure of the file (as revealed by HDSTRACE) is shown below. The layout is

     NAME(dimensions)    <TYPE>     VALUE(S)

and each level down the hierarchy is indented. Note that scalar objects have no dimensions

   EXAMPLE  <NDF>

      DATA_ARRAY     <ARRAY>         {structure}
         DATA(856)      <_REAL>         *,0.2284551,-2.040089,45.84504,56.47374,
                                        ... 746.2602,820.8976,570.0729,*,449.574
         ORIGIN(1)      <_INTEGER>      4
         BAD_PIXEL      <_LOGICAL>      FALSE

      TITLE          <_CHAR*30>      'HR6259 - AAT fibre data'
      LABEL          <_CHAR*20>      'Flux'
      UNITS          <_CHAR*20>      'Counts/s'
      QUALITY        <QUALITY>       {structure}
         BADBITS        <_UBYTE>        1
         QUALITY(856)   <_UBYTE>        1,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,
                                        ... 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0

      VARIANCE       <ARRAY>         {structure}
         DATA(856)      <_REAL>         2.1,0.1713413,1.5301,34.38378,42.35531,
                                        ... 615.6732,427.5547,353.9127,337.1805
         ORIGIN(1)      <_INTEGER>      4
         BAD_PIXEL      <_LOGICAL>      FALSE

      AXIS(1)        <AXIS>          {structure}

      Contents of AXIS(1)
         DATA_ARRAY(856)  <_REAL>       3847.142,3847.672,3848.201,3848.731,
                                        ... 4298.309,4298.838,4299.368,4299.897
         LABEL          <_CHAR*20>      'Wavelength'
         UNITS          <_CHAR*20>      'Angstroms'

      HISTORY        <HISTORY>       {structure}
         CREATED        <_CHAR*30>      '1990-DEC-12 08:21:02.324'
         CURRENT_RECORD  <_INTEGER>     3
         RECORDS(10)    <HIST_REC>      {array of structures}

         Contents of RECORDS(1)
            TEXT           <_CHAR*40>      'Extracted spectrum from fibre data.'
            DATE           <_CHAR*25>      '1990-DEC-19 08:43:03.08'
            COMMAND        <_CHAR*30>      'FIGARO V2.4 FINDSP command'


      MORE           <EXT>           {structure}
         FIGARO         <EXT>           {structure}
            TIME           <_REAL>         1275
            SECZ           <_REAL>         2.13

   End of Trace.

Of course, this is only an example format. There are various ways of representing some of the components. These variants are described in SGP/38, but not all are currently supported.

The components are considered in detail below. The names (in bold typeface) are significant as they are used by the NDF access routines to identify the components.

DATA

-- the main data array (called DATA_ARRAY for historical reasons) is the only component which must be present in an NDF. In the case of example.sdf, is a one-dimensional array of real type with 856 elements. It can have up to seven dimensions. It is particularly referenced via parameter names IN, OUT, and NDF.

If neither origin nor bad-pixel flag were present, the DATA component could have been a one-dimensional array like this,

      DATA_ARRAY(856)  <_REAL>       *,0.2284551,-2.040089,45.84504,56.47374,
                                     ... 746.2602,820.8976,570.0729,*,449.574

rather than the structure shown above.

ORIGIN

-- an array of the indices of the first pixel along each axis, defaulting to 1. See Pixel Indices, Pixel Co-ordinates, and Grid Co-ordinates  for further information and a graphic. ORIGIN may be set with task SETORIGIN, or after processing an NDF section.

BAD_PIXEL

-- the bad-pixel flag indicating whether there may be bad pixels present. See Bad-pixel Flag Values for further information. The flag may be set with task SETBAD.

TITLE

-- the character string "HR6259 - AAT fibre data" describes the contents of the NDF. The NDF's TITLE might be used as the title of a graph etc. It may be set with task SETTITLE. Applications that create an NDF assign a TITLE to the NDF via a parameter, called TITLE unless the application generates several NDFs.

LABEL

-- the character string "Flux" describes the quantity represented in the NDF's main data array. The LABEL is intended for use on the axis of graphs etc. It may be set using the task SETLABEL.

UNITS

-- this character string describes the physical units of the quantity stored in the main data array, in this case, "Counts/s". It may be set via the command SETUNITS.

QUALITY

-- this component is used to indicate the quality of each element in the main data array, for example whether each pixel is vignetted or not. The quality structure contains a quality array and a BADBITS value, both of which must be of type _UBYTE. The quality array has the same shape and size as the main data array and is used in conjunction with the BADBITS value to decide the quality of a pixel in the main data array. In example.sdf the BADBITS component has value 1. This means that a value of 1 in the quality array indicates a bad pixel in the main data array, whereas any other value indicates that the associated pixel is good. (Note that the pixel is bad if the bit-wise comparison QUALITY "AND" BADBITS is non-zero). The meanings of the QUALITY bits are arbitrary. See the task SETBB. To enter new quality information, use the SETQUAL command.

VARIANCE

-- the variance array is the same shape and size as the main data array and contains the errors associated with the individual data values. These are stored as variance estimates for each pixel. VARIANCE may be set with the SETVAR command.

AXIS

-- the AXIS structure may contain axis information for any dimension of the NDF's main array. In this case, the main data array is only one-dimensional, therefore only the AXIS(1) structure is present. This structure contains the actual axis data array of pixel centres, and also label and units information. KAPPA uses the label and units for axis annotations. Not shown in this example are optional array components for storing pixel widths and the variance of the axis centres. All axes or none must be present. Use SETAXIS to set the values of an AXIS array component; AXLABEL and AXUNITS to set an axis LABEL or UNITS  component; and SETNORM to set an axis normalisation flag. For more information see `Co-ordinate Frames, Axes and Domains'.

WCS

-- this component provides an alternative, and superior, method for storing co-ordinate information. The AXIS structure described above has the severe limitation that it can only describe co-ordinate systems in which all axes are independent (i.e. the value on any axis can be changed without needing to change the values on any other axes). This means for instance that axes cannot be rotated (other than by multiples of 90$^\circ$ ), and cannot be used to describe celestial co-ordinates over a large field, or near a pole. The WCS component overcomes these restrictions. It contains descriptions of an arbitrary number of different co-ordinate Frames, together with the mappings required to convert positions between these Frames. All NDFs have four default Frames available; these are known as the GRID, FRACTION, PIXEL, and AXIS Frames. The PIXEL Frame corresponds to pixel co-ordinates. The AXIS Frame corresponds to the co-ordinate Frame implied by the NDF AXIS structures. The GRID Frame is similar to the PIXEL Frame, the main difference being that it has a different origin; the centre of the first (`bottom left') pixel is always $(1.0,1.0)$ in the GRID Frame regardless of the pixel origin of the NDF. The FRACTION Frame corresponds to normalised PIXEL or GRID co-ordinates, scaled from zero to one. Other Frames can be added to the WCS component in various ways, for instance, by importing them from appropriate FITS headers (FITSIN, FITSDIN), or using an appropriate application to create them from scratch (e.g. WCSADD, SETSKY or GAIA). See Using World Co-ordinate Systems for more information.

HISTORY

-- this component provides a record of the processing history of the NDF. Only the first of three records is shown for example.sdf. This indicates that the spectrum was extracted from fibre data using the FIGARO FINDSP command on 1990 December 19. The history recording level is set by task HISSET. This task also allows you to switch off history recording or delete the history records. HISLIST lists an NDF's history. You can add commentary with HISCOM.

EXTENSIONs

-- the purpose of extensions is to store non-standard items. These auxiliary data could be information about the original observing setup, such as the airmass during the observation or the temperature of the detector; they may be calibration data or results produced during processing of the data array, e.g. spectral-line fits. The extensions are located within the MORE top-level component. example.sdf began life as a FIGARO file. It was converted to an NDF using the command DST2NDF. It contains values for the airmass and exposure time associated with the observations. These are stored in the FIGARO extension, and the intention is that the FIGARO applications which use these values will know where to find them. Task SETEXT lets the contents of extensions be listed, created, deleted, renamed and assigned new values.

One extension that is used by KAPPA is the FITS extension. This holds the FITS headers as an array of 80-character elements, i.e. one FITS card image per array element. You can extract the values of ancillary items from the FITS extension to a non-standard extension via FITSIMP. Use FITSEXP to do the reverse operation. The extension can be listed via the command FITSLIST. FITSEDIT allows you to edit the headers prior to export of the dataset to another format such as FITS or IRAF.

KAPPA uses a PROVENANCE extension to record details of the ancestor NDFs used to generate an NDF. This lets you determine how the NDF before you, came to be. For each ancestor NDF, the provenance information includes its path at creation time, the creation epoch, the software that generated the ancestor, and indices to its parent NDFs (if any exist) within the PROVENANCE extension. There is also a MORE component within PROVENANCE that can be filled with arbitrary additional information. Provenance recording is controlled through the AUTOPROV environment variable: set it to 1 to enable recording, and set it to any other value to disable recording. When AUTOPROV is undefined, then an output NDF will have provenance only if at least one of the input NDFs has provenance. You can examine provenance with PROVSHOW, and modify it with PROVADD and PROVMOD. Selected provenance may be removed with PROVREM.