Since the idea of some completely general data format has been rejected (for reasons already presented), even for astronomical data, a number of composite data formats will be defined for various classes or forms of data as required.
The only current example of a composite data format, the NDF, is presented in Table . (NDF is short for Extensible n-Dimensional-Data Format and will be described fully here.) The NDF is based on the -dimensional data array, which is the most natural way to express many sorts of astronomical data set--notably spectra, pictures and time series.
Within an HDS container file, the NDF structure resides usually, though not necessarily, at the top level. For example, the top-level structure within a container file might be built from several NDFs, each an observation of the same source but through a different filter.
|Component Name||TYPE||Brief Description|
|[VARIANT]||_CHAR||variant of the NDF|
|[TITLE]||_CHAR||title of [DATA_ARRAY]|
|[DATA_ARRAY]||various||NAXIS-dimensional data array|
|[LABEL]||_CHAR||label describing the data array|
|[UNITS]||_CHAR||units of the data array|
|[VARIANCE]||s_array||variance of the data array|
|[QUALITY]||various||quality of the data array|
|[AXIS(NAXIS)]||AXIS||axis values, labels, units and errors|
(The and signs are not actually part of the TYPE, nor are the brackets around the NAME. They are just a notation convention to distinguish TYPE from NAME.)
There are several low-level objects within the NDF, each with a NAME for access and identification, and a TYPE to control the general processing. They comprise:
Only the [DATA_ARRAY] is mandatory--so a primitive HDS object containing a 2-D array of numbers (for example) is valid as the only component of an NDF. A full description of the components is given here and the meaning of the TYPEs in Naming, Types and Variants.
The omission from the NDF of the maximum and minimum values of the data array, and other quantities which can be deduced from the data, is deliberate. This is because experience has shown that sooner or later applications come along which fail to keep these numbers up to date.
In designing the NDF, efforts were made to retain some compatibility with the original Wright-Giddings proposals. A limited but useful degree of compatibility was achieved--the former location of the main data array (i.e. at the top level) is still recognised, and the name has been retained--but it proved impossible to accommodate more of the original proposals without enormously adding to the complexity of the processing rules.
As was mentioned earlier, the NDF is a simple structure devoid of any obviously astronomical items but which can be used to express many different varieties of astronomical data. IPCS spectra, CCD pictures and Taurus datacubes, for example, are all essentially -dimensional arrays together with some ancillary items, and fit naturally into the form of the NDF, simple though it is. In the next section, we will look at the general question of layering--how the elaborate requirements of any particular data type can be broken down into different levels of generality. We will then go on to consider the topics of substructures (how common building blocks may be identified and exploited), extensibility (how items peculiar to a data type or an application package may be accommodated), and various processing considerations including the propagation of extensions.
Starlink Standard Data Structures