Specific tips

The following tricks are applicable when using several of the Starlink applications. To use some of the commands in the examples you will need to start KAPPA (see SUN/95[6]) by typing kappa at the C shell prompt. These commands (erase, ndftrace, parget, settype, ndfcopy, paste, ardmask, compave, compadd and compick) are described fully in SUN/95; but by way of a quick explanation, the ndftrace, parget pair tells you one thing about the NDF being queried.

Omit variances:

variance information doubles the size of NDFs on disk and for many of the applications substantially increases the CPU and memory usage. Often it is not required, or if it is can be satisfactorily estimated from the data themselves. If you do not need it, then do not generate and/or propagate it. To omit the variance set parameter genvar=false either in ccdsetup or in debias and makebias. If you wish to remove the VARIANCE component from a frame which already contains it, you can use the KAPPA command erase:

ndftrace frame quiet
parget variance ndftrace
TRUE
erase frame.variance
OK - The HDS object FRAME.VARIANCE is to be erased. OK ? /NO/ > yes
ndftrace frame quiet
parget variance ndftrace
FALSE

Use an appropriate data type:

possible data types for storage of the pixel values in NDFs are:

Data Type	Size (bytes)
_BYTE, _UBYTE	1
_WORD, _UWORD	2
_INTEGER, _REAL	4
_DOUBLE	8

The type _WORD is usually sufficient for storage of most of the intermediate NDFs required in a data reduction sequence (the exception is makeflat which always generates a master flat field of type _REAL or _DOUBLE). If your data type is _INTEGER, _REAL or _DOUBLE therefore it can be worth reducing it to one of the smaller types. The KAPPA programs ndftrace and settype can be used to determine and modify respectively the type of data in an NDF, as in this example:

ndftrace frame quiet
parget type ndftrace
_REAL
settype frame _WORD
ndftrace frame quiet
parget type ndftrace
_WORD

Reducing the size of the data type may increase or reduce the CPU time requirements of the program, but should reduce the memory and I/O requirements. Under certain circumstances using a two-byte type can lead to overflow errors however, so some caution should be exercised.

Compact NDFs:

sometimes applications generate or modify NDFs to contain additional empty space; you can tell if this is the case by examining the file using ndftrace to work out the approximate size it should be and comparing this with the size shown by ls. If disk space is very tight, and you do not want to delete files, such oversized NDFs can be compacted using the KAPPA application ndfcopy. For example, using the file of reduced data type created above:

ls -s frame.sdf
2047 frame.sdf
ndfcopy frame compacted_frame
ls -s compacted_frame.sdf
1027 compacted_frame.sdf
mv compacted_frame.sdf frame.sdf

Treat images in sections:

when faced with really large images, the only way to process them may be by breaking them up into sections. This can be done using NDF sections as described in SUN/95[6], for example:

flatcor in=huge"(:,:4000)" flat=master_flat"(:,:4000)" out=bottom
flatcor in=huge"(:,4001:)" flat=master_flat"(:,4001:)" out=top
paste bottom top out=huge_flatcor

Reduce image resolution:

in the event that image resolution is better than required, the size of the frames can be reduced by using one of the KAPPA applications compave, compadd or compick. If the averaged pixels are still small enough to under-sample the image point spread function this approach will be ok; otherwise it is rather a waste of good data, but may be useful for taking a quick look at oversized frames.

Finally, we list the most demanding of the CCDPACK applications with some notes about each one.

debias:

debias is the heaviest user of memory and so is where problems are most likely to arise. The following suggestions are possible ways of limiting the resources used:

Masking:

if the mask parameter is set (to the name of an image or ARD file) then more memory is required. It can therefore be more efficient to apply the mask explicitly elsewhere in the reduction sequence, for example, to the bias frame prior to de-biassing:

ardmask in=master_bias out=masked_bias ardfile=mask.dat
debias in="data?" out="debias_*" bias=masked_bias \
         getmask=FALSE

instead of:

debias in="data?" out="debias_*" bias=master_bias \
         getmask=TRUE mask=mask.dat

Variances:

using variances will also have a big impact on the requirements of debias, and so should be avoided (using genvar=false) if possible.

Bias frames:

de-biassing using the bias strips rather than a master bias frame (see Sections  and ) reduces the work done by debias and also makes it unnecessary to process the bias frames at all. This technique will lead to inferior de-biassing, but can represent significant savings, and using frames from modern CCDs may give quite satisfactory results.

makemos:

the normalisation part of makemos, if performed, is usually the most CPU intensive part of the data reduction process, although this depends on how numerous and how large the regions of overlap between frames are. The process should therefore be omitted if it is not required. Normalisation is performed only if one or both of the parameters scale and zero is true (both default to false): set scale=true only if multiplicative corrections might be required (for example, if the individual input images have differing exposure times) and set zero=true only if additive corrections might be required (for example, if the images have different background levels). If it must be performed, the following measures may decrease execution time, possibly at the expense of accuracy:

• if scale but not zero is being used and the images have variance information then set cmpvar=false,

• if there are many multiply-overlapping frames then set the parameter optov (optimum number of overlaps) to a small number (such as one),

• the normalisation is usually an iterative process, so it is possible to tweak the parameters controlling the iteration ( maxit, tols, tolz).

findoff:

this application uses one of two algorithms to match objects between frames for determining their relative offset. The first algorithm scales as $n^2$ (where $n$ is the number of objects found by findobj), but if this fails it normally falls back on a more reliable algorithm which scales as $n^3 \ln n$. In this case, and if there are many objects, findoff can be very slow indeed and come to dominate the whole reduction process. Failure of the fast algorithm is also more likely when there are very many objects. For both these reasons it can be a good idea to limit the number of objects found by findobj - a few tens of objects in the overlap region is about right. You can control the number of objects found by by modifying the minpix parameter: the higher this threshold is set the fewer objects findobj will identify in the image.

More detailed information on each of these applications can be found in SUN/139[10].