Arithmetic Precision

When a transformation is created, a precision specification is associated with it (i.e. via the PREC argument to TRN_NEW - Section ) and this subsequently determines the type of arithmetic (integer, real or double precision) which will be used to evaluate the transformation functions. The correct choice of this specification is important if the desired behaviour is to be achieved. Two considerations normally apply:

1. The type of calculation being performed. For instance, integer arithmetic might be required if effects due to rounding were being exploited, whereas transformation functions representing a 2-dimensional rotation would probably need to use real arithmetic, even if the coordinate data were to be stored as integers. Some functions may also require double precision arithmetic to reduce internal rounding errors.

2. The type of data being transformed. It would clearly be inappropriate to use real arithmetic on double precision data but, conversely, it would be inefficient to use a higher precision than was necessary to preserve the accuracy of real data.

It can be seen that an appropriate arithmetic precision cannot necessarily be selected solely on the basis of the type of calculation to be performed, because the type of coordinate data to be processed may also be relevant. Unfortunately, this latter information may not be available to the application which creates the transformation. Furthermore, whenever transformations are concatenated (Section ), each must be able to accommodate data passed to it by neighbouring transformations although it may not have any advance knowledge of the type of arithmetic its neighbours will be using.

To accommodate these possibilities without unnecessary loss of data precision, transformations are allowed some flexibility in adapting their internal arithmetic to the external data being processed. This is controlled by the precision specification, which is a character string taking one of the following six values:

Fixed precisions $\left\{ \mbox{ \begin{tabular}{l} \lq {\tt\_INTEGER}'\\ \lq {\tt\_REAL}'\\ \lq {\tt\_DOUBLE}' \end{tabular} } \right.$

Elastic precisions $\left\{ \mbox{ \begin{tabular}{l} \lq {\tt\_INTEGER:}'\\ \lq {\tt\_REAL:}'\\ \lq {\tt\_DOUBLE:}' \end{tabular} } \right.$

Fixed precisions specify the type of arithmetic to be used explicitly. In this case, the data used in the calculation (both incoming coordinates and explicit numerical constants in the transformation functions) are first converted to the specified numerical type (integer, real or double precision) and the transformation functions are then evaluated using the appropriate type of arithmetic.

N.B. It is currently assumed that the data types of all the input and output coordinates are the same and all the transformation functions are therefore evaluated using the same type of arithmetic. This may change in future to allow coordinates to have mixed data types. The present arrangement is designed so that changes to existing applications and datasets will not generally be necessary.

Elastic precisions are similar, except that some subsequent adjustment is also allowed. In this case, the precision specification indicates the minimum precision of the arithmetic to be used, but this may be increased if the data type warrants it. Thus, a precision specification of `_REAL:' would request that real arithmetic be used unless double precision data were being processed (in which case double precision arithmetic would be used instead). In general, the type of arithmetic used is related to the precision specification and the type of data being processed as follows:

	Elastic Precision Specification
Data Type	`_INTEGER:'	`_REAL:'	`_DOUBLE:'
_INTEGER	integer	real	double precision
_REAL	real	real	double precision
_DOUBLE	double precision	double precision	double precision

Once the arithmetic precision has been decided, the transformation functions are evaluated in the same way as for a fixed precision (i.e. by converting all incoming data and constants to the appropriate data type and then performing the arithmetic).

When several transformations have been concatenated, the type of arithmetic to be used is determined individually for each stage in the overall transformation (data type conversion being performed automatically between each stage as necessary). If an elastic precision (which is sensitive to the input data type) has been specified, then the input data type is taken to correspond with the type of arithmetic used in the previous mapping to be evaluated (or the input data type itself if appropriate). This arrangement avoids any unnecessary loss of data precision regardless of the order in which transformations are concatenated.