General tips

A full discussion of maximising performance for large jobs is beyond the scope of this document, but the following are good common-sense rules of thumb.

Run on a large machine:

usually the more memory available the faster the job will run, since this reduces the amount of disk I/O needed.

Use local disks:

disks attached to the machine running the job will be much faster than disks attached to another machine which are accessed remotely via the local network. It can also make a big difference to use a disk which no other process is making heavy use of at the time. Your system manager may be able to advise on choice of disk.

Be economical with disk space:

while it may make sense to retain all intermediate files (for example, de-biassed, flat fielded, re-sampled frames) for small images, these can take up excessive disk space for large images. Scripts can be written to remove files as they go along, or appropriate options of the applications can be used (for example, keepin=false in CCDPACK's debias and flatcor). When thinking about disk space requirements, remember that large temporary files can be created by some of the applications. These files have names like t123.sdf and are created in the directory pointed to by the environment variable HDS_SCRATCH, or in the current directory if HDS_SCRATCH is not defined.

Discuss with your system manager and/or other users:

if your job could have a serious effect on the system's performance it might be polite to ask if there are recommended ways of going about it, or to warn other users.

Run at off-peak times:

if you can run your job at a time when few or no other processes are running on the machine in question it will run faster and inconvenience other users less. The Unix at command can be used to start a job at a given time, or there may be other queuing software installed at your site.

Be nice to other processes:

on Unix the commands nice or renice should be used when running CPU-intensive jobs on multi-user machines. In the C shell typing:

nice +18 reduce_script

would run the script reduce_script at a `niceness' of 18. This setting means that the job will be less aggressive in requesting CPU time, thus making it run slower, but causing less disruption to other processes (presumably ones with more moderate requirements). The higher the niceness, the less demanding the job is, with 18 often a sensible maximum. Ask your system manager for more details; there may be locally recommended values for certain kinds of job. Note however that the only resource usage this affects is CPU time, so that even a maximally niced job can cause major disruption.

Keep an eye on the job:

if your job might push the system to its limits, especially if you have not run one of similar size before, it is a good idea to monitor its progress, for instance to check that the system's swap space or file system is not filling up (using, for example, top and df respectively).