Search NASASearch

Engineering topics

Berner, Jeffrey B.

Publications and source records attributed to Berner, Jeffrey B..

Reducing Truncation Error In Integer Processing

Improved method of rounding off (truncation of least-significant bits) in integer processing of data devised. Provides for reduction, to extremely low value, of numerical bias otherwise generated by accumulation of truncation errors from many arithmetic operations. Devised for use in integer signal processing, in which rescaling and truncation usually performed to reduce number of bits, which typically builds up in sequence of operations. Essence of method to alternate direction of roundoff (plus, then minus) on alternate occurrences of truncated values contributing to bias.

Thomas, J. Brooks