Search NASASearch

NASA NTRS · 19950065240

Reducing Truncation Error In Integer Processing

Abstract

Improved method of rounding off (truncation of least-significant bits) in integer processing of data devised. Provides for reduction, to extremely low value, of numerical bias otherwise generated by accumulation of truncation errors from many arithmetic operations. Devised for use in integer signal processing, in which rescaling and truncation usually performed to reduce number of bits, which typically builds up in sequence of operations. Essence of method to alternate direction of roundoff (plus, then minus) on alternate occurrences of truncated values contributing to bias.

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Thomas, J. Brooks, Berner, Jeffrey B., Graham, J. Scott. 1995-02-01. Reducing Truncation Error In Integer Processing. https://ntrs.nasa.gov/citations/19950065240

Cite the original work for its findings. Save a collection to share your selection of sources.