Analyzing high dimensional data

Abstract

Problems encountered in analyzing high dimensional data are discussed and possible solutions are proposed. The increased importance of second-order statistics in analyzing high dimensional data and the shortcoming of the minimum distance classifier in high dimensional data are recognized. By investigating characteristics of high dimensional data, it is shown that second-order statistics must be taken into account in high dimensional data. There is a need to represent second order statistics effectively. As the data dimensionality increases, it becomes more difficult to perceive and compare information present in statistics derived from data. In order to overcome this problem, a method to visualize statistics using color code is proposed. By representing statistics using a color code, the first and the second statistics can be more readily compared.