SEARCH · Search NASA

Results for “Statistical Algorithms”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 397 records · Page 22

All sources NASA NTRS DOE OSTI

NASA NTRS2015

Managing Large Datasets for Atmospheric Research

Since the mid-1980s, airborne and ground measurements have been widely used to provide comprehensive characterization of atmospheric composition and processes. Field campaigns have generated a wealth of insitu data and have grown considerably over the years in terms of both the number of measured parameters and the data volume. This can largely be attributed to the rapid advances in instrument development and computing power. The users of field data may face a number of challenges spanning data access, understanding, and proper use in scientific analysis. This tutorial is designed to provide an introduction to using data sets, with a focus on airborne measurements, for atmospheric research. The first part of the tutorial provides an overview of airborne measurements and data discovery. This will be followed by a discussion on the understanding of airborne data files. An actual data file will be used to illustrate how data are reported, including the use of data flags to indicate missing data and limits of detection. Retrieving information from the file header will be discussed, which is essential to properly interpreting the data. Field measurements are typically reported as a function of sampling time, but different instruments often have different sampling intervals. To create a combined data set, the data merge process (interpolation of all data to a common time base) will be discussed in terms of the algorithm, data merge products available from airborne studies, and their application in research. Statistical treatment of missing data and data flagged for limit of detection will also be covered in this section. These basic data processing techniques are applicable to both airborne and ground-based observational data sets. Finally, the recently developed Toolsets for Airborne Data (TAD) will be introduced. TAD (tad.larc.nasa.gov) is an airborne data portal offering tools to create user defined merged data products with the capability to provide descriptive statistics and the option to treat measurement uncertainty.

Chen, Gao↗

NASA NTRS2017

Finding the Needles in the Haystacks: High-Fidelity Models of the Modern and Archean Solar System for Simulating Exoplanet Observations

We present two state-of-the-art models of the solar system, one corresponding to the present day and one to the Archean Eon 3.5 billion years ago. Each model contains spatial and spectral information for the star, the planets, and the interplanetary dust, extending to 50 au from the Sun and covering the wavelength range 0.3-2.5 micron. In addition, we created a spectral image cube representative of the astronomical backgrounds that will be seen behind deep observations of extrasolar planetary systems, including galaxies and Milky Way stars. These models are intended as inputs to high-fidelity simulations of direct observations of exoplanetary systems using telescopes equipped with high-contrast capability. They will help improve the realism of observation and instrument parameters that are required inputs to statistical observatory yield calculations, as well as guide development of post-processing algorithms for telescopes capable of directly imaging Earth-like planets.

extrasolar planetary systems↗

NASA NTRS2018

(GO)2-SIM: a GCM-Oriented Ground-Observation Forward-Simulator Framework for Objective Evaluation of Cloud and Precipitation Phase

General circulation model (GCM) evaluation using ground-based observations is complicated by inconsistencies in hydrometeor and phase definitions. Here we describe (GO)2-SIM, a forward simulator designed for objective hydrometeor-phase evaluation, and assess its performance over the North Slope of Alaska using a 1-year GCM simulation. For uncertainty assessment, 18 empirical relationships are used to convert model grid-average hydrometeor (liquid and ice, cloud, and precipitation) water contents to zenith polarimetric micropulse lidar and Ka-band Doppler radar measurements, producing an ensemble of 576 forward-simulation realizations. Sensor limitations are represented in forward space to objectively remove from consideration model grid cells with undetectable hydrometeor mixing ratios, some of which may correspond to numerical noise.Phase classification in forward space is complicated by the inability of sensors to measure ice and liquid signals distinctly. However, signatures exist in lidarâ€“radar space such that thresholds on observables can be objectively estimated and related to hydrometeor phase. The proposed phase-classification technique leads to misclassification in fewer than 8% of hydrometeor-containing grid cells. Such misclassifications arise because, while the radar is capable of detecting mixed-phase conditions, it can mistake water- for ice-dominated layers. However, applying the same classification algorithm to forward-simulated and observed fields should generate hydrometeor-phase statistics with similar uncertainty. Alternatively, choosing to disregard how sensors define hydrometeor phase leads to frequency of occurrence discrepancies of up to 40%. So, while hydrometeor-phase maps determined in forward space are very different from model "reality" they capture the information sensors can provide and thereby enable objective model evaluation.

Lamer, K.↗

NASA NTRS2021

Assessment of the Advanced Very High Resolution Radiometer (AVHRR) for Snowfall Retrieval in High Latitudes Using CloudSat and Machine Learning

Precipitation retrieval is a challenging topic, especially in high latitudes (HL), and current precipitation products face ample challenges over these regions. This study investigates the potential of the Advanced Very High Resolution Radiometer (AVHRR) for snowfall retrieval in HL using CloudSat radar information and machine learning (ML). With all the known limitations, AVHRR observations should be considered for HL snowfall retrieval because 1) AVHRR data have been continuously collected for about four decades on multiple platforms with global coverage, and similar observations will likely continue in the future; 2) current passive microwave satellite precipitation products have several issues over snow and ice surfaces; and 3) good coincident observations between AVHRR and CloudSat are available for training ML algorithms. Using ML, snowfall rate was retrieved from AVHRR’s brightness temperature and cloud probability, as well as auxiliary information provided by numerical reanalysis. The results indicate that the ML-based retrieval algorithm is capable of detection and estimation of snowfall with comparable or better statistical scores than those obtained from the Atmospheric Infrared Sounder (AIRS) and two passive microwave sensors contributing to the Global Precipitation Measurement (GPM) mission constellation. The outcomes also suggest that AVHRR-based snowfall retrievals are spatially and temporally reasonable and can be considered as a quantitatively useful input to the merged precipitation products that require frequent sampling or long-term records.

Mohammad Reza↗

NASA NTRS2019

Advancements in the Aerosol Robotic Network (AERONET) Version 3 database – automated near-real-time quality control algorithm with improved cloud screening for Sun photometer aerosol optical depth (AOD) measurements

The Aerosol Robotic Network (AERONET) has provided highly accurate, ground-truth measurements of the aerosol optical depth (AOD) using Cimel Electronique Sun–sky radiometers for more than 25 years. In Version 2 (V2) of the AERONET database, the near-real-time AOD was semiautomatically quality controlled utilizing mainly cloud-screening methodology, while additional AOD data contaminated by clouds or affected by instrument anomalies were removed manually before attaining quality-assured status (Level 2.0). The large growth in the number of AERONET sites over the past 25 years resulted in significant burden to the manual quality control of millions of measurements in a consistent manner. The AERONET Version 3 (V3) algorithm provides fully automatic cloud screening and instrument anomaly quality controls. All of these new algorithm updates apply to near-real-time data as well as post-field-deployment processed data, and AERONET reprocessed the database in 2018. A full algorithm redevelopment provided the opportunity to improve data inputs and corrections such as unique filter-specific temperature characterizations for all visible and near-infrared wavelengths, updated gaseous and water vapor absorption coefficients, and ancillary data sets. The Level 2.0 AOD quality-assured data set is now available within a month after post-field calibration, reducing the lag time from up to several months. Near-real-time estimated uncertainty is determined using data qualified as V3 Level 2.0 AOD and considering the difference between the AOD computed with the pre-field calibration and AOD computed with pre-field and post-field calibration. This assessment provides a near-real-time uncertainty estimate for which average differences of AOD suggest a +0.02 bias and one sigma uncertainty of 0.02, spectrally, but the bias and uncertainty can be significantly larger for specific instrument deployments. Long-term monthly averages analyzed for the entire V3 and V2 databases produced average differences (V3–V2) of +0.002 with a ±0.02 SD (standard deviation), yet monthly averages calculated using time-matched observations in both databases were analyzed to compute an average difference of −0.002 with a ±0.004 SD. The high statistical agreement in multiyear monthly averaged AOD validates the advanced automatic data quality control algorithms and suggests that migrating research to the V3 database will corroborate most V2 research conclusions and likely lead to more accurate results in some cases.

David M. Giles↗

NASA NTRS2011

Calibrated Noise Measurements with Induced Receiver Gain Fluctuations

The lack of well-developed techniques for modeling changing statistical moments in our observations has stymied the application of stochastic process theory in science and engineering. These limitations were encountered when modeling the performance of radiometer calibration architectures and algorithms in the presence of non stationary receiver fluctuations. Analyses of measured signals have traditionally been limited to a single measurement series. Whereas in a radiometer that samples a set of noise references, the data collection can be treated as an ensemble set of measurements of the receiver state. Noise Assisted Data Analysis is a growing field of study with significant potential for aiding the understanding and modeling of non stationary processes. Typically, NADA entails adding noise to a signal to produce an ensemble set on which statistical analysis is performed. Alternatively as in radiometric measurements, mixing a signal with calibrated noise provides, through the calibration process, the means to detect deviations from the stationary assumption and thereby a measurement tool to characterize the signal's non stationary properties. Data sets comprised of calibrated noise measurements have been limited to those collected with naturally occurring fluctuations in the radiometer receiver. To examine the application of NADA using calibrated noise, a Receiver Gain Modulation Circuit (RGMC) was designed and built to modulate the gain of a radiometer receiver using an external signal. In 2010, an RGMC was installed and operated at the National Institute of Standards and Techniques (NIST) using their Noise Figure Radiometer (NFRad) and national standard noise references. The data collected is the first known set of calibrated noise measurements from a receiver with an externally modulated gain. As an initial step, sinusoidal and step-function signals were used to modulate the receiver gain, to evaluate the circuit characteristics and to study the performance of a variety of calibration algorithms. The receiver noise temperature and time-bandwidth product of the NFRad are calculated from the data. Statistical analysis using temporal-dependent calibration algorithms reveals that the natural occurring fluctuations in the receiver are stationary over long intervals (100s of seconds); however the receiver exhibits local non stationarity over the interval over which one set of reference measurements are collected. A variety of calibration algorithms have been applied to the data to assess algorithms' performance with the gain fluctuation signals. This presentation will describe the RGMC, experiment design and a comparative analysis of calibration algorithms.

Racette, Paul↗

NASA NTRS1996

Automated clustering-based workload characterization

The demands placed on the mass storage systems at various federal agencies and national laboratories are continuously increasing in intensity. This forces system managers to constantly monitor the system, evaluate the demand placed on it, and tune it appropriately using either heuristics based on experience or analytic models. Performance models require an accurate workload characterization. This can be a laborious and time consuming process. It became evident from our experience that a tool is necessary to automate the workload characterization process. This paper presents the design and discusses the implementation of a tool for workload characterization of mass storage systems. The main features of the tool discussed here are: (1)Automatic support for peak-period determination. Histograms of system activity are generated and presented to the user for peak-period determination; (2) Automatic clustering analysis. The data collected from the mass storage system logs is clustered using clustering algorithms and tightness measures to limit the number of generated clusters; (3) Reporting of varied file statistics. The tool computes several statistics on file sizes such as average, standard deviation, minimum, maximum, frequency, as well as average transfer time. These statistics are given on a per cluster basis; (4) Portability. The tool can easily be used to characterize the workload in mass storage systems of different vendors. The user needs to specify through a simple log description language how the a specific log should be interpreted. The rest of this paper is organized as follows. Section two presents basic concepts in workload characterization as they apply to mass storage systems. Section three describes clustering algorithms and tightness measures. The following section presents the architecture of the tool. Section five presents some results of workload characterization using the tool.Finally, section six presents some concluding remarks.

Pentakalos, Odysseas I.↗

NASA NTRS2019

Application of High-Dimensional Fuzzy K-Means Cluster Analysis to CALIOP/CALIPSO Version 4.1 Cloud-Aerosol Discrimination

This study applies fuzzy k-means (FKM) cluster analyses to a subset of the parameters reported in the CALIPSO lidar level 2 data products in order to classify the layers detected as either clouds or aerosols. The results obtained are used to assess the reliability of the cloud–aerosol discrimination (CAD) scores reported in the version 4.1 release of the CALIPSO data products. FKM is an unsupervised learning algorithm, whereas the CALIPSO operational CAD algorithm (COCA) takes a highly supervised approach. Despite these substantial computational and architectural differences, our statistical analyses show that the FKM classifications agree with the COCA classifications for more than 94 % of the cases in the troposphere. This high degree of similarity is achieved because the lidar-measured signatures of the majority of the clouds and the aerosols are naturally distinct, and hence objective methods can independently and effectively separate the two classes in most cases. Classification differences most often occur in complex scenes (e.g., evaporating water cloud filaments embedded in dense aerosol) or when observing diffuse features that occur only intermittently (e.g., volcanic ash in the tropical tropopause layer). The two methods examined in this study establish overall classification correctness boundaries due to their differing algorithm uncertainties. In addition to comparing the outputs from the two algorithms, analysis of sampling, data training, performance measurements, fuzzy linear discriminants, defuzzification, error propagation, and key parameters in feature type discrimination with the FKM method are further discussed in order to better understand the utility and limits of the application of clustering algorithms to space lidar measurements. In general, we find that both FKM and COCA classification uncertainties are only minimally affected by noise in the CALIPSO measurements, though both algorithms can be challenged by especially complex scenes containing mixtures of discrete layer types. Our analysis results show that attenuated backscatter and color ratio are the driving factors that separate water clouds from aerosols; backscatter intensity, depolarization, and mid-layer altitude are most useful in discriminating between aerosols and ice clouds; and the joint distribution of backscatter intensity and depolarization ratio is critically important for distinguishing ice clouds from water clouds.

Zeng, Shan↗

NASA NTRS2005

Development and Testing of Data Mining Algorithms for Earth Observation

The new algorithms developed under this project included a principled procedure for classification of objects, events or circumstances according to a target variable when a very large number of potential predictor variables is available but the number of cases that can be used for training a classifier is relatively small. These "high dimensional" problems require finding a minimal set of variables -called the Markov Blanket-- sufficient for predicting the value of the target variable. An algorithm, the Markov Blanket Fan Search, was developed, implemented and tested on both simulated and real data in conjunction with a graphical model classifier, which was also implemented. Another algorithm developed and implemented in TETRAD IV for time series elaborated on work by C. Granger and N. Swanson, which in turn exploited some of our earlier work. The algorithms in question learn a linear time series model from data. Given such a time series, the simultaneous residual covariances, after factoring out time dependencies, may provide information about causal processes that occur more rapidly than the time series representation allow, so called simultaneous or contemporaneous causal processes. Working with A. Monetta, a graduate student from Italy, we produced the correct statistics for estimating the contemporaneous causal structure from time series data using the TETRAD IV suite of algorithms. Two economists, David Bessler and Kevin Hoover, have independently published applications using TETRAD style algorithms to the same purpose. These implementations and algorithmic developments were separately used in two kinds of studies of climate data: Short time series of geographically proximate climate variables predicting agricultural effects in California, and longer duration climate measurements of temperature teleconnections.

Glymour, Clark↗

NASA NTRSDate not supplied

Using Machine Learning to Infer Material Properties of Debris Fragments from X-ray Images in the DebriSat Project

The DebriSat project is a collaboration effort with the NASA Orbital Debris Program Office, the U.S. Space Force Space Systems Command Center, The Aerospace Corporation, and the University of Florida. To date, over 200,000 fragments from this ground-based, hypervelocity impact experiment have been collected, and processing is underway to determine their physical characteristics, such as material, shape, color, characteristic length, and average cross-sectional area. The x-ray process is primarily used to identify the location of the fragments and estimated size for extraction, so that these physical characteristics can be assessed. This paper proposes a machine learning-based approach to characterize materials from x-ray images of debris fragments embedded in soft-catch foam used in the DebriSat project. The novel methodology discussed in this paper will highlight the use of x-ray imagery data to characterize these fragments without extraction or a human-in-the-loop. Both supervised and unsupervised machine learning techniques are utilized with this approach to infer the physical parameters of the fragments embedded in the soft-catch foam panels used in the impact experiment based on x-ray images of the foam panels. Additionally, 3D reconstructions of the extracted fragments are created with images taken from two different angles using the structure from motion (SfM) method. The characteristic lengths and shape from the 3D reconstruction, alongside the physical characteristics of the debris, are used in the inference of the material type. To develop and test the approach, a dataset of x-ray images of debris fragments of varying sizes and materials is collected. Supervised learning methods such as convolutional neural networks (CNNs), support vector machines (SVM), decision trees, and random forest classifiers are used due to the high-dimensional feature spaces of the debris and nonlinear decision boundaries for material categorization. Given the limited pre-labeled data of embedded debris materials smaller than 10 mm, unsupervised machine learning techniques such as clustering algorithms and autoencoders are used, in addition to supervised learning methods. The clustering algorithms group similar fragments together based on their physical properties, and autoencoders reduce the dimensionality of the x ray images and extract relevant features. The performance of the proposed approach's is analyzed using a range of statistical methods, including confusion matrices, receiver operating characteristic curves, and precision-recall curves. The results are compared with those obtained using a baseline approach that relies on manual identification and classification of debris fragments. To evaluate the effectiveness of different machine learning methods, statistical tests such as t-tests, ANOVA, and cross-validation are performed, comparing the performance of CNNs, SVMs, clustering algorithms, and autoencoders. Additional analysis needs to be conducted to identify any sources of bias or variability that may affect the results, such as variations in imaging conditions or fragmentation patterns. Other topics explored are limitations, refinements, and the potential use of semi-supervised learning techniques, such as self-training to label unlabeled datasets and co-training using x-ray images taken from two different angles as two different models.

Saik Anam Siam↗

NASA NTRS2023

Using Machine Learning to Infer Material Properties of Debris Fragments from X-ray Images in the DebriSat Project

Saik Anam Siam↗

NASA NTRS2005

An Ensemble Approach to Building Mercer Kernels with Prior Information

This paper presents a new methodology for automatic knowledge driven data mining based on the theory of Mercer Kernels, which are highly nonlinear symmetric positive definite mappings from the original image space to a very high, possibly dimensional feature space. we describe a new method called Mixture Density Mercer Kernels to learn kernel function directly from data, rather than using pre-defined kernels. These data adaptive kernels can encode prior knowledge in the kernel using a Bayesian formulation, thus allowing for physical information to be encoded in the model. Specifically, we demonstrate the use of the algorithm in situations with extremely small samples of data. We compare the results with existing algorithms on data from the Sloan Digital Sky Survey (SDSS) and demonstrate the method's superior performance against standard methods. The code for these experiments has been generated with the AUTOBAYES tool, which automatically generates efficient and documented C/C++ code from abstract statistical model specifications. The core of the system is a schema library which contains templates for learning and knowledge discovery algorithms like different versions of EM, or numeric optimization methods like conjugate gradient methods. The template instantiation is supported by symbolic-algebraic computations, which allows AUTOBAYES to find closed-form solutions and, where possible, to integrate them into the code.

Srivastava, Ashok N.↗

NASA NTRS1979

Compensation for atmospheric effects in LANDSAT data

Preprocessing algorithms were developed to remove or reduce the variations in multispectral data caused by variations in Sun angle and by changes in the atmospheric aerosol and water vapor levels. The two most significant algorithms developed by using mathematical models to define interrelations between the required multiplicative and additive correction factors so that just a few statistical characteristics of a LANDSAT distribution model would be sufficient to drive the mathematical model and to calculate the preprocessing corrections are examined. These are the atmospheric correction (ATCOR) computer program and the XSTAR haze correction algorithm. Neither the ATCOR nor the XSTAR algorithm provides an explicit compensation for the effects of changing LANDSAT view angle. Development efforts are underway to address this aspect of the preprocessing problem.

Lambeck, P. F.↗

NASA NTRS2015

Automatic Event Detection in Search for Inter-Moss Loops in IRIS Si IV Slit-Jaw Images

The high-resolution capabilities of the Interface Region Imaging Spectrometer (IRIS) mission have allowed the exploration of the finer details of the solar magnetic structure from the chromosphere to the lower corona that have previously been unresolved. Of particular interest to us are the relatively short-lived, low-lying magnetic loops that have foot points in neighboring moss regions. These inter-moss loops have also appeared in several AIA pass bands, which are generally associated with temperatures that are at least an order of magnitude higher than that of the Si IV emission seen in the 1400 angstrom pass band of IRIS. While the emission lines seen in these pass bands can be associated with a range of temperatures, the simultaneous appearance of these loops in IRIS 1400 and AIA 171, 193, and 211 suggest that they are not in ionization equilibrium. To study these structures in detail, we have developed a series of algorithms to automatically detect signal brightening or events on a pixel-by-pixel basis and group them together as structures for each of the above data sets. These algorithms have successfully picked out all activity fitting certain adjustable criteria. The resulting groups of events are then statistically analyzed to determine which characteristics can be used to distinguish the inter-moss loops from all other structures. While a few characteristic histograms reveal that manually selected inter-moss loops lie outside the norm, a combination of several characteristics will need to be used to determine the statistical likelihood that a group of events be categorized automatically as a loop of interest. The goal of this project is to be able to automatically pick out inter-moss loops from an entire data set and calculate the characteristics that have previously been determined manually, such as length, intensity, and lifetime. We will discuss the algorithms, preliminary results, and current progress of automatic characterization.

loops↗

NASA NTRS1991

Small convolution kernels for high-fidelity image restoration

An algorithm is developed for computing the mean-square-optimal values for small, image-restoration kernels. The algorithm is based on a comprehensive, end-to-end imaging system model that accounts for the important components of the imaging process: the statistics of the scene, the point-spread function of the image-gathering device, sampling effects, noise, and display reconstruction. Subject to constraints on the spatial support of the kernel, the algorithm generates the kernel values that restore the image with maximum fidelity, that is, the kernel minimizes the expected mean-square restoration error. The algorithm is consistent with the derivation of the spatially unconstrained Wiener filter, but leads to a small, spatially constrained kernel that, unlike the unconstrained filter, can be efficiently implemented by convolution. Simulation experiments demonstrate that for a wide range of imaging systems these small kernels can restore images with fidelity comparable to images restored with the unconstrained Wiener filter.

Reichenbach, Stephen E.↗

NASA NTRS1979

Radiometric correction and equalization of satellite digital data

Satellite digital data from Landsat and NOAA satellites is often marred by striping or streaking errors due to variations in the response of the radiometric sensors. In this paper, we discuss the equalization of the digital data as a preprocessing step, prior to image enhancement or automatic classification. The methods described make use of statistics of the data itself to generate nonlinear or linear memory-less equalization algorithms. These algorithms, by contrast to multidimensional filtering, do not result in a loss of spatial resolution. Examples of applications to Landsat and NOAA-3 thermal infrared data are given and illustrated.

Algazi, V. R.↗

NASA NTRS1987

Optimal mapping of irregular finite element domains to parallel processors

Mapping the solution domain of n-finite elements into N-subdomains that may be processed in parallel by N-processors is an optimal one if the subdomain decomposition results in a well-balanced workload distribution among the processors. The problem is discussed in the context of irregular finite element domains as an important aspect of the efficient utilization of the capabilities of emerging multiprocessor computers. Finding the optimal mapping is an intractable combinatorial optimization problem, for which a satisfactory approximate solution is obtained here by analogy to a method used in statistical mechanics for simulating the annealing process in solids. The simulated annealing analogy and algorithm are described, and numerical results are given for mapping an irregular two-dimensional finite element domain containing a singularity onto the Hypercube computer.

Flower, J.↗

NASA NTRS1990

Adaptive state estimation for control of flexible structures

This paper proposes a new approach of obtaining adaptive state estimation of a system in the presence of unknown system disturbances and measurement noise. In the beginning, a non-optimal Kalman filter with arbitrary initial guess for the process and measurement noises is implemented. At the same time, an adaptive transversal predictor (ATP) based on the recursive least-squares (RLS) algorithm is used to yield optimal one- to p- step-ahead output predictions using the previous input/output data. Referring to these optimal predictions the Kalman filter gain is updated and the performance of the state estimation is thus improved. If forgetting factor is implemented in the recursive least-squares algorithm, this method is also capable of dealing with the situation when the noise statistics are slowly time-varying. This feature makes this new approach especially suitable for the control of flexible structures. A numerical example demonstrates the feasibility of this real time adaptive state estimation method.

Chen, Chung-Wen↗