Search NASA⌕ Search

SEARCH · Search NASA

Results for “Sparse data analytics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

An Automated Scanning Transmission Electron Microscope Guided by Sparse Data Analytics

Abstract Artificial intelligence (AI) promises to reshape scientific inquiry and enable breakthrough discoveries in areas such as energy storage, quantum computing, and biomedicine. Scanning transmission electron microscopy (STEM), a cornerstone of the study of chemical and materials systems, stands to benefit greatly from AI-driven automation. However, present barriers to low-level instrument control, as well as generalizable and interpretable feature detection, make truly automated microscopy impractical. Here, we discuss the design of a closed-loop instrument control platform guided by emerging sparse data analytics. We hypothesize that a centralized controller, informed by machine learning combining limited a priori knowledge and task-based discrimination, could drive on-the-fly experimental decision-making. This platform may unlock practical, automated analysis of a variety of material features, enabling new high-throughput and statistical studies.

47 OTHER INSTRUMENTATION↗
DOE OSTI2022

Design of a graphical user interface for few-shot machine learning classification of electron microscopy data

The recent growth in data generation by modern electron microscopes requires rapid, scalable, and flexible approaches to image segmentation and analysis. Few-shot machine learning, which can richly classify images from a handful of user-provided examples, is a promising route to high-throughput analysis. However, current command-line implementations of such approaches can be slow and unintuitive to use, lacking the real-time feedback necessary to perform effective classification. Here we report on the development of a Python-based graphical user interface that enables end users to easily conduct and visualize the output of few-shot learning models. This interface is portable and can be hosted locally or on the web, providing the opportunity to reproducibly conduct, share, and crowd-source few-shot analyses.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

A High Performance Sparse Tensor Algebra Compiler in MLIR

Sparse tensor algebra is widely used in many applications, including scientific computing, machine learning, and data analytics. The performance of sparse tensor algebra kernels strongly depends on the intrinsic characteristics of the input tensors, hence many storage formats are designed for tensors to achieve optimal performance for particular applications/architectures, which makes it challenging to implement and optimize every tensor operation of interest on a given architecture. We propose a tensor algebra domain-specific language (DSL) and compiler framework to automatically generate kernels for mixed sparse-dense tensor algebra operations. The proposed DSL provides high-level programming abstractions that resemble the familiar Einstein notation to represent tensor algebra operations. The compiler introduces a new Sparse Tensor Algebra dialect built on top of LLVM's extensible MLIR compiler infrastructure for efficient code generation while covering a wide range of tensor storage formats. Our compiler also leverages input-dependent code optimization to enhance data locality for better performance. Our results show that the performance of automatically generated kernels outperforms the state-of-the-art sparse tensor algebra compiler, with up to 20.92x, 6.39x, and 13.9x performance improvement over state-of-the-art tensor algebra compilers, for parallel SpMV, SpMM, and TTM, respectively.

Tian, Ruiqin↗
DOE OSTI2023

Encoding nonlinear and unsteady aerodynamics of limit cycle oscillations using nonlinear sparse Bayesian learning

This article investigates the applicability of a recently proposed, nonlinear sparse Bayesian learning (NSBL) algorithm to identify and estimate the complex aerodynamics of limit cycle oscillations. NSBL provides a semi-analytical framework for determining the data-optimal sparse model nested within a (potentially) over-parameterized model. This is particularly relevant to nonlinear dynamical systems where modelling approaches involve the use of physics-based and data-driven components. In such cases, the data-driven components, where analytical descriptions of the physical processes are not readily available, are often prone to overfitting, meaning that the empirical aspects of these models will often involve the calibration of an unnecessarily large number of parameters. While an overparameterized model may fit the observed data well, such models may be inadequate for making predictions in regimes that are different from those wherein the data were recorded. In view of this, it is desirable to not only calibrate the model parameters, but also identify the optimal compromise between data fit and model complexity. In this article, we exhibit the optimal model discovery for an aeroelastic system wherein the structural dynamics are well-known and described by a differential equation model, coupled with a semi-empirical aerodynamic model for laminar separation flutter, resulting in low-amplitude limit cycle oscillations (LCO). To illustrate the performance of the algorithm, in this article, we use synthetic data and demonstrate the ability of the algorithm to correctly rediscover the optimal model and model parameters, given a known data-generating model. The synthetic data are generated from a forward simulation of a known differential equation model with parameters selected so as to mimic the dynamics observed in wind-tunnel experiments. Subsequently, we demonstrate the performance of the algorithm for model selection using noisy LCO data from wind tunnel experiments. As there is no ground truth available for the experimental data case, we provide a comparison between NSBL and Bayesian model selection to validate the results, and demonstrate the use of NSBL as an efficient alternative to traditional methods.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Exploration of Domain Aware Machine Learning for Grid Analytics: Transfer-Learnt Energy Models to Assist Buildings Control with Sparse Field Data

Buildings are a primary consumer of energy in the United States and are also increasingly being perceived as providers of grid services such as load shifting, shedding and modulation. High fidelity models of building energy consumption are needed to set appropriate baselines for measurement and verification (M&V) of controllers designed for energy efficient operation of buildings and to enable buildings to provide grid services via. participation in demand response programs. State-of-the-art building energy modeling techniques either rely on Physics based models, or extensive instrumentation of the building envelope to gather “big” data to train machine learning based models such as deep neural networks. While Physics based models are often limited by their accuracy, it is not always feasible to gather a significant amount of field data required to train machine learning based models with sufficient accuracy. In this paper, we explore the use of transfer learning-based strategies to address unsatisfactory accuracy of models for estimating building energy consumption when available field data for training is sparse or of unacceptable quality. In particular, we transfer knowledge in the form of data and parameters, from Physics based simulation frameworks to the field to improve the model accuracy, thus resulting in a Physics-informed Machine Learning framework. We evaluated the efficacy of our approach on field data collected from six commercial buildings and our results indicate that the proposed transfer learning based models provide comparative (and in some cases better) accuracy than state-of-the-art machine learning and deep learning solutions, with just one month of field data.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2021

A-SST Initial Specification

The U.S. Army Research Office (ARO), in partnership with IARPA, are investigating innovative, efficient, and scalable computer architectures that are capable of executing next-generation large scale data-analytic applications. These applications are increasingly sparse, unstructured, non-local, and heterogeneous. Under the Advanced Graphic Intelligence Logical computing Environment (AGILE) program, Performer teams will be asked to design computer architectures to meet the future needs of the DoD and the Intelligence Community (IC). This design effort will require flexible, scalable, and detailed simulation to assess the performance, efficiency, and validity of their designs. To support AGILE, Sandia National Labs will be providing the AGILE-enhanced Structural Simulation Toolkit (A-SST). This toolkit is a computer architecture simulation framework designed to support fast, parallel, and multi-scale simulation of novel architectures. This document describes the A-SST framework, some of its library of simulation models, and how it may be used by AGILE Performers.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1985

Iterative solution of large, sparse linear systems on a static data flow architecture - Performance studies

The applicability of static data flow architectures to the iterative solution of sparse linear systems of equations is investigated. An analytic performance model of a static data flow computation is developed. This model includes both spatial parallelism, concurrent execution in multiple PE's, and pipelining, the streaming of data from array memories through the PE's. The performance model is used to analyze a row partitioned iterative algorithm for solving sparse linear systems of algebraic equations. Based on this analysis, design parameters for the static data flow architecture as a function of matrix sparsity and dimension are proposed.

Reed, D. A.↗
DOE OSTI2026

Stoichiometrically-informed symbolic regression for extracting chemical reaction mechanisms from data

A data-driven computational method is introduced to extract chemical reaction mechanisms from time series chemical concentration data. It is realized through the use of dynamic symbolic regression in which a sparse analytical form for a dynamical system is discoverable from the underlying data. We specifically develop the stoichiometrically-informed symbolic regression (SISR) method to address a standing challenge in complex chemical reaction networks: given a time-series dataset of concentrations of several components, what is the mechanism and the associated rate constants? SISR finds the optimal mechanism, kinetic equations and rate constants by combining differential optimization with a genetic optimization approach that searches a symbolic space of possible reaction mechanisms. Use of SISR in several paradigmatic examples spanning linear and nonlinear reaction schemes results in excellent agreement between true and predicted mechanisms, including when the method is applied to noisy data. The advantages of a stoichiometrically-informed approach such as SISR to address reaction discovery is illustrated through comparison with the use of generic state-of-the-art data-driven approaches.

36 MATERIALS SCIENCE↗
DOE OSTI2020

Evaluation of Graph Analytics Frameworks Using the GAP Benchmark Suite

The analysis of connected data is an increasingly important application in high-performance computing. Such analyses can reveal fraudulent patterns in financial transactions, optimize telecommunications networks, predict information flow in social networks, etc. However, the landscape of graph analytics is highly diverse. Graph algorithms stress processor architectures differently, and no one graph can represent all topologies. Consequently, no single approach or framework is expected to be optimal for all graph analytics problems. To help make sense of this diverse landscape, we evaluated four approaches to graph analytics: GraphBLAS, Galois, BGL17, GraphIt; and compare them against hand-tuned implementations that take advantage of hardware features on our test platform. Graph- BLAS formulates graph analytics as sparse linear algebra. Galois provides syntactic constructs for data parallelism over irregular data structures. BGL17 is a generic C++ template library for implementing graph algorithms. GraphIt provides a domain- specific language to describe and optimize graph algorithms. We use the GAP Benchmark Suite to establish baseline performance and guide the side-by-side evaluation of each framework. GAP consists of 30 tests: six graph analytics algorithms (breadth- first search, single-source shortest path, PageRank, betweenness centrality, connected components, and triangle counting) run on five graphs, each with different topological characteristics (e.g., high diameter, skewed degree distribution, high average degree). High-performance reference implementations are included for each benchmark algorithm. Because a graph can be loaded into memory a number of ways (e.g., flat file on disk, compressed sparse format, data frames, retrieved from SQL or NoSQL databases), our evaluation focused on computational performance rather than I/O. Our results show the relative strengths of each framework.

Graph algorithms, Benchmarking, shared-memory prog↗
DOE OSTI2025

Hyperspectral segmentation of plants in fabricated ecosystems

Hyperspectral imaging provides a powerful tool for analyzing above-ground plant characteristics in fabricated ecosystems, offering rich spectral information across diverse wavelengths. This study presents an efficient workflow for hyperspectral data segmentation and subsequent data analytics, minimizing the need for user annotation through the use of ensembles of sparse mixed scale convolution neural networks. The segmentation process leverages the diversity of ensembles to achieve high accuracy with minimal labeled data, reducing labor-intensive annotation efforts. To further enhance robustness, we incorporate image alignment techniques to address spatial variability in the dataset. Downstream analysis focuses on using the segmented data for processing spectral data, enabling monitoring of plant health. This approach provides a scalable solution for spectral segmentation, and facilitates actionable insights into plant conditions in complex, controlled environments. Our results demonstrate the utility of combining advanced machine learning techniques with hyperspectral analytics for high-throughput plant monitoring.

Zwart, Petrus H.↗
DOE OSTI2023

Prediction of hemiwicking dynamics in micropillar arrays

Dynamic hemiwicking behavior is observable in both nature and a wide range of industrial applications ranging from biomedical devices to thermal management. We present a semi-analytical modeling framework (without empirical fitting coefficients) to predict transient capillary-driven hemiwicking behavior of a liquid through a nano/microstructured surface, specifically a micropillar array. In our model framework, the liquid domain is discretized into micropillar unit cells to enable the time marching of the hemiwicking front. A simplified linear pressure drop is assumed along the hemiwicking length such that the local meniscus curvature, contact angle, and effective liquid height are determined at each time step in our transient model. This semi-analytical model is validated with experimental data from our own experiments and from published literature for different fluids. Our model predicts hemiwicking dynamics with <20% error over a broad range of micropillar geometries with height-to-pitch ratio ranging between ≈0.34 and 6.7 and diameter-to-pitch ratio in the range of ≈0.25–0.7 and without any fitting parameters. For lower diameter-to-pitch ratio data points related to sparse micropillar array arrangements, we suggest modifications to the semi-analytical model. This work sheds light on complex and dynamic solid–liquid–vapor interfacial interactions which could serve as a guide for the design of textured surfaces for wicking enhancement in multi-phase thermal and mass transport technologies and applications.

Mechanics↗
DOE OSTI2020

Frequency Event Categorization in Power Distribution Systems Using Micro PMU Measurements

Frequency is an important metric for power stability. To enhance the situational awareness in power distribution systems, this article proposes a data-driven approach that utilizes the data from micro phasor measurement units (μPMU) to categorize frequency events observed in distribution feeders. The proposed approach has three layers, each of which analyzes if a frequency event is propagated from the transmission system, formed inside a distribution feeder or falsely produced by instantaneous frequency (IF) estimation algorithms, respectively. In the first layer, the Granger causality is utilized to compare the causal relations from measurements across different locations to decide if an event is propagated from the transmission side. In the second layer, a trained sparse coding dictionary is utilized to unmix the physical causes of a frequency event. In the third layer, a moving z-score based event-pair detector is applied to identify if a frequency event is algorithm-induced. The proposed approach is tested using the μPMU measurements collected from Lawrence Berkeley National Laboratory (LBNL) and Riverside Public Utility (RPU).

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2023

An Edge Alignment-Based Orientation Selection Method for Neutron Tomography

Neutron computed tomography (nCT) is a 3D char-acterization technique used to image the internal morphology or chemical composition of samples in biology and materials sciences. A typical workflow involves placing the sample in the path of a neutron beam, acquiring projection data at a predefined set of orientations, and processing the resulting data using an analytic reconstruction algorithm. Typical nCT scans require hours to days to complete and are then processed using conventional filtered back-projection (FBP), which performs poorly with sparse views or noisy data. Hence, the main methods in order to reduce overall acquisition time are the use of an improved sampling strategy combined with the use of advanced reconstruction methods such as model-based iterative reconstruction (MBIR). In this paper, we propose an adaptive orientation selection method in which an MBIR reconstruction on previously-acquired measurements is used to define an objective function on orientations that balances a data-fitting term promoting edge alignment and a regularization term promoting orientation diversity. Using simulated and experimental data, we demonstrate that our method produces high-quality reconstructions using significantly fewer total measurements than the conventional approach.

Yang, Diyu↗
DOE OSTI2020

A Sparse Tensor Benchmark Suite for CPUs and GPUs

Tensor computations present significant performance chal- lenges that impact a wide spectrum of applications ranging from machine learning, healthcare analytics, social network analysis, data mining to quantum chemistry and signal processing. Efforts to improve the perfor- mance of tensor computations include exploring data layout, execution scheduling, and parallelism in common tensor kernels. This work presents a benchmark suite for arbitrary-order sparse tensor kernels using state- of-the-art tensor formats: coordinate (COO) and hierarchical coordinate (HiCOO) on CPUs and GPUs. It presents a set of reference tensor kernel implementations that are compatible with real-world tensors and power law tensors extended from synthetic graph generation techniques. We also propose Roofline performance models for these kernels to provide insights of computer platforms from sparse tensor view. This benchmark suite along with the synthetic tensor generator is publicly available.

Li, Jiajia↗
DOE OSTI2020

Imagery Applications for Advanced Event Analytics (WBS 24.3.1.3.3-IDC FY202 Final Project Report)

Accurate event locations and replicability of location analyses are essential for assessing the nature of an event, its context, ambient site conditions, and proximity to relevant facilities and infrastructure. Additionally, accurate event locations provide valuable information that reduce uncertainties, improve confidence in event analyses, and inform in-field verification activities. However, event location/relocation and replicability are difficult due to a number of factors, including spatially-sparse network coverage in some areas of the globe and variability in seismic data processing. This team proposed that the incorporation of high-fidelity imagery as a data backbone to the analytical assessment of a suspected underground explosion and/or an advanced seismic event bulletin produced by the International Data Centre (IDC) of the Preparatory Commission for the Comprehensive Nuclear-Test-Ban Treaty Organization (CTBTO PrepCom) could reduce uncertainties and improve confidence in analyses. Specifically, temporally-separated images can reduce uncertainty by identifying areas where change has occurred (e.g., building construction or demolition, road or facilities improvements). The primary goal of this project was to develop an automated geospatial processing script for imagery change detection to better reflect needs of the technical community (including the IDC) and to make the use of such a tool accessible in a variety of settings across platforms. Technical experts at Los Alamos National Laboratory successfully built GAIA: the Geospatial Automated Imagery Analysis tool, to fill this need. GAIA combines five tool components to produce orthorectified time separated imagery and imagery change detection maps. Our toolkit (1) reduces error by providing a standardized workflow for image analyses and (2) significantly reduces processing time from between 7 and 24+ hours to approximately 5 minutes. Technical experts at Sandia National Laboratories supported GAIA via beta-testing and by introducing a web-based system approach for increased applicability. To test the function, performance, broad application, and ease-of-use of GAIA, we applied it to four separate test cases. The results of this preliminary investigation show promise in reducing uncertainty in seismic event locations: if satellite imagery can show regions where operations that produce seismic activity likely occurred, then pursuing imagery to locate epicenters of seismic nuclear events could reduce the time needed to find the true epicenter location.

58 GEOSCIENCES↗
DOE OSTI2020

Imagery Applications for Advanced Event Analytics (WBS 24.3.1.3.3-IDC FY2020 Final Project Report)

Accurate event locations and replicability of location analyses are essential for assessing the nature of an event, its context, ambient site conditions, and proximity to relevant facilities and infrastructure. Additionally, accurate event locations provide valuable information that reduce uncertainties, improve confidence in event analyses, and inform in-field verification activities. However, event location/relocation and replicability are difficult due to a number of factors, including spatially-sparse network coverage in some areas of the globe and variability in seismic data processing. This team proposed that the incorporation of high-fidelity imagery as a data backbone to the analytical assessment of a suspected underground explosion and/or an advanced seismic event bulletin produced by the International Data Centre (IDC) of the Preparatory Commission for the Comprehensive Nuclear-Test-Ban Treaty Organization (CTBTO PrepCom) could reduce uncertainties and improve confidence in analyses. Specifically, temporally-separated images can reduce uncertainty by identifying areas where change has occurred (e.g., building construction or demolition, road or facilities improvements). The primary goal of this project was to develop an automated geospatial processing script for imagery change detection to better reflect needs of the technical community (including the IDC) and to make the use of such a tool accessible in a variety of settings across platforms. Technical experts at Los Alamos National Laboratory successfully built GAIA: the Geospatial Automated Imagery Analysis tool, to fill this need. GAIA combines five tool components to produce orthorectified time-separated imagery and imagery change detection maps. Our toolkit (1) reduces error by providing a standardized workflow for image analyses and (2) significantly reduces processing time from between 7 and 24+ hours to approximately 5 minutes. Technical experts at Sandia National Laboratories supported GAIA via beta-testing and by introducing a web-based system approach for increased applicability. To test the function, performance, broad application, and ease-of-use of GAIA, we applied it to four separate test cases. The results of this preliminary investigation show promise in reducing uncertainty in seismic event locations: if satellite imagery can show regions where operations that produce seismic activity likely occurred, then pursuing imagery to locate epicenters of seismic nuclear events could reduce the time needed to find the true epicenter location.

58 GEOSCIENCES↗
NASA NTRS2010

The OMPS Limb Profiler Instrument: Two-Dimensional Retrieval Algorithm

The upcoming Ozone Mapper and Profiler Suite (OMPS), which will be launched on the NPOESS Preparatory Project (NPP) platform in early 2011, will continue monitoring the global distribution of the Earth's middle atmosphere ozone and aerosol. OMPS is composed of three instruments, namely the Total Column Mapper (heritage: TOMS, OMI), the Nadir Profiler (heritage: SBUV) and the Limb Profiler (heritage: SOLSE/LORE, OSIRIS, SCIAMACHY, SAGE III). The ultimate goal of the mission is to better understand and quantify the rate of stratospheric ozone recovery. The focus of the paper will be on the Limb Profiler (LP) instrument. The LP instrument will measure the Earth's limb radiance (which is due to the scattering of solar photons by air molecules, aerosol and Earth surface) in the ultra-violet (UV), visible and near infrared, from 285 to 1000 nm. The LP simultaneously images the whole vertical extent of the Earth's limb through three vertical slits, each covering a vertical tangent height range of 100 km and each horizontally spaced by 250 km in the cross-track direction. Measurements are made every 19 seconds along the orbit track, which corresponds to a distance of about 150km. Several data analysis tools are presently being constructed and tested to retrieve ozone and aerosol vertical distribution from limb radiance measurements. The primary NASA algorithm is based on earlier algorithms developed for the SOLSE/LORE and SAGE III limb scatter missions. All the existing retrieval algorithms rely on a spherical symmetry assumption for the atmosphere structure. While this assumption is reasonable in most of the stratosphere, it is no longer valid in regions of prime scientific interest, such as polar vortex and UTLS regions. The paper will describe a two-dimensional retrieval algorithm whereby the ozone distribution is simultaneously retrieved vertically and horizontally for a whole orbit. The retrieval code relies on (1) a forward 2D Radiative Transfer code (to model limb radiances within a non-uniform atmosphere and evaluate 2D analytical partial derivatives) and (2) an optimal estimator inversion routine. The algorithm uses the typically sparse nature of the kernel matrices as well as fast matrix inversion techniques to allow for fast inversion of limb data with efficient memory management (as was done for MIPAS data processing). While the method has so far only been developed in the context of Single Scatter, the paper will show how the CPU intensive Multiple Scatter modeling can be implemented using parallel CPU processing. Initial results will be presented in terms of retrieved ozone profiles and code performance.

Rault, Didier F.↗
DOE OSTI2023

Deep Learning Based Workflow for Accelerated Industrial X-Ray Computed Tomography

X-ray computed tomography (XCT) is an important tool for high-resolution non-destructive characterization of additively-manufactured metal components. XCT reconstructions of metal components may have beam hardening artifacts such as cupping and streaking which makes reliable detection of flaws and defects challenging. Furthermore, traditional workflows based on using analytic reconstruction algorithms require a large number of projections for accurate characterization - leading to longer measurement times and hindering the adoption of XCT for in-line inspections. In this paper, we introduce a new workflow based on the use of two neural networks to obtain high-quality accelerated reconstructions from sparse-view XCT scans of single material metal parts. The first network, implemented using fully-connected layers, helps reduce the impact of BH in the projection data without the need of any calibration or knowledge of the component material. The second network, a convolutional neural network, maps a low-quality analytic 3D reconstruction to a high-quality reconstruction. Using experimental data, we demonstrate that our method robustly generalizes across several alloys, and for a range of sparsity levels without any need for retraining the networks thereby enabling accurate and fast industrial XCT inspections.

Rahman, Obaid↗