Search NASA⌕ Search

SEARCH · Search NASA

Results for “online data analysis and reduction”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Online data analysis and reduction: An important co-design motif for extreme-scale computers

A growing disparity between supercomputer computation speeds and I/O rates means that it is rapidly becoming infeasible to analyze supercomputer application output only after that output has been written to a file system. Instead, data-generating applications must run concurrently with data reduction and/or analysis operations, with which they exchange information via high-speed methods such as interprocess communications. The resulting parallel computing motif, online data analysis and reduction (ODAR), has important implications for both application and HPC systems design. Here we introduce the ODAR motif and its co-design concerns, describe a co-design process for identifying and addressing those concerns, present tools that assist in the co-design process, and present case studies to illustrate the use of the process and tools in practical settings.

Data Analysis↗
DOE OSTI2021

A Co-design Framework for Online Data Analysis and Reduction

Science applications preparing for the exascale era are increasingly exploring in situ computations comprising of simulation-analysis-reduction pipelines coupled in-memory. Efficient composition and execution of such complex pipelines for a target platform is a codesign process that evaluates the impact and tradeoffs of various application- and system-specific parameters. In this article, we describe a toolset for automating performance studies of composed HPC applications that perform online data reduction and analysis. We describe Cheetah, a new framework for composing parametric studies on coupled applications, and Savanna, a runtime engine for orchestrating and executing campaigns of codesign experiments. Furthermore, this toolset facilitates understanding the impact of various factors such as process placement, synchronicity of algorithms, and storage versus compute requirements for online analysis of large data. Ultimately, we aim to create a catalog of performance results that can help scientists understand tradeoffs when designing next-generation simulations that make use of online processing techniques. We illustrate the design of Cheetah and Savanna, and present application examples that use this framework to conduct codesign studies on small clusters as well as leadership class supercomputers.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Performance analysis and data reduction for exascale scientific workflows

Chimbuko is the first in situ, scalable, workflow-level performance analysis tool for trace-level analysis and visualization of application performance. This tool was developed by the Co-design Center for Online Data Analysis and Reduction and funded by the U.S. Department of Energy’s Exascale Computing Project. We provide a detailed description of Chimbuko’s architecture and illustrate our online and offline visualization with multiple use cases. We also present results for the deployment and scalability of the tool as applied to a high-energy physics workflow running at large scale on the Frontier supercomputer.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

An integrated manifold learning approach for high-dimensional data feature extractions and its applications to online process monitoring of additive manufacturing

As an effective dimension reduction and feature extraction technique, manifold learning has been successfully applied to high-dimensional data analysis. With the rapid development of sensor technology, a large amount of high-dimensional data such as image streams can be easily available. Thus, a promising application of manifold learning is in the field of sensor signal analysis, particular for the applications of online process monitoring and control using high-dimensional data. The objective of this study is to develop a manifold learning-based feature extraction method for process monitoring of Additive Manufacturing (AM) using online sensor data. Due to the non-parametric nature of most existing manifold learning methods, their performance in terms of computational efficiency, as well as noise resistance has yet to be improved. To address this issue, this study proposes an integrated manifold learning approach termed multi-kernel metric learning embedded isometric feature mapping (MKML-ISOMAP) for dimension reduction and feature extraction of online high-dimensional sensor data such as images. Based on the extracted features with the utilization of supervised classification and regression methods, an online process monitoring methodology for AM is implemented to identify the actual process quality status. Finally, in the numerical simulation and real-world case studies, the proposed method demonstrates excellent performance in both prediction accuracy and computational efficiency.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Online learning of quadratic manifolds from streaming data for nonlinear dimensionality reduction and nonlinear model reduction

Here, this work introduces an online greedy method for constructing quadratic manifolds from streaming data, designed to enable in situ analysis of numerical simulation data on the Petabyte scale. Unlike traditional batch methods, which require all data to be available upfront and take multiple passes over the data, the proposed online greedy method incrementally updates quadratic manifolds in one pass as data points are received, eliminating the need for expensive disk input/output operations as well as storing and loading data points once they have been processed. A range of numerical examples demonstrate that the online greedy method learns accurate quadratic manifold embeddings while being capable of processing data that far exceed common disk input/output capabilities and volumes as well as main-memory sizes.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Data reduction through optimized scalar quantization for more compact neural networks

Raw data generation for several existing and planned large physics experiments now exceeds TB/s rates, generating untenable data sets in very little time. Those data often demonstrate high dimensionality while containing limited information. Meanwhile, Machine Learning algorithms are now becoming an essential part of data processing and data analysis. Those algorithms can be used offline for post processing and post data analysis, or they can be used online for real time processing providing ultra low latency experiment monitoring. Both use cases would benefit from data throughput reduction while preserving relevant information: one by reducing the offline storage requirements by several orders of magnitude and the other by allowing ultra fast online inferencing with low complexity Machine Learning models. Moreover, reducing the data source throughput also reduces material cost, power and data management requirements. In this work we demonstrate optimized nonuniform scalar quantization for data source reduction. This data reduction allows lower dimensional representations while preserving the relevant information of the data, thus enabling high accuracy Tiny Machine Learning classifier models for online fast inferences. We demonstrate this approach with an initial proof of concept targeting the CookieBox, an array of electron spectrometers used for angular streaking, that was developed for LCLS-II as an online beam diagnostic tool. We used the Lloyd-Max algorithm with the CookieBox dataset to design an optimized nonuniform scalar quantizer. Optimized quantization lets us reduce input data volume by 69% with no significant impact on inference accuracy. When we tolerate a 2% loss on inference accuracy, we achieved 81% of input data reduction. Finally, the change from a 7-bit to a 3-bit input data quantization reduces our neural network size by 38%.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

A variant selection framework for genome graphs

Abstract Motivation Variation graph representations are projected to either replace or supplement conventional single genome references due to their ability to capture population genetic diversity and reduce reference bias. Vast catalogues of genetic variants for many species now exist, and it is natural to ask which among these are crucial to circumvent reference bias during read mapping. Results In this work, we propose a novel mathematical framework for variant selection, by casting it in terms of minimizing variation graph size subject to preserving paths of length α with at most δ differences. This framework leads to a rich set of problems based on the types of variants [e.g. single nucleotide polymorphisms (SNPs), indels or structural variants (SVs)], and whether the goal is to minimize the number of positions at which variants are listed or to minimize the total number of variants listed. We classify the computational complexity of these problems and provide efficient algorithms along with their software implementation when feasible. We empirically evaluate the magnitude of graph reduction achieved in human chromosome variation graphs using multiple α and δ parameter values corresponding to short and long-read resequencing characteristics. When our algorithm is run with parameter settings amenable to long-read mapping (α = 10 kbp, δ = 1000), 99.99% SNPs and 73% SVs can be safely excluded from human chromosome 1 variation graph. The graph size reduction can benefit downstream pan-genome analysis. Availability and implementation https://github.com/AT-CG/VF. Supplementary information Supplementary data are available at Bioinformatics online.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

Intervening alcohol marketing to reduce harmful alcohol use and lessons learned from the theory of changes: Case studies in Thailand

Objectives Globally, the burden of disease caused by alcohol use has been steadily increasing, including in Thailand. In this study, we aim to test the effectiveness of Anderson et al.‘s suggested three approaches to change the collective social norms, which comprise of: (1) providing information and an understanding about alcohol use behaviour, its causes and distribution; (2) focusing strategies on groups rather than individuals; and (3) strengthening supportive laws, regulations and approaches. Study design We employed a mixed-methods approach. Evidence was gathered from literature review and in-depth interviews with key individuals who are responsible for community-based interventions to alcohol marketing strategies in Thailand. Methods We chose to focus on two case studies in Nan and Surin provinces, where hospital-based longitudinal data (8 years) were available. Changes in casualties related to the harmful use of alcohol, resulting from interactions between community-based interventions and alcohol marketing during the time of annual festivals were investigated. We employed the theory of change (ToC) defined by Vogel to guide the data collection and analysis. We reviewed literature from online databases and grey literature to generate causal-loop diagrams. Results We created a causal-loop diagram to describe the complexity of harmful alcohol use, its related factors, context, interventions and outcomes. Over the decade between 2006 and 2015, community-based strategies led to a substantial reduction of casualties (initially a 50% reduction, rising to an 80–90% reduction by the end of the study period) during the time of the festivals. Conclusions The reduction in injuries and fatalities could be a result of the concerted actions, including legal sanctions of alcohol beverage sales and advertisement, and public education to raise awareness and impart knowledge of the harmful use of alcohol. The actions were organised by a coalition of civil society, health professionals, public authorities and community leaders using hospital-based data on the adverse effects of harmful alcohol use to mobilise political support at the provincial level. The availability of long-term financial support as a catalytic source of funds and the presence of a comprehensive alcohol control act enabled framing and mobilisation of local resources and political support.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2022

The concentration of BTEX in selected urban areas of Malaysia during the COVID-19 pandemic lockdown

Volatile organic compounds (VOCs) such as benzene, toluene, ethylbenzene and xylene (BTEX) are air pollutants that harm human health. This study aims to identify BTEX concentrations before the lockdown known as the Movement Control Order was imposed (BMCO), during the implementation of the Movement Control Order (MCO), and then during the Conditional Movement Control Order (CMCO). These orders were introduced during the COVID-19 pandemic in Malaysia. The study utilised data measured by the continuous monitoring of BTEX using online gas chromatography instruments located at three urban area stations. Here, the results showed that the BTEX concentrations reduced by between –38% and –46% during the MCO compared to the BMCO period. The reduction of human mobility during the MCO and CMCO influenced the lower BTEX concentrations recorded at a station within the Kuala Lumpur area. The results of the BTEX diagnostic ratios and principal component analysis showed that the major source of BTEX, especially during the BMCO and CMCO periods, was motor vehicle emissions. Further investigation, using correlation analysis and polar plots, showed that the BTEX concentrations were also influenced by meteorological variables such as wind speed, air temperature and relative humidity.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Block-Structured Operator Inference for Coupled Multiphysics Model Reduction

This work presents a block-structured formulation of Operator Inference as a way to learn structured reduced-order models for multiphysics systems. The approach specifies the governing equation structure for each physics component and the structure of the coupling terms. Once the multiphysics structure is specified, the reduced-order model is learned from snapshot data following the nonintrusive Operator Inference methodology. In addition to preserving physical system structure, which in turn permits preservation of system properties such as stability and second-order structure, the block-structured approach has the advantages of reducing the overall dimensionality of the learning problem and admitting tailored regularization for each physics component. The numerical advantages of the block-structured formulation over a monolithic Operator Inference formulation are demonstrated for aeroelastic analysis, which couples aerodynamic and structural models. For the benchmark test case of the AGARD 445.6 wing, block-structured Operator Inference provides an average 20% online prediction speedup over monolithic Operator Inference across subsonic and supersonic flow conditions in both the stable and fluttering parameter regimes while preserving the accuracy achieved with monolithic Operator Inference.

42 ENGINEERING↗
DOE OSTI2021

Model fusion with physics-guided machine learning: Projection-based reduced-order modeling

The unprecedented amount of data generated from experiments, field observations, and large-scale numerical simulations at a wide range of spatiotemporal scales has enabled the rapid advancement of data-driven and especially deep learning models in the field of fluid mechanics. Although these methods are proven successful for many applications, there is a grand challenge of improving their generalizability. This is particularly essential when data-driven models are employed within outer-loop applications like optimization. In this work, we put forth a physics-guided machine learning (PGML) framework that leverages the interpretable physics-based model with a deep learning model. Leveraging a concatenated neural network design from multi-modal data sources, the PGML framework is capable of enhancing the generalizability of data-driven models and effectively protects against or inform about the inaccurate predictions resulting from extrapolation. We apply the PGML framework as a novel model fusion approach combining the physics-based Galerkin projection model and long- to short-term memory (LSTM) network for parametric model order reduction of fluid flows. We demonstrate the improved generalizability of the PGML framework against a purely data-driven approach through the injection of physics features into intermediate LSTM layers. Our quantitative analysis shows that the overall model uncertainty can be reduced through the PGML approach, especially for test data coming from a distribution different than the training data. Moreover, we demonstrate that our approach can be used as an inverse diagnostic tool providing a confidence score associated with models and observations. The proposed framework also allows for multi-fidelity computing by making use of low-fidelity models in the online deployment of quantified data-driven models.

42 ENGINEERING↗
DOE OSTI2020

Overcoming Oxidation State-Dependent Spectral Interferences: Online Monitoring of U(VI) Reduction to U(IV) via Raman and UV–vis Spectroscopy

Analytical characterization of chemical processes is an essential component of both industrial processes and fundamental studies. Applicable techniques abound, but optical spectroscopy has the unique combination of being easily incorporated into on-line (in-situ) monitoring probes and can also provide abundant chemical information (concentration, oxidation state, speciation). However, this abundance of chemical information can make applying optical spectroscopy to complex chemical processes challenging. This is particularly evident in the bulk electrolysis of U(VI) to U(IV), an essential preparatory step in some nuclear fuel recycling schemes. Starting and product materials have interfering spectral signatures in both electronic and vibrational spectroscopies, making accurate and real-time analysis difficult to achieve. In this paper, we discuss the application of optical spectroscopy to the on-line monitoring of a bulk electrolysis process and the methods utilized to analyze data as well as characterize the efficiency and mechanism of U(IV) generation.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

Hierarchical Model Reduction Driven by Machine Learning for Parametric Advection-Diffusion-Reaction Problems in the Presence of Noisy Data

Abstract We propose a new approach to generate a reliable reduced model for a parametric elliptic problem, in the presence of noisy data. The reference model reduction procedure is the directional HiPOD method, which combines Hierarchical Model reduction with a standard Proper Orthogonal Decomposition, according to an offline/online paradigm. In this paper we show that directional HiPOD looses in terms of accuracy when problem data are affected by noise. This is due to the interpolation driving the online phase, since it replicates, by definition, the noise trend. To overcome this limit, we replace interpolation with Machine Learning fitting models which better discriminate relevant physical features in the data from irrelevant unstructured noise. The numerical assessment, although preliminary, confirms the potentialities of the new approach.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

SNS: A Solution-Based Nonlinear Subspace Method for Time-Dependent Model Order Reduction

Several reduced order models have been successfully developed for nonlinear dynamical systems. To achieve a considerable speed-up, a hyper-reduction step is needed to reduce the computational complexity due to nonlinear terms. Many hyper-reduction techniques require the construction of nonlinear term basis, which introduces a computationally expensive offline phase. A novel way of constructing nonlinear term basis within the hyper-reduction process is introduced. In contrast to the traditional hyper-reduction techniques where the collection of nonlinear term snapshots is required, the SNS method avoids collecting the nonlinear term snapshots. Instead, it uses the solution snapshots that are used for building a solution basis, which enables avoiding an extra data compression of nonlinear term snapshots. As a result, the SNS method provides a more efficient offline strategy than the traditional model order reduction techniques, such as the DEIM, GNAT, and ST-GNAT methods. The SNS method is theoretically justified by the conforming subspace condition and the subspace inclusion relation. It is useful for model order reduction of large-scale nonlinear dynamical problems to reduce the offline cost. It is especially useful for ST-GNAT that has shown promising results, such as a good accuracy with a considerable online speed-up for hyperbolic problems in a recent paper by Choi and Carlberg [SIAM J. Sci. Comput., 41 (2019), pp. A26--A58], because ST-GNAT involves an expensive offline cost related to collecting nonlinear term snapshots. Error analysis for the SNS method is presented. Numerical results support that the accuracy of the solution from the SNS method is comparable to the traditional methods and a considerable speed-up (i.e., a factor of two to a hundred) is achieved in the offline phase.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Data-Driven Event Detection of Power Systems Based on Unequal-Interval Reduction of PMU Data and Local Outlier Factor

With the deployment of phasor measurement units (PMU) and wide area measurement system (WAMS), it is feasible to have an insight into the events occurred in power systems based on measured data. Thus, a novel data-driven algorithm based on local outlier factor (LOF) is proposed in this work to detect and locate events in power systems using reduced PMU data. First, the unequal-interval reduction method is presented to reduce the scale of PMU data in sub-stations and reconstruct it in master station of WAMS, which can relieve the burden of communication systems. Then, principle component analysis (PCA)-based similarity search method is proposed to measure the differences of operation state between any two buses. Next, LOF is presented to detect the abnormal events in power systems, and employed to determine the region of the event source. Finally, six cases from the Western electricity coordinating council (WECC) 179-bus power system, a case from the South China power system (SCPS), and a case from the Guangdong power system (GDPS) are utilized to demonstrate the effectiveness of the proposed algorithm. Overall, the results show that proposed algorithm is effective and can be applied to event detection, event location, and online monitoring, which can enhance the situation awareness ability of power system operators.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2024

BioXTAS RAW 2 : new developments for a free open-source program for small-angle scattering data reduction and analysis

BioXTAS RAW is a free open-source program for reduction, analysis and modelling of biological small-angle scattering data. Here, the new developments in RAW version 2 are described. These include improved data reduction using pyFAI ; updated automated Guinier fitting and D max finding algorithms; automated series ( e.g. size-exclusion chromatography coupled small-angle X-ray scattering or SEC-SAXS) buffer- and sample-region finding algorithms; linear and integral baseline correction for series; deconvolution of series data using regularized alternating least squares ( REGALS ); creation of electron-density reconstructions using electron density via solution scattering ( DENSS ); a comparison window showing residuals, ratios and statistical comparisons between profiles; and generation of PDF reports with summary plots and tables for all analysis. Furthermore, there is now a RAW API, which can be used without the graphical user interface (GUI), providing full access to all of the functionality found in the GUI. In addition to these new capabilities, RAW has undergone significant technical updates, such as adding Python 3 compatibility, and has entirely new documentation available both online and in the program.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Design and Operation of a Multi-Bed Catalytic Micro-Reactor for the Study of Co-Processing of Bio-Oils with VGO

An industry wide shift from fossil-based fuel to renewable fuel sources including biomass, municipal waste, and plastics will require new process monitoring methods to minimize transitional risks including off specification product formation and catalyst deactivation. This project aims to provide a machine learning based process monitoring tool composed of online, slipstream mass spectra for use in biomass refineries and co-processing in existing refineries allowing operators to monitor product qualities and adjust process conditions accordingly. In order to maximize the robustness of the tool, large volumes of data must be collected to fine tune model parameters which consists of both micro and pilot scale mass spectral data. Micro-scale data is collected with a multi-tube micro-reactor housing up to six catalysts in horizontal beds, coupled with a molecular beam mass spectrometer. A pyrolizer equipped with an auto-sampler streamlines the micro-scale data collection process. This type of pyrolizer/micro-reactor configuration does not exist on the market, and therefore had to be created for the purposes of this project. The design and commissioning of this reactor will be presented in detail. This reactor set-up is highly flexible and increases throughput of analysis. For catalyst testing, each bed can be individually selected simply by turning valves. For catalyst reduction and regeneration, simultaneous flow through all six beds is used. The reproducibility of the system was first assessed with whole biomass pyrolysis along with pyrolysis of calibration standards. Initial work on this system evaluated two FCC catalysts, equilibrium catalyst (E-cat), and a proprietary catalyst from Johnson Matthey specifically design for co-processing of bio-oil with vacuum gas oil (VGO). This work used model compounds and VGO which illuminated differences in products produced by the catalysts.

biomass↗
DOE OSTI2026

Toward real-time optimization through model reduction and model discrepancy sensitivities

Optimization problems arise in a range of scenarios, from optimal control to model parameter estimation. In many applications, such as the development of digital twins, it is essential to solve these optimization problems within wall-clock-time limitations. However, this is often unattainable for complex systems, such as those modeled by nonlinear partial differential equations. One strategy for mitigating this issue is to construct a reduced-order model (ROM) that enables more rapid optimization. In particular, the use of nonintrusive ROMs—those that do not require access to the full-order model at evaluation time—is popular because they facilitate the computation of optimization solutions within the wall-clock time requirements. However, the optimization solution will be unreliable if the iterates move outside the ROM training data. This article proposes the use of hyper-differential sensitivity analysis with respect to model discrepancy (HDSA-MD) as a computationally efficient tool to augment ROM-constrained optimization and improve its reliability. The proposed approach consists of two phases: (i) an offline phase where several full-order model evaluations are computed to train the ROM, and (ii) an online phase where a ROM-constrained optimization problem is solved, a limited number of full-order model evaluations are computed, and HDSA-MD is used to enhance the optimization solution. Numerical results are demonstrated for two examples, atmospheric contaminant control and wildfire ignition location estimation, in which a ROM is trained offline using inaccurate atmospheric data. In conclusion, the HDSA-MD update yields a significant improvement in the ROM-constrained optimization solution using only one full-order model evaluation online with corrected atmospheric data.

PDE-constrained optimization↗