Search NASA⌕ Search

SEARCH · Search NASA

Results for “in situ data analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

In Situ Data Analysis Through Physics-informed Tensor Decompositions (LDRD Final Report)

We introduce a new low-dimensional model of high-dimensional numerical simulation data based on low-rank tensor decompositions. Our new model aims to minimize differences between the model data and simulation data as well as functions of the model data and functions of the simulation data. This novel approach to dimensionality reduction of simulation data provides a means of directly incorporating quantities of interests and invariants associated with conservation principles associated with the simulation data into the low-dimensional model, thus enabling more accurate analysis of the simulation without requiring access to the full set of high-dimensional data. Computational results of applying this approach to two standard low-rank tensor decompositions of data arising from simulation of combustion and plasma physics are presented.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1981

In situ data analysis on high altitude balloons using microprocessors

It is pointed out that systems currently in use permit an uplink data rate from scientific balloons to satellites of only 100 bit/min. On-board microprocessors, which are able to condense data, can address this problem. An experiment is described that demonstrates the capabilities of inflight data processing by an on-board computer for electric field experiments. It was possible to reduce the data rate by two orders of magnitude with no diminution of data quality.

Schroeder, K. R.↗
DOE OSTI2021

Scoreboard

Emerging HPC machines have given rise to enhanced compute power that far outstrips the machine's ability to save large scale results for post-processing. To combat this, in situ data analysis techniques are slowly being adopted. With in situ data management favoring workflows composed of multiple simulations and analyses connected in transit on heterogeneous machines, scientists and engineers need a tool that enables them to create data extracts, visualizations, and interactively monitor and steer their simulations. Scoreboard Phase II is a next generation analysis software that supports composite in transit workflows on heterogeneous architectures and restores interactivity to in situ data analysis through simulation monitoring and computational steering. Scoreboard provides a simulation dashboard with graphs of metrics over time, controls for setting custom simulation steering parameters, controls for managing the set of data extracts being produced in the simulation, as well as the ability to explore data extracts, all from a web browser. Realizing the vision outlined in this project required research into making a system that integrates end to end from simulations all the way to the user. In situ tools generally suffer from complexity and excessive software dependencies. Scoreboard, by contrast, is easy to build and integrate into simulation codes and it provides first class FORTRAN support. The Scoreboard library is capable of in situ and in transit data analysis that can produce data extracts commonly needed for Computational Fluid Dynamics (CFD) analysis. Simulations can transparently stage data in transit to a Scoreboard Endpoint program, which can accept their data and produce the requested data extracts. This lets simulations return to their work while the Endpoint works on the analysis. Efficiently staging the data at scale was a topic of this research. Scoreboard provides the means to let the user manage data extracts and monitor/steer many simulations from a web browser. This area of the research focused on discovery of in transit network components to expose and control their steering parameters within an interactive browser-based user interface that includes: system topology, gathered metrics, notifications, dynamically-generated steering controls, and exploration of visualization data products.

Whitlock, BradJoseph [Intelligent Light] (00000001↗
NASA NTRS1991

Integrated data analysis of July 7, 1990 microburst

Integrated data analysis of July 7, 1990 microburst is presented in the form of view-graphs. Topics included are: summer 1990 TDWR flight experiment; least square estimate of linear shear; event synopsis; TDWR (terminal Doppler weather radar) data analysis; citation aircraft in situ data analysis; infrared data analysis; and summary of July 7 event.

Hinton, Dave↗
DOE OSTI2021

Scalable, In-situ Data Clustering Data Analysis for Extreme Scale Scientific Computing (Final Report)

The objective of this project is to address challenges in the design and development of scalable in-situ data clustering and analytics algorithms and software. Our goal is to develop parallel software consisting of a set of spatio-temporal data clustering and anomaly detection functions, both of which are very important for large-scale analysis and have wide applicability for in-situ runs as well as post-processing analysis. Our design principles for in-situ analysis consider the following: (1) identify parts of the computation can be done close to the data within the nodes, while it is still in memory; (2) extract analysis components can (and should) be performed in remote staging and analysis nodes; (3) develop error-bound approximation methods for applications tolerable for small errors; (4) identify the type of derived distributions and statistics, for spatio-temporal data, that can be kept locally in order to both accelerate computations and meet energy constraints in subsequent iterations and phases; (5) use a self-describing data format so that data can be consistent and understood among local storage (memory and SSDs) and at staging and analysis nodes, thereby providing portability and flexibility; (6) develop service-oriented functions that can schedule in-situ and post-hoc analysis tasks based on the dynamic requirements of applications. Our development focus is to produce the parallel data analysis software/library that will be scalable, reusable, extensible, and generic for applications in different disciplines. The software will be able to run in-situ with the simulations as well as post-hoc analysis. This approach will satisfy many synergistic requirements for data intensive applications executed on data coming from instruments and experiments. In particular, the proposed multilevel approach is directly applicable to perform design tradeoffs for running part of the algorithms near the instruments and the rest on remote (analysis) systems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

In-Situ Inference: Bringing Advanced Data Science Into Exascale Simulations

As simulations generate ever-increasing amounts of data, there are correspondingly richer opportunities for analysis and scientific discovery - discoveries that will be missed if most of the data must be discarded before it is analyzed. Because future exascale architectures will be increasingly storage-limited, it will not be possible to save the vast majority of simulation data for later analysis, requiring analysis to occur “in-situ” within the simulation. However, existing in-situ data analysis frameworks provide little or no support for one of the most sophisticated forms of data science: probabilistic statistical modeling or uncertainty quantification (UQ), and the accompanying challenge of inference - fitting those statistical models to massive simulation output. Our goal is to develop the fundamental statistical algorithms and computer science needed to perform statistical inference in-situ (in HPC simulations) to the full stream of data those simulations generate.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Performance Understanding and Analysis for Exascale Data Management Workflows

The goal of the Exascale Data Management Workflows project is to create tools to support techniques for in-situ data analysis and data management. Online data management co-running with simulations accelerates the scientific processes being carried out, provides rapid and timely scientific insights, and can help avoid unnecessary and scientifically invalid simulation computations, particularly when combined with or able to utilize data from experimental instrument and/or from past simulation runs. The University of Oregon team is contributing to the performance instrumentation and analysis aspect of this project. Our focus has been on efficient performance monitoring of data management at scale. The general approach of the MONA project is depicted in Figure 1 below. The figure shows performance monitoring applied to an I/O workflow associated with a scientific simulation, with online measurements captured from this workflow informing methods for workflow reconfiguration and dynamic adjustment. The goal is to maintain suitable levels of Quality of Service for workflow execution, by understanding the under lying causes of workflow performance and behavior. One outcome is workflow performance models able to characterize realistic workflows. Another outcome is performance ’mini-apps’ implementing such workflow behavior. Out of scope for this project are advanced methods for online workflow control.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

A Unit Testing Framework for Scientific Legacy Code

Large-scale scientific applications play important roles in supporting research. However, it is often very expensive and time-consuming to make changes to, maintain and evolve the scientific code due to its complexity and poor programming skills of researchers. Therefore, in order to visualize scientific code architecture to optimize software design, understand undocumented source code, and analyze software flow and functionality, we first introduce a unit testing framework (UTF). Then, because such infrastructure's performance is very crucial in practical use since the scientific legacy applications simulate instances in a long period of time, we improve the UTF by applying Message Passing based Parallelization and parallel I/O operations. Furthermore, due to the scientific code has enormous state data and the I/O capacity on the server is limited, we apply in situ data analysis method to encounter fewer resource limitations, and adopt signal processing to greatly reduce data transfer. Last, we demonstrated the correctness and high-efficiency of our framework for legacy Earth model on Titan supercomputer.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

Improving Mixed-phase Cloud Parameterization in Climate Model with the ACRF Measurements

Mixed-phase cloud microphysical and dynamical processes are still poorly understood, and their representation in GCMs is a major source of uncertainties in overall cloud feedback in GCMs. Thus improving mixed-phase cloud parameterizations in climate models is critical to reducing the climate forecast uncertainties. This study aims at providing improved knowledge of mixed-phase cloud properties from the long-term ACRF observations and improving mixed-phase clouds simulations in the NCAR Community Atmosphere Model version 5 (CAM5). The key accomplishments are: 1) An improved retrieval algorithm was developed to provide liquid droplet concentration for drizzling or mixed-phase stratiform clouds. 2) A new ice concentration retrieval algorithm for stratiform mixed-phase clouds was developed. 3) A strong seasonal aerosol impact on ice generation in Arctic mixed-phase clouds was identified, which is mainly attributed to the high dust occurrence during the spring season. 4) A suite of multi-senor algorithms was applied to long-term ARM observations at the Barrow site to provide a complete dataset (LWC and effective radius profile for liquid phase, and IWC, Dge profiles and ice concentration for ice phase) to characterize Arctic stratiform mixed-phase clouds. This multi-year stratiform mixed-phase cloud dataset provides necessary information to study related processes, evaluate model stratiform mixed-phase cloud simulations, and improve model stratiform mixed-phase cloud parameterization. 5). A new in situ data analysis method was developed to quantify liquid mass partition in convective mixed-phase clouds. For the first time, we reliably compared liquid mass partitions in stratiform and convective mixed-phase clouds. Due to the different dynamics in stratiform and convective mixed-phase clouds, the temperature dependencies of liquid mass partitions are significantly different due to much higher ice concentrations in convective mixed phase clouds. 6) Systematic evaluations of mixed-phase cloud simulations by CAM5 were performed. Measurement results indicate that ice concentrations control stratiform mixed-phase cloud properties. The improvement of ice concentration parameterization in the CAM5 was done in close collaboration with Dr. Xiaohong Liu, PNNL (now at University of Wyoming).

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2001

Numerical Simulations of Precipitation Processes, Microphysics, and Microwave Radiative Properties of flood Producing Storms in Mediterranean & Adriatic Basins

A comprehensive understanding of the meteorological and microphysical nature of Mediterranean storms requires a combination of in situ data analysis, radar data analysis, and satellite data analysis, effectively integrated with numerical modeling studies at various scales. An important aspect of understanding microphysical controls of severe storms, is first understanding the meteorological controls under which a storm has evolved, and then using that information to help characterize the dominant microphysical processes. For hazardous Mediterranean storms, highlighted by the October 5-6, 1998 Friuli flood event in northern Italy, a comprehensive microphysical interpretation requires an understanding of the multiple phases of storm evolution. This involves intense convective development, Sratiform decay, orographic lifting, and sloped frontal lifting processes, as well as the associated vertical motions and thermodynamical instabilities governing physical processes that effect details of the size distributions and fall rates of the various types of hydrometeors found within the storm environment. This talk overviews the microphysical elements of a severe Mediterranean storm in such a context, investigated with the aid of TRMM satellite and other remote sensing measurements, but guided by a nonhydrostatic mesoscale model simulation of the Friuli flood event. The data analysis for this paper was conducted by my research groups at the Global Hydrology and Climate Center in Huntsville, AL and Florida State University in Tallahassee, and in collaboration with Dr. Alberto Mugnai's research group at the Institute of Atmospheric Physics in Rome. The numerical modeling was conducted by Professor Oreg Tripoli and Ms. Giulia Panegrossi at the University of Wisconsin in Madison, using Professor Tripoli's nonhydrostatic modeling system (NMS). This is a scalable, fully nested mesoscale model capable of resolving nonhydrostatic circulations from regional scale down to cloud scale and below.

Smith, Eric A.↗
NASA NTRS2004

Investigation of the Seasonal Freeze/Thaw Cycle of Soils in the GAPP Regions

The objective of this project is to understand the current and future impacts and feedbacks of seasonally frozen ground on the regional and global climate system and the ecosystem as a whole. More specifically, we proposed to: investigate the dynamics of the seasonally freezing and thawing processes and their relations to current climatic conditions through in situ data analysis and numerical modeling; develop a comprehensive algorithm to detect surface soil freeze/thaw status for snow-free land surface using passive microwave remote sensing data and for snow-covered land surface using numerical modeling, and validate the algorithm and numerical model using available ground-based measurements; investigate seasonal and inter-annual variations of frozen soils in the GAPP regions using data generated from the new validated algorithm and their relations to the environmental factors such as air temperature, snow cover, surface morphology, soil type and soil moisture; generate a frozen soil data set which describes the timing, duration, thickness, and area extent of seasonally frozen ground for the period from 1978 to present. The proposed research will be accomplished through data analysis, remote sensing, and numerical modeling.

FREEZE-THAW TESTS↗
DOE OSTI2019

Ex situ metrology and data analysis for optimization of beamline performance of aspherical pre-shaped x-ray mirrors at the advanced light source

Super high quality aspherical x-ray mirrors with a residual slope error of ~100 nrad (root-mean-square) and a height error of ~1-2 nm (peak-to-valley), and even lower, are now available from a number of the most advanced vendors utilizing deterministic polishing techniques. The mirror specification for the fabrication is based on the simulations of the desired performance of the mirror in the beamline optical system and is normally given with the acceptable level of deviation of the mirror figure and finish from the desired ideal shape. For example, in the case of aspherical x-ray mirrors designed for the Advanced Light Source (ALS) QERLIN beamline, the ideal shape is defined with the beamline application (conjugate) parameters and their tolerances. Here, we first introduce an original procedure and dedicated software developed at the ALS X-Ray Optics Laboratory (XROL) for optimization of beamline performance of pre-shaped hyperbolic and elliptical mirrors. The optimization is based on results of ex situ surface slope metrology and consists in minimization of the mirror shape error by determining the conjugate parameters of the best-fit ideal shape within the specified tolerances. We describe novel optical metrology instrumentation, measuring techniques, and analytical methods used at the XROL for acquisition of surface slope data and optimization of the optic’s beamline performance. The high efficacy of the developed experimental methods and data analysis procedures is reflected in results of measurements with and performance optimization of hyperbolic and elliptical cylinder mirrors designed and fabricated for the ALS QERLIN beamline.

47 OTHER INSTRUMENTATION↗
DOE OSTI2016

Trigger Detection for Adaptive Scientific Workflows Using Percentile Sampling

The increasing complexity of both scientific simulations and high-performance computing system architectures are driving the need for adaptive workflows, in which the composition and execution of computational and data manipulation steps dynamically depend on the evolutionary state of the simulation itself. Consider, for example, the frequency of data storage. Critical phases of the simulation should be captured with high frequency and with high fidelity for postanalysis; however, we cannot afford to retain the same frequency for the full simulation due to the high cost of data movement. We can instead look for triggers, indicators that the simulation will be entering a critical phase, and adapt the workflow accordingly. In this paper, we present a methodology for detecting triggers and demonstrate its use in the context of direct numerical simulations of turbulent combustion using S3D. We show that chemical explosive mode analysis (CEMA) can be used to devise a noise-tolerant indicator for rapid increase in heat release. However, exhaustive computation of CEMA values dominates the total simulation, and thus is prohibitively expensive. To overcome this computational bottleneck, we propose a quantile sampling approach. Our sampling-based algorithm comes with provable error/confidence bounds, as a function of the number of samples. Most importantly, the number of samples is independent of the problem size, and thus our proposed sampling algorithm offers perfect scalability. Our experiments on homogeneous charge compression ignition and reactivity controlled compression ignition simulations show that the proposed method can detect rapid increases in heat release, and its computational overhead is negligible. Our results will be used to make dynamic workflow decisions regarding data storage and mesh resolution in future combustion simulations.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

A Data Analysis Framework for Earth System Simulation within an In-Situ Infrastructure

This paper presents a generic procedure to implement a scalable and high performance data analysis framework for large-scale scientific simulation within an in-situ infrastructure. It demonstrates a unique capability for global Earth system simulations using advanced computing technologies (i.e ., automated code analysis and instrumentation), in-situ infrastructure (i.e ., ADIOS) and big data analysis engines (i.e ., SciKit-learn). This paper also includes a useful case that analyzes a globe Earth System simulations with the integration of scalable in-situ infrastructure and advanced data processing package. Finally, the in-situ data analysis framework can provides new insights on scientific discoveries in multiscale modeling paradigms.

97 MATHEMATICS AND COMPUTING↗