Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Analytics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 649 records · Page 36
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Scalable Data-Intensive Geocomputation: A Design for Real-Time Continental Flood Inundation Mapping

The convergence of data-intensive and extreme-scale computing enables an integrated software and data ecosystem for scientific discovery. Developments in this realm will fuel transformative research in data-driven interdisciplinary domains. Geocomputation provides computing paradigms in Geographic Information Systems (GIS) for interactive computing of geographic data, processes, models, and maps. Because GIS is data-driven, the computational scalability of a geocomputation workflow is directly related to the scale of the GIS data layers, their resolution and extent, as well as the velocity of the geo-located data streams to be processed. Unique in high user interactivity and low end-to-end latency requirements, geocomputation applications will dramatically benefit from the convergence of high-end data analytics (HDA) and high-performance computing (HPC). The application level challenge, however, is to identify and eliminate computational bottlenecks that arise along a geocomputation workflow. Indeed, poor scalability at any of the workflow components is detrimental to the entire end-to-end pipeline. Here, we study a large geocomputation use case in flood inundation mapping that handles multiple national-scale geospatial datasets and targets low end-to-end latency. We discuss benefits and challenges for harnessing both HDA and HPC for data-intensive geospatial data processing and intensive numerical modeling of geographic processes. We propose an HDA+HPC geocomputation architecture design that couples HDA (e.g., Spark)-based spatial data handling and HPC-based parallel data modeling. Key techniques for coupling HDA and HPC to bridge the two different software stacks are reviewed and discussed.

Liu, Yan↗
DOE OSTI2022

Warning Signs of Potential Black Swan Outbreaks in Infectious Disease

Black swan events in infectious disease describe rare but devastatingly large outbreaks. While experts are skeptical that such events are predictable, it might be possible to identify the warning signs of a black swan event. Specifically, following the initiation of an outbreak, key differentiating features could serve as alerts. Such features could be derived from meta-analyses of large outbreaks for multiple infectious diseases. We hypothesized there may be common features among the pathogen, environment, and host epidemiological triad that characterize an infectious disease black swan event. Using Los Alamos National Laboratory’s tool, Analytics for Investigation of Disease Outbreaks, we investigated historical disease outbreak information and anomalous events for several infectious diseases. By studying 32 different infectious diseases and global outbreaks, we observed that in the past 20–30 years, there have been potential black swan events in the majority of infectious diseases analyzed. Importantly, these potential black swan events cannot be attributed to the first introduction of the disease to a susceptible host population. This paper describes our observations and perspectives and illustrates the value of broad analysis of data across the infectious disease realm, providing insights that may not be possible when we focus on singular infectious agents or diseases. Data analytics could be developed to warn health authorities at the beginning of an outbreak of an impending black swan event. Such tools could complement traditional epidemiological modeling to help forecast future large outbreaks and facilitate timely warning and effective, targeted resource allocation for mitigation efforts.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2020

Cross-Cutting Software Solutions in Support of Experimental Analysis Challenges at National Scattering Facilities

Rapid improvements in instrumentation at neutron and light source combined with new capabilities in computing, data analytics, and machine learning offer unprecedented scientific capabilities to address fundamental and applied science questions in a wide area of subjects from quantum materials to biological systems. In order to make these capabilities accessible to the broader scientific community, significant developments are needed related to workflow spanning from experimental control to advanced materials simulations as well as the underpinning scientific software. In this chapter, we explore these needs and recent developments at the Spallation Neutron Source (SNS) at Oak Ridge National Laboratory. A central part of the workflow at the SNS is the data reduction framework Mantid which is developed as a partnership between European and US neutron-scattering facilities.

Proffen, Thomas↗
DOE OSTI2016

Data Validation Package - July 2016 Groundwater Sampling at the Gunnison, Colorado, Disposal Site

Groundwater sampling at the Gunnison, Colorado, Disposal Site is conducted every 5 years to monitor disposal cell performance. During this event, samples were collected from eight monitoring wells as specified in the 1997 Long-Term Surveillance Plan for the Gunnison, Colorado, Disposal Site. Sampling and analyses were conducted as specified in the Sampling and Analysis Plan for US Department of Energy Office of Legacy Management Sites (LMS/PRO/S04351, continually updated, http://energy.gov/lm/downloads/sampling-and­ analysis-plan-us-department-energy-office-legacy-management-sites). Planned monitoring locations are shown in Attachment 1, Sampling and Analysis Work Order. A duplicate sample was collected from location 0723. Water levels were measured at all monitoring wells that were sampled and seven additional wells. The analytical data and associated qualifiers can be viewed in environmental database reports and are also available for viewing with dynamic mapping via the GEMS (Geospatial Environmental Mapping System) website at http://gems.lm.doe.gov/#. No issues were identified during the data validation process that require additional action or follow-up.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

A KBase Case Study on Genome-wide Transcriptomics and Plant Primary Metabolism in Sorghum

A better understanding of the genetic and metabolic mechanisms that confer stress resistance and tolerance in plants is key to engineering new crops through advanced breeding technologies. This requires a systems biology approach that builds on a genome-wide understanding of the regulation of gene expression, plant metabolism, physiology and growth. In this study, we examine the response to drought stress in Sorghum, as we leverage the tools for transcriptomics and plant metabolic modeling we have implemented at the U.S. Department of Energy Systems Biology Knowledgebase (KBase). KBase enables researchers worldwide to collaborate and advance research by allowing them to upload private or public data into the KBase Narrative Interface, empowering them to analyze this data using a rich, extensible array of computational and data-analytics tools, and allows them to securely share scientific workflows and conclusions. We demonstrate how to use the current RNA-seq tools in KBase, applicable to both plants and microbes, to assemble and quantify long transcripts and identify differentially expressed genes effectively. More specifically, we demonstrate the utility of the platform by identifying key genes that are differentially expressed during drought-stress in Sorghum bicolor, which is an important sustainable production crop plant. We then show how to use KBase tools to predict the membership of genes in metabolic pathways and examine expression data in the context of metabolic subsystems. We demonstrate the power of the platform by making the data analysis and interpretation available to the biologists in the reproducible, re-usable, point-and-click format of a KBase Narrative thus promoting FAIR (Findable, Accessible, Interoperable and Reusable) guiding principles for scientific data management and stewardship.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

A Hybrid Energy System Workflow for Energy Portfolio Optimization

This manuscript develops a workflow, driven by data analytics algorithms, to support the optimization of the economic performance of an Integrated Energy System. The goal is to determine the optimum mix of capacities from a set of different energy producers (e.g., nuclear, gas, wind and solar). A stochastic-based optimizer is employed, based on Gaussian Process Modeling, which requires numerous samples for its training. Each sample represents a time series describing the demand, load, or other operational and economic profiles for various types of energy producers. These samples are synthetically generated using a reduced order modeling algorithm that reads a limited set of historical data, such as demand and load data from past years. Numerous data analysis methods are employed to construct the reduced order models, including, for example, the Auto Regressive Moving Average, Fourier series decomposition, and the peak detection algorithm. All these algorithms are designed to detrend the data and extract features that can be employed to generate synthetic time histories that preserve the statistical properties of the original limited historical data. The optimization cost function is based on an economic model that assesses the effective cost of energy based on two figures of merit: the specific cash flow stream for each energy producer and the total Net Present Value. An initial guess for the optimal capacities is obtained using the screening curve method. The results of the Gaussian Process model-based optimization are assessed using an exhaustive Monte Carlo search, with the results indicating reasonable optimization results. The workflow has been implemented inside the Idaho National Laboratory’s Risk Analysis and Virtual Environment (RAVEN) framework. The main contribution of this study addresses several challenges in the current optimization methods of the energy portfolios in IES: First, the feasibility of generating the synthetic time series of the periodic peak data; Second, the computational burden of the conventional stochastic optimization of the energy portfolio, associated with the need for repeated executions of system models; Third, the inadequacies of previous studies in terms of the comparisons of the impact of the economic parameters. The proposed workflow can provide a scientifically defendable strategy to support decision-making in the electricity market and to help energy distributors develop a better understanding of the performance of integrated energy systems.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2019

Statistical change detection of building energy consumption: Applications to savings estimation

The surge in interval meter data availability and associated activity in energy data analytics has inspired new interest in advanced methods for building efficiency savings estimation. Statistical and machine learning approaches are being explored to improve the energy baseline models used to measure and verify savings. One outstanding challenge is the ability to identify and account for operational changes that may confound savings estimates. In the measurement and verification (M&V) context, ‘non-routine events’ (NREs) cause changes in building energy use that are not attributable to installed efficiency measures, and not accounted for in the baseline model’s independent variables. In the M&V process NREs must be accounted for as ‘adjustments’ to appropriately attribute the estimated energy savings to the specific efficiency interventions that were implemented. Currently this is a manual and custom process, conducted using professional judgment and engineering expertise. As such it remains a barrier in scaling and standardizing meter-based savings estimation. Here in this work, a data driven methodology was developed to (partially) automate, and therefore streamline the process of detecting NREs in the post-retrofit period and making associated savings adjustments. The proposed NRE detection algorithm is based on a statistical change point detection method and a dissimilarity metric. The dissimilarity metric measures the proximity between the actual time series of the post-retrofit energy consumption and the projected baseline, which is generated using a statistical baseline model. The suggested approach for NRE adjustment involves the NRE detection algorithm, the M&V practitioner, and a regression modeling algorithm. The performance of the detection and adjustment algorithm was evaluated using a simulation-generated test data set, and two benchmark algorithms. Results show a high true positive detection rate (75%-100% across the test cases), higher than ideal false positive detection rates (20%-70%), and low errors in energy adjustment (<0.7%). These results indicate that the algorithm holds for helping M&V practitioners to streamline the process of handling NREs. Moreover, the change point algorithm and underlying statistical principles could prove valuable for other building analytics applications such as anomaly detection and fault diagnostics.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2024

Tutorial on In Situ and Operando (Scanning) Transmission Electron Microscopy for Analysis of Nanoscale Structure–Property Relationships

In situ and operando (scanning) transmission electron microscopy [(S)TEM] is a powerful characterization technique that uses imaging, diffraction, and spectroscopy to gain nano-to-atomic scale insights into the structure–property relationships in materials. This technique is both customizable and complex because many factors impact the ability to collect structural, compositional, and bonding information from a sample during environmental exposure or under application of an external stimulus. In the past two decades, in situ and operando (S)TEM methods have diversified and grown to encompass additional capabilities, higher degrees of precision, dynamic tracking abilities, enhanced reproducibility, and improved analytical tools. Much of this growth has been shared through the community and within commercialized products that enable rapid adoption and training in this approach. This tutorial aims to serve as a guide for students, collaborators, and nonspecialists to learn the important factors that impact the success of in situ and operando (S)TEM experiments and assess the value of the results obtained. As this is not a step-by-step guide, readers are encouraged to seek out the many comprehensive resources available for gaining a deeper understanding of in situ and operando (S)TEM methods, property measurements, data acquisition, reproducibility, and data analytics.

(S)TEM↗
DOE OSTI2026

Comparability of Liquid Chromatography Tandem Mass Spectrometry Analysis of Dissolved Organic Matter across Laboratories

Non-targeted liquid chromatography tandem highresolution mass spectrometry (LC−MS/MS) is increasingly applied for the structure-resolved chemical analysis of dissolved organic matter (DOM). With new developments in MS instrumentation and analysis software, the approach has gained substantial momentum over the past decade. However, achieving high-quality analytical data that is reproducible and comparable across laboratories can be a bottleneck in non-targeted metabolomics and organic matter chemical analysis, especially for data reuse in repository-scale analyses. Understanding the capabilities as well as challenges of comparing LC−MS/MS data from different laboratories is necessary for inferring global trends from public data sets. To illuminate instrumentation factors that drive differences and variability, we used a standardized data analysis pipeline, including classical (CMN) and featurebased molecular networking (FBMN), to analyze data from a ring trial by 24 laboratories on identical sample sets of algal and DOM extracts that were mixed in predefined concentrations and spiked with standards. Our results showed that data sets from similar mass spectrometer types with unified instrument parameters were qualitatively comparable, resolving the same general trends and shared mass spectral features. Interlaboratory comparability was best for high-intensity features, while low-intensity features showed greater detection variability. Our analysis also highlights challenges when comparing data from instruments with different acquisition rates or operating with less standardized methods. Lastly, we provide recommendations for data integration, public data sharing, standardization, and best practices for standardized LC−MS/MS data acquisition, which will be critical for long-term time series and intercomparability of DOM chemical analyses.

DOM↗
DOE OSTI2024

Analytical capabilities for iodine detection: Review of possibilities for different applications

This Review summarizes a range of analytical techniques that can be used to detect, quantify, and/or distinguish between isotopes of iodine (e.g., long-lived 129 I, short-lived 131 I, stable 127 I). One reason this is of interest is that understanding potential radioiodine release from nuclear processes is crucial to prevent environmental contamination and to protect human health as it can incorporate into the thyroid leading to cancer. It is also of interest for evaluating iodine retention performances of next-generation iodine off-gas capture materials and long-term waste forms for immobilizing radioiodine for disposal in geologic repositories. Depending upon the form of iodine (e.g., molecules, elemental, and ionic) and the matter state (i.e., solid, liquid, and gaseous), the available options can vary. In addition, several other key parameters vary between the methods discussed herein, including the destructive vs nondestructive nature of the measurement process (including in situ vs ex situ measurement options), the analytical data collection times, and the amount of sample required for analysis.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2024

Emulation and detection of physical faults and cyber-attacks on building energy systems through real-time hardware-in-the-loop experiments

The increasing use of remote or mobile access, integrated wearable technologies, data exchange, and cloud-based data analytics in modern smart buildings is steering the building industry towards open communication technologies. The increased connectivity and accessibility could lead to more cyber-attacks in smart buildings. On the other hand, physical faults (e.g., HVAC -heating, ventilation, and air-conditioning faults) may have similar adverse impacts as those from the cyber-attacks on building energy systems, such as occupant discomfort, energy wastage, and equipment downtime. However, current physical behavior-based anomaly detection methods fail to differentiate between cyber-attacks and physical faults in building energy systems. Moreover, the challenge in collecting real-world threat data with ground truth has led researchers to rely on numerical models with user-defined assumptions, which may not accurately reflect real-world conditions due to the lack of in-situ experimental datasets. To address these challenges and gaps, this paper presents a flexible hardware-in-the-loop (HIL) testbed for generating cyber-attack and physical fault datasets and demonstrating threat detection algorithms in a real building automation system (BAS) environment. This testbed combines hardware (i.e., real BAS with local HVAC controllers and a physical network) with software (i.e., high-fidelity models to represent behaviors of building envelope and HVAC energy systems), enabling emulations of realistic threats. Five HIL experiments, including one baseline without any threats, two with physical faults, and two with cyber-attacks, were conducted to generate datasets containing detailed network traffic and system states. A joint classification framework, incorporating a network analyzer and a physical HVAC fault detector, was proposed to automatically detect cyber-physical abnormalities on BAS at both the network and the physical HVAC levels. The network analyzer comprises a conditional random fields (CRF) based command validator and a statistics-based detection strategy. The fault detector employs a weather and schedule-based pattern matching and feature-based principal component analysis (WPM-FPCA) method. Evaluation of the classification using four metrics from the multi-class confusion matrix revealed an average accuracy of 90.2%, recall of 89.7%, precision of 88.5% and F1-score of 89.2%. Finally, these results demonstrate that the proposed joint classification framework can effectively differentiate between specific types of cyber-attacks (e.g., device reinitialization attack, network Denial-of-Service attack) and physical faults (e.g., air handling unit operational fault, cooling coil valve stuck) in real time for improved building energy management.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2018

A Deterministic and Probabilistic Framework for Forecasting of Time-Series Damage States and Associated End-of-lIfe of a Pressurizer Water Reactor Surge Line under Design-Basis and Grid-Load-Following Loading Conditions

Argonne National Laboratory in support of the Department of Energy’s Light Water Reactor Sustainability (LWRS) program. The present semi-annual report covers the period between May 2018 and September 2018. In this report, we present the work performed to further improve the capability for structural integrity prediction. Part of this work focused on validating a computational fluid dynamics model for modeling thermal stratification. We also present finite element (FE) model based thermal-mechanical stress analysis of a pressurized water reactor (PWR) surge line under grid-load-following loading condition. In addition, we present work related to end-of-life prediction of a PWR surge line based on ASME code and NUREG-6909 deterministic approaches under design-basis and grid-load-following conditions. Our research in this report period also majorly involved with the development of a strategy for probabilistic fatigue life estimation. First, we applied a Weibull probabilistic modeling approach based on end-of-life fatigue data (traditional S~N data under in-air and PWR water conditions) to estimate the probabilistic life for a given strain amplitude. Second, we developed a data analytics based Markov-Chain-Monte-Carlo (MCMC) model for probabilistic modeling of time-series fatigue damage states and the associated probabilistic end-of-life of a component. Unlike the Weibull type approach, the MCMC type may not require a large number of fatigue tests for estimating the end-of-life for given loading and environmental conditions. We demonstrated the use of this framework through symmetric fatigue tests (stress- or strain-ontrolled tests either under in-air or PWR-water conditions) obtained from our earlier experiments. The MCMC model can use base time-series data either obtained from a model or fatigue experiment. Since experimental data are typically more accurate than model-calculated data, we used experiment-based time-series data to demonstrate the MCMC framework. However, when performing a complex experiment is not possible (due to time and costs involved), a model-based approach can be used to generate the time-series data. Similar model-based data are presented in our earlier report (e.g. for cyclic hardening damage states of a PWR surge line under constant amplitude and design-basis loading [1]). However, for the MCMC work in this report, we used experiment-based time-series damage states obtained from fatigue experiments under design-basis and grid-load-following loading conditions. The experimental loading inputs were based on thermal-mechanical FE model results of a typical PWR surge line. Then, we used the experimental time-series damage state data for forecasting the probabilistic time-series damage states and end-of-life of the PWR surge line. Based on this MCMC modeling strategy, and with the assumed loading, environment and thermal-mechanical boundary conditions, we have estimated that a 316SS-pure base metal PWR surge line would survive at least 159 years with zero failure probability.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2021

Scalable Technologies Achieving Risk-Informed Condition-Based Predictive Maintenance Enhancing the Economic Performance of Operating Nuclear Power Plants

The primary objective of the research presented in this report is to develop scalable technologies that are deployable across plant assets and across the nuclear fleet to achieve risk-informed predictive maintenance (PdM) strategies at commercial nuclear power plants (NPPs). Over the years, the nuclear fleet has relied on labor-intensive and time-consuming preventive maintenance (PM) programs, driving up operation and maintenance (O&M) costs to achieve high capacity factors. A well-constructed risk-informed PdM approach for an identified plant asset has been developed in this research, taking advantage of advancements in data analytics, machine learning (ML), artificial intelligence (AI), physics-informed modeling, and visualization. These technologies would allow commercial NPPs to reliably transition from current labor-intensive PM programs to a technology driven PdM program, eliminating unnecessary O&M costs. The work presented in the report is being developed as part of a collaborative research effort between Idaho National Laboratory and Public Service Enterprise Group Nuclear, LLC. This report (1) reflects the results of work by LWRS Program researchers with PSEG, Nuclear LLC-owned Salem and Hope Creek Nuclear Power Plants; (2) presents utilization of circulating water system (CWS) heterogeneous data and fault modes from both the Salem and Hope Creek nuclear power plant sites to develop salient fault signatures associated with each fault mode; (3) describes the integration of component-level predictive models into a robust system-level model enabled by the federated-transfer learning; (4) describes the development of physics-informed model of circulating water pump and motor; (5) develops a scalable risk and economic model; and (6) outlines the development of a user-centric visualization application. The outcomes presented in this report lays the foundation and provides a much-needed technical basis to focus on explainability and trustworthiness of ML and AI-based technologies, as part of future research.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2017

Helium Ion Microscopy for Imaging and Quantifying Porosity at the Nanoscale

Nanoporous materials are essential components in a vast number of applications from energy to drug delivery and to agriculture. Yet, the number of ways to analytically quantify the salient features of these materials, for example: surface structure, pore shape, and size, remain limited. The most common approach is gas absorption, where volumetric gas absorption and desorption are measured. This technique has some fundamental drawbacks such as low sample throughput and a lack of direct surface visualization. In this work, we demonstrate Helium Ion Microscopy (HIM) as a tool for imaging and quantification of pores in industrially relevant SiO 2 catalyst supports. We start with the fundamental principles of ion-sample interaction, and build on this knowledge to experimentally observe and quantify surface pores by using the HIM and image data analytics. We contrast our experimental results to gas absorption and demonstrate full statistical agreement between two techniques. The principles surrounding the theoretical, experimental, and analytical framework presented herein offer an automated framework for visualization and quantification of pore structures in a wide variety of materials.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2018

Time resolved surface photovoltage measurements using a big data capture approach to KPFM

Optoelectronic behavior in materials such as organic/inorganic hybrid perovskites depend on a complex interplay between fast (electronic) and slower (ionic) processes. These processes are thought to be influenced by structural inhomogeneities (e.g. interfaces and grain boundaries) bringing forward the necessity for development of techniques capable of correlating nanostructure and photo-transport behavior. While Kelvin probe force microscopy (KPFM) is ideally suited to map surface potentials on relevant length scales, it lacks sufficient temporal resolution to extract the meaningful system dynamics. Here, we develop a time resolved surface photovoltage (SPV) measurement based on full information capture of the photodetector stream during open loop KPFM operation. G-Mode, or G-KPFM allows quantification of SPV with microsecond temporal and nanoscale spatial resolution. Using this technique, we observe concurrent spatial and fast temporal variations in the SPV generated across a methylammonium lead bromide (MAPbBr3) thin film, a possible indicator relating microstructure with heterogenous photo-transport behavior. Here, we further demonstrate the advantage of adopting big data analytics including unsupervised clustering methods to quickly discern spatial variability in the information rich SPV dataset. Beyond G-KPFM, such clustering methods will be useful for interpretation of the multidimensional datasets arising from the growing number of time resolved KPFM approaches now available.

36 MATERIALS SCIENCE↗
DOE OSTI2017

NBSR reactor control room upgrade - 215

The maintenance and modernization of the National Institute of Standards and Technology (NIST) Center for Neutron Research (NCNR) reactor, known as the NBSR, is an ongoing process to endure until the eventual decommissioning of the reactor. As part of the renovation effort, the reactor control system is to be upgraded. Following some initial outsourcing efforts, the NBSR reactor control system upgrade has been restructured to be implemented by in-house means. The beginning stage of the upgrade involved the development of a simulator environment based on a PC system and digitization of the majority of nuclear reactor monitoring data. The NBSR reactor surveillance data consists of approximately thirteen hundred input/output points, including analog/digital sensor signals, signal conditioners, and relay logic. The aim of the current phase is to implement a new reactor data acquisition and display system (RDS) and lay out the infrastructure for the upgrade of the reactor control system (RCS). Key design criteria are to implement a reliable, redundant, diverse, modular and candid system where any complex digital manipulations are taken outside of the control room. Basically, the data analytics and custom codes are separated from reactor operation control and decision making logic. Furthermore, the new RDS is designed such that it could be maintained internally by NCNR personnel. The RDS is unique in design and modularity, minimizing custom codes and testing, making it ideal for research reactors. (authors)

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2019

FY2018 Analysis Annual Progress Report

During fiscal year 2018 (FY 2018), the U.S. Department of Energy Vehicle Technologies Office (VTO) funded analysis projects supportive of VTO’s goals to pursue early stage research in vehicle and mobility system technologies to improve energy efficiency, increase energy reliability and security, improve transportation affordability, and promote economic growth. VTO analysis projects result in a foundation of data, analytical models, and applied analyses that provide insights into critical transportation energy problems and assist in research investment prioritization and portfolio planning.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2022

Adaptive elasticity policies for staging-based in situ visualization

In situ processing aims to alleviate the growing gap between computation and I/O capabilities by performing data processing close to the data source. In situ processing is widely used to process data generated by multiple data sources, including observation data from edge devices or scientific observational facilities and the simulation data generated by scientific computation on a high-performance computing (HPC) platform. For a scientific workflow that is run on an HPC platform and composed of a simulation program and an in situ data analytics or visualization (abbreviated as ana/vis) task, there is an implicit assumption that the computing resources assigned to the workflow keep static during the workflow execution. However, with the converging trend between the HPC and cloud computing platform, running the in situ ana/vis task in an elastic way is promising to decrease its overhead and improve its resource utilization rate. Resource elasticity represents the ability to change resource configurations such as the number of computing nodes/processes during workflow execution. An elastic job may dynamically adjust resource configurations; it may use a few resources at the beginning and more resources toward the end of the job when interesting data appear. However, it is hard to predict a priori how many computing nodes/processes need to be added/removed during the workflow execution to adapt to changing workflow needs. How to efficiently guide elasticity operations, such as growing or shrinking the number of processes used for in situ analysis during workflow execution, is an open-ended research question. In this article, we present adaptive elasticity policies that adopt workflow runtime information collected during workflow execution to predict how to trigger the addition/removal of processes in order to minimize in situ processing overhead. Taking in situ visualization tasks as an example, we integrate the presented elasticity policies into a staging-based elastic workflow and evaluate its efficiency in multiple elasticity scenarios. Compared with the situation without elasticity or with a static elasticity policy that uses a fixed number of processes for each rescaling operation, the adaptive elasticity policy can save overhead in finding a proper resource configuration and improve resource utilization efficiency. Furthermore, one experiment illustrates that the adaptive elasticity policy saves 41% of core-hours compared with the situation without the resource elasticity.

97 MATHEMATICS AND COMPUTING↗