Search NASA⌕ Search

SEARCH · Search NASA

Results for “intelligence analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 739 records · Page 41
All sourcesNASA NTRSDOE OSTI
DOE OSTI2016

NDMAS implementation and data qualification for accident tolerant fuel experiments

In order to meet quality and regulatory requirements associated with the U.S. Department of Energy Accident Tolerant Fuels (ATF) experiments, all data and documentation associated with ATF component fabrication, characterization, traceability, testing and post irradiation examination are being managed using the Nuclear Data Management and Analysis System (NDMAS). NDMAS melds commercial off-the-shelf software to provide the full range of needed functionality. The core component of NDMAS is the SAS Enterprise Business Intelligence software, which provides data manipulation, statistical and graphical capabilities, and delivery of live content to web pages. Content delivery to both Idaho National Laboratory researchers and external users via a web interface is managed using SharePoint, which is access-controlled. Additionally, scripts, macros, and commercial wrappers may be used or written by the NDMAS team to control interfaces between SAS and the structured query language server, where much of the data reside. ATF experiment data and documentation stored in the NDMAS database may be utilized to support Nuclear Regulatory Commission licensing of ATF concepts for commercial Light Water Reactor demonstration. Therefore, Data Qualification Plans are being developed to screen and tag all document and data input for ATR water loop testing, Transient Reactor Testing (TREAT), and post irradiation examination. A project-wide document library has been created in the ATF NDMAS SharePoint Site for general unrestricted document and information access. Restricted libraries have also been created to allow individual customer access to restricted documents. Data from ATR water loop instrumented lead experimentation, TREAT, and post-irradiation examination will be available to program participants on the SharePoint site as they are collected. This offers a unique capability for team collaboration while these tests are being conducted. (authors)

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2024

A data-driven framework for predicting machining stability: employing simulated data, operational modal analysis, and enhanced transfer learning

Chatter, a self-excited vibration phenomenon, presents a significant challenge in machining operations, particularly in high-speed milling, where it can degrade tool life, reduce material removal efficiency, and compromise workpiece quality. Addressing this challenge requires a reliable predictive model that can accommodate the complex dynamics of various machining scenarios. This study introduces a novel, data-driven approach to predicting machining stability, leveraging over 140,000 simulated datasets and employing advanced techniques such as operational modal analysis (OMA), enhanced transfer learning (TL), and receptance coupling substructure analysis (RCSA). By integrating these methodologies, the framework effectively classifies and predicts chatter across diverse operational modes, achieving robust and accurate outcomes. Our model utilizes a Random Forest (RF) classifier trained with the comprehensive dataset, which demonstrates substantial improvements in both predictive accuracy and robustness. Specifically, the RF model achieved an accuracy rate of 85%, an area under the curve (AUC) of 0.90, and an F1 score of 0.88, underscoring its capability to adapt to varying machining configurations. These results highlight the framework’s potential to enhance operational efficiency and machining quality by providing reliable chatter predictions across a broad range of machining parameters. In conclusion, this research thus offers a significant advancement in predictive maintenance for machining processes, enabling more stable and efficient manufacturing operations.

42 ENGINEERING↗
NASA NTRS1987

A conceptual framework for intelligent real-time information processing

By combining artificial intelligence concepts with the human information processing model of Rasmussen, a conceptual framework was developed for real time artificial intelligence systems which provides a foundation for system organization, control and validation. The approach is based on the description of system processing terms of an abstraction hierarchy of states of knowledge. The states of knowledge are organized along one dimension which corresponds to the extent to which the concepts are expressed in terms of the system inouts or in terms of the system response. Thus organized, the useful states form a generally triangular shape with the sensors and effectors forming the lower two vertices and the full evaluated set of courses of action the apex. Within the triangle boundaries are numerous processing paths which shortcut the detailed processing, by connecting incomplete levels of analysis to partially defined responses. Shortcuts at different levels of abstraction include reflexes, sensory motor control, rule based behavior, and satisficing. This approach was used in the design of a real time tactical decision aiding system, and in defining an intelligent aiding system for transport pilots.

Schudy, Robert↗
DOE OSTI2015

Towards intelligent diagnostic system employing integration of mathematical and engineering model

The development of medical diagnostic system has been one of the main research fields during years. The goal of the medical diagnostic system is to place a nosological system that could ease the diagnostic evaluation normally performed by scientists and doctors. Efficient diagnostic evaluation is essentials and requires broad knowledge in order to improve conventional diagnostic system. Several approaches on developing the medical diagnostic system have been designed and tested since the earliest 60s. Attempts on improving their performance have been made which utilizes the fields of artificial intelligence, statistical analyses, mathematical model and engineering theories. With the availability of the microcomputer and software development as well as the promising aforementioned fields, medical diagnostic prototypes could be developed. In general, the medical diagnostic system consists of several stages, namely the 1) data acquisition, 2) feature extraction, 3) feature selection, and 4) classifications stages. Data acquisition stage plays an important role in converting the inputs measured from the real world physical conditions to the digital numeric values that can be manipulated by the computer system. One of the common medical inputs could be medical microscopic images, radiographic images, magnetic resonance image (MRI) as well as medical signals such as electrocardiogram (ECG) and electroencephalogram (EEG). Normally, the scientist or doctors have to deal with myriad of data and redundant to be processed. In order to reduce the complexity of the diagnosis process, only the significant features of the raw data such as peak value of the ECG signal or size of lesion in the mammogram images will be extracted and considered in the subsequent stages. Mathematical models and statistical analyses will be performed to select the most significant features to be classified. The statistical analyses such as principal component analysis and discriminant analysis as well as mathematical model of clustering technique have been widely used in developing the medical diagnostic systems. The selected features will be classified using mathematical models that embedded engineering theory such as artificial intelligence, support vector machine, neural network and fuzzy-neuro system. These classifiers will provide the diagnostic results without human intervention. Among many publishable researches, several prototypes have been developed namely NeuralPap, Neural Mammo, and Cervix Kit. The former system (NeuralPap) is an automatic intelligent diagnostic system for classifying and distinguishing between the normal and cervical cancerous cells. Meanwhile, the Cervix Kit is a portable Field-programmable gate array (FPGA)-based cervical diagnostic kit that could automatically diagnose the cancerous cell based on the images obtained during sampling test. Besides the cervical diagnostic system, the Neural Mammo system is developed to specifically aid the diagnosis of breast cancer using a fine needle aspiration image.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
NASA NTRS1992

A demonstration of an intelligent control system for a reusable rocket engine

An Intelligent Control System for reusable rocket engines is under development at NASA Lewis Research Center. The primary objective is to extend the useful life of a reusable rocket propulsion system while minimizing between flight maintenance and maximizing engine life and performance through improved control and monitoring algorithms and additional sensing and actuation. This paper describes current progress towards proof-of-concept of an Intelligent Control System for the Space Shuttle Main Engine. A subset of identifiable and accommodatable engine failure modes is selected for preliminary demonstration. Failure models are developed retaining only first order effects and included in a simplified nonlinear simulation of the rocket engine for analysis under closed loop control. The engine level coordinator acts as an interface between the diagnostic and control systems, and translates thrust and mixture ratio commands dictated by mission requirements, and engine status (health) into engine operational strategies carried out by a multivariable control. Control reconfiguration achieves fault tolerance if the nominal (healthy engine) control cannot. Each of the aforementioned functionalities is discussed in the context of an example to illustrate the operation of the system in the context of a representative failure. A graphical user interface allows the researcher to monitor the Intelligent Control System and engine performance under various failure modes selected for demonstration.

Musgrave, Jeffrey L.↗
DOE OSTI2025

Techno-economics of hydrocarbon fuel production and recyclables recovery from landfill-destined municipal solid waste: AI-enhanced materials recovery facility design

Sustainable aviation fuels (SAF) production from cellulosic paper fractions of municipal solid waste (MSW) destined for landfills has strong potential to advance environmental, social, and economic sustainability across the aviation and waste sectors. This study proposes an artificial intelligence-enabled material recovery facility (AI-MRF) design to efficiently characterize, separate, process, and convert recovered paper waste from MSW into intermediate chemicals and SAF. The AI-MRF, designed to process 233,091 metric tons of MSW annually, integrates smart manufacturing technologies including AI, visual and hyperspectral imaging, multi-sensor data, and traditional sorting systems. Well-characterized and sorted cellulosic paper waste was utilized for chemical and fuel production scenarios, while clean plastics, metals, and glass were considered for recycling. Conversion of paper waste into intermediate sugars achieved a net present value (NPV) of up to $\$67$ million. For sugar-to-SAF production scenarios, the minimum fuel selling price (MFSP) was calculated at $\$6.11$ per gasoline gallon equivalent (GGE) when excluding recyclable revenue, and $\$4.03$ per GGE when halving recyclable revenue. The MFSP was further reduced to $\$1.96$ per GGE when accounting for SAF sales and recyclables. Nationally, this approach could yield about 2 billion GGE of hydrocarbon fuel annually from available MSW in the United States.

09 BIOMASS FUELS↗
DOE OSTI2023

An autonomous laboratory for the accelerated synthesis of novel materials

To close the gap between the rates of computational screening and experimental realization of novel materials, we introduce the A-Lab, an autonomous laboratory for the solid-state synthesis of inorganic powders. This platform uses computations, historical data from the literature, machine learning (ML) and active learning to plan and interpret the outcomes of experiments performed using robotics. Over 17 days of continuous operation, the A-Lab realized 41 novel compounds from a set of 58 targets including a variety of oxides and phosphates that were identified using large-scale ab initio phase-stability data from the Materials Project and Google DeepMind. Synthesis recipes were proposed by natural-language models trained on the literature and optimized using an active-learning approach grounded in thermodynamics. Analysis of the failed syntheses provides direct and actionable suggestions to improve current techniques for materials screening and synthesis design. The high success rate demonstrates the effectiveness of artificial-intelligence-driven platforms for autonomous materials discovery and motivates further integration of computations, historical knowledge and robotics.

36 MATERIALS SCIENCE↗
NASA NTRS1988

Autonomous control systems - Architecture and fundamental issues

A hierarchical functional autonomous controller architecture is introduced. In particular, the architecture for the control of future space vehicles is described in detail; it is designed to ensure the autonomous operation of the control system and it allows interaction with the pilot and crew/ground station, and the systems on board the autonomous vehicle. The fundamental issues in autonomous control system modeling and analysis are discussed. It is proposed to utilize a hybrid approach to modeling and analysis of autonomous systems. This will incorporate conventional control methods based on differential equations and techniques for the analysis of systems described with a symbolic formalism. In this way, the theory of conventional control can be fully utilized. It is stressed that autonomy is the design requirement and intelligent control methods appear at present, to offer some of the necessary tools to achieve autonomy. A conventional approach may evolve and replace some or all of the `intelligent' functions. It is shown that in addition to conventional controllers, the autonomous control system incorporates planning, learning, and FDI (fault detection and identification).

Antsaklis, P. J.↗
NASA NTRS1996

Method of Testing and Predicting Failures of Electronic Mechanical Systems

A method employing a knowledge base of human expertise comprising a reliability model analysis implemented for diagnostic routines is disclosed. The reliability analysis comprises digraph models that determine target events created by hardware failures human actions, and other factors affecting the system operation. The reliability analysis contains a wealth of human expertise information that is used to build automatic diagnostic routines and which provides a knowledge base that can be used to solve other artificial intelligence problems.

Iverson, David L.↗
NASA NTRS2026

Machine Learning for Predicting Team Functioning in HERA Missions

Team functioning is integral to success in future long term space exploration missions. Proactively detecting declines in team functioning can mitigate conflict and ensure mission success. This project developed a speech-based artificial intelligence (AI) system that unobtrusively predicts degradation in team functioning, including performance and cohesion, in the Human Exploration Research Analog (HERA) Campaigns 4 and 5. The AI system conducted automated analysis of the prosodic (tone of voice) and linguistic (language content) components of speech, modeling interpersonal dynamics at both the turn-taking and day-wide levels. We investigated team functioning via observing structured interactions (i.e., multi-mission space exploration vehicle-extra vehicular activity [MMSEV-EVA], team interaction battery [TIB]) and unstructured interactions before the MMSEV-EVA task. We developed machine learning models to predict team functioning (objective task accuracy, self reported team efficacy and self reported team cohesion) by analyzing OpenSmile acoustic features, linguistic descriptors extracted via the linguistic inquiry and word count (LIWC) dictionary, and semantic embeddings. In the TIB, static models using logistic regression and random forests were not able to predict task accuracy, but predicted team efficacy and cohesion during both the decision making and relational tasks to a moderate level (60-70%). Majority voting on the individual turns to predict day long team efficacy further increased accuracies (70-80%). Finally, long short-term memory (LSTM) models showed the best performance across all variables (80-91%), including task performance. In the MMSEV-EVA, static models achieved an accuracy of 60% with majority voting, which increased to 80% through the incorporation of mission day as a variable, accounting for the learning effect. A key finding across both tasks was the "team-dependent" nature of these interactions; models achieved much higher accuracy when trained on prior days of the same team's data rather than attempting to generalize across entirely different teams, with even 1-2 days of prior data per team achieving 5-15% improvement over team-independent models. In addition, the incorporation of pre-task data from the same team also improves model performance, e.g., incorporating data from the decision-making task of the TIB, which preceded the relational task, improved the prediction of team efficacy and cohesion during the latter. We compared model performance when trained on machine-generated data compared to data that had been further corrected by human annotators. Overall, models trained on human-corrected data exhibited a modest improvement in performance, particularly when acoustic features were used. We found no significant correlation between word error rate (WER) and model accuracy (r(55) = -0.08, p = 0.51), but model’s accuracy was significantly higher for medium/high quality transcription (0.74 (SD = 0.48)) compared to the low-quality group (0.64 (SD = 0.36)) (t(63)=2.82, p = 0.006). Based on these, several design recommendation emerge, that could inform Standards at NASA. Models predicting team functioning should incorporate at least one to two days of historical interaction data, include brief pre-task discussions, and explicitly model temporal learning effects, especially for longer operational tasks. Minimum quality standards for automated speech-processing pipelines are needed, given the performance gains observed with manually corrected acoustic data. Finally, systems should leverage both acoustic features and language embeddings in complementary ways, with modality choices and fusion strategies tailored to mission context, task demands, and data quality requirements.

Shrivatsa Mishra↗
DOE OSTI2022

Toward autonomous laboratories: Convergence of artificial intelligence and experimental automation

The ever-increasing demand for novel materials with superior properties inspires retrofitting traditional research paradigms in the era of artificial intelligence and automation. An autonomous experimental platform (AEP) has emerged as an exciting research frontier that achieves full autonomy via integrating data-driven algorithms such as machine learning (ML) with experimental automation in the material development loop from synthesis, characterization, and analysis, to decision making. In this review, we started with a primer to describe how to develop data-driven algorithms for solving material problems. Then, we systematically summarized recent progress on automated material synthesis, ML-enabled data analysis, and decision-making. Finally, we discussed the challenges and opportunities in an endeavor to develop the next-generation AEP for ultimately realizing an autonomous or self-driving laboratory. In conclusion, this review will provide insights for researchers aiming to learn the frontier of ML in materials science and deploy AEP in their labs for accelerating material development.

36 MATERIALS SCIENCE↗
DOE OSTI2024

Computing with a Chemical Reservoir

Contemporary computation is expensive, with large language models and artificial intelligence becoming more common in daily life. However, high-performance computing is reaching the limits in speed and energy expenditure, and domain science requires ever-increasing computational capacity, with simulations and data analysis pipelines ever-growing in complexity. As we progress towards post-exascale computation, with the associated high energy costs, new methods of energy-conscious computation are required. Novel analog and hybrid digital-analog systems can overcome these challenges, and chemical reactions offer a promising avenue. Computers based on chemistry can provide compact desktop devices with immense computational power. These devices are readily scalable by considering greater reaction systems or vessels, meeting the high-performance requirements for scientific workflows. In this article, we present ChemComp, a compilation pipeline for the conversion of ordinary differential equations into implementable chemical reactions. We then demonstrate the solving capabilities of ChemComp by emulating a potential chemical reservoir device. We leverage the multi-layer intermediate representation (MLIR) compiler framework to implement an expressive chemical reaction abstraction and propose a path for chemical reaction networks (CRNs) to represent mathematical problems effectively. Combined, we demonstrate a potential workflow that can harness chemistry’s computing power to create energy-efficient, high-performance computation systems for contemporary computing needs.

artificial intelligence↗
NASA NTRS1999

Aerospace Engineering Systems and the Advanced Design Technologies Testbed Experience

Continuous improvement of aerospace product development processes is a driving requirement across much of the aerospace community. As up to 90% of the cost of an aerospace product is committed during the first 10% of the development cycle, there is a strong emphasis on capturing, creating, and communicating better information (both requirements and performance) early in the product development process. The community has responded by pursuing the development of computer-based systems designed to enhance the decision-making capabilities of product development individuals and teams. Recently, the historical foci on sharing the geometrical representation and on configuration management are being augmented: 1) Physics-based analysis tools for filling the design space database; 2) Distributed computational resources to reduce response time and cost; 3) Web-based technologies to relieve machine-dependence; and 4) Artificial intelligence technologies to accelerate processes and reduce process variability. The Advanced Design Technologies Testbed (ADTT) activity at NASA Ames Research Center was initiated to study the strengths and weaknesses of the technologies supporting each of these trends, as well as the overall impact of the combination of these trends on a product development event. Lessons learned and recommendations for future activities are reported.

VanDalsem, William R.↗
NASA NTRS2003

Flight Investigation of Prescribed Simultaneous Independent Surface Excitations for Real-Time Parameter Identification

Near real-time stability and control derivative extraction is required to support flight demonstration of Intelligent Flight Control System (IFCS) concepts being developed by NASA, academia, and industry. Traditionally, flight maneuvers would be designed and flown to obtain stability and control derivative estimates using a postflight analysis technique. The goal of the IFCS concept is to be able to modify the control laws in real time for an aircraft that has been damaged in flight. In some IFCS implementations, real-time parameter identification (PID) of the stability and control derivatives of the damaged aircraft is necessary for successfully reconfiguring the control system. This report investigates the usefulness of Prescribed Simultaneous Independent Surface Excitations (PreSISE) to provide data for rapidly obtaining estimates of the stability and control derivatives. Flight test data were analyzed using both equation-error and output-error PID techniques. The equation-error PID technique is known as Fourier Transform Regression (FTR) and is a frequency-domain real-time implementation. Selected results were compared with a time-domain output-error technique. The real-time equation-error technique combined with the PreSISE maneuvers provided excellent derivative estimation in the longitudinal axis. However, the PreSISE maneuvers as presently defined were not adequate for accurate estimation of the lateral-directional derivatives.

Moes, Timothy R.↗
NASA NTRS2011

Network-Capable Application Process and Wireless Intelligent Sensors for ISHM

Intelligent sensor technology and systems are increasingly becoming attractive means to serve as frameworks for intelligent rocket test facilities with embedded intelligent sensor elements, distributed data acquisition elements, and onboard data acquisition elements. Networked intelligent processors enable users and systems integrators to automatically configure their measurement automation systems for analog sensors. NASA and leading sensor vendors are working together to apply the IEEE 1451 standard for adding plug-and-play capabilities for wireless analog transducers through the use of a Transducer Electronic Data Sheet (TEDS) in order to simplify sensor setup, use, and maintenance, to automatically obtain calibration data, and to eliminate manual data entry and error. A TEDS contains the critical information needed by an instrument or measurement system to identify, characterize, interface, and properly use the signal from an analog sensor. A TEDS is deployed for a sensor in one of two ways. First, the TEDS can reside in embedded, nonvolatile memory (typically flash memory) within the intelligent processor. Second, a virtual TEDS can exist as a separate file, downloadable from the Internet. This concept of virtual TEDS extends the benefits of the standardized TEDS to legacy sensors and applications where the embedded memory is not available. An HTML-based user interface provides a visual tool to interface with those distributed sensors that a TEDS is associated with, to automate the sensor management process. Implementing and deploying the IEEE 1451.1-based Network-Capable Application Process (NCAP) can achieve support for intelligent process in Integrated Systems Health Management (ISHM) for the purpose of monitoring, detection of anomalies, diagnosis of causes of anomalies, prediction of future anomalies, mitigation to maintain operability, and integrated awareness of system health by the operator. It can also support local data collection and storage. This invention enables wide-area sensing and employs numerous globally distributed sensing devices that observe the physical world through the existing sensor network. This innovation enables distributed storage, distributed processing, distributed intelligence, and the availability of DiaK (Data, Information, and Knowledge) to any element as needed. It also enables the simultaneous execution of multiple processes, and represents models that contribute to the determination of the condition and health of each element in the system. The NCAP (intelligent process) can configure data-collection and filtering processes in reaction to sensed data, allowing it to decide when and how to adapt collection and processing with regard to sophisticated analysis of data derived from multiple sensors. The user will be able to view the sensing device network as a single unit that supports a high-level query language. Each query would be able to operate over data collected from across the global sensor network just as a search query encompasses millions of Web pages. The sensor web can preserve ubiquitous information access between the querier and the queried data. Pervasive monitoring of the physical world raises significant data and privacy concerns. This innovation enables different authorities to control portions of the sensing infrastructure, and sensor service authors may wish to compose services across authority boundaries.

Figueroa, Fernando↗
DOE OSTI2020

A Hybrid AI/ML and Computational Mechanics Based Approach for Time-Series State and Fatigue Life Estimation of Nuclear Reactor Components

Environmental fatigue modeling is a complex problem due to multiple failure modes and their intermixing. The failure modes are function of various underlying causes in addition to the corrosive effect of reactor coolant environment. Some of the major causes are time-dependence of material associated with cyclic loading, load sequence effect associated with random/variable amplitude loading, effect of strain amplitude and rates, effect of varying temperature (along both temporal and spatial directions) and the effect of mean strain and stress. The nonlinear intermixing of failure modes associated with above mentioned causing parameters makes the environmental fatigue modeling is a challenging task. Because of this challenge, fatigue is traditionally being modeled based on experimental data. However, test based empirical approach often requires hundreds of fatigue tests to model the above-mentioned intermixing failure causes even for a single material system. The problem is further exaggerated for reactor component made from multi-material systems such as made from both carbon and stainless-steel base metals and their similar and dissimilar metal welds. With the difficulty of conducting hundreds of fatigue tests to capture the above-mentioned intermixing failure causes, fatigue modeling approaches often depends on empirical models based on limited available test data such as available through ASME code and NUREG 6909. However, these limited test-data-based models may not be enough to accurately predict the life of reactor components. Accurate prediction of life of reactor component would become a necessity, particularly when the license of the reactors to be extended for long-term-operation (LTO) that is for well beyond its original design life of 40 years. The requirement of extending the license of reactor under LTO requires hundreds of fatigue tests to be conducted to understand the mechanism associated with the above-mentioned interdependent failure causes. However, conducting large number of fatigue tests is not a feasibility due to the cost involved. To address this issues Argonne National Laboratory (ANL) with the sponsorship of DOE Light Water Reactor Sustainability (LWRS) program trying to develop a hybrid predictive modeling approach. This is based on limited experiment-data, Artificial-intelligence (AI) – Machine-Learning (ML) - Deep-Learning (DL) based techniques and Multiphysics-computational-mechanics based modeling tools. The hybrid approach not-only can improve the accuracy of the existing stress analysis and fatigue modeling approach but also can reduce the over-dependency on test-based approach. Towards this goal following are some of the major contributions based on ANL’s FY-20 environmental fatigue modeling activities: 1) A cyclic plasticity material model database for 82/182 dissimilar metal weld, which can be readily shared with US nuclear industry and regulatory agency on request. 2) A well validated analytical modeling methodology to perform cycle-by-cycle stress prediction under both constant amplitude fatigue loading and variable amplitude fatigue loading (with load-sequence effect). 3) An AI/ML/DL based methodology to predict unmeasurable cyclic strain based on other available sensor signals. This type of approach can be used for estimating strain in real reactor components from other sensor readings. 4) An AI/ML based approach to improve the US capability on environmental fatigue testing. This is by improving ANL’s existing environmental fatigue testing capacity to conduct ASME required strain-controlled tests (by controlling strain amplitudes and its rate), while not measuring the strain (due to the difficulty of placing an extensometer in a narrow autoclave in a PWR-water-test system). 5) A simulation and experiment based probabilistic modeling methodology for time-series fatigue state and life estimation of reactor metal such as dissimilar metal weld.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2023

Integrated End-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows

This report details recent progress for the ASCR funded project “Integrated End-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows”. We refer to the project as IPPD/2, reflecting the 2017 renewal under expanded scope and partners In IPPD/2, we increased our research scope to include data motion. We are focusing on three major aspects: a) observe how data is generated, distributed, and used; b) analyze how data is (repeatedly) consumed with a focus both on repeated patterns and anomalies; and c) explore how to optimize data motion. This new work on data motion will augment and complement IPPD/2’s research that focused on the computational aspects of tasks. We leverage and extend our existing tools and demonstrate our work on the Belle II workflow suite as well as on workflows from NSLS-II. The highlights of our work are as follows: Provenance for Workflows: Provenance is used to provide information enabling quality control, re-run computational workflows, and reproduce results. IPPD/2 has been building a scalable provenance management system that enables the capture of provenance from the high-level workflow through all relevant system levels in one integrated environment. Leveraging this work, our recent efforts have included using provenance as an enabling technique. Workload characterization: Leveraging provenance and analysis, we characterize data movement within network, storage, and memory over a variety of workloads. This characterization enables an understanding by performance analysts and application developers of the range of behaviors that could be expected. Performance Prediction for Workflows: The goal of modeling distributed workflows is to understand performance bottlenecks and enable more intelligent task scheduling to optimize selected metrics of interest (e.g., task throughput or output data rate). IPPD/2 has utilized both analytical and AI/ML modeling methodologies for performance modeling. Advanced Scheduling and Fault Modeling for Workflows: Scheduling of large-scale scientific workflows on geographically distributed resources is a challenging problem. To improve workflow throughput, we combined novel scheduling algorithms with task predictions from performance modeling and fault modeling. Dynamically Alleviating Bottlenecks in Workflows: Exploiting our provenance, analysis, and modeling efforts, we have explored and developed several techniques for dynamically detecting and alleviating bottlenecks in data movement. In particular, we have spent considerable effort demonstrating our techniques on production-like workflow configurations.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Rare Events via Cross-Entropy Population Monte Carlo

Rare events are events that happen with very low frequency. Estimating rare event probabilities using Monte Carlo techniques is computationally expensive, often to the point of intractability, and special methods are required. Importance sampling (IS) is a well known technique that uses a proposal distribution in place of a target distribution to lower the variance of the estimator. Key to the success of IS methods is the choice of a proposal distribution, or the parameters governing the distribution. Adaptive importance sampling improves the parameters of a family or population of proposal distributions iteratively through trials. We present a novel cross-entropy population Monte Carlo algorithm, which adapts the parameters of proposals through the cross-entropy method. The proposed method stands apart from previous work in that we are not optimizing a mixture distribution. Instead, we leverage deterministic mixture weights and optimize the distributions individually through a reinterpretation of the typical derivation of the cross-entropy method. Demonstrations on rare event examples show that the algorithm can outperform existing resampling based population Monte Carlo methods, especially for higher-dimensional problems. Finally, we also demonstrate efficacy on a conjunction analysis problem.

97 MATHEMATICS AND COMPUTING↗