Search NASA⌕ Search

SEARCH · Search NASA

Results for “Workflow Management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 199 records · Page 11

Three practical workflow schedulers for easy maximum parallelism

Runtime scheduling and workflow systems are an increasingly popular algorithmic component in HPC because they allow full system utilization with relaxed synchronization requirements. There are so many special-purpose tools for task scheduling, one might wonder why more are needed. Use cases seen on the Summit supercomputer needed better integration with MPI and greater flexibility in job launch configurations. Preparation, execution, and analysis of computational chemistry simulations at the scale of tens of thousands of processors revealed three distinct workflow patterns. A separate job scheduler was implemented for each one using extremely simple and robust designs: file-based, task-list based, and bulk-synchronous. Comparing to existing methods shows unique benefits of this work, including simplicity of design, suitability for HPC centers, short startup time, and well-understood per-task overhead. All three new tools have been shown to scale to full utilization of Summit, and have been made publicly available with tests and documentation. This work presents a complete characterization of the minimum effective task granularity for efficient scheduler usage scenarios. Here, these schedulers have the same bottlenecks, and hence similar task granularities as those reported for existing tools following comparable paradigms.

97 MATHEMATICS AND COMPUTING↗

Biofoundries: Principles, Tools, and Applications

This chapter aims to provide a broad overview of biofoundries and introduces the principles, concepts, and case studies. We first outline the underlying principles of the Design-Build-Test-Learn (DBTL) framework and the role of automation, digital integration, and standardization. The chapter then explores core biofoundry technologies including robotic liquid handlers, high-throughput analytical instruments, and digital infrastructure for data management and workflow scheduling. Case studies spanning DNA assembly, protein engineering, metabolic engineering, and mammalian cell culture demonstrate the practical applications of the biofoundries. Economic and societal impacts are assessed alongside current limitations. We discuss the emerging trends including artificial intelligence integration and cloud-based distributed facilities to highlight its potential for biotechnology and the bioeconomy.

Singh, Nilmani↗

Enabling machine learning-ready HPC ensembles with Merlin

With the growing complexity of computational and experimental facilities, many scientific researchers are turning to machine learning (ML) techniques to analyze large scale ensemble data. With complexities such as multi-component workflows, heterogeneous machine architectures, parallel file systems, and batch scheduling, care must be taken to facilitate this analysis in a high performance computing (HPC) environment. Here, we present Merlin, a workflow framework to enable large ML-friendly ensembles of scientific HPC simulations. By augmenting traditional HPC with distributed compute technologies, Merlin aims to lower the barrier for scientific subject matter experts to incorporate ML into their analysis. As a producer–consumer workflow model, Merlin enables multi-machine, cross-batch job, dynamically allocated yet persistent workflows capable of utilizing surge-compute resources. Key features of Merlin are a flexible HPC-centric interface, low per-task overhead, multi-tiered fault recovery, and a hierarchical sampling algorithm that allows for $\mathscr{O}$(N) task execution and $\mathscr{O}$(N ln N) task queuing to ensembles of millions of tasks. In addition to Merlin’s design, we test the algorithm’s performance in an HPC center and demonstrate the ability to enqueue 40 million simulations in 100 s, with a 30 millisecond per-task overhead that is independent of ensemble size. Finally, we describe some example applications that Merlin has enabled on leadership-class HPC resources, such as the ML-augmented optimization of nuclear fusion experiments and the calibration of infectious disease models to study the progression of and possible mitigation strategies for COVID-19.

97 MATHEMATICS AND COMPUTING↗

LCLS Big Data Handling – How I Learned to Stop Worrying and Love the Data Deluge

Advanced data and computing systems are vital to Linac Coherent Light Source (LCLS) operations, data interpretation and overall scientific productivity. The transition to MHz-era operation marks a fundamental change in scale that requires new infrastructure and architectures to link LCLS to the required scale of computing needed for scientific interpretation. The LCLS-II Data System meets big data challenges by implementing configurable data reduction that can adapt to multiple science areas, real-time analysis frameworks to provide visualization and fast feedback, and the ability to transfer data to local and remote computational facilities for near real time analysis at the appropriate scale. Feature extracted information generated in the data analysis pipeline - at the edge, local compute, or remote High-Performance Computing (HPC) resources - can be used to steer experiments and inform user decisions during beam time. Artificial Intelligence and Machine Learning (AI/ML) techniques present new opportunities to rapidly analyse large datasets and direct experiments, but create new challenges in scaling, adaptability, complexity, and trustworthiness. We describe how the LCLS-II Data System architecture addresses its data-driven challenges in the areas of data acquisition, data processing, data management, and workflow orchestration to decrease the overall time-to-science and provide a vision for future developments.

artificial intelligence↗

libEnsemble: A Library to Coordinate the Concurrent Evaluation of Dynamic Ensembles of Calculations

Almost all applications stop scaling at some point; those that don't are seldom performant when considering time to solution on anything but aspirational/unicorn resources. Recognizing these tradeoffs as well as greater user functionality in a near-term exascale computing era, we present libEnsemble, a library aimed at particular scalability- and capability-stretching uses. libEnsemble enables running concurrent instances of an application in dynamically allocated ensembles through an extensible Python library. Here, we highlight the structure, execution, and capabilities of the library on leading pre-exascale environments as well as advanced capabilities for exascale environments and beyond.

97 MATHEMATICS AND COMPUTING↗

ComStock™ 2024 Release 1 [SWR-19-33 and SWR-20-32]

ComStock™ is an NREL model of the U.S. commercial building stock. The model takes some building characteristics from the U.S. Department of Energy's (DOE's) Commercial Prototype Building Models and Commercial Reference Building. However, unlike many other building stock models, ComStock also combines these with a variety of additional public- and private-sector data sets. Collectively, this information provides high-fidelity building stock representation with a realistic diversity of building characteristics. This repository contains the source code used to build and execute ComStock models, including upgrade scenarios. In addition, the sampling of buildings characteristics used for the initial ComStock (V1.0) release is provided. The ComStock model is under active calibration and development, which is publicly visible on this repository. Execution of the ComStock workflow is managed through the buildstockbatch repository, a shared asset of ResStock™ and ComStock™ , specifically developed to scale to execution of tens of millions of simulations through multiple infrastructure providers. The dataset output from the initial ComStock (V1.0) release can be found at the accompanying ComStock data viewer website and additional information about ComStock found on the NREL Buildings Website. For more details about ongoing model development please consult the End Use Load Profiles website. ComStock is a direct result of the NREL residential stock modeling tool ResStock™ (recipient of a R&D100 award) and was inspired by the high-fidelity solar & storage adoption model dGen™. Additionally, this tool would not be possible without the decades of work undertaken by the OpenStudio® and EnergyPlus® visionaries and contributors, significant funding, feedback and support from the Los Angeles Department of Water and Power, and the Department of Energy's Building Technology Office ongoing support of and investment in building energy modeling software. is an analytic methodology for modeling the energy usage of the commercial building stock within the United States of America. The commercial building stock is represented through a sampling of complex probabilistic distributions of various features of interest for modeling energy usage within commercial buildings. Each sample from these distributions is converted into a building energy model based on the features of that specific sample. Each building energy model can be simulated as is, but additional changes can be made to the model through addition of energy conservation measures, component faults, or other desired alterations. The results of the simulations are then processed to provide insights for various stakeholders, including but not limited to policy makers, engineers, and marketers.

Horsey, Henry↗

Demystifying asynchronous I/O Interference in HPC applications

With increasing complexity of HPC workflows, data management services need to perform expensive I/O operations asynchronously in the background, aiming to overlap the I/O with the application runtime. However, this may cause interference due to competition for resources: CPU, memory/network bandwidth. The advent of multi-core architectures has exacerbated this problem, as many I/O operations are issued concurrently, thereby competing not only with the application but also among themselves. Furthermore, the interference patterns can dynamically change as a response to variations in application behavior and I/O subsystems (e.g. multiple users sharing a parallel file system). Without a thorough understanding, I/O operations may perform suboptimally, potentially even worse than in the blocking case. To fill this gap, here we investigate the causes and consequences of interference due to asynchronous I/O on HPC systems. Specifically, we focus on multi-core CPUs and memory bandwidth, isolating the interference due to each resource. Then, we perform an in-depth study to explain the interplay and contention in a variety of resource sharing scenarios such as varying priority and number of background I/O threads and different I/O strategies: sendfile, read/write, mmap/write underlining trade-offs. The insights from this study are important both to enable guided optimizations of existing background I/O, as well as to open new opportunities to design advanced asynchronous I/O strategies.

97 MATHEMATICS AND COMPUTING↗

Collaborative Project: The problem of bias in defining uncertainty in computationally enabled strategies for data-driven climate model development

The objective of the project is to develop strategies for better representing scientific sensibilities within statistical measures of model skill that then can be used within a Bayesian statistical framework for data-driven climate model development and improved measures of model scientific uncertainty. One of the thorny issues in model evaluation is quantifying the effect of biases on climate projections. While any bias is not desirable, only those biases that affect feedbacks affect scatter in climate projections. The effort at the University of Texas is to analyze previously calculated ensembles of CAM3.1 with perturbed parameters to discover how biases affect projections of global warming. The hypothesis is that compensating errors in the control model can be identified by their effect on a combination of processes and that developing metrics that are sensitive to dependencies among state variables would provide a way to select version of climate models that may reduce scatter in climate projections. Gabriel Huerta was responsible for leading the effort to develop statistical methods for evaluating these field dependencies. The UT effort will incorporate these developments into MECS, which is a set of python scripts being developed at the University of Texas for managing the workflow associated with data-driven climate model development over HPC resources. We regularly consult with Peter Gleckler (LLNL), Mark Taylor (SNL), and Rich Neale (NCAR) to provide additional insight into issues concerning CAM5 model development and multi-model evaluation of the AR5 archive. This grant was originally assigned to Indiana University where both the PI (Huerta) and the Postdoc (Nosedal) worked on the project from March 1, 2012-March 31, 2013.

97 MATHEMATICS AND COMPUTING↗

Collaborative Project: The problem of bias in defining uncertainty in computationally enabled strategies for data-driven climate model development. Final Technical Report.

The objective of the project is to develop strategies for better representing scientific sensibilities within statistical measures of model skill that then can be used within a Bayesian statistical framework for data-driven climate model development and improved measures of model scientific uncertainty. One of the thorny issues in model evaluation is quantifying the effect of biases on climate projections. While any bias is not desirable, only those biases that affect feedbacks affect scatter in climate projections. The effort at the University of Texas is to analyze previously calculated ensembles of CAM3.1 with perturbed parameters to discover how biases affect projections of global warming. The hypothesis is that compensating errors in the control model can be identified by their effect on a combination of processes and that developing metrics that are sensitive to dependencies among state variables would provide a way to select version of climate models that may reduce scatter in climate projections. Gabriel Huerta at the University of New Mexico is responsible for developing statistical methods for evaluating these field dependencies. The UT effort will incorporate these developments into MECS, which is a set of python scripts being developed at the University of Texas for managing the workflow associated with data-driven climate model development over HPC resources. This report reflects the main activities at the University of New Mexico where the PI (Huerta) and the Postdocs (Nosedal, Hattab and Karki) worked on the project.

58 GEOSCIENCES↗

Basic Energy Sciences Exascale Requirements Review. An Office of Science review sponsored jointly by Advanced Scientific Computing Research and Basic Energy Sciences, November 3-5, 2015, Rockville, Maryland

Computers have revolutionized every aspect of our lives. Yet in science, the most tantalizing applications of computing lie just beyond our reach. The current quest to build an exascale computer with one thousand times the capability of today’s fastest machines (and more than a million times that of a laptop) will take researchers over the next horizon. The field of materials, chemical reactions, and compounds is inherently complex. Imagine millions of new materials with new functionalities waiting to be discovered — while researchers also seek to extend those materials that are known to a dizzying number of new forms. We could translate massive amounts of data from high precision experiments into new understanding through data mining and analysis. We could have at our disposal the ability to predict the properties of these materials, to follow their transformations during reactions on an atom-by-atom basis, and to discover completely new chemical pathways or physical states of matter. Extending these predictions from the nanoscale to the mesoscale, from the ultrafast world of reactions to long-time simulations to predict the lifetime performance of materials, and to the discovery of new materials and processes will have a profound impact on energy technology. In addition, discovery of new materials is vital to move computing beyond Moore’s law. To realize this vision, more than hardware is needed. New algorithms to take advantage of the increase in computing power, new programming paradigms, and new ways of mining massive data sets are needed as well. This report summarizes the opportunities and the requisite computing ecosystem needed to realize the potential before us. In addition to pursuing new and more complete physical models and theoretical frameworks, this review found that the following broadly grouped areas relevant to the U.S. Department of Energy (DOE) Office of Advanced Scientific Computing Research (ASCR) would directly affect the Basic Energy Sciences (BES) mission need. Simulation, visualization, and data analysis are crucial for advances in energy science and technology. Revolutionary mathematical, software, and algorithm developments are required in all areas of BES science to take advantage of exascale computing architectures and to meet data analysis, management, and workflow needs. In partnership with ASCR, BES has an emerging and pressing need to develop new and disruptive capabilities in data science. More capable and larger high-performance computing (HPC) and data ecosystems are required to support priority research in BES. Continued success in BES research requires developing the next-generation workforce through education and training and by providing sustained career opportunities.

97 MATHEMATICS AND COMPUTING↗

Successful Completion of FY18/Q1 ASC L2 Milestone 6355: Electrical Analysis Calibration Workflow Capability Demonstration.

The Sandia Analysis Workbench (SAW) project has developed and deployed a production capability for SIERRA computational mechanics analysis workflows. However, the electrical analysis workflow capability requirements have only been demonstrated in early prototype states, with no real capability deployed for analysts’ use. This milestone aims to improve the electrical analysis workflow capability (via SAW and related tools) and deploy it for ongoing use. We propose to focus on a QASPR electrical analysis calibration workflow use case. We will include a number of new capabilities (versus today’s SAW), such as: 1) support for the XYCE code workflow component, 2) data management coupled to electrical workflow, 3) human-in-theloop workflow capability, and 4) electrical analysis workflow capability deployed on the restricted (and possibly classified) network at Sandia. While far from the complete set of capabilities required for electrical analysis workflow over the long term, this is a substantial first step toward full production support for the electrical analysts.

42 ENGINEERING↗

Alternative Technologies Adopters Outreach Survey Report

Brookhaven National Laboratory (BNL), in collaboration with Ioanna Iliopulos and Christopher Boyd (independent contractors), conducted quantitative and qualitative surveys to document views and opinions of scientists and researchers with firsthand experience operating x-ray devices for a wide range of research applications. The goal of the study was to fill critical information gaps and identify misconceptions widely held among the research community regarding the transition from Cesium-137 irradiators to x-ray devices. The survey instrument focused on three distinct yet interrelated areas that impact users' experience: operational impacts, including infrastructure needs, workflow, personnel management, and training, and regulatory compliance; research design and investigation impacts, including calibration, dosimetry, continuity of previous studies, development of new research agendas, and the outcomes for both; and funding impacts, specifically how grant sources and opportunities have responded to the transition.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

BigPanDA: PanDA Workload Management System and its Applications beyond ATLAS

Modern experiments collect peta-scale volumes of data and utilize vast, geographically distributed computing infrastructure that serves thousands of scientists around the world. Requirements for rapid, near real-time data processing, fast analysis cycles and need to run massive detector simulations to support data analysis pose special premium on efficient use of available computational resources. A sophisticated Workload Management System (WMS) is needed to coordinate the distribution and processing of data and jobs in such environment. The ATLAS experiment at CERN uses PanDA (Production and Data Analysis) Workload Management System for managing the workflow for all data processing on over 150 data centers. While PanDAcurrently uses more than 250,000 cores with a peak performance of 0.3 petaFLOPS, it runs around 2 million jobs per day on hundreds of Grid sites and serving thousands of ATLAS users. In 2017 about 1.5 exabytes of data were processed with PanDA.In 2012 BigPanDA project project was started with aim to introduce new types of computing resources into ATLAS computing infrastructure, but also to offering PanDA features to different data-intensive applications for projects and experiments outside of ATLAS and High-Energy and Nuclear Physics. In this article we will present accomplishments and discuss possible directions for future work.

Svirin, Pavlo↗

The Acoustic Emission signal acquired by the microphones placed in the CABRI test device along the fourteen last R.I.A. experiments: an example of reproducible research in nuclear science

The CABRI facility is an experimental nuclear reactor of the French Atomic Energy Commission. It is located at the Cadarache Research Centre in southern France and it is designed to act as a support to the French nuclear infrastructure. The purpose of the new testing programme termed, 'CABRI International Programme' (CIP) is to study the behaviour of PWR-type fuel rods at high burnup, equipped with an 'advanced' cladding, under Reactivity Initiated Accident (RIA) conditions (such as the scenario of a control rod ejection). Within the framework of this programme, piloted and funded by the French Institute of Nuclear Radioprotection and Safety (IRSN), ten tests are to be conducted with a frequency of two tests per year. The LPRE laboratory of the CEA which is in charge of the Preparation, realisation and breakdown of the test results studies the possibility to set up a new test analysis based on the processing of signals coming from sensors placed within the test equipment. During the experimental phase, the behaviour of the fuel element generates acoustic waves which can be detected by two microphones placed upstream and downstream of the test device. Studies showed the interest to realize temporal and spectral analyses on these signals by showing the existence of signatures which can be correlated with physical phenomena as the rod failure or the test shutdown (i.e. scram). The work presented in this article results from the will to consolidate these studies. Since the main phenomenon to be tracked is the fuel rod failure, the aim would be to highlight specific events which would have been precursors of the rod failure in order to use in the future these signals for further interpretation. Such an antecedent works resumption leads to a better understanding of the experimental needs and constitutes a good initial state to prepare the new very fast digital data acquisition systems. Although all the raw data are accessible in the form of text files, analyses and graphics representations were not straightforward to reproduce from the ancient studies since that, on one hand, people who were in charge of the original work left the laboratory and on the other hand because it is not easy when the time passes, even with our own work, to be able to remember the steps of data manipulations and the exact setup: - During the ancient experiments the use of analog data acquisition systems required to digitize tapes to be able to realize computer treatments. That had had for consequence to lose the initial dating. This one must be correctly edited to do temporal comparisons. - Analyses require functions for calculations whose parameters has to be well-known to reach the same results. We thus wished to manage our workflow in the idea that it can be easily reproducible on all the experiments. The object of the work presented in this article was to put in practice this strong bind between the data, treatments and generation of the document in order not to hesitate to do the iteration principle in action. We do not have to be afraid by the data driven analyses. According to the philosophy of the literate programming, the text of the technical document is woven with the computer code that produces all the printed output as tables, graphs for the study eliminating hence the unrealistic cut and paste. This difficulty is not specific to the nuclear domain. For many years, researchers have been worked out solutions to this mundane issue. And, presently, new technologies and high-level programming languages offer us actual answers. We will firstly present the tools applied in our laboratory to implement this workflow, then we will describe the global perception carried out to continue the study of the Acoustic Emission signals recorded by the two microphones during the fourteen last CABRI R.I.A. test.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Using cross-layer adaptations for dynamic data management in large scale coupled scientific workflows, In: SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis

As system scales and application complexity grow, managing and processing simulation data has become a significant challenge. While recent approaches based on data staging and in-situ/in-transit data processing are promising, dynamic data volumes and distributions, such as those occurring in AMR-based simulations, make the efficient use of these techniques challenging. In this paper we propose cross-layer adaptations that address these challenges and respond at runtime to dynamic data management requirements. Specifically we explore (1) adaptations of the spatial resolution at which the data is processed, (2) dynamic placement and scheduling of data processing kernels, and (3) dynamic allocation of in-transit resources. We also exploit coordinated approaches that dynamically combine these adaptations at the different layers. We evaluate the performance of our adaptive cross-layer management approach on the Intrepid IBM-BlueGene/P and Titan Cray-XK7 systems using Chombo-based AMR applications, and demonstrate its effectiveness in improving overall time-to-solution and increasing resource efficiency.

Computer Science↗