SEARCH · Search NASA

Results for “Analysis Ready Data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

All sources NASA NTRS DOE OSTI

NASA NTRS2017

AI and Autonomy Initiatives for NASA’s Deep Space Network (DSN)

NASA’s Deep Space Network (DSN) consists of thirteen large (34- and 70-meter) antennas that are used to communicate with approximately 40 NASA and partner spacecraft, all at great distance from the earth (generally at Lunar distances and beyond). The DSN has a long history — over 50 years — and has evolved with cutting edge, often custom, telecommunications equipment and associated software systems. In recent years, and in preparation for an increasing future demand, there has been an effort to invest in initiatives that will result in significant cost savings in the future. These efforts are building on, or augmenting, the recent deployment of “Follow-the-Sun” operations (day shift remote operational control of the entire network from each of the three antenna complexes in turn) — which is being deployed in 2017. This paper focuses on Adaptive Demand Access: in a paradigm shift from completely pre-planned operations, this concept calls for spacecraft to signal their intent (or not) for near-future contacts, in case they have science results of interest, or have experienced an anomaly. This would take advantage of a beacon tone transmission, which can be detected using smaller antennas. When a connection request is received, the DSN ground systems would adaptively accommodate the request, inserting the contact into the plan as soon as possible, subject to constraints and priorities. The demand access concept incorporates onboard data analysis and science data processing, so that beacon tones can be generated with maximum information. This area is representative of several where infusing AI technologies can lead to improved effectiveness of the DSN as the network readies for support of expanded Mars exploration efforts in the 2020’s and beyond.

Wyatt, E. Jay↗

DOE OSTI2025

A Business Case Evaluation of Gas Switching Reforming (GSR) Technology: A Promising Technology for Natural Gas Reforming with Integrated CO2 Capture

Hydrogen is essential in the transition to sustainable energy, and developing low-carbon production methods is a key research focus. Traditional steam methane reforming (SMR) dominates the hydrogen industry but contributes substantially to CO2 emissions. In response, Gas Switching Reforming (GSR) has emerged as a novel process that integrates carbon capture and utilizes process heat more efficiently. Unlike other reforming methods, GSR consolidates oxidation and reduction reactions within a single reactor, which minimizes external energy inputs and simplifies scaling. Like conventional steam methane reforming (SMR), GSR can be integrated with water-gas shift and pressure swing adsorption units for pure hydrogen production. This work presents a comprehensive business case analysis of GSR technology based on experimental results in Technology Readiness Level 3, Life Cycle Assessment (LCA) and Techno-Economic (TEA) evaluation incorporating ASPEN Plus process modeling considering different configurations and energy scenarios. The TEA incorporates data from kinetic experiments from various catalysts to evaluate the GSR process under various conditions. The goal of this work is to evaluate GSR’s potential to serve as a low-carbon alternative to SMR, focusing on global warming potential and additional impact categories to evaluate a wide spectrum of environmental impacts. Comparative assessments were conducted with SMR, chemical loop reforming (CLR), and proton exchange membrane (PEM) electrolysis to explore trade-offs across environmental metrics. The environmental impact assessment of this work encompasses the entire hydrogen production lifecycle from raw material extraction to plant decommissioning, using a cradle-to-gate boundary. Preliminary findings highlight that GSR, when integrated with low-carbon energy sources, could significantly reduce environmental impacts, making it a promising candidate for low-carbon hydrogen infrastructure. The insights from this business case evaluation aim to guide industry in scale-up and commercialization of this promising clean energy technology.

03 NATURAL GAS↗

DOE OSTI2025

Considerations for Distributed Edge Data Centers and Use of Building Loads to Support Large Interconnections

The rapid expansion of artificial intelligence (AI) and machine learning is driving unprecedented electricity demand from data centers. It is predicted that by 2030, 90% of AI workloads will be inference-based, requiring interconnection of multiple low-latency edge data centers (<20 MW) sited closer to end users - often on already constrained distribution feeders. Although individually small, these loads can aggregate to large loads per feeder, straining infrastructure, creating multi-year interconnection delays, and driving up customer costs. This paper proposes a data center-focused grid-integration framework that combines feeder hosting capacity analysis with building energy efficiency, building load flexibility, and waste heat reuse to expand effective feeder and substation headroom. Such approaches can reduce interconnection delays, lower costs for ratepayers, and accelerate AI-ready infrastructure deployment.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

NASA NTRS2009

Characterizing Space Environments with Long-Term Space Plasma Archive Resources

A significant scientific benefit of establishing and maintaining long-term space plasma data archives is the ready access the archives afford to resources required for characterizing spacecraft design environments. Space systems must be capable of operating in the mean environments driven by climatology as well as the extremes that occur during individual space weather events. Long- term time series are necessary to obtain quantitative information on environment variability and extremes that characterize the mean and worst case environments that may be encountered during a mission. In addition, analysis of large data sets are important to scientific studies of flux limiting processes that provide a basis for establishing upper limits to environment specifications used in radiation or charging analyses. We present applications using data from existing archives and highlight their contributions to space environment models developed at Marshall Space Flight Center including the Chandra Radiation Model, ionospheric plasma variability models, and plasma models of the L2 space environment.

Minow, Joseph I.↗

NASA NTRS2001

Assessment of Stirling Technology Has Provided Critical Data Leading Toward Flight Readiness of the Stirling Converter

The NASA Glenn Research Center is supporting the development of a Stirling converter with the Department of Energy (DOE, Germantown, Maryland) for an advanced Stirling Radioisotope Power System (SRPS) to provide spacecraft onboard electric power for NASA space science missions. A key technology assessment completed by Glenn and DOE has led to the SRPS being identified as a high-efficiency power source for such deep space missions as the Europa Orbiter and the Solar Probe. In addition, the Stirling system is now being considered for unmanned Mars rovers, especially where mission profiles may exclude the use of photovoltaic power systems, such as exploration at high Martian latitudes or for missions of long duration. The SRPS efficiency of over 20 percent will reduce the required amount of radioisotope by more than a factor of 3 in comparison to current radioisotope thermoelectric generators. This significantly reduces radioisotope cost, radiological inventory, and system cost, and it provides efficient use of scarce radioisotope resources. In support of this technology assessment, Glenn conducted a series of independent evaluations and tests to determine the technology readiness of a 55-We Stirling converter developed by Stirling Technology Company (Kennewick, Washington) and DOE. Key areas evaluated by Glenn included: 1) Radiation tolerance of materials; 2) Random vibration testing of the Stirling converter in Glenn's Structural Dynamics Lab to simulate operation in the launch environment; 3) Electromagnetic interference and compatibility (EMI/EMC) of the converter operating in Glenn's EMI lab; Independent failure modes, effects, and criticality analysis, and life and reliability 4. Independent failure modes, effects, and criticality analysis, and life and reliability assessment; and 5) SRPS cost estimate. The data from these evaluations were presented to NASA Headquarters and the Jet Propulsion Laboratory mission office by a joint industry/Government team consisting of DOE, Glenn, and Lockheed Martin Astronautics. This team concluded that there are no technical reasons that would rule out using the Stirling converter for deep space missions. As a direct result of the successful testing at Glenn, the DOE/Stirling Technology Company 55-We Stirling converter has been baselined for the SRPS. Glenn is now continuing an in-house project to assist in developing the Stirling converter for readiness for space qualification and mission implementation. As part of this effort, the Stirling converter will be further characterized under launch environment random vibration testing, methods to reduce converter EMI will be developed, and an independent performance verification will be completed. Converter life assessment and permanent magnet aging characterization tasks are also underway. Substitute organic materials for the linear alternator and piston bearing coatings for use in a high-radiation environment have been identified and have now been incorporated in Stirling converters built by Stirling Technology Company for Glenn. Electromagnetic and thermal finite element analyses for the alternator are also being conducted.

Thieme, Lanny G.↗

DOE OSTI2022

A Methodology for Evaluating Operator Usage of Machine Learning Recommendations for Power Grid Contingency Analysis

This work presents the application of a methodology to measure domain expert trust and workload, elicit feedback, and understand the technological usability and impact when a machine learning assistant is introduced into contingency analysis for real-time power grid simulation. The goal of this framework is to rapidly collect and analyze a broad variety of human factors data in order to accelerate the development and evaluation loop for deploying machine learning applications. We describe our methodology and analysis, and we discuss insights gained from a pilot participant about the current usability state of an early technology readiness level (TRL) artificial neural network (ANN) recommender.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

NASA NTRS2005

BEAM Technology Flight Demonstration

As technologies advance, their growing complexity makes them harder to maintain. Detection methods for isolating and identifying impending problems are needed to balance this complexity. Through comparison of signal pairs from onboard sensors, the Beacon-based Exception Analysis For Multimissions (BEAM) algorithm can identify and help classify deviations in system operation from a data-trained statistical model. The goal of this task is to mature BEAM and validate its performance on a flying test bed. A series of F-18 flight demonstrations with BEAM monitoring engine parameters in real time was used to demonstrate in-the-field readiness. Captured F-18 and simulated F-18 engine data were used in model creation and training. The algorithm was then ported to the embedded system with a data buffering, file writing, and data-time-stamp monitoring shell to reduce the impact of embedded system faults on BEAM'S ability to correctly identify engine faults. Embedded system testing identified hardware related restrictions and contributed to iterative improvements in the code's runtime performance. The system was flown with forced engine flameouts and other pilot induced faults to simulate operation out of the norm. Successful detection of these faults, confirmed through post-flight data analysis, helped BEAM achieve TRL6.

Wang, David↗

DOE OSTI2022

Application of Big Data Analytics and Machine Learning to Large-Scale Synchrophasor Datasets: Evaluation of Dataset ‘Machine Learning-Readiness’

Not Available

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

End-Use Savings Shapes Measure Documentation: LED Lighting

Building on the successfully completed effort to calibrate and validate the U.S. Department of Energy's ResStock and ComStock models over the past three years, the objective of this work is to produce national data sets that empower analysts working for federal, state, utility, city, and manufacturer stakeholders to answer a broad range of analysis questions. The goal of this work is to develop energy efficiency, electrification, and demand flexibility end-use load shapes (electricity, gas, propane, or fuel oil) that cover a majority of the high-impact, market-ready (or nearly market-ready) measures. "Measures" refers to energy efficiency variables that can be applied to buildings during modeling. An end-use savings shape is the difference in energy consumption between a baseline building and a building with an energy efficiency, electrification, or demand flexibility measure applied. It results in a timeseries profile that is broken down by end use and fuel (electricity or on-site gas, propane, or fuel oil use) at each timestep. ComStock is a highly granular, bottom-up model that uses multiple data sources, statistical sampling methods, and advanced building energy simulations to estimate the annual subhourly energy consumption of the commercial building stock across the United States. The baseline model intends to represent the U.S. commercial building stock as it existed in 2018. The methodology and results of the baseline model are discussed in the final technical report of the End-Use Load Profiles project. This documentation focuses on a single end-use savings shape measure - LED lighting. The LED lighting measure replaces interior lights with LEDs where applicable. The measure resulted in 1.25 TBtu energy savings across the building stock, with the majority of savings occurring in retail and warehouse buildings.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2022

Dataset for report: Non-Technical Barriers to Geothermal Development in California and Nevada

In California and Nevada, geothermal projects are subject to non-technical barriers, which may create development delays leading to higher project costs and risks and decreased competitiveness with other electricity generation technologies. These non-technical barriers may include federal and state permits, authorizations, environmental reviews, and other regulatory requirements that are applicable throughout different phases of geothermal project development. The 2022 National Renewable Energy Laboratory (NREL) report, "Non-Technical Barriers to Geothermal Development in California and Nevada" presents findings of a study on non-technical barriers that may influence geothermal project development. The set of data resources relied upon for the report includes: 1) interviews conducted with relevant geothermal stakeholders including regulators and project developers, 2) federal and state environmental review documents developed for specific projects, 3) a techno-economic analysis conducted using the NREL Annual Technology Baseline (ATB) framework, and 4) an analysis of the impacts of site-specific land access and permitting considerations on project development readiness conducted using the Geothermal Resource Portfolio Optimization and Reporting Technique (GeoRePORT) Socioeconomic Assessment Tool (SEAT).

15 GEOTHERMAL ENERGY↗

DOE OSTI2024

End-Use Savings Shapes Measure Documentation: Boiler Replacement with Air-Source Heat Pump Boiler and Electric Boiler Backup

Building on the successfully completed effort to calibrate and validate the U.S. Department of Energy's ResStock TM and ComStock TM models over the past three years, the objective of this work is to produce national data sets that empower analysts working for federal, state, utility, city, and manufacturer stakeholders to answer a broad range of analysis questions. The goal of this work is to develop energy efficiency, electrification, and demand flexibility end-use load shapes (electricity, gas, propane, or fuel oil) that cover a majority of the high-impact, market-ready (or nearly market-ready) measures. "Measures" refers to energy efficiency variables that can be applied to buildings during modeling. An end-use savings shape is the difference in energy consumption between a baseline building and a building with an energy efficiency, electrification, or demand flexibility measure applied. It results in a time-series profile that is broken down by end use and fuel (electricity or on-site gas, propane, or fuel oil use) at each timestep. ComStock is a highly granular, bottom-up model that uses multiple data sources, statistical sampling methods, and advanced building energy simulations to estimate the annual sub hourly energy consumption of the commercial building stock across the United States. The baseline model intends to represent the U.S. commercial building stock as it existed in 2018. The methodology and results of the baseline model are discussed in the final technical report of the End-Use Load Profiles project. This documentation focuses on a single end-use savings shape measure—boiler replacement by air-source heat pump boiler. This measure replaces space heating natural gas boilers by air-source heat pump boilers when applicable and helps quantify the decarbonization as well as the energy savings potential from the replacement. The measure resulted higher savings in natural gas consumption compared to the increase in electricity consumption, with a ratio of 2.9. The total natural gas energy consumption was reduced by 20%, whereas the total electricity consumption was increased by 2.5%.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2024

End-Use Savings Shapes Measure Documentation: Boiler Replacement with Air-Source Heat Pump Boiler and Natural Gas Boiler Backup

Building on the successfully completed effort to calibrate and validate the U.S. Department of Energy's ResStock (™) and ComStock (™) models over the past three years, the objective of this work is to produce national data sets that empower analysts working for federal, state, utility, city, and manufacturer stakeholders to answer a broad range of analysis questions. The goal of this work is to develop energy efficiency, electrification, and demand flexibility end-use load shapes (electricity, gas, propane, or fuel oil) that cover a majority of the high-impact, market-ready (or nearly market-ready) measures. "Measures" refers to energy efficiency variables that can be applied to buildings during modeling. An end-use savings shape is the difference in energy consumption between a baseline building and a building with an energy efficiency, electrification, or demand flexibility measure applied. It results in a time-series profile that is broken down by end use and fuel (electricity or on-site gas, propane, or fuel oil use) at each timestep. ComStock is a highly granular, bottom-up model that uses multiple data sources, statistical sampling methods, and advanced building energy simulations to estimate the annual subhourly energy consumption of the commercial building stock across the United States. The baseline model intends to represent the U.S. commercial building stock as it existed in 2018. The methodology and results of the baseline model are discussed in the final technical report of the End-Use Load Profiles project. This documentation focuses on a single end-use savings shape measure - boiler replacement with air-source heat pump boiler with natural gas boiler backup. This measure replaces natural gas boilers for HVAC application by air-source heat pump boilers when applicable and use natural gas boiler backup when the heat pump boiler could not operate due to outdoor air conditions which are below its cutoff temperature. This measure helps to quantify the decarbonization as well as the energy savings potential from the replacement. The measure resulted higher savings in natural gas consumption compared to the increase in electricity consumption, with a ratio of 3. The total natural gas energy consumption was reduced by 41%, whereas the total electricity consumption was increased by 5.3%.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2023

End-Use Savings Shapes Measure Documentation: Variable Refrigerant Flow with Heat Recovery and Dedicated Outdoor Air System

Building on the successfully completed effort to calibrate and validate the U.S. Department of Energy's ResStock™ and ComStock™ models over the past 3 years, the objective of this work is to produce national data sets that empower analysts working for federal, state, utility, city, and manufacturer stakeholders to answer a broad range of analysis questions. The goal of this work is to develop energy efficiency, electrification, and demand flexibility end-use load shapes (electricity, gas, propane, or fuel oil) that cover a majority of the high-impact, market-ready (or nearly market-ready) measures. "Measures" refers to energy efficiency variables that can be applied to buildings during modeling. An end-use savings shape is the difference in energy consumption between a baseline building and a building with an energy efficiency, electrification, or demand flexibility measure applied. It results in a time-series profile that is broken down by end use and fuel (electricity or on-site gas, propane, or fuel oil use) at each time step. ComStock is a highly granular, bottom-up model that uses multiple data sources, statistical sampling methods, and advanced building energy simulations to estimate the annual subhourly energy consumption of the commercial building stock across the United States. The baseline model intends to represent the U.S. commercial building stock as it existed in 2018. The methodology and results of the baseline model are discussed in the final technical report of the End-Use Load Profiles project. This documentation focuses on a single heating, ventilation, and air-conditioning (HVAC) end-use savings shape measure - a variable refrigerant flow with heat recovery (VRF HR) heating and cooling system coupled with a dedicated outdoor air system (DOAS) for ventilation. This measure replaces existing multi-zone variable air volume (VAV) systems or single-zone rooftop units (RTU) with a VRF HR system coupled with a DOAS that includes an energy/heat recovery ventilator (E/HRV). The measure covers 53% of exisiting building stock's floor area and is not applicable to HVAC system types using district heating or cooling or buildings/spaces that include high-ventilation spaces such as kitchens where the amount of exhaust air is large. A DOAS with E/HRV is used to provide required outdoor ventilation air to spaces since ventilation air is generally not supplied by a VRF HR system. An exhaust air energy recovery ventilator (ERV ) with sensible and latent heat exchange is added to humid climate zones while a heat recovery ventilator (HRV ) with sensible only exchange is added to drier climate zones. The ERV is modeled as a fixed membrane plate counterflow heat exchanger, while the HRV is modeled as a sensible-only fixed aluminum plate counterflow heat exchanger. Both systems include a bypass (for temperature control and economizer lockout) and minimum exhaust temperature control for frost prevention.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2024

End-Use Savings Shapes Measure Documentation: Economizers

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2024

End-Use Savings Shapes Upgrade Package Documentation: Wall and Roof Insulation and New Windows

Building on the successfully completed effort to calibrate and validate the U.S. Department of Energy's ResStock and ComStock models over the past 3 years, the objective of this work is to produce national data sets that empower analysts working for federal, state, utility, city, and manufacturer stakeholders to answer a broad range of analysis questions. The goal of this work is to develop energy efficiency, electrification, and demand flexibility end-use load shapes (electricity, gas, propane, or fuel oil) that cover a majority of the high-impact, market-ready (or nearly market-ready) upgrade measures, or upgrades. "Measures" refers to energy efficiency variables that can be applied to buildings during modeling. An end-use savings shape is the difference in energy consumption between a baseline building and a building with an energy efficiency, electrification, or demand flexibility upgrade applied. It results in a time-series profile that is broken down by end use and fuel (electricity or on-site gas, propane, or fuel oil use) at each time step. ComStock is a highly granular, bottom-up model that uses multiple data sources, statistical sampling methods, and advanced building energy simulations to estimate the annual subhourly energy consumption of the commercial building stock across the United States. The baseline model intends to represent the U.S. commercial building stock as it existed in 2018. The methodology and results of the baseline model are discussed in the final technical report of the End-Use Load Profiles project. This documentation focuses on an upgrade package of three end-use savings shapes upgrades - Window Replacement, Exterior Wall Insulation, and Roof Insulation, which we will refer to collectively as the "High Efficiency Envelope" package. More details on the individual upgrades can be found on the ComStock Measures Documentation page. An upgrade package applies two or more EUSS upgrades to a single building model simulation. Since ComStock is a bottom-up physics-based model, an upgrade package will go beyond aggregating or summing the individual upgrade results and produce novel results by simulating interactions between the upgrades. For example, pairing an envelope upgrade with an electrification upgrade would likely result in higher savings results than the sum of these upgrades individually, and the size of the heating, ventilating, and air conditioning (HVAC) equipment may be reduced if the envelope upgrade reduces the loads significantly.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

DOE OSTI2024

End-Use Savings Shapes Upgrade Package Documentation: LED Lighting, HP-RTU and ASHP-Boiler

Building on the successfully completed effort to calibrate and validate the U.S. Department of Energy's ResStock and ComStock models over the past 3 years, the objective of this work is to produce national data sets that empower analysts working for federal, state, utility, city, and manufacturer stakeholders to answer a broad range of analysis questions. The goal of this work is to develop energy efficiency, electrification, and demand flexibility end-use load shapes (electricity, gas, propane, or fuel oil) that cover a majority of the high-impact, market-ready (or nearly market-ready) upgrade measures, or upgrades. "Measures" refers to energy efficiency variables that can be applied to buildings during modeling. An end-use savings shape is the difference in energy consumption between a baseline building and a building with an energy efficiency, electrification, or demand flexibility upgrade applied. It results in a time-series profile that is broken down by end use and fuel (electricity or on-site gas, propane, or fuel oil use) at each time step. ComStock is a highly granular, bottom-up model that uses multiple data sources, statistical sampling methods, and advanced building energy simulations to estimate the annual subhourly energy consumption of the commercial building stock across the United States. The baseline model intends to represent the U.S. commercial building stock as it existed in 2018. The methodology and results of the baseline model are discussed in the final technical report of the End-Use Load Profiles project. This documentation focuses on an upgrade package of three end-use savings shapes upgrades - Light Emitting Diode (LED) Lighting, Heat Pump Rooftop Unit (RTU) (HP-RTU), and Air-Source Heat Pump (ASHP) Boiler, which we will refer to collectively as the "Interior Lighting and Heat Pump" package. More details on the individual upgrades can be found on the ComStock Measures Documentation page. An upgrade package applies two or more EUSS upgrades to a single building model simulation. Since ComStock is a bottom-up physics-based model, an upgrade package will go beyond aggregating or summing the individual upgrade results and produce novel results by simulating interactions between the upgrades. For example, pairing an envelope upgrade with an electrification upgrade would likely result in higher savings results than the sum of these upgrades individually, and the size of the heating, ventilating, and air conditioning (HVAC) equipment may be reduced if the envelope upgrade reduces the loads significantly.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

NASA NTRS2012

Stress Analysis and Testing at the Marshall Space Flight Center to Study Cause and Corrective Action of Space Shuttle External Tank Stringer Failures

After the launch scrub of Space Shuttle mission STS-133 on November 5, 2010, large cracks were discovered in two of the External Tank intertank stringers. The NASA Marshall Space Flight Center, as managing center for the External Tank Project, coordinated the ensuing failure investigation and repair activities with several organizations, including the manufacturer, Lockheed Martin. To support the investigation, the Marshall Space Flight Center formed an ad-hoc stress analysis team to complement the efforts of Lockheed Martin. The team undertook six major efforts to analyze or test the structural behavior of the stringers. Extensive finite element modeling was performed to characterize the local stresses in the stringers near the region of failure. Data from a full-scale tanking test and from several subcomponent static load tests were used to confirm the analytical conclusions. The analysis and test activities of the team are summarized. The root cause of the stringer failures and the flight readiness rationale for the repairs that were implemented are discussed.

Wingate, Robert J.↗

DOE OSTI2022

Enabling machine learning-ready HPC ensembles with Merlin

With the growing complexity of computational and experimental facilities, many scientific researchers are turning to machine learning (ML) techniques to analyze large scale ensemble data. With complexities such as multi-component workflows, heterogeneous machine architectures, parallel file systems, and batch scheduling, care must be taken to facilitate this analysis in a high performance computing (HPC) environment. Here, we present Merlin, a workflow framework to enable large ML-friendly ensembles of scientific HPC simulations. By augmenting traditional HPC with distributed compute technologies, Merlin aims to lower the barrier for scientific subject matter experts to incorporate ML into their analysis. As a producer–consumer workflow model, Merlin enables multi-machine, cross-batch job, dynamically allocated yet persistent workflows capable of utilizing surge-compute resources. Key features of Merlin are a flexible HPC-centric interface, low per-task overhead, multi-tiered fault recovery, and a hierarchical sampling algorithm that allows for $\mathscr{O}$(N) task execution and $\mathscr{O}$(N ln N) task queuing to ensembles of millions of tasks. In addition to Merlin’s design, we test the algorithm’s performance in an HPC center and demonstrate the ability to enqueue 40 million simulations in 100 s, with a 30 millisecond per-task overhead that is independent of ensemble size. Finally, we describe some example applications that Merlin has enabled on leadership-class HPC resources, such as the ML-augmented optimization of nuclear fusion experiments and the calibration of infectious disease models to study the progression of and possible mitigation strategies for COVID-19.

97 MATHEMATICS AND COMPUTING↗