Object-Centric Data Management in HPC Workflows - A Case Study
Explore the source record for details and available documents.
Computer science↗
SEARCH · Search NASA
Results for “Data science, data management”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
FY 2021 Mid-Year HPC Utilization Report M3UF-21IN0701018
Idaho National Laboratory (INL), supported by the Department of Energy Office of Nuclear Energy (DOE-NE) through the Nuclear Science User Facilities, provides access to supercomputer systems and data storage along with support staff for system management, software installation, cybersecurity and user support to the broader DOE-NE user community. Users include individuals at universities, industry, and government laboratories enabling a wide range of research and development and mission-supporting activities. The availability of high-performance computing (HPC) capabilities is a key foundation of collaboration and innovation in nuclear energy systems research. HPC resources and INL directly support the mission and objectives of DOE-NE. From October 2020 through March 2021, INL HPC capabilities were utilized by a diverse set of computing and applied researchers, for a wide range of research and engineering activities. This report focuses on current INL HPC systems and utilization from October 1, 2020 through March 31, 2021.
99 GENERAL AND MISCELLANEOUS↗
Post-Closure Report for Closed Resource Conservation and Recovery Act Corrective Action Units, Nevada National Security Site, Nevada, For Calendar Year 2021 (Revision 1)
This report serves as the combined annual report for post-closure activities in compliance with the requirements listed in Resource Conservation and Recovery Act (RCRA) Permit Number NEV HW0101 associated with the following closed corrective action units (CAUs): CAU 90, Area 2 Bitcutter Containment; CAU 91, Area 3 U-3fi Injection Well; CAU 92, Area 6 Decon Pond Facility; CAU 110, Area 3 WMD U-3ax/bl Crater; CAU 111, Area 5 WMD Retired Mixed Waste Pits; and CAU 112, Area 23 Hazardous Waste Trenches. This report will address all monitoring requirements noted in the RCRA Permit NEV HW0101, except groundwater monitoring. Groundwater monitoring is documented in the Nevada National Security Site Data Report: Groundwater Monitoring Program Area 5 Radioactive Waste Management Site.
54 ENVIRONMENTAL SCIENCES↗
Autonomous System Subversion Tactics: Prototypes and Recommended Countermeasures
One of the fielding requirements for Advanced and Small Modular Reactors (AR/SMR) is the ability to support remote and autonomous operations. Autonomous Control Systems (ACS) are found on platforms such as Autonomous Space Vehicles, Cruise Missiles, and advanced driver-assistance systems. Each of these ACS implementations depends upon a set of decision support subsystems responsible for supporting Autonomous Mission Managers (names vary based upon field and author preferences). These Autonomous Mission Managers receive inputs from system sensors (e.g., LIDAR collection from an automobile travelling down a street; transients from a nuclear reactor), and perform a set of classifications (e.g., Red Traffic Light; Small Pedestrian at 10m; Load Rejection; Single Coolant Pump Trip), and then use these classifications in combination with recommendation algorithms to achieve platform goals (e.g., Stop the Vehicle at the Traffic Light, Avoid the Small Pedestrian; Trip the Reactor to prevent a Safety Event). The design, implementation, and fielding of an ACS capability will alter the cyber-attack surface such that existing risk management plans will need to be updated to include how to protect and defend against data-science and decision-support-system attack classes. These attack classes would include protection of the design and training environments where algorithm selection and testing and training data would be obvious attack vectors. These attack classes would also require an informed set of detection and response procedures to identify anomalous behaviors and document best practices for anomaly assessment and vulnerability mitigation and remediation. Last year we published a Cyber Threat Assessment Methodology for Autonomous and Remote Operations for AR/SMRs along with a companion publication on Cyber Attack and Defense Use Cases. The focus of the methodology was on describing and enumerating ACS processes, components, and functions such that security engineers could: evaluate subversion options against the target; identify threat actor attributes and capabilities derived from each subversion option; and identify security controls and response countermeasures. The Use Cases document offered detailed methodology examples including an assessment of a Military Base SMR, an Autonomous System Decision Loop, and implementation of AR/SMR Machine Learning algorithms. Our proposal at the end of last year was to focus on implementation of subversion prototypes related to the last Use Case area: AR/SMR Machine Learning (ML) Algorithms. We included six attack scenarios in our Use Cases paper: a Poisoning Attack against ML functions implemented using an FPGA; a Trojaning Attack against ML classifiers exploiting the excitability of Nuclear Engineers; a Backdooring Attack against ML Training environments to ensure persistence of an attack vector; a False Positive Evasion Attack against multi-factor Access Control Systems using clever inputs; an Inference Attack against ML models by an Insider with access to the Operational environment; and an Adversarial Reprogramming Attack against a Material Access Control Video Surveillance System. At the beginning of this year these six attack scenarios were provided to our research teams at Georgia Tech and Idaho State University and each team successfully implemented a subversion attack against a ML implementation to include transient misclassifications. While this is a notable outcome from this type of research, this paper offers the reader insight into not only how to structure and execute these types of attacks, but into the thought process behind how the researcher investigated the problem space, performed initial algorithm implementation, and the trial-and-error behind arriving at the successful subversion prototypes. We include in this paper a set of associated Scenarios on how these subversion prototypes could be implemented and an initial set of guidance for AR/SMR architects, Nuclear Regulators, and Cyber Defenders to implement awareness and defense capabilities into their current operational portfolios.
42 ENGINEERING↗
US Department of Energy, Office of Science, High-Performance Computing Facility: 2023 Operational Assessment Oak Ridge Leadership Computing Facility
The Oak Ridge Leadership Computing Facility (OLCF) was established to accelerate scientific discovery by providing world-leading computational performance and advanced data infrastructure. As a US Department of Energy (DOE) Office of Science user facility, the OLCF has managed the successful deployment and operation of a succession of leadership-class resources dedicated to open science. In addition to these resources, the OLCF staff continually strive to develop innovative processes and technologies, improve security, and empower users through effective allocation management and comprehensive user support and training. These efforts support the advancement of science by the OLCF users and benefit high-performance computing (HPC) facilities around the world. In calendar year (CY) 2023, the OLCF supported 1,676 users and 598 projects and exceeded all targets for user satisfaction. The facility received an average satisfaction score of 4.52 out of 5 on the annual user survey, and 94% of respondents reported a high satisfaction rate with the OLCF overall. Of the 3,619 user tickets submitted in CY 2023, OLCF staff resolved 97% within 3 business days. The facility opened Frontier to full scientific operations this year. Two projects conducted on Frontier received the Association for Computing Machinery (ACM) Gordon Bell Prize and the Gordon Bell Special Prize for Climate Modeling, and a third earned a nomination as a Gordon Bell Prize finalist. The facility’s previous flagship machine, Summit, gained new life and was extended through 2024 in part to help provide resources to the Integrated Research Infrastructure (IRI) projects and the National Artificial Intelligence Research Resource (NAIRR) pilot program. The facility instantiated an Advanced Computing Ecosystem testbed in part to support IRI workflows. OLCF made interactivity easier and more accessible to users than ever through tools like Jupyter notebooks and workflows.
97 MATHEMATICS AND COMPUTING↗
Enabling the Write-Once Read-Many (WORM) Dashboard and Web Interface for HPC Data Storage
Idaho National Laboratory (INL), supported by the Department of Energy Office of Nuclear Energy (DOE-NE) through the Nuclear Science User Facilities, provides access to supercomputer systems and data storage along with support staff for system management, software installation, cybersecurity, and user support to the broader DOE-NE user community. A key component of this is providing long-term or perpetual data storage as part of efforts to improve reproducibility in modeling and simulation in addition to supporting simulations with regulatory compliance requirements. This milestone details the deployment of a new type of storage option for users requiring tamper resistant perpetual storage of data and the associated support for data control and release via the data storage web interface. The perpetual data storage system is integrated within the Open OnDemand framework deployed at INL and is called the ?WORM? in recognition of its write-once read-many characteristics.
99 GENERAL AND MISCELLANEOUS↗
Active Causal Machine Learning for Molecular Property Prediction
Predicting properties from molecular structures is paramount to design tasks in medicine, materials science, and environmental management. However, design rules derived from the structure-property relationships using correlative data-driven methods fail to elucidate underlying causal mechanisms controlling chemical phenomena. This preliminary work proposes a workflow to actively learn robust cause-effect relations between structural features and molecular property for a broad chemical space utilizing smaller subsets, entailing partial information.
Fox, Zach↗
Multiscale Modeling Meets Machine Learning: What Can We Learn?
Machine learning is increasingly recognized as a promising technology in the biological, biomedical, and behavioral sciences. There can be no argument that this technique is incredibly successful in image recognition with immediate applications in diagnostics including electrophysiology, radiology, or pathology, where we have access to massive amounts of annotated data. However, machine learning often performs poorly in prognosis, especially when dealing with sparse data. This is a field where classical physics-based simulation seems to remain irreplaceable. In this review, we identify areas in the biomedical sciences where machine learning and multiscale modeling can mutually benefit from one another: Machine learning can integrate physics-based knowledge in the form of governing equations, boundary conditions, or constraints to manage ill-posted problems and robustly handle sparse and noisy data; multiscale modeling can integrate machine learn- ing to create surrogate models, identify system dynamics and parameters, analyze sensitivities, and quantify uncertainty to bridge the scales and understand the emergence of function. With a view towards applications in the life sciences, we discuss the state of the art of combining machine learning and multiscale modeling, identify applications and opportunities, raise open questions, and address potential challenges and limitations. We anticipate that it will stimulate discussion within the community of computational mechanics and reach out to other disciplines including mathematics, statistics, computer science, artificial intelligence, biomedicine, systems biology, and precision medicine to join forces towards creating robust and efficient models for biological systems.
machine learning, multiscale modeling, physics-bas↗
Real-Time Topology Detection and State Estimation in Distribution Systems Using Micro-PMU and Smart Meter Data
Not provided.
Computer Science↗
FY22 Laboratory Directed Research and Development Annual Report
The Laboratory Directed Research and Development (LDRD) program yields foundational scientific research and development (R&D) essential to growing SRNL’s core competencies, in alignment with SRNL’s Strategic Plan to provide long-term benefits to the Department of Energy (DOE), the National Nuclear Security Administration (NNSA), and other customers and stakeholders. Five strategic goals are outlined in SRNL’s strategic plan: 1) Provide applied science and engineering for EM’s active clean-up sites and LM’s post closure management sites; 2) Provide science-based solutions for gaps identified in nonproliferation strategic vision and support the government in actives impacting national security; 3) Lead ST&E as the central technical authority for processing tritium loaded reservoirs and support production of plutonium pits; 4) Align science and energy security programs by focusing modern modeling, simulation, and data analytics tools on materials engineering and performance applications; 5) Build a workforce for the future.
12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
HAM: Hotspot-Aware Manager for Improving Communications with 3D-Stacked Memory
merging High-Performance Computing (HPC) workloads, such as graph analytics, machine learning, and big data science, are data-intensive. Data-intensive workloads usually present fine-grained memory accesses with limited or no data locality, and thus incur frequent cache misses and low utilization of memory bandwidth. 3D-stacked memory devices such as Hybrid Memory Cube (HMC) and High Bandwidth Memory (HBM) can provide significantly higher bandwidth than conventional memory modules. However, the traditional interfaces and optimization methods for JEDEC DDR devices do not allow to fully exploit the potential performance of 3D-stacked memory with the massive amount of irregular memory accesses of data-intensive applications. In this paper, we propose a novel Hotspot-Aware Manager (HAM) infrastructure for 3D-stacked memory devices capable of optimizing memory access streams via request aggregation, hotspot detection, and in-memory prefetching. %and an associated hotspot-aware page policy. We present the HAM design and implementation, and simulate it on a system using RISC-V embedded cores with attached HMC devices. We extensively evaluate HAM with over 12 benchmarks and applications representing diverse irregular memory access patterns. The results show that, on average, HAM reduces redundant requests by 37.51\% and increases the prefetch buffer hit rate by 4.2 times, compared to a baseline streaming prefetcher. On the selected benchmark set, HAM provides performance gains of 21.81\% in average (up to 34.28\%) and power savings of 35.07\% over a standard 3D-stacked memory.
Wang, Xi↗
FAIR principles for AI models with a practical application for accelerated high energy diffraction microscopy
Abstract A concise and measurable set of FAIR (Findable, Accessible, Interoperable and Reusable) principles for scientific data is transforming the state-of-practice for data management and stewardship, supporting and enabling discovery and innovation. Learning from this initiative, and acknowledging the impact of artificial intelligence (AI) in the practice of science and engineering, we introduce a set of practical, concise, and measurable FAIR principles for AI models. We showcase how to create and share FAIR data and AI models within a unified computational framework combining the following elements: the Advanced Photon Source at Argonne National Laboratory, the Materials Data Facility, the Data and Learning Hub for Science, and funcX, and the Argonne Leadership Computing Facility (ALCF), in particular the ThetaGPU supercomputer and the SambaNova DataScale ® system at the ALCF AI Testbed. We describe how this domain-agnostic computational framework may be harnessed to enable autonomous AI-driven discovery.
47 OTHER INSTRUMENTATION↗
Overview of the distributed image processing infrastructure to produce the Legacy Survey of Space and Time
The Vera C. Rubin Observatory is preparing to execute the most ambitious astronomical survey ever attempted, the Legacy Survey of Space and Time (LSST). Currently the final phase of construction is under way in the Chilean Andes, with the Observatory’s ten-year science mission scheduled to begin in 2025. Rubin’s 8.4-meter telescope will nightly scan the southern hemisphere collecting imagery in the wavelength range 320–1050 nm covering the entire observable sky every 4 nights using a 3.2 gigapixel camera, the largest imaging device ever built for astronomy. Automated detection and classification of celestial objects will be performed by sophisticated algorithms on high-resolution images to progressively produce an astronomical catalog eventually composed of 20 billion galaxies and 17 billion stars and their associated physical properties. In this article we present an overview of the system currently being constructed to perform data distribution as well as the annual campaigns which reprocess the entire image dataset collected since the beginning of the survey. These processing campaigns will utilize computing and storage resources provided by three Rubin data facilities (one in the US and two in Europe). Each year a Data Release will be produced and disseminated to science collaborations for use in studies comprising four main science pillars: probing dark matter and dark energy, taking inventory of solar system objects, exploring the transient optical sky and mapping the Milky Way. Also presented is the method by which we leverage some of the common tools and best practices used for management of large-scale distributed data processing projects in the high energy physics and astronomy communities. We also demonstrate how these tools and practices are utilized within the Rubin project in order to overcome the specific challenges faced by the Observatory.
79 ASTRONOMY AND ASTROPHYSICS↗
kynema-fmb [SWR-23-07]
Kynema-FMB (FKA: Kynema) is an open-source performance portable flexible multibody (FMB) dynamics solver designed for time-domain simulations. While originally tailored for wind turbine structural dynamics, the formulation and implementation are those of a general flexible-multidbody dynamics solver that can readily be applied to a wide range of systems. Kynema was designed with a narrow focus, namely to provide a lightweight, fast, accurate FMD solver for coupling to computational-fluid-dynamics (CFD) codes, especially the CFD codes in the Kynema suite, for fluid-structure-interaction (FSI) simulations. Kynema-FMB is equipped to model systems that can be represented as a collection of beams and rigid bodies that are connected through constraints. Degrees of freedom are defined in the inertial/global frame of reference and include displacements and rotations (formally as rotation matrices, but stored as quaternions). The underlying formulation is built on a Lie-group time integrator designed for index-3 differential-algebraic equations, which is second-order accurate in time (Bruls et al., 2012). Beam models are based on geometrically exact beam theory and are discretized as high-order spectral finite elements similar to those in BeamDyn (Wang et al., 2017). The governing equations for a FMD system like a wind turbine constitute a highly nonlinear system of constrained partial-differential equations. Kynema-FMB uses analytical Jacobians in the nonlinear-system solves in each time step. Linear systems use sparse storage and several third-party sparse-linear-system solvers are enabled. Ill conditioning of linear systems is mitigated with preconditioning described in Bottasso et al, 2008. Kynema-FMB is integrated with a simple open-source controller (ROSCO). There is an application programming interface (API) for coupling to geometry-resolved CFD (like that in Sharma et al., 2023) and actuator-force CFD (like that in Kuhn et al., 2025). In the latter, for actuator-line models, Kynema-FMB includes an internal blade-element solver that depends on user-provided lookup tables for coefficients of lift and drag, i.e., aerodynamic polars. Kynema-FMB is written in C++ and leverages Kokkos and Kokkos-Kernels (KokkosEcosystem) as its performance portability layer enabling simulations on both CPU and GPU systems. The repository is equipped with extensive automated testing at the unit and regression/system levels. The following describes the high-level development objectives conceived for Kynema: *Kynema will follow modern software development best practices, including test-driven development (TDD), version control, hierarchical automated testing, and continuous integration (CI) for a robust development environment. *The core data structures are memory efficient and enable vectorization and parallelization at multiple levels. *Data structures are data-oriented to exploit methods for accelerated computing including high utilization of chip resources (e.g., single instruction multiple data (SIMD) instruction sets) and parallelization using GP-GPUs. *The computational algorithms incorporate robust open-source libraries for mathematical operations, resource allocation, and data management. *The API design considers multiple stakeholder needs and ensure integration with existing and future ecosystems for data science, machine learning, and AI. *Kynema-FMB is written in modern C++ and leverages Kokkos as its performance-portability library with inspiration from the kynema stack.
Sprague, MichaelA.↗
U.S. Manufacturing Water Use Data and Estimates: Current State, Limitations, and Future Needs for Supporting Manufacturing Research and Development
Water is essential to manufacturing operations; without it, many facilities could not operate or meet production demands. Physical, reputational, and regulatory risks to water supplies compounded by climate change-induced impacts on hydrological conditions threaten the adequacy of water supplies for manufacturing. Manufacturing water use has not been a major focus of either water or manufacturing-related research. Research and development (R&D) aimed at helping manufacturers use water more sustainably and adapt to changing water conditions is needed to ensure a thriving sector and economy. However, the ability to identify R&D needs is severely limited due to a lack of current, statistically representative data on manufacturing water use and its environmental implications. In this Perspective, we outline four key questions to inform R&D on manufacturing use and highlight how the current state of water data in the United States does not support the adequate investigation of these questions. We make recommendations for the water data characteristics needed to explore the research questions and knowledgeably inform R&D on manufacturing water use.
McCall, James↗
How an Autonomous Offshore Power System Can Transform the Ocean Economy - A Hypothetical Case Study Utilizing an Autonomous Offshore Power System in Northern Lights Carbon Capture and Storage Project
An Autonomous Offshore Power System (AOPS) provides in-situ power, energy storage, real-time data and communications support, asset management, and other capabilities at sea. It has applications for all offshore industries: energy, defense and security, aquaculture, science and research, and communications. Furthermore, this paper highlights how an AOPS can reduce cost, complexity, and carbon-intensity for existing offshore operations and enable new capabilities for offshore industry leaders. This new AOPS technology has two primary advantages. First, it unlocks the autonomous, electric future of the ocean economy via ‘local’ power generation and energy storage, in addition to real-time connection to the data cloud. Second, it helps enable a material change in the global energy mix through cost-effective, reliable generation and storage technology for use cases including mobile/static data-gathering and reporting systems, operating equipment, and charging networks for uncrewed surface vessels. AOPS technology will help transform the ocean economy, and thus has implications for offshore industry leaders as they push to reduce costs today and make an autonomous and decarbonized future possible.
16 TIDAL AND WAVE POWER↗
Open Reproducible Electron Microscopy Data Analysis
Electron microscopy (EM) is a cornerstone technique in the materials and biological sciences capable of imaging structures at nano- to atomic-scale resolution. Advances in technologies mean that one acquires datasets at increasing data rates and sizes. These advancements present enormous opportunities for researchers to understand complex systems. However, processing the resulting large-scale, complex data in a reproducible and shareable way is a real challenge for researchers. The building, managing, and maintaining complex workflows in a reproducible manner requires extensive knowledge in several areas outside the researchers’ core skill sets, such as software engineering, data science, and high-performance computing (HPC). Our work demonstrates an innovative approach to solving these problems, enabling reproducible EM data analysis through container encapsulated pipelines. Using modern container technologies, we encapsulate processing elements and connect them using shared memory. We expose user-friendly, advanced algorithms and tools to allow end users to utilize without expert programming skills. The platform enables reproducible, scalable, shareable pipelines for the analysis and visualization of EM data. Focusing on interoperability, we leverage the DOE and other agencies’ existing investments to provide a powerful software platform for EM data analysis.
Harris, Christopher↗
An Indicator-based Approach to Sustainable Management of Natural Resources (Chapter 12)
Assessing the sustainability of natural resource management choices for agricultural and forest lands requires quantification of potential changes to a set of environmental and socioeconomic indicators selected to characterize reference scenarios relative to projected future scenarios. Correctly framing the questions with local stakeholders is a critical first step in the sustainability assessment, and the questions that can be addressed are often limited by data availability. Selecting and prioritizing indicators with stakeholders to address their needs and concerns improves the likelihood of investment in monitoring and evaluation of those indicators over time. Computational techniques for analyzing interactions between the selected indicators are inherently affected by the scales and formats of the assembled indicator datasets. Data analytics have the potential to improve understanding of the potential synergies and tradeoffs involved with meeting multiple environmental and socioeconomic goals simultaneously, but timely and appropriate indicator datasets are not always available—even in this new era of “big data.” Continued improvements in data science and data analytics are needed to broaden understanding and acceptance of problems and to provide valuable information for natural resource management. Advances in these areas will enable society to design future landscapes that meet multiple objectives, including the provisioning of agricultural and forest resources along with a variety of ecosystem services (e.g., clean water and healthy soils).
Parish, Esther↗