Search NASA⌕ Search

SEARCH · Search NASA

Results for “data challenges”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 415 records · Page 23
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Rucio beyond ATLAS: experiences from Belle II, CMS, DUNE, EISCAT3D, LIGO/VIRGO, SKA, XENON

For many scientific projects, data management is an increasingly complicated challenge. The number of data-intensive instruments generating unprecedented volumes of data is growing and their accompanying workflows are becoming more complex. Their storage and computing resources are heterogeneous and are distributed at numerous geographical locations belonging to different administrative domains and organisations. These locations do not necessarily coincide with the places where data is produced nor where data is stored, analysed by researchers, or archived for safe long-term storage. To fulfil these needs, the data management system Rucio has been developed to allow the high-energy physics experiment ATLAS at LHC to manage its large volumes of data in an efficient and scalable way. But ATLAS is not alone, and several diverse scientific projects have started evaluating, adopting, and adapting the Rucio system for their own needs. As the Rucio community has grown, many improvements have been introduced, customisations have been added, and many bugs have been fixed. Additionally, new dataflows have been investigated and operational experiences have been documented. In this article we collect and compare the common successes, pitfalls, and oddities that arose in the evaluation efforts of multiple diverse experiments, and compare them with the ATLAS experience. This includes the high-energy physics experiments Belle II and CMS, the neutrino experiment DUNE, the scattering radar experiment EISCAT3D, the gravitational wave observatories LIGO and VIRGO, the SKA radio telescope, and the dark matter search experiment XENON.

Lassnig, Mario↗
DOE OSTI2023

Exploring Future Storage Options for ATLAS at the BNL/SDCC facility

The ATLAS experiment is expected to deliver an unprecedented amount of scientific data in the High Luminosity(HL-LHC) era. As the demand for disk storage capacity in ATLAS continues to rise steadily, the BNL Scientific Data and Computing Center (SDCC) faces challenges in terms of cost implications for maintaining multiple disk copies and adapting to the coming ATLAS storage requirements. To address these challenges, the SDCC Storage team has undertaken a thorough analysis of the ATLAS experiment's requirements, matching them to suitable storage options and strategies, and has explored alternatives to enhance or replace the current storage solution. This paper aims to present the main challenges encountered while supporting big data experiments such as ATLAS. We describe the experiment's specific requirements and priorities, particularly focusing on the critical storage system characteristics of the high-luminosity run and how the key storage components provided by the Storage team work together: the dCache disk storage system; its archival back-end, HPSS; and its OS-level backend Storage. Specifically, we investigate a novel approach to integrate Lustre and XRootD. In this setup, Lustre serves as backend storage and XRootD acts as an access layer frontend, supporting various grid access protocols. Additionally, we also describe the validation and commissioning tests, including the performance comparison between dCache and XRootd. Furthermore, we provide a performance and cost analysis comparing OpenZFS and LINUX MD RAID, evaluate different storage software stacks, and showcase stress tests conducted to validate Third Party Copy (TPC) functionality.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2022

2.2.5.401 - Ocean Observing Prize

The Powering the Blue Economy: Ocean Observing Prize is a $2.4M contest that challenges innovators to develop solutions that integrate marine renewable energy with ocean observation platforms to revolutionize our ability to monitor, manage, and understand the ocean. Competitors are challenged to design, build, and test novel, wave-powered, self-charging autonomous underwater vehicle (AUV) systems that could be suitable for a 6-month deployment in the Atlantic Ocean to monitor hurricane formation. Through engagement with the end user ocean observing community, the team identified challenges in a lack of data on storm intensity, endangering coastal communities. This challenge could be addressed by collecting data before, during, and after a hurricane develops and strengthens at sea. Amassing this data, however, requires an ocean observing system that can be deployed at sea for long periods, waiting for approaching storms. To help solve this problem, competitors are working to integrate wave energy capture with AUV systems to collect this data, helping to better protect coastal communities from oncoming storms. Solutions developed through this prize aim to enable long-term collection of data before, during, and after a hurricane that will help better forecast storm intensity. The prize mechanism provides an environment to accelerate technology development for innovators and helps to improve awareness of marine energy and its ability to provide power to blue economy applications.

coastal communities↗
DOE OSTI2024

Types of data: terminology and examples

Data produced at wastewater utilities are obtained through a variety of devices, including actuators, control devices, and sensors. This results in data that can be highly variable in its structure. Dealing with the resulting heterogeneity of data formats can be a challenge when storing or interpreting the data. For this reason, this chapter serves as an overview for the most important structural aspects of data typically found at a utility. The specific aims of this chapter are to:

Plana, Queralt↗
DOE OSTI2026

Comparability of Liquid Chromatography Tandem Mass Spectrometry Analysis of Dissolved Organic Matter across Laboratories

Non-targeted liquid chromatography tandem highresolution mass spectrometry (LC−MS/MS) is increasingly applied for the structure-resolved chemical analysis of dissolved organic matter (DOM). With new developments in MS instrumentation and analysis software, the approach has gained substantial momentum over the past decade. However, achieving high-quality analytical data that is reproducible and comparable across laboratories can be a bottleneck in non-targeted metabolomics and organic matter chemical analysis, especially for data reuse in repository-scale analyses. Understanding the capabilities as well as challenges of comparing LC−MS/MS data from different laboratories is necessary for inferring global trends from public data sets. To illuminate instrumentation factors that drive differences and variability, we used a standardized data analysis pipeline, including classical (CMN) and featurebased molecular networking (FBMN), to analyze data from a ring trial by 24 laboratories on identical sample sets of algal and DOM extracts that were mixed in predefined concentrations and spiked with standards. Our results showed that data sets from similar mass spectrometer types with unified instrument parameters were qualitatively comparable, resolving the same general trends and shared mass spectral features. Interlaboratory comparability was best for high-intensity features, while low-intensity features showed greater detection variability. Our analysis also highlights challenges when comparing data from instruments with different acquisition rates or operating with less standardized methods. Lastly, we provide recommendations for data integration, public data sharing, standardization, and best practices for standardized LC−MS/MS data acquisition, which will be critical for long-term time series and intercomparability of DOM chemical analyses.

DOM↗
DOE OSTI2020

A genoscape–network model for conservation prioritization in a migratory bird

Migratory animals are declining worldwide and coordinated conservation efforts are needed to reverse current trends. We devised a novel genoscape-network model that combines genetic analyses with species distribution modeling and demographic data to overcome challenges with conceptualizing alternative risk factors in migratory species across their full annual cycle. We applied our method to the long distance, Neotropical migratory bird, Wilson's Warbler (Cardellina pusilla). Despite a lack of data from some wintering locations, we demonstrated how the results can be used to help prioritize conservation of breeding and wintering areas. For example, we showed that when genetic, demographic, and network modeling results were considered together it became clear that conservation recommendations will differ depending on whether the goal is to preserve unique genetic lineages or the largest number of birds per unit area. More specifically, if preservation of genetic lineages is the goal, then limited resources should be focused on preserving habitat in the California Sierra, Basin Rockies, or Coastal California, where the 3 most vulnerable genetic lineages breed, or in western Mexico, where 2 of the 3 most vulnerable lineages overwinter. Alternatively, if preservation of the largest number of individuals per unit area is the goal, then limited conservation dollars should be placed in the Pacific Northwest or Central America, where densities are estimated to be the highest. Overall, our results demonstrated the utility of adopting a genetically based network model for integrating multiple types of data across vast geographic scales and better inform conservation decision-making for migratory animals.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2025

CryoDRGN-AI: neural ab initio reconstruction of challenging cryo-EM and cryo-ET datasets

Proteins and other biomolecules form dynamic macromolecular machines that are tightly orchestrated to move, bind, and perform chemistry. Cryo-electron microscopy (cryo-EM) and cryo-electron tomography (cryo-ET) can access the intrinsic heterogeneity of these complexes and are therefore key tools for understanding their function. However, 3D reconstruction of the collected imaging data presents a challenging computational problem, especially without any starting information, a setting termed ab initio reconstruction. Here, in this study, we introduce cryoDRGN-AI, a method leveraging an expressive neural representation and combining an exhaustive search strategy with gradient-based optimization to process challenging heterogeneous datasets. Using cryoDRGN-AI, we reveal new conformational states in large datasets, reconstruct previously unresolved motions from unfiltered datasets, and demonstrate ab initio reconstruction of biomolecular complexes from in situ data. With this expressive and scalable model for structure determination, we hope to unlock the full potential of cryo-EM and cryo-ET as a high-throughput tool for structural biology and discovery.

Levy, Axel [Stanford Univ., CA (United States); SL↗
DOE OSTI2024

CLAS12 remote data-stream processing using ERSAP framework

Implementing a physics data processing application is relatively straightforward with the use of current containerization technologies and container image runtime services, which are prevalent in most high-performance computing (HPC) environments. However, the process is complicated by the challenges associated with data provisioning and migration, impacting the ease of workflow migration and deployment. Transitioning from traditional file-based batch processing to data-stream processing workflows is suggested as a method to streamline these workflows. This transition not only simplifies file provisioning and migration but also significantly reduces the necessity for extensive disk space. Data-stream processing is particularly effective for real-time processing during data acquisition, thereby enhancing data quality assurance. This paper introduces the integration of the JLAB CLAS12 event reconstruction application within the ERSAP data-stream processing framework that facilitates the execution of streaming event reconstruction at a remote data center and enables the return streaming of reconstructed events to JLAB while circumventing the need for temporary data storage throughout the process.

Gyurjyan, Vardan↗
DOE OSTI2021

Development of a Unified Taxonomy for HVAC System Faults

Detecting and diagnosing HVAC faults is critical for maintaining building operation performance, reducing energy waste, and ensuring indoor comfort. An increasing deployment of commercial fault detection and diagnostics (FDD) software tools in commercial buildings in the past decade has significantly increased buildings’ operational reliability and reduced energy consumption. A massive amount of data has been generated by the FDD software tools. However, efficiently utilizing FDD data for ‘big data’ analytics, algorithm improvement, and other data-driven applications is challenging because the format and naming conventions of those data are very customized, unstructured, and hard to interpret. This paper presents the development of a unified taxonomy for HVAC faults. A taxonomy is an orderly classification of HVAC faults according to their characteristics and causal relations. The taxonomy includes fault categorization, physical hierarchy, fault library, relation model, and naming/tagging scheme. The taxonomy employs both a physical hierarchy of HVAC equipment and a cause-effect relationship model to reveal the root causes of faults in HVAC systems. A structured and standardized vocabulary library is developed to increase data representability and interpretability. The developed fault taxonomy can be used for HVAC system ‘big data’ analytics such as HVAC system fault prevalence analysis or the development of an HVAC FDD software standard. A common type of HVAC equipment-packaged rooftop unit (RTU) is used as an example to demonstrate the application of the developed fault taxonomy. Two RTU FDD software tools are used to show that after mapping FDD data according to the taxonomy, the meta-analysis of the multiple FDD reports is possible and efficient.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2020

CoREC: Scalable and Resilient In-memory Data Staging for In-situ Workflows

The dramatic increase in the scale of current and planned high-end HPC systems is leading new challenges, such as the growing costs of data movement and IO, and the reduced mean time between failures (MTBF) of system components. In-situ workflows, i.e., executing the entire application workflows on the HPC system, have emerged as an attractive approach to address data-related challenges by moving computations closer to the data, and staging-based frameworks have been effectively used to support in-situ workflows at scale. However, the resilience of these staging-based solutions has not been addressed, and they remain susceptible to expensive data failures. Furthermore, naive use of data resilience techniques such as n-way replication and erasure codes can impact latency and/or result in significant storage overheads. In this article, we present CoREC, a scalable and resilient in-memory data staging runtime for large-scale in-situ workflows. CoREC uses a novel hybrid approach that combines dynamic replication with erasure coding based on data access patterns. It also leverages multiple levels of replications and erasure coding to support diverse data resiliency requirements. Furthermore, the article presents optimizations for load balancing and conflict-avoiding encoding, and a low overhead, lazy data recovery scheme. We have implemented the CoREC runtime and have deployed with the DataSpaces staging service on leadership class computing machines and present an experimental evaluation in the article. Here, the experiments demonstrate that CoREC can tolerate in-memory data failures while maintaining low latency and sustaining high overall storage efficiency at large scales.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Deep Learning-based Parameterization of Complex 3D CO2 Saturation Data in Large-scale Geological Carbon Storage

In deep learning (DL), dimension reduction plays a pivotal role in improving training efficiency and minimizing overfitting, especially when working with complex datasets like three-dimensional (3D) saturation data. In the context of geological carbon storage (GCS), 3D saturation data introduces unique challenges due to its sparse nature and sharp transitions at plume boundaries, known as shock fronts. To tackle these challenges, we developed a novel DL framework that combines dimension reduction with advanced 3D reconstruction techniques. Our approach utilizes latent variables derived from 2D average saturation fields to efficiently capture the essential features of high-dimensional data while reducing the number of variables. This enhances both the robustness and accuracy of DL models, making the framework more practical for real-world applications. By offering a tailored solution for modeling complex 3D saturation dynamics, this framework holds significant potential for environmental monitoring, energy storage, and other geological applications.

Wang, Hongsheng [University of Texas at Austin]↗
DOE OSTI2022

Case Study: NREL Campus Chilled Water Storage Potential: Benchmark Datasets Development and Applications, Task 4 - Use Case Demonstration

The Benchmark Datasets Development and Applications project is a three-year collaboration between the National Renewable Energy Laboratory (NREL), Oak Ridge National Laboratory, Pacific Northwest National Laboratory, and Lawrence Berkeley National Laboratory. The project seeks to collect and curate high-resolution, well-calibrated time series of building operational and indoor/outdoor environmental data, which are crucial to understanding and optimizing building energy efficiency performance and demand flexibility capabilities as well as benchmarking energy algorithms. Project outcomes include approximately twelve high-fidelity building datasets, enhanced data representation tools, and four case studies to illustrate example applications. The goal of these case studies is to define and execute analyses that demonstrate how one or more datasets collected through this project can address a data gap or challenge historically faced by building stakeholders. This technical paper summarizes the findings of one of these case studies, in which we studied the operational efficiencies of the central cooling system at NREL. We looked at three years of data from the three chillers in the Field Test Laboratory Building (FTLB), from 2019 to 2021, to compare equipment operation and demand throughout the time period. Our analysis indicates that all three chillers are operating at or below the optimal loading conditions for most of the operation time, and thus there was no efficiency drop due to loading of the chillers at full capacity. Our recommendation is that no chiller capacity increase is needed; instead, the central plant could benefit from adopting advanced control logics for optimal sequencing of chillers during part load operations. Analysis of adding chilled water thermal storage to the central plant indicated 34% savings in demand cost and 24.5% savings in total cost (energy consumption and demand charge cost). The payback period is estimated to be 11-22 years with an assumed TES cost of $\$$100-$200 per ton. This case study shows how a selected dataset is used to solve a practical building problem - learning the operational status of its components, analyzing the effectiveness of a proposed new technique, and aiding decision-making for the building operations and maintenance team.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2023

Synchrotron X-ray diffraction analysis of constituent phases in transition joint between nickel alloy 738LC and a MnFeCoNiCu alloy

Synchrotron x-ray diffraction (XRD) analysis was performed on transition joints between a single-phase MnFeCoNiCu alloy and Ni-base Alloy 738LC to efficiently identify the constituent phases across the interface, with different levels of material intermixing generated by laser-welding with variable power. Here, the intermixing extent was quantified by postmortem energy dispersive x-ray spectroscopy mapping. Diffraction-based analyses on complex composition spaces with limited prior data present significant challenges because elemental substitution in both disordered and ordered phases is more extensive than in conventional alloy spaces, which may effectuate relatively large shifts in the observed lattice parameters that convolute the analysis. Therefore, thermodynamic simulations and crystallographic literature data were employed to construct a system-specific diffraction library of twelve prospective phases for the composition space investigated. Subsequently, for predicted disordered cubic phases, statistical hard-sphere models were established to estimate the lattice parameters and predict diffraction peak positions for inclusion in the library. The library was then employed to analyze diffraction profiles measured from the variably intermixed transition joints, with focus on accounting for both high and low-intensity peaks. 99.0 % of diffraction peaks with relative intensity greater than 0.001 were assigned to phases from the system-specific library, exemplifying rigorous peak accounting and indicating that no unexpected phases were present. Up to six of the twelve library phases were experimentally found in the transition joints. The lattice parameters predicted by the statistical hard-sphere model based upon thermodynamic simulations agree reasonably well with the measured values for the disordered FCC matrix phase.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Toward a Generalizable Prediction Model of Molten Salt Mixture Density with Chemistry‐Informed Transfer Learning

Optimally designing applications of molten salts requires knowledge of their thermophysical properties over a wide range of temperatures and compositions. There exist significant gaps in existing databases and this data can be challenging to experimentally measure due to high temperatures, salt corrosivity, and salt hygroscopicity. Existing databases have been used to create Redlich–Kister (RK) models for mixture density showing improved accuracy with respect to ideal mixing assumptions, but these models require subcomponent data measurements for each new system, therefore lacking generality. In order to address generalizability and data sparsity, a transfer learning procedure is proposed to train deep neural networks (DNNs) using a combination of semi‐empirical relationships (RK), data from the thermophysical arm of the molten salt thermal properties database and universal ab initio properties of component mixtures taken from the joint automated repository for various integrated simulations (JARVIS) classical force‐field inspired descriptors database to predict density in molten salts. Herein, it is shown that DNNs predict molten salt density with an r 2 over 0.99 and a mean absolute percentage error under 1%, outperforming alternative methods.

inorganic materials↗
DOE OSTI2023

Derivative-based SINDy (DSINDy): Addressing the challenge of discovering governing equations from noisy data

Recent advances in the field of data-driven dynamics allow for the discovery of ODE systems using state measurements. One approach, known as Sparse Identification of Nonlinear Dynamics (SINDy), assumes the dynamics are sparse within a predetermined basis in the states and finds the expansion coefficients through linear regression with sparsity constraints. This approach requires an accurate estimation of the state time derivatives, which is not necessarily possible in the high-noise regime without additional constraints. We present an approach called Derivative-based SINDy (DSINDy) that combines two novel methods to improve ODE recovery at high-noise levels. First, we denoise the state variables by applying a projection operator that leverages the assumed basis for the system dynamics. Second, we use a second order cone program (SOCP) to find the derivative and governing equations simultaneously. We derive theoretical results for the projection-based denoising step, which allow us to estimate the values of hyperparameters used in the SOCP formulation. This underlying theory helps limit the number of required user-specified parameters. Finally, we present results demonstrating that our approach leads to improved system recovery for the Van der Pol oscillator, the Duffing oscillator, the Rössler attractor, and the Lorenz 96 model.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Leveraging Real-World Driving Data for Design and Impact Evaluation of Energy Efficient Control Strategies

Modeling and simulation are crucial in the development of advanced energy efficient control strategies. Utilizing real-world driving data as the underlying basis for control design and simulation lends veracity to projected real-world energy savings. Standardized drive cycles are limited in their utility for evaluating advanced driving strategies that utilize connectivity and on-vehicle sensing, primarily because they are typically intended for evaluating emissions and fuel economy under controlled conditions. Real-world driving data, because of its scale, is a useful representation of various road types, driving styles, and driving environments. The scale of real-world data also presents challenges in effectively using it in simulations. A fast and efficient simulation methodology is necessary to handle the large number of simulations performed for design analysis and impact evaluation of control strategies. In this study, two methods are presented of leveraging real-world data in both design optimization of energy efficient control strategies and in evaluating the real-world impact of those control strategies upon large-scale deployment. Through these methodologies, strategies with highest impact on energy savings were selected to be implemented as control algorithms. The developed algorithms were incorporated into a vehicle dynamics and powertrain control architecture implemented on a Cadillac CT6 demonstration vehicle. The control algorithms were then exercised on real-world driving scenarios to determine their impact on collective energy savings. The methodology utilizes the large-scale driving data sets maintained by the National Renewable Energy Laboratory to extract real-world driving scenarios and efficient simulation software tools. The insights obtained through this research help in guiding technology selection for energy efficient driving controls.

ARPA-E↗
DOE OSTI2026

The Art of Automation: Translating Electron Microscopy Workflows Into Automated Processes

Acquiring data using a scanning transmission electron microscope (STEM) is a complex, multi-step process. The intricacy of the process depends on the type of sample, composition of the material, desired results of the experiment, resolution requirement and other experimental factors. Each experiment presents unique complications, such as sample drift and contamination, that the microscopist must consider when acquiring data. All these challenges are handled fluidly and expertly by experienced microscopists, but to reach new levels of innovation in material development, including greater reproducibility, throughput, and precision, the automation of these workflows is essential. The initial phase of this work involved translating intuition-based workflows into discrete, programmable steps. Some common key stages in STEM workflows are the initial tuning, scanning the sample for areas of interest, and then acquiring the data. Each stage can be broken further into specific parameter adjustments, such as aberration correction and dwell time optimization, depending on the experiment. When deconstructing various experiments each step was assessed for automation feasibility based on the amount of real time operator decisions. There are steps that lend themselves to automation more readily than others, such as course focusing and sample screening, but there is potential for full automation of all stages with time. As an initial step, an automated montage routine was developed, allowing for the efficient acquisition of large portions of the sample without requiring continuous intervention from the operator. The automation of this small process of the procedure demonstrates the value of this capability. A major challenge in automation arises from discrepancies between commanded, reported and actual stage movements. Using systematic tests, stage movement was quantified. This error can be corrected algorithmically for more accurate workflows in the future. Expanding automation capabilities would result in larger, more efficient data acquisition which allows for more robust statistical analysis. Additionally, this work lays the groundwork for a closed loop system where machine learning algorithms would intake automatically acquired data and make real time decisions. By progressively automating this instrument, this work establishes the foundation for fully automated experimentation in transmission electron microscopy.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Advanced Materials & Manufacturing Technology (AMMT): Process understanding for qualifying LPBF 316H SS

Investigations were conducted in fiscal years (FY) 2023 and 2024 to gather relevant data sets addressing challenges related to qualifying 316H stainless steel (SS) for use in future nuclear reactors. This work was a collaborative effort involving researchers from Idaho National Laboratory (INL), Argonne National Laboratory (ANL), and Oak Ridge National Laboratory (ORNL). Key outcomes included: • Development of process-structure-property data sets to better understand the relationships between manufacturing processes, material structure, and performance characteristics. • Establishment of an in-situ monitoring system to link these various data sets together. • Detailed characterization of the raw material feedstock to further strengthen the understanding of process-structure and process-property relationships. Building on this foundation, in FY25 there was interest in exploring the behavior of additively manufactured 316H SS under different test conditions to support the overall material qualification process. Los Alamos National Laboratory (LANL) was tasked with providing specimen samples to the collaborating labs, who then conducted a round-robin study examining factors like selective heat treatment, low-cycle fatigue (LCF), and tensile-creep (T-C) properties. Additionally, high-temperature differential scanning calorimetry (DSC) was performed by LANL to better understand how the material's thermal characteristics change as it is heated up to the melting point. This provided a more comprehensive understanding of the material's behavior. The combined results from these investigations could potentially be used to support the inclusion of 316H stainless steel in Section III, Division 5 of the relevant codes and standards, allowing its use in future nuclear reactor applications.

36 MATERIALS SCIENCE↗