Search NASA⌕ Search

SEARCH · Search NASA

Results for “data challenges”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 685 records · Page 38
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Rucio beyond ATLAS: experiences from Belle II, CMS, DUNE, EISCAT3D, LIGO/VIRGO, SKA, XENON

For many scientific projects, data management is an increasingly complicated challenge. The number of data-intensive instruments generating unprecedented volumes of data is growing and their accompanying workflows are becoming more complex. Their storage and computing resources are heterogeneous and are distributed at numerous geographical locations belonging to different administrative domains and organisations. These locations do not necessarily coincide with the places where data is produced nor where data is stored, analysed by researchers, or archived for safe long-term storage. To fulfil these needs, the data management system Rucio has been developed to allow the high-energy physics experiment ATLAS at LHC to manage its large volumes of data in an efficient and scalable way. But ATLAS is not alone, and several diverse scientific projects have started evaluating, adopting, and adapting the Rucio system for their own needs. As the Rucio community has grown, many improvements have been introduced, customisations have been added, and many bugs have been fixed. Additionally, new dataflows have been investigated and operational experiences have been documented. In this article we collect and compare the common successes, pitfalls, and oddities that arose in the evaluation efforts of multiple diverse experiments, and compare them with the ATLAS experience. This includes the high-energy physics experiments Belle II and CMS, the neutrino experiment DUNE, the scattering radar experiment EISCAT3D, the gravitational wave observatories LIGO and VIRGO, the SKA radio telescope, and the dark matter search experiment XENON.

Lassnig, Mario↗
DOE OSTI2023

Exploring Future Storage Options for ATLAS at the BNL/SDCC facility

The ATLAS experiment is expected to deliver an unprecedented amount of scientific data in the High Luminosity(HL-LHC) era. As the demand for disk storage capacity in ATLAS continues to rise steadily, the BNL Scientific Data and Computing Center (SDCC) faces challenges in terms of cost implications for maintaining multiple disk copies and adapting to the coming ATLAS storage requirements. To address these challenges, the SDCC Storage team has undertaken a thorough analysis of the ATLAS experiment's requirements, matching them to suitable storage options and strategies, and has explored alternatives to enhance or replace the current storage solution. This paper aims to present the main challenges encountered while supporting big data experiments such as ATLAS. We describe the experiment's specific requirements and priorities, particularly focusing on the critical storage system characteristics of the high-luminosity run and how the key storage components provided by the Storage team work together: the dCache disk storage system; its archival back-end, HPSS; and its OS-level backend Storage. Specifically, we investigate a novel approach to integrate Lustre and XRootD. In this setup, Lustre serves as backend storage and XRootD acts as an access layer frontend, supporting various grid access protocols. Additionally, we also describe the validation and commissioning tests, including the performance comparison between dCache and XRootd. Furthermore, we provide a performance and cost analysis comparing OpenZFS and LINUX MD RAID, evaluate different storage software stacks, and showcase stress tests conducted to validate Third Party Copy (TPC) functionality.

99 GENERAL AND MISCELLANEOUS↗
NASA NTRS2005

Intelligent Systems: Terrestrial Observation and Prediction Using Remote Sensing Data

NASA has made science and technology investments to better utilize its large space-borne remote sensing data holdings of the Earth. With the launch of Terra, NASA created a data-rich environment where the challenge is to fully utilize the data collected from EOS however, despite unprecedented amounts of observed data, there is a need for increasing the frequency, resolution, and diversity of observations. Current terrestrial models that use remote sensing data were constructed in a relatively data and compute limited era and do not take full advantage of on-line learning methods and assimilation techniques that can exploit these data. NASA has invested in visualization, data mining and knowledge discovery methods which have facilitated data exploitation, but these methods are insufficient for improving Earth science models that have extensive background knowledge nor do these methods refine understanding of complex processes. Investing in interdisciplinary teams that include computational scientists can lead to new models and systems for online operation and analysis of data that can autonomously improve in prediction skill over time.

Coughlan, Joseph C.↗
NASA NTRS2017

Transforming Science Data for GIS: How to Find and Use NASA Earth Observation Data Without Being a Rocket Scientist

NASAs Earth Observing System Data Information System (EOSDIS) manages Earth Observation satellites and the Distributed Active Archive Centers (DAACs), where the data is stored and processed. The challenge is that Earth Observation data is complicated. There is plenty of data available, however, the science teams have had a top-down approach: define what it is you are trying to study -select a set of satellite(s) and sensor(s), and drill down for the data.Our alternative is to take a bottom-up approach using eight environmental fields of interest as defined by the Group on Earth Observations (GEO) called Societal Benefit Areas (SBAs): Disaster Resilience (DR) Public Health Surveillance (PHS) Energy and Mineral Resource Management (EMRM) Water Resources Management (WRM) Infrastructure and Transport Management (ITM) Sustainable Urban Development (SUD) Food Security and Sustainable Agriculture (FSSA) Biodiversity and Ecosystems Sustainability (BES).

DAAC↗
NASA NTRS2017

Giovanni - The Bridge Between Data and Science

This article describes new features in the Geospatial Interactive Online Visualization ANd aNalysis Infrastructure (Giovanni), a user-friendly online tool that enables visualization, analysis, and assessment of NASA Earth science data sets without downloading data and software. Since the satellite era began, data collected from Earth-observing satellites have been widely used in research and applications; however, using satellite-based data sets can still be a challenge to many. To facilitate data access and evaluation, as well as scientific exploration and discovery, the NASA Goddard Earth Sciences (GES) Data and Information Services Center (DISC) has developed Giovanni for a wide range of users around the world. This article describes the latest capabilities of Giovanni with examples, and discusses future plans for this innovative system.

GPM↗
NASA NTRS2018

Serving NASA GES DISC Multi-Spatiotemporal Earth Science Data to the GIS community

NASA Earth Science (ES) data is essential to a wide range of GIS research and applications. However, for many GIS users, searching, accessing, using and analyzing NASA ES data can be of a great challenge- ranging from the sheer data volumes, types of science parameters, and to the complexity of data encoding formats. As one of the twelve NASA Science Mission Directorate (SMD) Data Centers, Goddard Earth Sciences (GES) Data and Information Services Center (DISC) archives and distributes petabytes of ES parameters covering atmosphere, land, and ocean fields. Most data are multidimensional and multi-spatiotemporal in nature and are encoded in different science data formats (e.g, HDF, HDF-EOS, netCDF, GRIB, binary), which usually contain multiple variables and different metadata information. By far, GES DISC has been developing a number of services and online tools to help GIS users to easily explore our data products. In this presentation, we will describe our ArcGIS-based data accessing and visualization services and portals, which allow users directly exploring the multi-spatiotemporal ES data in ArcGIS clients without having to pre-download/import the data. The ArcGIS services are also compliant with the Open Geospatial Consortium (OGC) Web Coverage Service (WCS) and Web Map Service (WMS) protocols and can be accessed by any other WCS/WMS clients to get customized GES DISC EO data on-the-fly from such services.

Wei, Jennifer↗
NASA NTRS2021

An Evaluation of Remotely Sensed and In-Situ Data Sufficiency for SGMA-Scale Groundwater Studies in the Central Valley, California

California’s Central Valley aquifer is a critical freshwater resource for the state, providing drinking water to 6.5 million residents and irrigation water for more than half of the nation’s produce. Increasing demands have led to a steady decline of aquifer water levels, leading to irreversible compaction and land subsidence. The Sustainable Groundwater Management Act of California (SGMA), instituted in 2014, requires high-priority regions to establish sustainability plans by 2020-2022 that mitigate groundwater depletion and land subsidence. Many regions within the Central Valley lack groundwater monitoring wells or continuous well data, rendering in-situ monitoring challenging. When combined with other data, NASA’s Gravity Recovery and Climate Experiment (GRACE) and GRACE Follow On (GRACE-FO) missions provide valuable information about groundwater storage changes at a sub-basin scale. Interferometric Surface Aperture Radar (InSAR) measurements can map land subsidence, and Global Positioning System (GPS) can be used to estimate crustal uplift. Yet, all of these measurements have differences in resolution, coverage, discretization, and record length. Here we assess the various datasets that are potentially useful for Central Valley sub-basin groundwater monitoring and provide an analysis of gaps and other issues in regard to their suitability for SGMA-related analysis. Finally, we offer next steps and recommendations on data treatment and integration.

Kyra H Kim↗
DOE OSTI2022

2.2.5.401 - Ocean Observing Prize

The Powering the Blue Economy: Ocean Observing Prize is a $2.4M contest that challenges innovators to develop solutions that integrate marine renewable energy with ocean observation platforms to revolutionize our ability to monitor, manage, and understand the ocean. Competitors are challenged to design, build, and test novel, wave-powered, self-charging autonomous underwater vehicle (AUV) systems that could be suitable for a 6-month deployment in the Atlantic Ocean to monitor hurricane formation. Through engagement with the end user ocean observing community, the team identified challenges in a lack of data on storm intensity, endangering coastal communities. This challenge could be addressed by collecting data before, during, and after a hurricane develops and strengthens at sea. Amassing this data, however, requires an ocean observing system that can be deployed at sea for long periods, waiting for approaching storms. To help solve this problem, competitors are working to integrate wave energy capture with AUV systems to collect this data, helping to better protect coastal communities from oncoming storms. Solutions developed through this prize aim to enable long-term collection of data before, during, and after a hurricane that will help better forecast storm intensity. The prize mechanism provides an environment to accelerate technology development for innovators and helps to improve awareness of marine energy and its ability to provide power to blue economy applications.

coastal communities↗
DOE OSTI2024

Types of data: terminology and examples

Data produced at wastewater utilities are obtained through a variety of devices, including actuators, control devices, and sensors. This results in data that can be highly variable in its structure. Dealing with the resulting heterogeneity of data formats can be a challenge when storing or interpreting the data. For this reason, this chapter serves as an overview for the most important structural aspects of data typically found at a utility. The specific aims of this chapter are to:

Plana, Queralt↗
NASA NTRS2006

Analyzing Electric Field Morphology Through Data-Model Comparisons of the GEM IM/S Assessment Challenge Events

The storm-time inner magnetospheric electric field morphology and dynamics are assessed by comparing numerical modeling results of the plasmasphere and ring current with many in situ and remote sensing data sets. Two magnetic storms are analyzed, April 22,2001 and October 21-23,2001, which are the events selected for the Geospace Environment Modeling (GEM) Inner Magnetosphere/Storms (IM/S) Assessment Challenge (IMSAC). The IMSAC seeks to quantify the accuracy of inner magnetospheric models as well as synthesize our understanding of this region. For each storm, the ring current-atmosphere interaction model (RAM) and the dynamic global core plasma model (DGCPM) were run together with various settings for the large-scale convection electric field and the nightside ionospheric conductance. DGCPM plasmaspheric parameters were compared with IMAGE-EUV plasmapause extractions and LANL-MPA plume locations and velocities. RAM parameters were compared with Dst*, LANL-MPA fluxes and moments, IMAGE-MENA images, and IMAGE-HENA images. Both qualitative and quantitative comparisons were made to determine the electric field morphology that allows the model results to best fit the plasma data at various times during these events. The simulations with self-consistent electric fields were, in general, better than those with prescribed field choices. This indicates that the time-dependent modulation of the inner magnetospheric electric fields by the nightside ionosphere is quite significant for accurate determination of these fields (and their effects). It was determined that a shielded Volland-Stern field description driven by the 3-hour Kp index yields accurate results much of the time, but can be quite inconsistent. The modified Mcllwain field description clearly lagged in overall accuracy compared to the other fields, but matched some data sets (like Dst*) quite well. The rankings between the simulations varied depending on the storm and the individual data sets, indicating that each field description did well for some place, time, and energy range during the events, as well as doing less well in other places, times, and energies. Several unresolved issues regarding the storm-time inner magnetospheric electric field are discussed.

Liemohn, Michael W.↗
DOE OSTI2026

Comparability of Liquid Chromatography Tandem Mass Spectrometry Analysis of Dissolved Organic Matter across Laboratories

Non-targeted liquid chromatography tandem highresolution mass spectrometry (LC−MS/MS) is increasingly applied for the structure-resolved chemical analysis of dissolved organic matter (DOM). With new developments in MS instrumentation and analysis software, the approach has gained substantial momentum over the past decade. However, achieving high-quality analytical data that is reproducible and comparable across laboratories can be a bottleneck in non-targeted metabolomics and organic matter chemical analysis, especially for data reuse in repository-scale analyses. Understanding the capabilities as well as challenges of comparing LC−MS/MS data from different laboratories is necessary for inferring global trends from public data sets. To illuminate instrumentation factors that drive differences and variability, we used a standardized data analysis pipeline, including classical (CMN) and featurebased molecular networking (FBMN), to analyze data from a ring trial by 24 laboratories on identical sample sets of algal and DOM extracts that were mixed in predefined concentrations and spiked with standards. Our results showed that data sets from similar mass spectrometer types with unified instrument parameters were qualitatively comparable, resolving the same general trends and shared mass spectral features. Interlaboratory comparability was best for high-intensity features, while low-intensity features showed greater detection variability. Our analysis also highlights challenges when comparing data from instruments with different acquisition rates or operating with less standardized methods. Lastly, we provide recommendations for data integration, public data sharing, standardization, and best practices for standardized LC−MS/MS data acquisition, which will be critical for long-term time series and intercomparability of DOM chemical analyses.

DOM↗
NASA NTRS2012

Quantifying Interannual Variability of the UTLS Ozone Using Assimilation of Satellite Data

An accurate representation of spatial and temporal variability of the Upper Troposphere Lower Stratosphere (UTLS) ozone is essential for understanding both the tropospheric ozone budget and ozone s contribution to radiative forcing. The complex, dynamically driven structure of trace gas fields in the UTLS presents a challenge to data-based and modelling studies. Small features are not fully resolved in data from limb-sounding instruments such as the Microwave Limb Sounder on EOS-Aura (the EOS-MLS), but are captured in assimilation of those data as vertical structure is added from the assimilated meteorology. This will be demonstrated using a multi-year assimilation of EOS-MLS observations in the Goddard Earth Observing System, Version 5 (GEOS-5) data assimilation system. The results demonstrate the realism of the seasonal and year to year variability of laminar structures in the mid-latitudinal ozone field between years 2005-2007, for which independent validation data are available from the HIRDLS instrument. The analysis is done in the context of the underlying large scale dynamics. The lifetimes of most research instruments are too short for them to be used throughout the duration of long-term (at least 3 decades) reanalyses. For example, the EOS-MLS instrument has operated since mid-2004 until present. By contrast, Solar Backscatter Ultra Violet (SBUV) measurements provide continuous data since late 1978, but their vertical resolution is insufficient to represent the profile shape in the UTLS. Assimilation of these SBUV/2 observations in the GEOS-5 data assimilation system has hitherto not captured a realistic ozone structure in the UTLS, even though transport studies using GEOS-5 wind fields do show such structures. We show that careful construction of the background error covariance structure in GEOS-5 can lead to more realistic UTLS ozone fields when assimilating SBUV/2 observations. The reasoning behind this will be discussed, emphasizing the need to retain the sharp gradient of ozone concentrations across the tropopause. We analyze the UTLS ozone distributions in multi-year SBUV/2 assimilation experiments, comparing the results against the independent HIRDLSdataset and, for a longer period, with the MLS assimilation and discuss the consequences for tropospheric ozone and radiative forcing.

Wargan, K.↗
DOE OSTI2020

A genoscape–network model for conservation prioritization in a migratory bird

Migratory animals are declining worldwide and coordinated conservation efforts are needed to reverse current trends. We devised a novel genoscape-network model that combines genetic analyses with species distribution modeling and demographic data to overcome challenges with conceptualizing alternative risk factors in migratory species across their full annual cycle. We applied our method to the long distance, Neotropical migratory bird, Wilson's Warbler (Cardellina pusilla). Despite a lack of data from some wintering locations, we demonstrated how the results can be used to help prioritize conservation of breeding and wintering areas. For example, we showed that when genetic, demographic, and network modeling results were considered together it became clear that conservation recommendations will differ depending on whether the goal is to preserve unique genetic lineages or the largest number of birds per unit area. More specifically, if preservation of genetic lineages is the goal, then limited resources should be focused on preserving habitat in the California Sierra, Basin Rockies, or Coastal California, where the 3 most vulnerable genetic lineages breed, or in western Mexico, where 2 of the 3 most vulnerable lineages overwinter. Alternatively, if preservation of the largest number of individuals per unit area is the goal, then limited conservation dollars should be placed in the Pacific Northwest or Central America, where densities are estimated to be the highest. Overall, our results demonstrated the utility of adopting a genetically based network model for integrating multiple types of data across vast geographic scales and better inform conservation decision-making for migratory animals.

60 APPLIED LIFE SCIENCES↗
NASA NTRS2012

Solar Thermal Upper Stage Liquid Hydrogen Pressure Control Testing and Analytical Modeling

The demonstration of a unique liquid hydrogen (LH2) storage and feed system concept for solar thermal upper stage was cooperatively accomplished by a Boeing/NASA Marshall Space Flight Center team. The strategy was to balance thermodynamic venting with the engine thrusting timeline during a representative 30-day mission, thereby, assuring no vent losses. Using a 2 cubic m (71 cubic ft) LH2 tank, proof-of-concept testing consisted of an engineering checkout followed by a 30-day mission simulation. The data were used to anchor a combination of standard analyses and computational fluid dynamics (CFD) modeling. Dependence on orbital testing has been incrementally reduced as CFD codes, combined with standard modeling, continue to be challenged with test data such as this.

Olsen, A. D.↗
NASA NTRS2002

Software Engineering Support of the Third Round of Scientific Grand Challenge Investigations: An Earth Modeling System Software Framework Strawman Design that Integrates Cactus and UCLA/UCB Distributed Data Broker

One of the most significant challenges in large-scale climate modeling, as well as in high-performance computing in other scientific fields, is that of effectively integrating many software models from multiple contributors. A software framework facilitates the integration task. both in the development and runtime stages of the simulation. Effective software frameworks reduce the programming burden for the investigators, freeing them to focus more on the science and less on the parallel communication implementation, while maintaining high performance across numerous supercomputer and workstation architectures. This document proposes a strawman framework design for the climate community based on the integration of Cactus, from the relativistic physics community, and UCLA/UCB Distributed Data Broker (DDB) from the climate community. This design is the result of an extensive survey of climate models and frameworks in the climate community as well as frameworks from many other scientific communities. The design addresses fundamental development and runtime needs using Cactus, a framework with interfaces for FORTRAN and C-based languages, and high-performance model communication needs using DDB. This document also specifically explores object-oriented design issues in the context of climate modeling as well as climate modeling issues in terms of object-oriented design.

Talbot, Bryan↗
DOE OSTI2025

CryoDRGN-AI: neural ab initio reconstruction of challenging cryo-EM and cryo-ET datasets

Proteins and other biomolecules form dynamic macromolecular machines that are tightly orchestrated to move, bind, and perform chemistry. Cryo-electron microscopy (cryo-EM) and cryo-electron tomography (cryo-ET) can access the intrinsic heterogeneity of these complexes and are therefore key tools for understanding their function. However, 3D reconstruction of the collected imaging data presents a challenging computational problem, especially without any starting information, a setting termed ab initio reconstruction. Here, in this study, we introduce cryoDRGN-AI, a method leveraging an expressive neural representation and combining an exhaustive search strategy with gradient-based optimization to process challenging heterogeneous datasets. Using cryoDRGN-AI, we reveal new conformational states in large datasets, reconstruct previously unresolved motions from unfiltered datasets, and demonstrate ab initio reconstruction of biomolecular complexes from in situ data. With this expressive and scalable model for structure determination, we hope to unlock the full potential of cryo-EM and cryo-ET as a high-throughput tool for structural biology and discovery.

Levy, Axel [Stanford Univ., CA (United States); SL↗
DOE OSTI2024

CLAS12 remote data-stream processing using ERSAP framework

Implementing a physics data processing application is relatively straightforward with the use of current containerization technologies and container image runtime services, which are prevalent in most high-performance computing (HPC) environments. However, the process is complicated by the challenges associated with data provisioning and migration, impacting the ease of workflow migration and deployment. Transitioning from traditional file-based batch processing to data-stream processing workflows is suggested as a method to streamline these workflows. This transition not only simplifies file provisioning and migration but also significantly reduces the necessity for extensive disk space. Data-stream processing is particularly effective for real-time processing during data acquisition, thereby enhancing data quality assurance. This paper introduces the integration of the JLAB CLAS12 event reconstruction application within the ERSAP data-stream processing framework that facilitates the execution of streaming event reconstruction at a remote data center and enables the return streaming of reconstructed events to JLAB while circumventing the need for temporary data storage throughout the process.

Gyurjyan, Vardan↗
DOE OSTI2021

Development of a Unified Taxonomy for HVAC System Faults

Detecting and diagnosing HVAC faults is critical for maintaining building operation performance, reducing energy waste, and ensuring indoor comfort. An increasing deployment of commercial fault detection and diagnostics (FDD) software tools in commercial buildings in the past decade has significantly increased buildings’ operational reliability and reduced energy consumption. A massive amount of data has been generated by the FDD software tools. However, efficiently utilizing FDD data for ‘big data’ analytics, algorithm improvement, and other data-driven applications is challenging because the format and naming conventions of those data are very customized, unstructured, and hard to interpret. This paper presents the development of a unified taxonomy for HVAC faults. A taxonomy is an orderly classification of HVAC faults according to their characteristics and causal relations. The taxonomy includes fault categorization, physical hierarchy, fault library, relation model, and naming/tagging scheme. The taxonomy employs both a physical hierarchy of HVAC equipment and a cause-effect relationship model to reveal the root causes of faults in HVAC systems. A structured and standardized vocabulary library is developed to increase data representability and interpretability. The developed fault taxonomy can be used for HVAC system ‘big data’ analytics such as HVAC system fault prevalence analysis or the development of an HVAC FDD software standard. A common type of HVAC equipment-packaged rooftop unit (RTU) is used as an example to demonstrate the application of the developed fault taxonomy. Two RTU FDD software tools are used to show that after mapping FDD data according to the taxonomy, the meta-analysis of the multiple FDD reports is possible and efficient.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗