Search NASA⌕ Search

SEARCH · Search NASA

Results for “data management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13

A National Infrastructure for Artificial Intelligence on the Grid (NI4AI) (Final Scientific/Technical Report)

Electric utilities have traditionally taken a very pragmatic yet myopic approach with grid sensors and the resulting collected data. Sensors are purchased and deployed to solve a specific, known problem that has risen to sufficient awareness as to justify the effort of deploying sensors and the needed capital investment. This sensor data flows into proprietary software packages with limited functionality intended only to address the initial problem. This approach aligns with the financial incentives of the utility to deploy capital into fixed hardware assets for which the corporations earn a rate of return. This mentality stands in stark contrast to the big data revolution that started nearly 25 years ago with the rise of Google. In this worldview, data is a fundamental business asset; successful organizations collect, store, explore, merge, and exploit as much data as possible to not only solve problems well understood today but also to tackle new problems that will inevitably rise tomorrow. The ARPA-E Open Innovation 2018 project entitled A National Infrastructure for Artificial Intelligence on the Grid or NI4AI for short was designed to demonstrate this alternative paradigm for using data. To do this, the project was composed of three key thrust areas. The first major component deployed a variety of high-frequency grid sensors and captured terabytes of both wide-scale and localized grid measurements, generating high-value datasets for grid research and algorithm development. The second aspect made available PingThings’ PredictiveGridTM, a horizontally scalable, cloud-based data management and AI platform built for time series data to explore and exploit the collected data. Finally, the project fostered a diverse and open research community composed of experts from numerous fields through focused educational content, code sharing, and data science competitions. Shifting away from “single use” sensors and closed data silos within electric utilities is a major benefit to the public at large. This legacy approach to data is incredibly (1) capital intensive (new sensors must be deployed for each new problem and problems tend to arise continuously) and (2) painfully slow (new problems must be identified first and then new sensors must be deployed to collect data to begin to address the issue). The transition to a carbon neutral grid requires a massive transformation of the existing grid infrastructure and will continue to challenge the legacy grid in unforeseen ways. The only way to make the energy transition cost effective is for utilities to abandon this dated data paradigm and adopt more contemporary approaches. NI4AI has shown that it is technically possible and economically feasible to ingest, explore, and exploit grid data collected from even very high frequency sensing, such as continuous point on wave sensors collecting measurements 10,000 times a second. In fact, the PredictiveGrid platform used is commercially available and deployed at several utilities in the United States. Project accomplishments were numerous and included (1) making available a state of the art time series platform to the community, (2) collecting over 520 streams of time series data from grid sensors totaling over 1 trillion grid measurements, and (3) developing and nurturing a community within the industry focused on the use of data to create value for utilities and, ultimately, end consumers.

97 MATHEMATICS AND COMPUTING↗

TORONE: Total Characterisation by Remote Observation of Nuclear Environments - 20591

This paper describes the autonomous robotic systems developed in the TORONE project (Total Characterisation by Remote Observation of Nuclear Environments) for combined characterisation by remote techniques in nuclear environments. There are four main components to the system. Firstly, robot operation which uses a Robot Operating System (ROS) allowing ease of communication between hardware and software alongside robot vision systems such as Lidar and/or photogrammetry. Secondly, radiation detection instrumentation for gamma and neutron sources which is also collimated for spatial resolution. Thirdly, photonic characterisation instrumentation for remote material identification by means of Raman spectroscopy and Laser induced breakdown spectroscopy (LIBS). Finally, a data management system is described which contains the TORONE characterisation data with the ability to overlay material characterisation information onto 3D maps of the nuclear environment. Preliminary results are described alongside discussion of some of the synergies of the combined approach. There are many applications of such an autonomous system in nuclear decommissioning, waste management and asset management. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Spatialyze: A Geospatial Video Analytics System with Spatial-Aware Optimizations

Videos that are shot using commodity hardware such as phones and surveillance cameras record various metadata such as time and location. We encounter suchgeospatial videoson a daily basis and such videos have been growing in volume significantly. Yet, we do not have data management systems that allow users to interact with such data effectively. In this paper, we describe Spatialyze, a new framework for end-to-end querying of geospatial videos. Spatialyze comes with a domain-specific language where users can construct geospatial video analytic workflows using a 3-step, declarative,build-filter-observeparadigm. Internally, Spatialyze leverages the declarative nature of such workflows, the temporal-spatial metadata stored with videos, and physical behavior of real-world objects to optimize the execution of workflows. Our results using real-world videos and workflows show that Spatialyze can reduce execution time by up to 5.3×, while maintaining up to 97.1% accuracy compared to unoptimized execution.

Computer Science↗

New Architecture to Support Integration and Processing of Seismic Data from Heterogeneous Sources

The Geophysical Monitoring Program (GMP) at Lawrence Livermore National Lab (LLNL) maintains a database and supporting infrastructure for geophysical data used in support of the Nuclear Detonation Detection mission. This database includes data from multiple sources, many of which do not distribute data to the public or for which there is no automated means of access. For example, Figure 1 shows (left) the distribution of waveform data in our database by source. The Incorporated Research Institutions for Seismology Data Management Center (IRISDMC) is our major source of waveform data and those data may be retrieved at will using the Federated Digital Seismograph Networks FDSN web Application Programming Interface (API). However, the next 6 most important sources of waveform data have no or only limited automated access to waveforms. As Figure 1 (right) shows, it is very common for waveform records associate with an event in our database to come from two or more sources, and in some cases data come from 10 sources. This diversity of data sources drives our need for efficient and correct integration of metadata, parametric data, and waveform data.

58 GEOSCIENCES↗

Atmospheric Radiation Measurement (ARM) User Facility 2020 Decadal Vision

The Atmospheric Radiation Measurement (ARM) user facility was established in 1989 by the U.S Department of Energy (DOE) Office of Biological and Environmental Research (BER) to provide an observational basis for studying the Earth’s climate. ARM began collecting observations in 1992 and was designated a user facility in 2003. The facility includes a network of extensively instrumented long-term fixed-location observatories and mobile facilities. The ARM facility also includes an aerial component to augment these ground-based measurements. Because of the diversity of in situ and remotely obtained observations, ARM’s data management infrastructure is designed to collect, process, and deliver data to the research community.

54 ENVIRONMENTAL SCIENCES↗

The Neurodata Without Borders ecosystem for neurophysiological data science

The neurophysiology of cells and tissues are monitored electrophysiologically and optically in diverse experiments and species, ranging from flies to humans. Understanding the brain requires integration of data across this diversity, and thus these data must be findable, accessible, interoperable, and reusable (FAIR). This requires a standard language for data and metadata that can coevolve with neuroscience. We describe design and implementation principles for a language for neurophysiology data. Our open-source software (Neurodata Without Borders, NWB) defines and modularizes the interdependent, yet separable, components of a data language. We demonstrate NWB’s impact through unified description of neurophysiology data across diverse modalities and species. NWB exists in an ecosystem, which includes data management, analysis, visualization, and archive tools. Thus, the NWB data language enables reproduction, interchange, and reuse of diverse neurophysiology data. More broadly, the design principles of NWB are generally applicable to enhance discovery across biology through data FAIRness.

59 BASIC BIOLOGICAL SCIENCES↗

NIRP Core Software Suite v.5.1

SAND2023-06610O The Nuclear Incident Response Program (NIRP) Core Software Suite is a set of code that supports multiple applications. It includes miscellaneous base code for data objects, mathematic equations, and user interface components that are used for developing large applications, including those used for data management. Among the types of applications: • Analyst Manager is used for managing contact information, which is also often included in reports and identifies the owners of calculations. • Radionuclide Viewer provides access to the Dose Coefficient File Package (DCFPAK) radiological data, and it complements the Mixture Manager tool. • Mixture Manager creates and manages radionuclides mixtures that are commonly used in other applications. • High Explosive Manager manages explosives and properties. • Chart Viewer is used for data charts such as meteorology charts and other charts specific to data needs. • Unit Converter converts numeric values to/from different units. This package includes object-oriented base code such as data objects describing the date, time, and location of an event; weather conditions; data types that support predefined ranges of values; a unit library, and others. The package also includes mathematic equations such as determining the distance between two points; erf functions; factorial functions; incidence fractions of probit curves; and others. Other features are customized user interface components for text fields; unit selectors (combo boxes); menus; panels; bug-tracking; and automated checks for software updates. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Collins, Ian↗

Distribution Development for Residual Inventory at the New York West Valley Site - 20400

The New York State Energy Research and Development Authority (NYSERDA) is the owner of the Western New York Nuclear Service Center (WNYNSC), a 1,351 ha site located approximately 48 km south of Buffalo, New York. In 1962, Nuclear Fuel Services, Inc. (NFS) entered into Agreements with the Atomic Energy Commission and New York State to construct the first commercial reprocessing plant of nuclear fuel in the United States. NFS, a private company, built and operated the spent fuel reprocessing plant and waste disposal facilities, processing 640 Mg of spent nuclear fuel from 1966 to 1972 under an Atomic Energy Commission license. Nuclear fuel reprocessing operations ended in 1972 and never reopened, leaving behind radioactive and chemical wastes. Operations led to contamination in a number of facilities and locations. Some of that contamination has migrated from waste disposal zones to other layers, formations, and features on and off the WNYNSC. Phase I decommissioning activities are ongoing and involve the removal of a number of areas and structures that have been associated with contamination. The purpose of this work is to outline the approach for characterizing contamination not associated with disposed wastes, contaminated structures, or specific releases. In this work, the term, residual radiological activity, is used to describe environmental contamination that exists subsequent to the completion of Phase I decommissioning activities, that is not associated with disposed wastes, contaminated structures, or specific releases. Contamination from the Site was quantified relative to data that characterize the concentrations of radionuclides that exist in background. Background concentrations are those present in the area but having no influence from Site related activities. The existence of residual radiological activity that is elevated relative to background has the potential to contribute to future risks to human health and the environment. As a consequence, the residual inventory information is used to inform the West Valley Probabilistic Performance Assessment (PPA) model to characterize potential future risks to human health and the environment. The centralized West Valley Data Management System (DMS) was the source of information for the data assembled in this analysis. The DMS is a fairly large compilation consisting of thousands of records from investigation studies, with sample dates ranging from 1990 to present. Samples from monitoring wells, boreholes, geoprobe studies, surface water, surface soils, storm water outfalls, ventilation stack filters, plant and animal tissues, and more are included in the DMS. Results are typically reported in units of activity per unit volume. For the purpose of the analyses presented here, all results were converted into consistent units of pCi per unit volume. Since 1990, data have been collected from various locations across the WNYNSC at different times with varying frequency over the course of several decades. As a consequence, a number of potential issues can arise with respect to the assembly of a dataset that is deemed adequate for the characterization of residual radiological activity. These issues were assessed and resolved to the extent possible through careful consideration of the properties of the distributions. The intent was to use data which characterize the current state of the Site. Radionuclides can be designated to one of several groups depending on their origin. In this work the groups considered were 1) Naturally Occurring Radioactive Material (NORM), 2) fallout, and 3) Other (including power plant, medical research, etc). This grouping is a useful construct with respect to the interpretation of fixed laboratory results. For example, NORM radionuclides that exist within a decay chain should have approximately equivalent distributions of concentrations if they are representative of background conditions. Insights such as these can be used as a check to identify sample results that need to be further investigated or omitted due to issues associated with reported results from fixed laboratory analyses. This type of analysis provided a foundation for the assessment of the adequacy of sample results for use in subsequent components of an assessment. The general process for the assessment of residual radiological contamination at the Site consists of a sequence of several steps. First, for each analyte, several statistical tests were performed to assess the weight of evidence against the null hypothesis that the mean of the distribution of concentrations was equal to zero. If the mean of the distribution of concentrations for a given radionuclide was not found to be greater than zero, then it was removed from consideration as a component of the residual radiological contamination. If there was significant evidence to reject the hypothesis of the mean being equal to zero, the second step was to compare the distribution of the data from the Site to that of the corresponding background. A suite of tests was used to compare the distributions of the site and background data. The results of these tests were collectively used to determine if site data are elevated relative to background. The third step was to develop distributions using a Bayesian framework to characterize the distribution of mean of the increment present above background for each of the radionuclides. The Bayesian model implemented allowed for the comparison of site-specific records to background concentrations to better approximate contamination attributed to the Site. A final screening step was employed for radionuclides that exceed background. This screening step compared 95% upper confidence limits (UCLs) from the increment distribution developed in the previous step to the risk screening levels. This approach yields a list of analytes that were determined to be elevated relative to background.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Streaming Large-Scale Microscopy Data to a Supercomputing Facility

Data management is a critical component of modern experimental workflows. As data generation rates increase, transferring data from acquisition servers to processing servers via conventional file-based methods is becoming increasingly impractical. The 4D Camera at the National Center for Electron Microscopy generates data at a nominal rate of 480 Gbit s -1 (87,000 frames s -1 ⁠), producing a 700 GB dataset in 15 s. To address the challenges associated with storing and processing such quantities of data, we developed a streaming workflow that utilizes a high-speed network to connect the 4D Camera’s data acquisition system to supercomputing nodes at the National Energy Research Scientific Computing Center, bypassing intermediate file storage entirely. In this work, we demonstrate the effectiveness of our streaming pipeline in a production setting through an hour-long experiment that generated over 10 TB of raw data, yielding high-quality datasets suitable for advanced analyses. Additionally, we compare the efficacy of this streaming workflow against the conventional file-transfer workflow by conducting a postmortem analysis on historical data from experiments performed by real users. Our findings show that the streaming workflow significantly improves data turnaround time, enables real-time decision-making, and minimizes the potential for human error by eliminating manual user interactions.

4D-STEM↗

A Computational Review of Privacy-Preserving Mechanisms for the Smart Grid

Smart grid technologies have rapidly become one of the largest and most comprehensive sources of data for the modern utility. For the most part, data streams are seen as an essential tool that enable utilities to carry their day-to-day business operations, but they also create the need for efficient and secure data management strategies. In the context of the smart grid, ensuring data privacy is becoming an increasing concern due to a combination of factors that range from shifts in operational paradigms and rapid technology evolution to changes in legislation. Furthermore, researchers have highlighted the risks associated with improperly protected energy records. For example, energy consumption data from homes could be used to infer the behaviors and habits of home occupants through activity recognition or user profiling (Fan, 2017), which may lead to unfair service pricing, targeted advertising, or other personal security violations. Similarly, Electric Vehicles’ (EVs) charging metadata could be used to reveal private information about the owner such as their payment methods, preferred charging stations, and other locational and timing information that could be used to reconstruct the vehicle owner’s behaviors. The privacy of user data, even when used for statistical analysis or machine learning training processes, also needs to be carefully considered, as an individual’s private traits may still be vulnerable if their inclusion/exclusion greatly impacts the result or could be linked to a public dataset through cross-reference. The breach of user privacy also has severe impacts for organizations that store, transmit, or work on the data in the form of diminishing the public’s trust in them while potentially incurring legal consequences (e.g., fines and suspensions under the European Union General Data Protection Regulation, Health Insurance Portability and Accountability Act, etc.). Because of these risks, several privacy-preserving mechanisms are available to help organizations comply with privacy legislations and prevent the unauthorized and malicious use of user data. In light of these concerns, this report focuses on performing a computational review of privacy-preserving mechanisms that have received a significant amount of interest in literature. It specifically focuses on 1) homomorphic encryption, 2) zero-knowledge proofs, 3) differential privacy, and 4) federated learning. It is worth noting that although many of the methods presented in this document rely on cryptographic primitives, their intent is not to provide perfect secrecy, but rather to enable users to maintain privacy, and thus they shall not be compared or equated to other constructs that are aimed to address cybersecurity constructs.

24 POWER TRANSMISSION AND DISTRIBUTION↗

dCache: The Storage System of Choice for Data-Intensive Applications

The ever-increasing volumes of data produced by modern scientific facilities like EuXFEL and LHC put significant stress on data management infrastructure operated by laboratories and research centers. The challenges to be addressed span the entire data life cycle, from ingest and efficient data analysis to long-term preservation, typically involving large tape libraries. dCache, a storage system developed in collaboration between the Deutsches Elektronen-Synchrotron (DESY), Fermi National Accelerator Laboratory, and Nordic e-Infrastructure Collaboration (NeIC), is designed to manage a large number of disk servers and to facilitate transparent data migration to and from archival storage. Its multifaceted approach offers a unified method to support a variety of scientific use cases with the same storage infrastructure, including high-throughput data ingest, data sharing over wide area networks, efficient access from HPC clusters, and long-term data preservation on tertiary storage. Initially developed for high energy physics (HEP) experiments, dCache is now used by various scientific communities, including astrophysics, biomedical research, and life sciences, each having specific requirements. This paper presents architecture, deployment strategies, performance and scalability enhancements, and recent advancements in dCache addressing the needs of scientific communities. Finally, we touch on the development and release process, ensuring the software’s high quality.

DCache↗

Provenance infrastructure for multimodal x-ray experiments and reproducible analysis

Exploring relationships between material structures and desired properties for material discovery and synthesis requires mining machine-readable databases and linking experimental data with sufficiently rich annotations. There are vast gaps both in these data sources and in the connections between them. We present a provenance-based data management and analysis framework that enables capturing, persisting, and reanalyzing experimental data from the BNL National Synchrotron Light Source II (NSLS-II). Our system leverages NSLS-II Bluesky for experimental data acquisition, captures analysis parameters, and enriches the search space with results from the scientific literature. We describe in detail the challenges and benefits of data-driven approaches when comparing computational and experimental data, such as the multiple possible scenarios of interest in often ill-posed inverse problems and missing data. Our infrastructure utilizes the NSLS-II event model and includes discovery mechanisms across multiple beamlines. A text and data mining module using natural language processing classification techniques extracts relevant information from the literature that is added to data sources. We describe a provenance acquisition and analysis rerun workflow based on a directed acyclic graph that keeps track of process and data provenance and allows to replay analysis.

97 MATHEMATICS AND COMPUTING↗

RBDMS, FracFocus, State Support, and Produced Water Initiatives

Award DE-FE-0027702 from the Department of Energy to the Ground Water Protection Council (GWPC) focused on state and federal priorities in the areas of state Risk Based Data Management System (RBDMS) development, connectivity between state systems and FracFocus.org, and data sharing initiatives across agencies. The primary objective was to enhance the RBDMS by adding new components relevant to current environmental topics such as hydraulic fracturing, increasing field inspection capabilities, creating linkages between FracFocus and state programs, upgrading eForm capabilities, and analyzing potential for data sharing. The recipient worked with state agencies developing RBDMS module(s) that meet these needs.

54 ENVIRONMENTAL SCIENCES↗

MDDC Multi-Length Scale Data Architecture Contribution Report – PNNL, INL, ANL, LANL and ORNL

This report offers a comprehensive view of data streams currently generated at Pacific Northwest National Laboratory, Idaho National Laboratory, Argonne National Laboratory, Los Alamos National Laboratory, and Oak Ridge National Laboratory set to integrate into the evolving Multi-Dimensional Data Correlation framework at Oak Ridge National Laboratory. Developed by the Advanced Materials and Manufacturing Technologies program, the Multi-Dimensional Data Correlation framework serves as a cutting-edge software to manage data relevant to advanced manufacturing and material behavior in advanced reactors. The report defines data streams, highlights their generation methods and visualization methods both for experimental and computational aspects relevant to the Advanced Materials and Manufacturing Technologies project. A logical next step for this work is to integrate the MDDC framework into PNNL’s, INL’s, ANL’s, LANL’s and ORNL’s fabrication, experimentation, and modelling workflows. This would require setting up the MDDC framework at PNNL, INL, ANL, and LANL and integrating it into the data collection and storage for these different activities.

36 MATERIALS SCIENCE↗

Seamless integration of commercial Clouds with ATLAS Distributed Computing

The CERN ATLAS Experiment successfully uses a worldwide dis-tributed computing Grid infrastructure to support its physics programme at the Large Hadron Collider (LHC). The Grid workflow system PanDA routinely manages up to 700,000 concurrently running production and analysis jobs to process simulation and detector data. In total more than 500 PB of data are distributed over more than 150 sites in the WLCG and handled by the ATLAS data management system Rucio. To prepare for the ever growing data rate in future LHC runs new developments are underway to embrace industry accepted protocols and technologies, and utilize opportunistic resources in a standard way. This paper reviews how the Google and Amazon Cloud computing ser-vices have been seamlessly integrated as a Grid site within PanDA and Rucio. Performance and brief cost evaluations will be discussed. Such setups could offer advanced Cloud tool-sets and provide added value for analysis facilities that are under discussions for LHC Run-4.

97 MATHEMATICS AND COMPUTING↗

Investigating Scientific Data Change with User Research Methods

Scientific datasets are continually expanding and changing due to fluctuations with instruments, quality assessment and quality control processes, and modifications to software pipelines. Datasets include minimal information about these changes or their effects requiring scientists manually assess modifications through a number of labor intensive and ad-hoc steps. The Deduce project is investigating data change to develop metrics, methods, and tools that will help scientists systematically identify and make decisions around data changes. Currently, there is a lack of understanding, and common practices, for identifying and evaluating changes in datasets since systematically measuring and managing data change is under explored in scientific work. We are conducting user research to address this need by exploring scientist's conceptualizations, behaviors, needs, and motivations when dealing with changing datasets. Our user research utilizes multiple methods to produce foundational, generative insights and evaluate research products produced by our team. In this paper, we detail our user research process and outline our findings about data change that emerge from our studies. Our work illustrates how scientific software teams can push beyond just usability testing user interfaces or tools to better probe the underlying ideas they are developing solutions to address.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

A digital twin platform for building performance monitoring and optimization: Performance simulation and case studies

Advancements in sensor technology, data analytics, affordable compute, and communication infrastructure have paved the way for Digital Twin technology in optimizing building operations and controls. This study presents the development of an open and interoperable web-based Digital Twin platform for integrating diverse data streams and facilitating effective user interactions. The platform utilizes modern technologies for the web framework and time-series data management, ensuring scalability and responsiveness. The backend supports seamless integration of diverse data sources and emulators, incorporating data from building sensors and meters, external weather Application Programming Interfaces, and advanced EnergyPlus simulation models of the building and its energy systems including the Distributed Energy Resources that are formulated in Functional Mockup Units. A simulation case study was conducted with FlexLab, a test facility on Lawrence Berkeley National Laboratory campus. The case study includes normal operations, Distributed Energy Resource integration, and power outage scenarios, to illustrate the Digital Twin’s ability to provide critical insights into energy performance and thermal resilience. The results demonstrated the platform’s potential as a decision-support tool for optimizing building energy performance and enhancing resilience against extreme weather events. Future work will focus on deploying the Digital Twin platform to a real building for field validation, extending its capabilities to cover more scenarios such as bidirectional Electric Vehicle interactions, and enhancing user engagement.

EnergyPlus↗

HydraGNN v3.0

New or improved capabilities included in v3.0 release are as follows: 1. Enhancement in message passing layers through generalization of the class inheritance to enable the inclusion of a broader set of message passing policies Inclusion of equivariant message passing layers from the original implementations of: SchNet (https://pubs.aip.org/aip/jcp/article/148/24/241722/962591/SchNet-A-deep-learning-architecture-for-molecules); DimeNet++ (https://arxiv.org/abs/2011.14115); EGNN models (https://arxiv.org/pdf/2102.09844.pdf) 2. Restructuring of class inheritance for data management 3. Support of DDStore https://github.com/ORNL/DDStore capabilities for improved distributed data parallelism on large volumes of data that cannot fit on intra-node memory capacities 4. Large-scale system support for OLCF-Crusher and OLCF-Frontier

Lupo Pasini, Massimiliano [Oak Ridge National Labo↗