Search NASA⌕ Search

SEARCH · Search NASA

Results for “data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13

Real-Time XFEL Data Analysis at SLAC and NERSC: a Trial Run of Nascent Exascale Experimental Data Analysis

X-ray scattering experiments using Free Electron Lasers (XFELs) are a powerful tool to determine the molecular structure and function of unknown samples (such as COVID-19 viral proteins). XFEL experiments are a challenge to computing in two ways: i) due to the high cost of running XFELs, a fast turnaround time from data acquisition to data analysis is essential to make informed decisions on experimental protocols; ii) data collection rates are growing exponentially, requiring new scalable algorithms. Here we report our experiences analyzing data from two experiments at the Linac Coherent Light Source (LCLS) during September 2020. Raw data were analyzed on NERSC's Cori XC40 system, using the Superfacility paradigm: our workflow automatically moves raw data between LCLS and NERSC, where it is analyzed using the software package CCTBX. We achieved real time data analysis with a turnaround time from data acquisition to full molecular reconstruction in as little as 10 min -- sufficient time for the experiment's operators to make informed decisions. By hosting the data analysis on Cori, and by automating LCLS-NERSC interoperability, we achieved a data analysis rate which matches the data acquisition rate. Furthermore, completing data analysis with 10 mins is a first for XFEL experiments and an important milestone if we are to keep up with data collection trends.

Blaschke, Johannes P.↗

A Vision for Coupling Operation of US Fusion Facilities with HPC Systems and the Implications for Workflows and Data Management

The operation of large US Department of Energy (DOE) research facilities, like the DIII-D National Fusion Facility, results in the collection of complex multi-dimensional scientific datasets, both experimental and model-generated. In the future, it is envisioned that integrated data analysis coupled with large-scale high performance computing (HPC) simulations will be used to improve experimental planning and operation. Practically, massive data sets from these simulations provide the physics basis for generation of both reduced semi-analytic and machine-learning-based models. Storage of both HPC simulation datasets (generated from US DOE leadership computing facilities) and experimental datasets presents significant challenges. In this paper, we present a vision for a DOE-wide data management workflow that integrates US DOE fusion facilities with leadership computing facilities. Data persistence and long-term availability beyond the length of allocated projects is essential, particularly for verification and recalibration of artificial intelligence and machine learning (AI/ML) models. Because these data sets are often generated and shared among hundreds of users across multiple leadership computing facility centers, they would benefit from cross-platform accessibility, persistent identifiers (e.g. DOI, or digital object identifier), and provenance tracking. Here, the ability to handle different data access patterns suggests that a combination of low cost, high latency (e.g. for storing ML training sets) and high cost, low latency systems (e.g. for real-time, integrated machine control feedback) may be needed.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

BEAM: A Computational Workflow System for Managing and Modeling Material Characterization Data in HPC Environments

Improvements in scientific instrumentation allow imaging at mesoscopic to atomic length scales, many spectroscopic modes, and now—with the rise of multimodal acquisition systems and the associated processing capability—the era of multidimensional, informationally dense data sets has arrived. Technical issues in these combinatorial scientific fields are exacerbated by computational challenges best summarized as a necessity for drastic improvement in the capability to transfer, store, and analyze large volumes of data. The Bellerophon Environment for Analysis of Materials (BEAM) platform provides material scientists the capability to directly leverage the integrated computational and analytical power of High Performance Computing (HPC) to perform scalable data analysis and simulation via an intuitive, cross-platform client user interface. This framework delivers authenticated, “push-button” execution of complex user workflows that deploy data analysis algorithms and computational simulations utilizing the converged compute-and-data infrastructure at Oak Ridge National Laboratory's (ORNL) Compute and Data Environment for Science (CADES) and HPC environments like Titan at the Oak Ridge Leadership Computing Facility (OLCF). In this work we address the underlying HPC needs for characterization in the material science community, elaborate how BEAM's design and infrastructure tackle those needs, and present a small sub-set of user cases where scientists utilized BEAM across a broad range of analytical techniques and analysis modes.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

LCLS Big Data Handling – How I Learned to Stop Worrying and Love the Data Deluge

Advanced data and computing systems are vital to Linac Coherent Light Source (LCLS) operations, data interpretation and overall scientific productivity. The transition to MHz-era operation marks a fundamental change in scale that requires new infrastructure and architectures to link LCLS to the required scale of computing needed for scientific interpretation. The LCLS-II Data System meets big data challenges by implementing configurable data reduction that can adapt to multiple science areas, real-time analysis frameworks to provide visualization and fast feedback, and the ability to transfer data to local and remote computational facilities for near real time analysis at the appropriate scale. Feature extracted information generated in the data analysis pipeline - at the edge, local compute, or remote High-Performance Computing (HPC) resources - can be used to steer experiments and inform user decisions during beam time. Artificial Intelligence and Machine Learning (AI/ML) techniques present new opportunities to rapidly analyse large datasets and direct experiments, but create new challenges in scaling, adaptability, complexity, and trustworthiness. We describe how the LCLS-II Data System architecture addresses its data-driven challenges in the areas of data acquisition, data processing, data management, and workflow orchestration to decrease the overall time-to-science and provide a vision for future developments.

artificial intelligence↗

Open Research Challenges with Big Data - A Data-Scientist s Perspective

In this paper, we discuss data-driven discovery challenges of the Big Data era. We observe that recent innovations in being able to collect, access, organize, integrate, and query massive amounts of data from a wide variety of data sources have brought statistical data mining and machine learning under more scrutiny and evaluation for gleaning insights from the data than ever before. In that context, we pose and debate the question - Are data mining algorithms scaling with the ability to store and compute? If yes, how? If not, why not? We survey recent developments in the state-of-the-art to discuss emerging and outstanding challenges in the design and implementation of machine learning algorithms at scale. We leverage experience from real-world Big Data knowledge discovery projects across domains of national security, healthcare and manufacturing to suggest our efforts be focused along the following axes: (i) the data science challenge - designing scalable and flexible computational architectures for machine learning (beyond just data-retrieval); (ii) the science of data challenge the ability to understand characteristics of data before applying machine learning algorithms and tools; and (iii) the scalable predictive functions challenge the ability to construct, learn and infer with increasing sample size, dimensionality, and categories of labels. We conclude with a discussion of opportunities and directions for future research.

Big data↗

The LLNL nuclear data infrastructure for the GNDS data format

The next generation of nuclear data infrastructure tools at the Livermore National Laboratory (LLNL) consists of pipeline of codes that read and process nuclear data from evaluated files saved in the new GNDS (Generalised Nuclear Data Structure) nuclear data format. The processing code FUDGE (For Updating Data and Generating Evaluations) is at the front-end of this pipeline as it reads and process the evaluated data for use in downstream transport codes. FUDGE is Python based with C and C++ extensions for computationally intensive tasks. As is the case for the evaluated data, the processed output is also saved in the GNDS format and the GIDI+ API is provided as the interface between the processed data and the transport codes. GIDI+ is a C++ based suite of codes and it includes GIDI (General Interaction Data Interface), a library for reading and writing GNDS data, and MCGIDI which is the cross section lookup, and reaction and product distribution sampling interface between Monte Carlo transport codes and the GNDS data. GIDI provides methods for easy access to the multi-group processed GNDS data and this is demonstrated through its implementation in ARDRA, the LLNL deterministic transport code. The evaluation and sampling methods in MCGIDI are available as both CPU and GPU methods which facilitates the use of MCGIDI in both traditional CPU-based as well as the next generation mixed model computational architectures. This is demonstrated through the GIDI+ implementation in MERCURY, the LLNL Monte Carlo transport code. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Data-Centric AI and the Open Energy Data Initiative (OEDI)

This presentation emphasizes the critical importance of data-centric AI. The limitations of model-centric AI when dealing with poor or insufficient data are highlighted, and it is illustrated how training models on inaccurate or noisy data leads to suboptimal results. This talk advocates for a hybrid approach that combines a focus on data quality and model parameters to achieve optimal results. The Open Energy Data Initiative (OEDI) is introduced as a valuable resource for obtaining high-quality energy-related datasets, hosting nearly 2,000 publicly accessible datasets, including 99 solar-related datasets, totaling over 2.7 petabytes of data. OEDI's data lakes enable users to query and work with data without extensive transfers. In conclusion, the significance of data-centric AI and adherence to data curation best practices is emphasized, positioning OEDI as a prime source of high-quality data for AI and machine learning in the renewable energy sector.

AI↗

Using Skyline to Analyze Data-Containing Liquid Chromatography, Ion Mobility Spectrometry, and Mass Spectrometry Dimensions

Recent advances in ion mobility spectrometry (IMS) have illustrated its power in determining the structural characteristics of a molecule, especially when coupled with other separations dimensions such as liquid chromatography (LC) and mass spectrometry (MS). However, these three separation techniques together greatly complicate data analyses, making better informatics tools essential for assessing the resulting data. In this manuscript, Skyline was adapted to analyze LC-IMS-CID-MS data from numerous instrument vendor datasets and determine the effect of adding the IMS dimension into the normal LC-MS molecular pipeline. For the initial evaluation, a tryptic digest of bovine serum albumin (BSA) was spiked into a yeast protein digest at seven different concentrations, and Skyline was able to rapidly analyze the MS and CID-MS data for 38 of the BSA peptides. Calibration curves for the precursor and fragment ions were assessed with and without the IMS dimension. In all cases, addition of the IMS dimension removed noise from co-eluting peptides with close m/z values, resulting in calibration curves with greater linearity and lower detection limits. This study presents an important informatics development since to date LC-IMS-CID-MS data from the different instrument vendors is often assessed manually and cannot be analyzed quickly. Because these evaluations require days for the analysis of only a few target molecules in a limited number of samples, it is unfeasible to evaluate hundreds of targets in numerous samples. Thus, this study showcases Skyline’s ability to work with the multidimensional LC-IMS-CID-MS data and provide biological and environmental insights rapidly. .

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Equation-based object-oriented modeling and simulation of data center cooling systems

In this paper, we introduce a newly developed open source data center package in the Modelica Buildings library to support modeling and simulation of cooling and control systems of data centers. The data center package contains major thermal and control component models, such as Computer Room Air Handler, Computer Room Air Conditioner, models of different subsystem configurations such as chillers with differently configured waterside economizers, as well as templates for different systems. Two case studies based on this package are performed to investigate the performances of the cooling and electrical system under normal conditions and emergency situations such as a blackout: one is for a data center powered by conventional energy, and the other is for a data center powered by renewable energy. Simulation results show that the dynamic modeling and multi-domain simulation in the Modelica-based tool make it convenient for users to investigate both normal and emergent operations for conventional and renewable data centers.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

A continuous measure of gross primary production for the conterminous United States derived from MODIS and AmeriFlux data

The quantification of carbon fluxes between the terrestrial biosphere and the atmosphere is of scientific importance and also relevant to climate-policy making. Eddy covariance flux towers provide continuous measurements of ecosystem-level exchange of carbon dioxide spanning diurnal, synoptic, seasonal, and interannual time scales. However, these measurements only represent the fluxes at the scale of the tower footprint. Here we used remotely sensed data from the Moderate Resolution Imaging Spectroradiometer (MODIS) to upscale gross primary productivity (GPP) data from eddy covariance flux towers to the continental scale. We first combined GPP and MODIS data for 42 AmeriFlux towers encompassing a wide range of ecosystem and climate types to develop a predictive GPP model using a regression tree approach. The predictive model was trained using observed GPP over the period 2000–2004, and was validated using observed GPP over the period 2005–2006 and leave-one-out cross-validation. Our model predicted GPP fairly well at the site level. We then used the model to estimate GPP for each 1 km×1 km cell across the U.S. for each 8-day interval over the period from February 2000 to December 2006 using MODIS data. Our GPP estimates provide a spatially and temporally continuous measure of gross primary production for the U.S. that is a highly constrained by eddy covariance flux data. Our study demonstrated that our empirical approach is effective for upscaling eddy flux GPP data to the continental scale and producing continuous GPP estimates across multiple biomes. With these estimates, we then examined the patterns, magnitude, and interannual variability of GPP. We estimated a gross carbon uptake between 6.91 and 7.33 Pg C yr -1 for the conterminous U.S. Drought, fires, and hurricanes reduced annual GPP at regional scales and could have a significant impact on the U.S. net ecosystem carbon exchange. The sources of the interannual variability of U.S. GPP were dominated by these extreme climate events and disturbances.

54 ENVIRONMENTAL SCIENCES↗

PHOTOMETRIC REDSHIFTS AND QUASAR PROBABILITIES FROM A SINGLE, DATA-DRIVEN GENERATIVE MODEL

We describe a technique for simultaneously classifying and estimating the redshift of quasars. It can separate quasars from stars in arbitrary redshift ranges, estimate full posterior distribution functions for the redshift, and naturally incorporate flux uncertainties, missing data, and multi-wavelength photometry. We build models of quasars in flux-redshift space by applying the extreme deconvolution technique to estimate the underlying density. By integrating this density over redshift, one can obtain quasar flux densities in different redshift ranges. This approach allows for efficient, consistent, and fast classification and photometric redshift estimation. This is achieved by combining the speed obtained by choosing simple analytical forms as the basis of our density model with the flexibility of non-parametric models through the use of many simple components with many parameters. We show that this technique is competitive with the best photometric quasar classification techniques-which are limited to fixed, broad redshift ranges and high signal-to-noise ratio data-and with the best photometric redshift techniques when applied to broadband optical data. We demonstrate that the inclusion of UV and NIR data significantly improves photometric quasar-star separation and essentially resolves all of the redshift degeneracies for quasars inherent to the ugriz filter system, even when included data have a low signal-to-noise ratio. For quasars spectroscopically confirmed by the SDSS 84% and 97% of the objects with Galaxy Evolution Explorer UV and UKIDSS NIR data have photometric redshifts within 0.1 and 0.3, respectively, of the spectroscopic redshift; this amounts to about a factor of three improvement over ugriz-only photometric redshifts. Our code to calculate quasar probabilities and redshift probability distributions is publicly available.

79 ASTRONOMY AND ASTROPHYSICS↗

High-Performance Data Analytics Beyond the Relational and Graph Data Models with GEMS

Graphs represent an increasingly popular data model for data-analytics, since they can naturally represent relationships and interactions between entities. Relational databases and their pure table-based data model are not well suitable to store and process sparse data. Consequently, graph databases have gained interest in the last few years and the Resource Description Framework (RDF) became the standard data model for graph data. Nevertheless, while RDF is well suited to analyze the relationships between the entities, it is not efficient in representing their attributes and properties. In this work we propose the adoption of a new hybrid data model, based on attributed graphs, that aims at overcoming the limitations of the pure relational and graph data models. We present how we have re-designed the GEMS data-analytics framework to fully take advantage of the proposed hybrid data model. To improve analysts productivity, in addition to a C++ API for applications development, we adopt GraQL as input query language. We validate our approach implementing a set of queries on net-flow data and we compare our framework performance against Neo4j. Experimental results show significant performance improvement over Neo4j, up to several orders of magnitude when increasing the size of the input data.

Data Analytics↗

A Nonrelational Data Warehouse for the Analysis of Field and Laboratory Data From Multiple Heterogeneous Photovoltaic Test Sites

A nonrelational, distributed computing, data warehouse, and analytics environment (Energy-CRADLE) was developed for the analysis of field and laboratory data from multiple heterogeneous photovoltaic (PV) test sites. This data informatics and analytics infrastructure was designed to process diverse formats of PV performance data and climatic telemetry time-series data collected from a PV outdoor test network, i.e., the Solar Durability and Lifetime Extension global SunFarm network, as well as point-in-time laboratory spectral and image measurements of PV material samples. Using Hadoop/HBase for the distributed data warehouse, Energy-CRADLE does not have a predefined data table schema, which enables ingestion of data in diverse and changing formats. For easy data ingestion and data retrieval, Energy-CRADLE utilizes Hadoop streaming to enable Python MapReduce and provides a graphical user interface, i.e., py-CRADLE. By developing the Hadoop distributed computing platform and the HBase NoSQL database schema for solar energy, Energy-CRADLE exemplifies an integrated, scalable, secure, and user-friendly data informatics and analytics system for PV researchers. Furthermore, an example of Energy-CRADLE enabled scalable, data-driven, analytics is presented, where machine learning is used for anomaly detection across 2.2 million real-world current-voltage (I–V) curves of PV modules in three distinct Koppen–Geiger climatic zones.

14 SOLAR ENERGY↗

The Modern Research Data Portal: a design pattern for networked, data-intensive science

We describe best practices for providing convenient, high-speed, secure access to large data via research data portals. Here, we capture these best practices in a new design pattern, the Modern Research Data Portal, that disaggregates the traditional monolithic web-based data portal to achieve orders-of-magnitude increases in data transfer performance, support new deployment architectures that decouple control logic from data storage, and reduce development and operations costs. We introduce the design pattern; explain how it leverages high-performance data enclaves and cloud-based data management services; review representative examples at research laboratories and universities, including both experimental facilities and supercomputer sites; describe how to leverage Python APIs for authentication, authorization, data transfer, and data sharing; and use coding examples to demonstrate how these APIs can be used to implement a range of research data portal capabilities. Sample code at a companion web site, https://docs.globus.org/mrdp , provides application skeletons that readers can adapt to realize their own research data portals.

97 MATHEMATICS AND COMPUTING↗

Tsdat: An Open-Source Data Standardization Framework for Marine Energy and Beyond

Many organizations are tasked with the collection and processing of large quantities of data from various measurement devices. Data reported from these sources are often not interoperable with datasets and software used by analysts and other organizations in the same domain, introducing barriers for collaboration on large-scale projects. This poses a particular problem for cross-device comparisons and machine learning applications, which rely on large quantities of data from multiple sources. To address these challenges, the open-source Time-Series Data Pipelines (Tsdat) Python framework was developed by Pacific Northwest National Laboratory, with strategic guidance and direction provided by the National Renewable Energy Laboratory and Sandia National Laboratories to facilitate collaboration and accelerate advancements in the marine energy domain through the development of an open-source ecosystem of tools. This paper will describe the Tsdat framework and the data standards within which it operates. A beta version of Tsdat has been released and is being used by several projects in marine energy, wind energy, and building energy systems.

big data↗

BASIN-3D: A brokering framework to integrate diverse environmental data

Diverse observational and simulation datasets are needed to understand and predict complex ecosystem behavior over seasonal to decadal and century time-scales. Integration of these datasets poses a major barrier towards advancing environmental science, particularly due to differences in the structure and formats of data provided by various sources. Here, we describe BASIN-3D (Broker for Assimilation, Synthesis and Integration of eNvironmental Diverse, Distributed Datasets), a data integration framework designed to dynamically retrieve and transform heterogeneous data from different sources into a common format to provide an integrated view. BASIN-3D enables users to adopt a standardized approach for data retrieval and avoid customizations for the data type or source. We demonstrate the value of BASIN-3D with two use cases that require integration of data from regional to watershed spatial scales. The first application uses the BASIN-3D Python library to integrate time-series hydrological and meteorological data to provide standardized inputs to analytical and machine learning codes in order to predict the impacts of hydrological disturbances on large river corridors of the United States. The second application uses the BASIN-3D Django framework to integrate diverse time-series data in a mountainous watershed in East River, Colorado, United States to enable scientific researchers to explore and download data through an interactive web portal. Thus, BASIN-3D can be used to support data integration for both web-based tools, as well as data analytics using Python scripting and extensions like Jupyter notebooks. The framework is expected to be transferable to and useful for many other field and modeling studies.

Varadharajan, C↗

Challenges for monitoring and data analytics in a leadership public data repository

The availability and disposition of data has assumed increasing importance in large-scale computational science. Data repositories are evolving to meet new classes of requirements: compliance with government access guidelines, support for reproducibility of experimental results, and long-term availability of data products. The Constellation public data repository at the Oak Ridge Leadership Computing Facility faces these issues while being situated in one of the most productive data centers in the world. While monitoring and operational data analysis are ingrained in the operation of the OLCF’s large-scale high performance computing platforms, data repositories do not have this history of support. Problems faced by Constellation range from data size (over 7 petabytes in current holdings) to analytic complexity (detailed curation is both absolutely necessary for many data sets and absolutely impossible for humans to accomplish in any practical manner) to deployment environment (OLCF storage resources are oriented toward the needs of the compute platforms). In this paper we describe some of the challenges for collecting monitoring and analytic data from a leadership public data repository. We also discuss various strategies we are pursuing in order to address these challenges, from manual data collection to plans for introducing machine learning-based curatorial techniques.

Widener, Patrick [ORNL] (ORCID:0000000258820816)↗