Search NASA⌕ Search

SEARCH · Search NASA

Results for “Big data processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Where Is the Provenance? Ethical Replicability and Reproducibility in GIScience and Its Critical Applications

As replicability and reproducibility (R&R) crises develop within emerging convergent inquiry, ethical use of provenance information is central to the establishment and preservation of trust in critical applications of GIScience and geospatial technologies. Today large volumes of geospatial data are generated at high velocity from satellite sensors and unmanned aircraft systems, citizen sensors, geolocation-based data services, global navigation satellite systems, and so on. The extensive use of these data for applications such as disaster and humanitarian response raises the issue of R&R from competing perspectives of location privacy and geospatial data quality. Although geospatial data can be integrated and linked with contextual information to identify individuals’ movements, steps taken to ensure privacy can complicate the multiuser development of high-quality geospatial workflows. Provenance information as digital records of historical (retrospective) and potential future (prospective) geospatial processes is often overlooked, misunderstood, or inadequately addressed. We explore the relationship between provenance information, location privacy, and geospatial data quality in the context of R&R with a focus on disaster analytics. Here, we argue that in the era of big data and deep learning, GIScientists and associated institutions bear greater responsibility both for geospatial workflow quality and for location privacy. Given vastly heterogenous computational landscapes, we provide practical recommendations for ethically driven provenance and R&R research and development within the GIScience community and beyond.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2018

Towards an Open (Data) Science Analytics-Hub for Reproducible Multi-Model Climate Analysis at Scale

Open Science is key to future scientific research and promotes a deep transformation in the whole scientific research process encouraging the adoption of transparent and collaborative scientific approaches aimed at knowledge sharing. Open Science is increasingly gaining attention in the current and future research agenda worldwide. To effectively address Open Science goals, besides Open Access to results and data, it is also paramount to provide tools or environments to support the whole research process, in particular the design, execution and sharing of transparent and reproducible experiments, including data provenance (or lineage) tracking. This work introduces the Climate Analytics-Hub, a new component on top of the Earth System Grid Federation (ESGF), which joins big data approaches and parallel computing paradigms to provide an Open Science environment for reproducible multi-model climate change data analytics experiments at scale. An operational implementation has been set up at the SuperComputing Centre of the Euro- Mediterranean Center on Climate Change, with the main goal of becoming a reference Open Science hub in the climate community regarding the multi-model analysis based on the Coupled Model Intercomparison Project (CMIP).

analytics-hub↗
DOE OSTI2019

Condition Monitoring of Wind Turbine Planetary Gearboxes Under Different Operating Conditions

Digitally enhanced services for wind power could reduce Operations and Maintenance costs as well as the Levelised Cost Of Energy. Therefore, there is a continuous need for advanced monitoring techniques which can exploit the opportunities of Internet of Things and Big Data Analytics. The heart of wind turbines is an epicyclic gearbox and rolling element bearings are often responsible for machinery stops. The vibration signatures of bearings are rather weak compared to other components. As a result a number of signal processing techniques have been proposed, focusing towards accurate and early fault detection with limited false alarms and missed detections. In Envelope Analysis an envelope of the vibration signal is estimated usually after filtering around a frequency band. Different tools, such as Kurtogram, have been proposed to select the optimum filter parameters. Monitoring techniques have reached a mature level for steady speed and load operating conditions. On the other hand, under chaniging operating conditions, it is still unclear whether the change of the monitoring indicators is due to the change of the machinery's health or due to the change of operating parameters. Recently, the authors have proposed a new tool called IESFOgram, based on Cyclic Spectral Coherence for the automatic selection of the filtering band. In this paper, the performance of the tool is evaluated on the National Renewable Energy Laboratory wind turbine gearbox vibration condition monitoring benchmarking data set which includes various faults with different levels of diagnostic complexity as well as various speed and load operating conditions.

17 WIND ENERGY↗
DOE OSTI2022

Comparing Model Representations of Physiological Limits on Transpiration at a Semi-arid Ponderosa Pine Site

Mechanistic representations of biogeochemical processes in ecosystem models are rapidly advancing, requiring advancements in model evaluation approaches. Here we quantify multiple aspects of model functional performance to evaluate improved process representations in ecosystem models. We compare semi-empirical stomatal models with hydraulic constraints against more mechanistic representations of stomatal and hydraulic functioning at a semi-arid pine site using a suite of metrics and analytical tools. We find that models generally perform similarly under unstressed conditions, but performance diverges under atmospheric and soil drought. The more empirical models better capture synergistic information flows between soil water potential and vapor pressure deficit to transpiration, while the more mechanistic models are overly deterministic. Although models can be parameterized to yield similar functional performance, alternate parameterizations could not overcome structural model constraints that underestimate the unique information contained in soil water potential about transpiration. Additionally, both multilayer canopy and big-leaf models were unable to capture the magnitude of canopy temperature divergence from air temperature, and we demonstrate that errors in leaf temperature can propagate to considerable error in simulated transpiration. This study demonstrates the value of merging underutilized observational data streams with emerging analytical tools to characterize ecosystem function and discriminate among model process representations.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Big Data and AI at DoE's Legacy Sites - 20546

More than 30 years have passed since DOE started the decommissioning of nuclear weapon complexes and the clean-up of soil and groundwater. All the sites have been collecting and archiving soil and groundwater monitoring datasets; particularly contaminant concentration time-series. These datasets provide unparalleled opportunities to understand the system behavior (including more fundamental hydrological and geochemical processes, the response to various perturbations, the long-term trend and environmental decay rate towards the regulatory limit). This understanding is critical for providing multiple lines of evidences that can support site closure. In this study, we explore the machine learning (ML) and artificial intelligence (AI) applications to the long-term soil and groundwater management at DoE's legacy sites. ML can improve our understanding of the subsurface systems, which is critical for long-term monitoring and management of the sites, while AI can automate or support some of decision-making processes (e.g., anomaly detection, monitoring well placements). The particular focuses are to develop general algorithms to: (1) to identify distinct spatiotemporal patterns and to identify several groups that have similar temporal behaviors, using unsupervised clustering methods, (2) identify the different temporal scales of hydrological responses to climate perturbations by time-series analysis, and (3) reduce the number of monitoring wells by identifying the minimum sufficient number of wells to capture the heterogeneity of the groundwater contaminant plume and concentration distribution, using the Gaussian Process model. We demonstrate our methodology at the Savannah River Site F-Area. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2023

A differentiable, physics-informed ecosystem modeling and learning framework for large-scale inverse problems: demonstration with photosynthesis simulations

Photosynthesis plays an important role in carbon, nitrogen, and water cycles. Ecosystem models for photosynthesis are characterized by many parameters that are obtained from limited in situ measurements and applied to the same plant types. Previous site-by-site calibration approaches could not leverage big data and faced issues like overfitting or parameter non-uniqueness. Here we developed an end-to-end programmatically differentiable (meaning gradients of outputs to variables used in the model can be obtained efficiently and accurately) version of the photosynthesis process representation within the Functionally Assembled Terrestrial Ecosystem Simulator (FATES) model. As a genre of physics-informed machine learning (ML), differentiable models couple physics-based formulations to neural networks (NNs) that learn parameterizations (and potentially processes) from observations, here photosynthesis rates. We first demonstrated that the framework was able to correctly recover multiple assumed parameter values concurrently using synthetic training data. Then, using a real-world dataset consisting of many different plant functional types (PFTs), we learned parameters that performed substantially better and greatly reduced biases compared to literature values. Further, the framework allowed us to gain insights at a large scale. Our results showed that the carboxylation rate at 25 °C (V c,max25 ) was more impactful than a factor representing water limitation, although tuning both was helpful in addressing biases with the default values. This framework could potentially enable substantial improvement in our capability to learn parameters and reduce biases for ecosystem modeling at large scales.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

Understanding the Thermal Physics and Metallurgy of Metal Big Area Additive Manufacturing

The research goal of this EPSCoR-DOE partnership is to mitigate defects in parts made using a new type of additive manufacturing (AM) process called metal Big Area Additive Manufacturing (m-BAAM). To realize this goal, the PIs will detect and correct defects in the part as it is being printed by combining fundamental knowledge of the thermal physics and metallurgy of m-BAAM with in-process sensor data. Developed at the DOE-funded Manufacturing Demonstration Facility at Oak Ridge National Laboratory, the m-BAAM process involves one or more robots working together to produce a part by fusing metal wire layer-by-layer using arc welding. The process can print large metal parts such as turbine blades, which is not possible using other AM processes. In addition, m-BAAM production rates are more than ten times faster than other AM processes while requiring one-tenth of the material cost. Despite their potential to become a critical force multiplier in the energy generation industry, m-BAAM parts may fail to print accurately due to retention of heat and uneven cooling. Overheating and anomalous cooling rates in turn can cause inconsistencies in the microstructure, leading to sudden failure when used in safety-critical applications. In other words, flaw formation in m-BAAM parts is governed by the thermal history – intensity and spatial distribution of heat inside the part during printing. The thermal history is a complex function of the part shape and process settings such as welding energy, path taken by the welding torch for deposition (tool path), wire feed rate, among others.

36 MATERIALS SCIENCE↗
DOE OSTI2019

Enabling real-time multi-messenger astrophysics discoveries with deep learning

Multi-messenger astrophysics is a fast-growing, interdisciplinary field that combines data, which vary in volume and speed of data processing, from many different instruments that probe the Universe using different cosmic messengers: electromagnetic waves, cosmic rays, gravitational waves and neutrinos. In this Expert Recommendation, we review the key challenges of real-time observations of gravitational wave sources and their electromagnetic and astroparticle counterparts, and make a number of recommendations to maximize their potential for scientific discovery. These recommendations refer to the design of scalable and computationally efficient machine learning algorithms; the cyber-infrastructure to numerically simulate astrophysical sources, and to process and interpret multi-messenger astrophysics data; the management of gravitational wave detections to trigger real-time alerts for electromagnetic and astroparticle follow-ups; a vision to harness future developments of machine learning and cyber-infrastructure resources to cope with the big-data requirements; and finally, the need to build a community of experts to realize the goals of multi-messenger astrophysics.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2021

Automatic Search of Cataclysmic Variables Based on LightGBM in LAMOST-DR7

The search for special and rare celestial objects has always played an important role in astronomy. Cataclysmic Variables (CVs) are special and rare binary systems with accretion disks. Most CVs are in the quiescent period, and their spectra have the emission lines of Balmer series, HeI, and HeII. A few CVs in the outburst period have the absorption lines of Balmer series. Owing to the scarcity of numbers, expanding the spectral data of CVs is of positive significance for studying the formation of accretion disks and the evolution of binary star system models. At present, the research for astronomical spectra has entered the era of Big Data. The Large Sky Area Multi-Object Fiber Spectroscopy Telescope (LAMOST) has produced more than tens of millions of spectral data. the latest released LAMOST-DR7 includes 10.6 million low-resolution spectral data in 4926 sky regions, providing ideal data support for searching CV candidates. To process and analyze the massive amounts of spectral data, this study employed the Light Gradient Boosting Machine (LightGBM) algorithm, which is based on the ensemble tree model to automatically conduct the search in LAMOST-DR7. Finally, 225 CV candidates were found and four new CV candidates were verified by SIMBAD and published catalogs. This study also built the Gradient Boosting Decision Tree (GBDT), Adaptive Boosting (AdaBoost), and eXtreme Gradient Boosting (XGBoost) models and used Accuracy, Precision, Recall, the F1-score, and the ROC curve to compare the four models comprehensively. Experimental results showed that LightGBM is more efficient. The search for CVs based on LightGBM not only enriches the existing CV spectral library, but also provides a reference for the data mining of other rare celestial objects in massive spectral data.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2020

Next-generation geospatial-temporal information technologies for disaster management

Traditional geographic information systems (GIS) have been disrupted by the emergence of Big Data in the form of geo-coded raster, vector, and time-series Internet-of-Things data. This article discusses the application of new scalable technologies that go far beyond relational databases and file-based storage on spinning disk or tape to incorporate both storage and processing data in the same platform. The roles of the Apache Hadoop Distributed File Systems and NoSQL key-value stores such as the Apache Hbase are discussed, along with indexing schemes that optimally support geospatial-temporal use. Here, we highlight how this new approach can rapidly search multiple GIS data layers to obtain insights in the context of early warning, impact evaluation, response, and recovery to earthquake and wildfire disasters.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Realized ecological forecast through an interactive Ecological Platform for Assimilating Data (EcoPAD, v1.0) into models

Abstract. Predicting future changes in ecosystem services is not only highly desirable but is also becoming feasible as several forces (e.g., available big data, developed data assimilation (DA) techniques, and advanced cyber-infrastructure) are converging to transform ecological research into quantitative forecasting. To realize ecological forecasting, we have developed an Ecological Platform for Assimilating Data (EcoPAD, v1.0) into models. EcoPAD (v1.0) is a web-based software system that automates data transfer and processing from sensor networks to ecological forecasting through data management, model simulation, data assimilation, forecasting, and visualization. It facilitates interactive data–model integration from which the model is recursively improved through updated data while data are systematically refined under the guidance of model. EcoPAD (v1.0) relies on data from observations, process-oriented models, DA techniques, and the web-based workflow. We applied EcoPAD (v1.0) to the Spruce and Peatland Responses Under Climatic and Environmental change (SPRUCE) experiment in northern Minnesota. The EcoPAD-SPRUCE realizes fully automated data transfer, feeds meteorological data to drive model simulations, assimilates both manually measured and automated sensor data into the Terrestrial ECOsystem (TECO) model, and recursively forecasts the responses of various biophysical and biogeochemical processes to five temperature and two CO 2 treatments in near-real time (weekly). Forecasting with EcoPAD-SPRUCE has revealed that mismatches in forecasting carbon pool dynamics are more related to model (e.g., model structure, parameter, and initial value) than forcing variables, opposite to forecasting flux variables. EcoPAD-SPRUCE quantified acclimations of methane production in response to warming treatments through shifted posterior distributions of the CH 4 :CO 2 ratio and the temperature sensitivity ( Q 10 ) of methane production towards lower values. Different case studies indicated that realistic forecasting of carbon dynamics relies on appropriate model structure, correct parameterization, and accurate external forcing. Moreover, EcoPAD-SPRUCE stimulated active feedbacks between experimenters and modelers to identify model components to be improved and additional measurements to be taken. It has become an interactive model–experiment (ModEx) system and opens a novel avenue for interactive dialogue between modelers and experimenters. Altogether, EcoPAD (v1.0) acts to integrate multiple sources of information and knowledge to best inform ecological forecasting.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2018

PySFD: comprehensive molecular insights from significant feature differences detected among many simulated ensembles

Motivation Many modeling analyses of molecular dynamics (MD) simulations are based on a definition of states that can be (groups of) clusters of simulation frames in a feature space composed of molecular coordinates. With increasing dimension of this feature space (due to the increasing size or complexity of a simulated molecule), it becomes very difficult to cluster the underlying MD data and estimate a statistically robust model. To mitigate this “curse of dimensionality”, one can reduce the feature space, e.g., with principal component or time-lagged independent component analysis transformations, focusing the analysis on the most important modes of transitions. In practice, however, all these reduction strategies may neglect important molecular details that are susceptible to experimental verification. Results To recover such molecular details, I have developed PySFD (Significant Feature Differences analyzer for Python), a multi-processing software package that efficiently selects significantly different features of any user-defined feature type among potentially many different simulated state ensembles, such as meta-stable states of a Markov State Model (MSM). Applying PySFD on MSMs of an aggregate of 300 microseconds MD simulations recently performed on the major histocompatibility complex class II (MHCII) protein, I demonstrate how this toolkit can extract and visualize valuable mechanistic information from big MD simulation data, e.g., in form of networks of dynamic interaction changes connecting functionally relevant sites of a protein complex.

Biochemistry & Molecular Biology↗
DOE OSTI2026

A network of soil moisture, soil temperature, air temperature, net radiation, ground heat flux and ground water for Chicago, Illinois

This dataset contains environmental monitoring data collected using solar-powered Multi-Function Research (MFR) Long Range Wide Area (LoRaWAN)-enabled nodes at 11 sites in Chicago, Illinois, as part of the DOE Urban Integrated Field Lab CROCUS project. The MFR node system consists of an Input/Output Digital Input Module (IB8) interface box (ICT International) providing wired connections for environmental sensors and an MFR-Node-L data logger that manages power, data processing, and LoRaWAN communication. The wireless data are ingested via Sage network (https://sagecontinuum.org/) nodes that contain LoRaWAN antennae. Measurements were collected from 11 MFR nodes deployed across Chicago State University (CSU), Northeastern Illinois University (NEIU), Northwestern University (NU), University of Illinois Chicago (UIC), West Woodlawn "Blacks in Green" (BIG), and Indian Boundary Prairies (IBP). Each MFR node supports a consistent suite of sensors measuring atmospheric, soil, and hydrological variables. Atmospheric measurements include 2m air temperature (°C), 2m vapor pressure deficit (kPa), and 2m shortwave/longwave radiation (incoming and outgoing, W/m²) measured using ATH-VPD and Apogee SN500 sensors. Soil measurements include volumetric water content (VWC, %) and temperature (°C) at four depths (15, 30, 45, and 60 cm below surface) using Meter Teros54 sensors, and heat flux (W/m²) at 10 cm depth using Huske HFP01-05 sensors. At selected locations, Meter Hydros21 sensors measure groundwater depth (mm), specific conductivity (dS/m), and temperature (°C). The dataset includes timestamps, site identifiers with location names, device IDs, Global Positioning System (GPS) coordinates, variable names with units, measurement depths, values, sensor names, and Sage node identifiers. All timestamps are in local Chicago time (CDT/CST). Quality control flags are provided using a 6-bit binary system indicating physical range violations, step spikes, 24-hour flat-line conditions, 6-hour jitter, 7-day ultra-low variance, and persistent high offset. Data is provided in CSV and CF-compliant NetCDF formats. This dataset is part of a larger collection of CROCUS environmental monitoring data, including linked datasets from Air Quality Transmitter (AQT) sensors, Weather Transmitter (WXT) sensors, and Sap Flow Meter (SFM1x) sensors.

Chicago↗
DOE OSTI2022

Scalable algorithms for physics-informed neural and graph networks

Physics-informed machine learning (PIML) has emerged as a promising new approach for simulating complex physical and biological systems that are governed by complex multiscale processes for which some data are also available. In some instances, the objective is to discover part of the hidden physics from the available data, and PIML has been shown to be particularly effective for such problems for which conventional methods may fail. Unlike commercial machine learning where training of deep neural networks requires big data, in PIML big data are not available. Instead, we can train such networks from additional information obtained by employing the physical laws and evaluating them at random points in the space–time domain. Such PIML integrates multimodality and multifidelity data with mathematical models, and implements them using neural networks or graph networks. Here, we review some of the prevailing trends in embedding physics into machine learning, using physics-informed neural networks (PINNs) based primarily on feed-forward neural networks and automatic differentiation. For more complex systems or systems of systems and unstructured data, graph neural networks (GNNs) present some distinct advantages, and here we review how physics-informed learning can be accomplished with GNNs based on graph exterior calculus to construct differential operators; we refer to these architectures as physics-informed graph networks (PIGNs). We present representative examples for both forward and inverse problems and discuss what advances are needed to scale up PINNs, PIGNs and more broadly GNNs for large-scale engineering problems.

42 ENGINEERING↗
DOE OSTI2022

Towards a robust out-of-the-box neural network model for genomic data

The accurate prediction of biological features from genomic data is paramount for precision medicine and sustainable agriculture. For decades, neural network models have been widely popular in fields like computer vision, astrophysics and targeted marketing given their prediction accuracy and their robust performance under big data settings. Yet neural network models have not made a successful transition into the medical and biological world due to the ubiquitous characteristics of biological data such as modest sample sizes, sparsity, and extreme heterogeneity. Here, we investigate the robustness, generalization potential and prediction accuracy of widely used convolutional neural network and natural language processing models with a variety of heterogeneous genomic datasets. Mainly, recurrent neural network models outperform convolutional neural network models in terms of prediction accuracy, overfitting and transferability across the datasets under study. While the perspective of a robust out-of-the-box neural network model is out of reach, we identify certain model characteristics that translate well across datasets and could serve as a baseline model for translational researchers.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

LCLS Big Data Handling – How I Learned to Stop Worrying and Love the Data Deluge

Advanced data and computing systems are vital to Linac Coherent Light Source (LCLS) operations, data interpretation and overall scientific productivity. The transition to MHz-era operation marks a fundamental change in scale that requires new infrastructure and architectures to link LCLS to the required scale of computing needed for scientific interpretation. The LCLS-II Data System meets big data challenges by implementing configurable data reduction that can adapt to multiple science areas, real-time analysis frameworks to provide visualization and fast feedback, and the ability to transfer data to local and remote computational facilities for near real time analysis at the appropriate scale. Feature extracted information generated in the data analysis pipeline - at the edge, local compute, or remote High-Performance Computing (HPC) resources - can be used to steer experiments and inform user decisions during beam time. Artificial Intelligence and Machine Learning (AI/ML) techniques present new opportunities to rapidly analyse large datasets and direct experiments, but create new challenges in scaling, adaptability, complexity, and trustworthiness. We describe how the LCLS-II Data System architecture addresses its data-driven challenges in the areas of data acquisition, data processing, data management, and workflow orchestration to decrease the overall time-to-science and provide a vision for future developments.

artificial intelligence↗
DOE OSTI2026

Hyaloscypha finlandica Metabolome Repository

This repository provides the curated data tables, manuscript figure and table exports, dependency records, and workflow scripts supporting an integrated comparative genomics and untargeted LC-MS/MS metabolomics analysis of Hyaloscypha finlandica strain PMI 746, a root-associated dark septate endophyte of poplar. The repository includes genome-mining summaries from antiSMASH, FunBGCeX, BGC-Prophet, and BiG-SCAPE; processed metabolomics inputs; metabolite annotation evidence; statistical outputs; and publication-facing figures and tables. Raw LC-MS/MS spectra, full genome/protein downloads, and large generated tool outputs are referenced through public archive/accession records and are not stored in Git.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2018

Big-Data Multi-Energy Iterative Volumetric Reconstruction Methods for As-Built Validation & Verification Applications

This document archives the results developed by the Lab Directed Research and Development (LDRD) project sponsored by Sandia National Laboratories (SNL). In this work, it is shown that SNL has developed the first known high-energy hyperspectral computed tomography system for industrial and security applications. The main results gained from this work include dramatic beam-hardening artifact reduction by using the hyperspectral reconstruction as a bandpass filter without the need for any other computation or pre-processing; additionally, this work demonstrated the ability to use supervised and unsupervised learning methods on the hyperspectral reconstruction data for the application of materials characterization and identification which is not possible using traditional computed tomography systems or approaches.

47 OTHER INSTRUMENTATION↗