Search NASA⌕ Search

SEARCH · Search NASA

Results for “Big data processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 217 records · Page 12
All sourcesNASA NTRSDOE OSTI
NASA NTRS1974

The origin of the diffuse background gamma radiation

Recent observations provided evidence for diffuse background gamma radiation extending to energies beyond 100 MeV, and evidence of isotropy and implied cosmological origin. Significant features in the spectrum of this background radiation were observed which provide evidence for its origin in nuclear processes in the early stages of big-bang cosmology, and connect these processes with the galaxy formation theory. A test of the theory is in future observations of the background radiation in the 100 MeK to 100 GeV energy range which are made with large orbiting spark-chamber satellite detectors. The theoretical interpretations of present data, their connection with baryon-symmetric cosmology and galaxy formation theory, and the need for future observations are discussed.

Stecker, F. W.↗
DOE OSTI2022

Scalable algorithms for physics-informed neural and graph networks

Physics-informed machine learning (PIML) has emerged as a promising new approach for simulating complex physical and biological systems that are governed by complex multiscale processes for which some data are also available. In some instances, the objective is to discover part of the hidden physics from the available data, and PIML has been shown to be particularly effective for such problems for which conventional methods may fail. Unlike commercial machine learning where training of deep neural networks requires big data, in PIML big data are not available. Instead, we can train such networks from additional information obtained by employing the physical laws and evaluating them at random points in the space–time domain. Such PIML integrates multimodality and multifidelity data with mathematical models, and implements them using neural networks or graph networks. Here, we review some of the prevailing trends in embedding physics into machine learning, using physics-informed neural networks (PINNs) based primarily on feed-forward neural networks and automatic differentiation. For more complex systems or systems of systems and unstructured data, graph neural networks (GNNs) present some distinct advantages, and here we review how physics-informed learning can be accomplished with GNNs based on graph exterior calculus to construct differential operators; we refer to these architectures as physics-informed graph networks (PIGNs). We present representative examples for both forward and inverse problems and discuss what advances are needed to scale up PINNs, PIGNs and more broadly GNNs for large-scale engineering problems.

42 ENGINEERING↗
DOE OSTI2022

Towards a robust out-of-the-box neural network model for genomic data

The accurate prediction of biological features from genomic data is paramount for precision medicine and sustainable agriculture. For decades, neural network models have been widely popular in fields like computer vision, astrophysics and targeted marketing given their prediction accuracy and their robust performance under big data settings. Yet neural network models have not made a successful transition into the medical and biological world due to the ubiquitous characteristics of biological data such as modest sample sizes, sparsity, and extreme heterogeneity. Here, we investigate the robustness, generalization potential and prediction accuracy of widely used convolutional neural network and natural language processing models with a variety of heterogeneous genomic datasets. Mainly, recurrent neural network models outperform convolutional neural network models in terms of prediction accuracy, overfitting and transferability across the datasets under study. While the perspective of a robust out-of-the-box neural network model is out of reach, we identify certain model characteristics that translate well across datasets and could serve as a baseline model for translational researchers.

59 BASIC BIOLOGICAL SCIENCES↗
NASA NTRS1999

Ocean Wave Studies with Applications to Ocean Modeling and Improvement of Satellite Altimeter Measurements

Combining analysis of satellite data (altimeter, scatterometer, high-resolution visible and infrared images, etc.) with mathematical modeling of non-linear wave processes, we investigate various ocean wave fields (on scales from capillary to planetary), their role in ocean dynamics and turbulent transport (of heat and biogeochemical quantities), and their effects on satellite altimeter measuring accuracy. In 1998 my attention was focused on long internal gravity waves (10 to 1000 km), known also as baroclinic inertia-gravity (BIG) waves. We found these waves to be a major factor of altimeter measurements "noise," resulting in a greater uncertainty [up to 10 cm in terms of sea surface height (SSH) amplitude] in the measured SSH signal than that caused by the sea state bias variations (up to 5 cm or so). This effect still remains largely overlooked by the satellite altimeter community. Our studies of BIG waves address not only their influence on altimeter measurements but also their role in global ocean dynamics and in transport and turbulent diffusion of biogeochemical quantities. In particular, in collaboration with Prof Peter Weichman, Caltech, we developed a theory of turbulent diffusion caused by wave motions of most general nature. Applied to the problem of horizontal turbulent diffusion in the ocean, the theory yielded the effective diffusion coefficient as a function of BIG wave parameters obtainable from satellite altimeter data. This effort, begun in 1997, has been successfully completed in 1998. We also developed a theory that relates spatial fluctuations of scalar fields (such as sea surface temperature, chlorophyll concentration, drifting ice concentration, etc.) to statistical characteristics of BIG waves obtainable from altimeter measurements. A manuscript is in the final stages of preparation. In order to verify the theoretical predictions and apply them to observations, we are now analyzing Sea-viewing Wide Field of view Sensor (SeaWiFS) and Field of view Sensor (SeaWiFS) and Advanced Very High-Resolution Radiometer (AVHRR) data on sea surface temperature (SST) and chlorophyll concentration jointly with TOPEX/POSEIDON data on SSH variations.

Glazman, Roman E.↗
NASA NTRS1993

Use of negotiated rulemaking in developing technical rules for low-Earth orbit mobile satellite systems

Technical innovations have converged with the exploding market demand for mobile telecommunications to create the impetus for low-earth orbit (LEO) communications satellite systems. The so-called 'Little LEO's' propose use of VHF and UHF spectrum to provide position - location and data messaging services. The so-called 'Big LEO's' propose to utilize the RDSS bands to provide voice and data services. In the United States, several applications were filed with the U.S. Federal Communications Commission (FCC) to construct and operate these mobile satellite systems. To enable the prompt introduction of such new technology services, the FCC is using innovative approaches to process the applications. Traditionally, when the FCC is faced with 'mutually exclusive' applications, e.g. a grant of one would preclude a grant of the others, it uses selection mechanisms such as comparative hearings or lotteries. In the case of the LEO systems, the FCC has sought to avoid these time-consuming approaches by using negotiated rulemakings. The FCC's objective is to enable the multiple applicants and other interested parties to agree on technical and service rules which will enable the grant of all qualified applications. With regard to the VHF/UHF systems, the Advisory Committee submitted a consensus report to the FCC. The process for the systems operating in the bands above 1 GHz involved more parties and more issues but still provided the FCC useful technical information to guide the adoption of rules for the new mobile satellite service.

Taylor, Leslie A.↗
NASA NTRS2018

Introduction to NASA Goddard Workshop on Artificial Intelligence

Artificial Intelligence (AI) is a collection of advanced technologies that allows machines to think and act, both humanly and rationally, through sensing, comprehending, acting and learning. AI's foundations lie at the intersection of several traditional fields Philosophy, Mathematics, Economics, Neuroscience, Psychology and Computer Science. Although the inception of AI started in the 1950's, it has recently made a strong comeback in all aspects of society and all over the world; this is mainly due to the timely combination of increased data volumes, advanced and mature algorithms, and improvements in computing power and storage. Current AI applications include big data analytics, robotics, intelligent sensing, assisted decision making, and speech recognition just to name a few.This workshop will be investigating how AI technologies can be adapted or developed to address the following challenges: Discover events of interest and correlations in large amounts of science data; improve the outcomes of science modeling and data assimilation using improved data processing, integration, and analysis. Design advisors for mission planning and operations, including anomaly detection and spacecraft health monitoring. Develop tools for engineering support, including advanced manufacturing, orbit determination, new component design and system engineering. Customize intelligent user interfaces, including visual analytics and natural language processing.

Le Moigne, Jacqueline↗
DOE OSTI2025

LCLS Big Data Handling – How I Learned to Stop Worrying and Love the Data Deluge

Advanced data and computing systems are vital to Linac Coherent Light Source (LCLS) operations, data interpretation and overall scientific productivity. The transition to MHz-era operation marks a fundamental change in scale that requires new infrastructure and architectures to link LCLS to the required scale of computing needed for scientific interpretation. The LCLS-II Data System meets big data challenges by implementing configurable data reduction that can adapt to multiple science areas, real-time analysis frameworks to provide visualization and fast feedback, and the ability to transfer data to local and remote computational facilities for near real time analysis at the appropriate scale. Feature extracted information generated in the data analysis pipeline - at the edge, local compute, or remote High-Performance Computing (HPC) resources - can be used to steer experiments and inform user decisions during beam time. Artificial Intelligence and Machine Learning (AI/ML) techniques present new opportunities to rapidly analyse large datasets and direct experiments, but create new challenges in scaling, adaptability, complexity, and trustworthiness. We describe how the LCLS-II Data System architecture addresses its data-driven challenges in the areas of data acquisition, data processing, data management, and workflow orchestration to decrease the overall time-to-science and provide a vision for future developments.

artificial intelligence↗
NASA NTRS1974

The origin of the diffuse background gamma-radiation

Recent observations have now provided evidence for diffuse background gamma radiation extending to energies beyond 100 MeV. There is some evidence of isotropy and implied cosmological origin. Significant features in the spectrum of this background radiation have been observed which provide evidence for its origin in nuclear processes in the early stages of the big-band cosmology and tie in these processes with galaxy fromation theory. A crucial test of the theory may lie in future observations of the background radiation in the 100 MeV to 100 GeV energy range which may be made with large orbiting spark-chamber satellite detectors. A discussion of the theoretical interpretations of present data, their connection with baryon symmetric cosmology and galaxy formation theory, and the need for future observations are given.

Stecker, F. W.↗
NASA NTRSDate not supplied

Pyroscopegridding: Geo-Leo Aerosol Data Fusion (an Open-Source Package)

The retrieval of aerosol optical depths (AODs) from sun-synchronous polar orbiting satellites, such as MODISs, VIIRSs, OMI, TROPOMI, etc., has been widely adopted as a method for obtaining information regarding particulate matter (PM) and related atmospheric processes. However, the advent of recently launched geostationary satellites, such as GOES-16/17/18, Himawari-8/9, and Meteosat Third Generation (MTG), has led to an increased temporal resolution of AOD observations (order of 10 minutes), resulting in typically one or more images per hour during daylight hours, compared to the once-per-day observations obtained from LEO satellites. By integrating these observations, the diurnal cycle of global AOD can be characterized at local, regional, and global scales. The scientific community is still examining the novel data from geostationary satellite observations and evaluating methods for effectively merging these observations with differing spatial and temporal resolutions. This presents a significant ""Big Data"" challenge, encompassing not only data storage, but also data discoverability, accessibility, and migration within cloud computing environments. This study presents our attempts at fusing Level 2 aerosol data from six satellites, three of which are geostationary (GOES-16/17 and Himawari-8) and three of which are polar orbiting (TERRA/MODIS, AQUA/MODIS, and SNPP-VIIRS), using the Dark Target aerosol retrieval algorithm. The ability to fuse remote sensing products on demand into desired temporal and spatial domains empowers researchers and practitioners to more efficiently work with satellite and sensor data. It is our hope that through making our open-source package and accompanying functionality available, the scientific community will have improved access to aerosol data processing resources.

Jennifer Wei↗
DOE OSTI2026

Hyaloscypha finlandica Metabolome Repository

This repository provides the curated data tables, manuscript figure and table exports, dependency records, and workflow scripts supporting an integrated comparative genomics and untargeted LC-MS/MS metabolomics analysis of Hyaloscypha finlandica strain PMI 746, a root-associated dark septate endophyte of poplar. The repository includes genome-mining summaries from antiSMASH, FunBGCeX, BGC-Prophet, and BiG-SCAPE; processed metabolomics inputs; metabolite annotation evidence; statistical outputs; and publication-facing figures and tables. Raw LC-MS/MS spectra, full genome/protein downloads, and large generated tool outputs are referenced through public archive/accession records and are not stored in Git.

59 BASIC BIOLOGICAL SCIENCES↗
NASA NTRS1988

Stellar nucleosynthesis and chemical evolution of the solar neighborhood

Current theoretical models of nucleosynthesis (N) in stars are reviewed, with an emphasis on their implications for Galactic chemical evolution. Topics addressed include the Galactic population II red giants and early N; N in the big bang; star formation, stellar evolution, and the ejection of thermonuclearly evolved debris; the chemical evolution of an idealized disk galaxy; analytical solutions for a closed-box model with continuous infall; and nuclear burning processes and yields. Consideration is given to shell N in massive stars, N related to degenerate cores, and the types of observational data used to constrain N models. Extensive diagrams, graphs, and tables of numerical data are provided.

Clayton, Donald D.↗
DOE OSTI2018

Big-Data Multi-Energy Iterative Volumetric Reconstruction Methods for As-Built Validation & Verification Applications

This document archives the results developed by the Lab Directed Research and Development (LDRD) project sponsored by Sandia National Laboratories (SNL). In this work, it is shown that SNL has developed the first known high-energy hyperspectral computed tomography system for industrial and security applications. The main results gained from this work include dramatic beam-hardening artifact reduction by using the hyperspectral reconstruction as a bandpass filter without the need for any other computation or pre-processing; additionally, this work demonstrated the ability to use supervised and unsupervised learning methods on the hyperspectral reconstruction data for the application of materials characterization and identification which is not possible using traditional computed tomography systems or approaches.

47 OTHER INSTRUMENTATION↗
DOE OSTI2021

GOOML Big Kahuna Forecast Modeling and Genetic Optimization Files

This submission includes example files associated with the Geothermal Operational Optimization using Machine Learning (GOOML) Big Kahuna fictional power plant, which uses synthetic data to model a fictional power plant. A forecast was produced using the GOOML data model framework and fictional input data, and a genetic optimization is included which determines optimal flash plant parameters. The inputs and outputs associated with the forecast and genetic optimization are included. The input and output files consist of data, configuration files, and plots. A link to the Physics-Guided Neural Networks (phygnn) GitHub repository is also included, which augments a traditional neural network loss function with a generic loss term that can be used to guide the neural network to learn physical or theoretical constraints. phygnn is used by the GOOML framework to help integrate its machine learning models into the relevant physics and engineering applications. Note that the data included in this submission are intended to provide a demonstration of GOOML's capabilities. Additional files that have not been released to the public are needed for users to run these models and reproduce these results. Units can be found in the readme data resource.

15 GEOTHERMAL ENERGY↗
NASA NTRS2018

NASA Pilot-Engaged Expert Response Using IBM Watson Technology: Prototype Evaluation of Knowledge Retrieval System

NASA Langley Research Center and IBM have been investigating the use of IBM Watson technology in aerospace research and development. One application of Watson technology is the Pilot-Engaged Expert Response (PEER) use case. The PEER system is envisioned as an in-cockpit advisor that will act as a source of situationally-relevant information for pilots and other flight crew members to assist in decision making about real-time events and situations that arise in the course of aircraft operations. PEER will make available vast stores of knowledge and information quickly and directly, putting important informational resources where they are needed most. IBM has worked with NASA to develop an architecture and articulate a roadmap for the development of the PEER system. That vision is built around Watson Discovery Advisor (WDA) software solution, derived from IBM's Jeopardy!-winning automatic question answering system. PEER makes use of WDA's sophisticated question-answering capabilities as its core, adding important User Interface components and other customizations for the cockpit environment, including communication with flight systems and other external data sources. The development plan for PEER includes four development stages, with the current project constituting the first phase. In this project, a prototype instance of PEER was successfully adapted to the aviation domain, enabling users to ask questions about aviation topics and receive useful and accurate answers to these questions. Major tasks accomplished include the development of procedures for domain adaptation through automatic lexicon extraction from domain glossaries; generation of question-answer training data which was used to train the system; and assessment of the effectiveness of domain adaptation, which showed a dramatic improvement in the ability of the PEER system to answer domain-relevant questions. In addition, the vision for the PEER system was pushed forward by the articulation of a plan for the automatic enhancement of question-answering with contextual information. This initial phase focused on two main goals: 1) the targeted domain adaptation of the underlying WDA system to the aviation domain; and, 2) the design of the software systems needed to leverage flight-contextual data. Domain adaptation of the WDA system proceeds via three main activities: Domain data ingestion, lexical customization and model training. A textual corpus consisting of 1,147 individual documents with more than 7.5 million words of text was ingested into the system and this served as the basis of all further development. A domain lexicon of over 3,500 aviation-domain terms was semi-automatically generated from domain documents and used to train the system. In addition, a set of over 500 question-answer (QA) pairs relevant to the PEER use case was developed; these were used to train and assess the system. These important first steps established the basis for the PEER system. In addition, steps were taken towards the integration of the PEER system into the cockpit environment with the development of a functional design for the Contextual Data Augmentation (CDA) subsystem. This subsystem brings to bear contextual data to improve system responses. It has three main submodules: the Contextual Data Collection module, the Contextual Data Selection module, and the Contextual QA Augmentation module. These modules form a processing pipeline that addresses the problems associated with automatically integrating information from external resources into the knowledge-retrieval mechanism.

Machine learning↗
DOE OSTI2020

Generation and representation of synthetic smart meter data

Advanced energy algorithms running at big-data scale will be necessary to identify, realize, and verify energy savings to meet government and utility goals of building energy efficiency. Any algorithm must be well characterized and validated before it is trusted to run at these scales. Smart meter data from real buildings will ultimately be required for the development, testing, and validation of these energy algorithms and processes. However, for initial development and testing, smart meter data are difficult to work with due to privacy restrictions, noise from unknown sources, data accessibility, and other concerns which can complicate algorithm development and validation. This paper describes a new methodology to generate synthetic smart meter data of electricity use in buildings using detailed building energy modeling, which aims to capture the variability and stochastics of real energy use in buildings. The methodology can create datasets tailored to represent specific scenarios with known truth and controllable amounts of synthetic noise. Knowledge of ground truth also allows the development and validation of enhanced processes which leverage building metadata, such as building type or size (floor area), in addition to smart meter data. The methodology described in this paper includes the key influencing factors of real-world building energy use including weather data, occupant-driven loads, building operation and maintenance practices, and special events. Data formats to support workflows leveraging both synthetic meter data and associated metadata are proposed and discussed. Finally, example use cases of the synthetic meter data are described to illustrate potential applications.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2021

Unleashing the Power of Industrial Big Data through Scalable Manual Labeling

Big Data plays a central role in the remarkable results achieved by Machine Learning (ML) and especially Deep Learning (DL) in the recent years. However, the difficulty in obtaining a reasonable amount of labeled samples limits ML/DL application in various domains, including industrial equipment and system monitoring. In this paper the need for methods that turn manual labeling into a scalable process is highlighted. A real world problem is analyzed for which weak supervision methods, successfully employed in other domains, did not produce acceptable results. An alternative approach based on clustering ensembles is described and tested, achieving good performance.

Paes Leao, Bruno↗
NASA NTRSDate not supplied

Open Source Application of Fusing Aerosol Products from GEO and LEO Satellites

Retrieving aerosol optical depths (AODs) from sun-synchronous polar orbiting (aka low earth orbit, LEO) satellites, such as MODISs, and VIIRSs, OMI, TROPOMI, etc, has become well-established as a tool for extracting information on particulate matter (PM) and related processes in the atmosphere. However, with recently launched geostationary satellites (GEO), such as GOES-16/17/18, and Himawari-8/9, and Meteosat Third Generation (MTG) they provide a much higher temporal resolution (order of 10 minutes), typically an image once or more per hour during daylight compared to LEO once per day. By combining these observations, we may be able to characterize the diurnal cycle of global AOD at the local, regional and global scale. While the science community is still exploring the new data from GEO observations, we have been thinking about how to properly combine/merge/fuse those data considering differences in their spatial and temporal resolutions. However, this poses a “Big Data” challenge. The big data challenge is not just about data storage, but also about data discoverability, and accessibility, and even more, about data migration/mirroring in the cloud-computing environment. This paper is merely showing some of the efforts and approaches we have attempted in fusing six satellites’ Level 2 aerosol data (three are from GEO (GOES-16/17 and Himawari-8), and the other three are from LEO (TERRA/MODIS, AQUA/MODIS, SNPP-VIIRS) from Dark Target (DT) aerosol retrieval algorithm. Having the on-demand capability of fusing remote sensing products onto the desired temporal and spatial domain enables researchers and application practitioners to better manipulate and work with satellite and sensor data. It is our hopeWe hope that by making such an open-source package, and the accompanying functionality, the scientific community will be granted easier access to aerosol data processing resources. The MEaSUREs Program (Making Earth System Data Records for Use in Research Environments) expands our understanding of the Earth's current system through atmospheric and surface measurements. In an effort to aid the scientific research component and improve open source methods, this project developed Python code for fusing six satellite Level 2 aerosol data (three are from geostationary satellites (GEO), and the other three are from low earth orbital satellites (LEO)) from Dark Target Aerosol Retrieval Algorithm.

Jennifer Wei↗
NASA NTRS2017

GeoDash: Assisting Visual Image Interpretation in Collect Earth Online by Leveraging Big Data on Google Earth Engine

Collect Earth Online (CEO) is a free and open online implementation of the FAO Collect Earth system for collaboratively collecting environmental data through the visual interpretation of Earth observation imagery. The primary collection mechanism in CEO is human interpretation of land surface characteristics in imagery served via Web Map Services (WMS). However, interpreters may not have enough contextual information to classify samples by only viewing the imagery served via WMS, be they high resolution or otherwise. To assist in the interpretation and collection processes in CEO, SERVIR, a joint NASA-USAID initiative that brings Earth observations to improve environmental decision making in developing countries, developed the GeoDash system, an embedded and critical component of CEO. GeoDash leverages Google Earth Engine (GEE) by allowing users to set up custom browser-based widgets that pull from GEE's massive public data catalog. These widgets can be quick looks of other satellite imagery, time series graphs of environmental variables, and statistics panels of the same. Users can customize widgets with any of GEE's image collections, such as the historical Landsat collection with data available since the 1970s, select date ranges, image stretch parameters, graph characteristics, and create custom layouts, all on-the-fly to support plot interpretation in CEO. This presentation focuses on the implementation and potential applications, including the back-end links to GEE and the user interface with custom widget building. GeoDash takes large data volumes and condenses them into meaningful, relevant information for interpreters. While designed initially with national and global forest resource assessments in mind, the system will complement disaster assessments, agriculture management, project monitoring and evaluation, and more.

SERVI↗