Search NASA⌕ Search

SEARCH · Search NASA

Results for “big data analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
NASA NTRS2020

Near 40 Years MERRA-2 Data at NASA GES DISC -Opportunity and Challenge to Support Extremes Study

To the end of 2019, 40 years NASA climate reanalysis data sets from the Modern Era Retrospective-analysis for Research and Applications, Version 2 (MERRA-2) will be available at NASA Goddard Earth Sciences Data and Information Services Center (GES DISC). MERRA-2 consists atmosphere, land, and ocean data, which may be used for the studies ranging from the short scale weather events to the large scale decadal vulnerabilities. The hourly products, such as precipitation, soil moisture, temperature, and aerosols etc., have been used widely to study extreme events.In supporting users from broad communities, GES DISC have developed various data access services, including subsetter for downloading only data of interest with preferred format; OPeNDAP - for machine-to-machine data access; and Giovanni- for online visualization and analysis, etc. A big challenge for extreme study is to downloading and processing long-term hourly or daily data. The data downloading performance is not very satisfied by many users with current services and the native archived data structure. Late June 2019, many people in Europe had experienced extreme heat waves. The temperatures in several countries exceeded 40°C (104°F). For example, MERRA-2 shows that the near surface daily maximum temperature of June 28 2019 over Marseille, a city in southern France, reached 41.1 °C (106°F), which is the record breaking temperature in the last 40 years. GES DISC is working together with domain science experts to improve the performance of long time series access, making analysis ready data sets in supporting application researches, such as extreme study. In this presentation, using Europe heat wave as an example, we will show prototype of the in developing service for finding extremes from near 40 years MERRA-2 data at a given location. MERRA-2 data can be accessed from NASA GES DISC(https://disc.gsfc.nasa.gov/ ) by search keyword "MERRA-2".

Shen, Suhung↗
DOE OSTI2024

Time series methods for the analysis of soundscapes and other cyclical ecological data

Biodiversity monitoring has entered an era of ‘big data’, exemplified by a near-continuous collection of sounds, images, chemical and other signals from organisms in diverse ecosystems. Such data streams have the potential to help identify new threats, assess the effectiveness of conservation interventions, as well as generate new ecological insights. However, appropriate analytical methods are often still missing, particularly with respect to characterizing cyclical temporal patterns. Here, we present a framework for characterizing and analysing ecological responses that represent nonstationary, complex temporal patterns and demonstrate the value of using Fourier transforms to decorrelate continuous data points. In our example, we use a framework based on three approaches (spectral analysis, magnitude squared coherence, and principal component analysis) to characterize differences in tropical forest soundscapes within and across sites and seasons in Gabon. By reconstructing the underlying, cyclic behaviour of the soundscape for each site, we show how one can identify circadian patterns in acoustic activity. Soundscapes in the dry season had a complex diel cycle, requiring multiple harmonics to represent daily variation, while in the wet season there was less variance attributable to the daily cyclic patterns. Our framework can be applied to most continuous, or near-continuous ecological data collected at a fine temporal resolution, allowing ecologists to explore patterns of temporal autocorrelation at multiple levels for biologically meaningful trends. Such methods will become indispensable as biological big data are used to understand the impact of anthropogenic pressures on biodiversity and to inform efforts to mitigate them.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Cognitive analysis of metabolomics data for systems biology

Cognitive computing is revolutionizing the way big data are processed and integrated, with artificial intelligence (AI) natural language processing (NLP) platforms helping researchers to efficiently search and digest the vast scientific literature. Most available platforms have been developed for biomedical researchers, but new NLP tools are emerging for biologists in other fields and an important example is metabolomics. NLP provides literature-based contextualization of metabolic features that decreases the time and expert-level subject knowledge required during the prioritization, identification and interpretation steps in the metabolomics data analysis pipeline. Here, we describe and demonstrate four workflows that combine metabolomics data with NLP-based literature searches of scientific databases to aid in the analysis of metabolomics data and their biological interpretation. Additionally, the four procedures can be used in isolation or consecutively, depending on the research questions. The first, used for initial metabolite annotation and prioritization, creates a list of metabolites that would be interesting for follow-up. The second workflow finds literature evidence of the activity of metabolites and metabolic pathways in governing the biological condition on a systems biology level. The third is used to identify candidate biomarkers, and the fourth looks for metabolic conditions or drug-repurposing targets that the two diseases have in common. The protocol can take 1–4 h or more to complete, depending on the processing time of the various software used.

59 BASIC BIOLOGICAL SCIENCES↗
NASA NTRS2022

An Advanced Open-Source Platform for Air Quality Analysis, Visualization, and Prediction

Ambient air pollution is the largest environmental health risk factor, leading to several million premature deaths globally per year. The challenge of combating poor air quality is exacerbated by growing urban populations, changing emissions, and a warming climate. While there have been many advances monitoring and modeling of atmospheric composition, reflected in the dramatic increase in archived Earth Observations, there is no single measurement or method that alone can provide an accurate depiction of the entire atmosphere. The rapidly growing collections of observational and modeling data require us to be smarter about what data to include, and how such data is used. In recent years, NASA has invested significantly in advancing the concepts for Analytics Collaborative Framework (ACF) [5] and New Observing Strategies (NOS) [4] to tackle our software infrastructure need for harmonized data management and dynamic acquisition of diverse measurements for on-demand, interactive, multivariate analysis, and access [3]. It is not enough to have a big data, standalone analytics solution; it is critical that we start integrating data from remote sensing, modeling, and in-situ networks in a harmonized manner that enables timely and data-driven decision-making for air quality management. This work presents the design and development of an Air Quality Analytics Collaborative Framework (AQ ACF), as part of NASA’s Advanced Information Systems Technology (AIST) effort, to establish a data, machine-learning, and numerically driven platform for air quality analysis, visualization, and prediction.

Liu, Qian↗
DOE OSTI2020

Application of Systems Engineering Principles and Techniques in Biological Big Data Analytics: A Review

In the past few decades, we have witnessed tremendous advancements in biology, life sciences and healthcare. These advancements are due in no small part to the big data made available by various high-throughput technologies, the ever-advancing computing power, and the algorithmic advancements in machine learning. Specifically, big data analytics such as statistical and machine learning has become an essential tool in these rapidly developing fields. As a result, the subject has drawn increased attention and many review papers have been published in just the past few years on the subject. Different from all existing reviews, this work focuses on the application of systems, engineering principles and techniques in addressing some of the common challenges in big data analytics for biological, biomedical and healthcare applications. Specifically, this review focuses on the following three key areas in biological big data analytics where systems engineering principles and techniques have been playing important roles: the principle of parsimony in addressing overfitting, the dynamic analysis of biological data, and the role of domain knowledge in biological data analytics.

dynamic analysis↗
NASA NTRS2018

Advanced Analytics and Big Earth Data

NASA's Earth Science Data Systems process, archive and distribute petabytes of Earth Observation data to a variety of end users. These end users will face dramatically increased data size in the near future, bringing about new challenges and opportunities in analyzing those data. One area of particular ferment currently is Machine Learning. Many Machine Learning methods are black boxes, limiting direct insight into the data's properties. However, they can be used for a variety of data enhancement purposes, such as parameter retrieval, data fusion and image classification and segmentation. The Earth Observing System Data and Information System is also evolving to host large data volumes in the cloud, enabling data proximal analysis. As part of this effort, an Analytics framework is being developed to support and enhance user analysis of the data. By using standards based services in the framework, diverse user communities can be served, while also allowing inter-system collaboration in the analysis process.

Cloud Computing↗
NASA NTRS2004

Statistical Analysis of CMC Constituent and Processing Data

Ceramic Matrix Composites (CMCs) are the next "big thing" in high-temperature structural materials. In the case of jet engines, it is widely believed that the metallic superalloys currently being utilized for hot structures (combustors, shrouds, turbine vanes and blades) are nearing their potential limits of improvement. In order to allow for increased turbine temperatures to increase engine efficiency, material scientists have begun looking toward advanced CMCs and SiC/SiC composites in particular. Ceramic composites provide greater strength-to-weight ratios at higher temperatures than metallic alloys, but at the same time require greater challenges in micro-structural optimization that in turn increases the cost of the material as well as increases the risk of variability in the material s thermo-structural behavior. to model various potential CMC engine materials and examines the current variability in these properties due to variability in component processing conditions and constituent materials; then, to see how processing and constituent variations effect key strength, stiffness, and thermal properties of the finished components. Basically, this means trying to model variations in the component s behavior by knowing what went into creating it. inter-phase and manufactured by chemical vapor infiltration (CVI) and melt infiltration (MI) were considered. Examinations of: (1) the percent constituents by volume, (2) the inter-phase thickness, (3) variations in the total porosity, and (4) variations in the chemical composition of the Sic fiber are carried out and modeled using various codes used here at NASA-Glenn (PCGina, NASALife, CEMCAN, etc...). The effects of these variations and the ranking of their respective influences on the various thermo-mechanical material properties are studied and compared to available test data. The properties of the materials as well as minor changes to geometry are then made to the computer model and the detrimental effects observed using statistical analysis software. The ultimate purpose of this study is to determine what variations in material processing can lead to the most critical changes in the materials property. The work I have taken part in this summer explores, in general, the key properties needed In this study SiC/SiC composites of varying architectures, utilizing a boron-nitride (BN)

Fornuff, Jonathan↗
DOE OSTI2020

Hawaii Play Fairway Analysis: Noble Gas Raw Data for Hawaii, Maui, Oahu, Kauai, and Lanai islands

Noble gas raw data for the Hawaiian islands of Big Island, Maui, Oahu, Lanai, and Kauai. Based on results from prior phases of the Hawaii Play Fairway Analysis, this project targeted 66 wells on the islands of Hawaii, Maui, Lanai, Oahu, and Kauai for sampling of dissolved noble gases, trace metals, common ions, and the stable isotopes 2H and 18O. Ultimately, 23 of the 66 well targets were sampled. Noble gas data from this study is supplemented with data shared by the United States Geologic Survey for the summit of Kilauea, and by the geothermal energy company Ormat Technologies Inc. for their geothermal power plant Puna Geothermal Venture on the Lower East Rift of Kilauea, and for their exploration of Kona and Hualalai on Hawaii, as well as the Southwest Rift of Haleakala on Maui. The noble gas helium is used as an indicator of geothermal heat when excess 3He and/or 4He is present when compared to the atmospheric ratio of those isotopes (R/Ra). R/Ra is minimally affected by dilution and transport, allowing even those wells not perfectly situated over a geothermal system to indicate a geothermal anomaly. R/Ra anomalies are present on every island in this study. There is a strong correlation between R/Ra anomalies and proximity to rift zones and calderas. The Hawaii Play Fairway project was funded by the U.S. Department of Energy Geothermal Technologies Office (award DE-EE0006729). For more information, see Colin Ferguson's Master of Science thesis "Exploration for Blind Geothermal Resources in the State of Hawaii Utilizing Dissolved Noble Gasses in Well Waters."

15 GEOTHERMAL ENERGY↗
DOE OSTI2024

Performance Analysis of Data Processing in Distributed File Systems with Near Data Processing

In the era of big data, the escalating volume and velocity of data generation pose significant challenges in data processing. Traditional systems like Spark and Hadoop manage the increasing amount and velocity of data by improving data placement and processing speeds. However, they face inherent limitations due to the essential data movement required for processing. In this paper, we explore the Skyhook framework, a novel extension of the Ceph distributed system, which significantly reduces the need for data movement. We present an extensive case study using the Skyhook framework, applying it with the TPC-H and K-means clustering algorithms. More specifically, we leverage the TPC-H benchmark to distinguish between CPU-intensive and I/O-intensive tasks. We explore the integration of K-means clustering into SQL, coupled with a near-data processing system to offload the computational burden of the K-means clustering algorithm to storage nodes. We conduct a comprehensive performance evaluation of distributed data processing applications across three processing approaches: traditional layout (baseline), optimized layout, and near-data processing. Additionally, we introduce the use of the FIO tool to simulate real-world system workloads, enabling the measurement of performance metrics such as average latency and CPU utilization. Our research is a significant advance in understanding how to optimize data processing systems to meet the demands of the modern data landscape.

Hou, Shiyue↗
DOE OSTI2021

Improving the Accessibility and Usability of Geothermal Information with Data Lakes and Data Pipelines on the Geothermal Data Repository: Preprint

The Geothermal Data Repository (GDR) provides universal access to data and information resulting from research and development activities funded by the Department of Energy (DOE). The GDR has extended this universal access to big data through integration with data lakes developed by the Open Energy Data Initiative (OEDI). Previously, large datasets such as seismic waveform or distributed acoustic sensing (DAS) data could only be accessed by institutions with high performance data storage and compute capabilities, effectively limiting the accessibility of big data to national labs, larger universities, and major corporations. Moreover, the time and resources needed to transport big data and configure them can produce additional barriers to use. Many of the standard formats used for structured data models (also known as content models) are incapable of handling big data and can introduce additional usability problems, often requiring data to be reformatted prior to use. This paper will explore how recent integrations between the GDR and the OEDI data lake have improved the accessibility and usability of geothermal data in a big way, making the data available to a broader audience, and enabling collaborative analysis and innovation across the greater geothermal industry.

access↗
NASA NTRS2018

Introduction to Big Earth Data Applications

Climate and weather modeling generate enormous volumes that make iterative analysis challenging, spurring the development of new ways to work with the data. A theme going across applications is the need to identify and highlight "interesting" data for the scientist to focus on. Operational applications often scale up from small, local studies to larger spatial scales with more analysis targets.

parallel processing (computers)↗
NASA NTRS2022

Introduction to Big Earth Data Applications

Climate and weather modeling generate enormous volumes that make iterative analysis challenging, spurring the development of new ways to work with the data. At the same time in the Earth Observation area, technology advances are enabling new sensors and satellites that will increase data volume, velocity and application variety. Scaling up can also be seen when operational applications expand from small, local studies to larger spatial scales with more analysis targets.

Christopher Lynnes↗
DOE OSTI2020

Measurement of the Li 7 ( γ , t ) He 4 ground-state cross section between E γ = 4.4 and 10 MeV

Here, the Li 7 ( γ , t ) He 4 ground state cross section was measured for the first time using monoenergetic γ rays with energies between 4.4 and 10 MeV at the High Intensity Gamma-ray Source. The reaction is important for the primordial Li problem and for testing our understanding of the mirror α -capture reactions H 3 ( α , γ ) Li 7 and He 3 ( α , γ ) Be 7 . Although over the last 30 years most measurements of the H 3 ( α , γ ) Li 7 reaction have concentrated in an energy range below E γ = 3.65 MeV, measurements at higher energies could potentially restrict the extrapolation to astrophysically important energies. The experimental arrangement for measuring the Li 7 ( γ , t ) He 4 reaction included a large-area silicon detector array and several beam characterization instruments. The experimental astrophysical S factor of H 3 ( α , γ ) calculated from the present data was fitted using the R -matrix formalism. The results are in disagreement with previous experimental measurements in the same energy range but the extrapolated S factor agrees with the potential model calculation and lower energy experimental data.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2020

Spatiotemporal Pattern Recognition in the PMU Signals in the WECC system

Phasor measurement unit (PMU) data has been used by multiple power system applications, including state estimation, post event analysis, oscillation detection, model validation, and many others. Still, due to its big data nature and availability to general research institutions, comprehensive understanding of the spatiotemporal patterns and underlying mechanisms are incomplete. This study applies a set of signal processing and machine learning approaches aiming at deciphering the characteristic behaviors of multiple phasor measurement units (PMUs) attributes (e.g., voltage, frequency, rate of change of frequency, phase angle), including their auto-correlation, cross-dependence, similarities and discrepancies across units and temporal scales, and distributions of anomalies and their linkages to potential external factors such as weather events. Data analytics are applied to PMUs from the U.S. Western Electricity Coordinating Council (WECC) system. The PMU measurements, recorded events, outages, and weather extremes are all from real world datasets. The findings from the study and mechanistic understanding of the PMU dynamics help provide guidance on system control or preventing blackouts. The derived metrics can be directly used for adjusting or filtering simulated PMU data used for advanced algorithm development.

Hou, Zhangshuan↗
DOE OSTI2020

On Performance Prediction of Big Data Transfer in High-performance Networks

Big data generated by large-scale scientific and industrial applications need to be transferred between different geographical locations for remote storage, processing, and analysis. High-speed dedicated connections provisioned in High-performance Networks (HPNs) are increasingly utilized to carry out such big data transfer. HPN management highly relies on an important capability of performance (mainly throughput) prediction to reserve sufficient bandwidth and meanwhile avoid over-provisioning that may result in unnecessary resource waste. This capability is critical to improving the resource (mainly bandwidth) utilization of dedicated connections and meeting various user requests for data transfer. Conventional methods conduct performance prediction by fitting prior observed transfer history with predefined loss functions, without considering unobservable latent factors such as competing loads on end hosts. Such latent factors also have a significant impact on the application-level data transfer performance, which may result in an inaccurate prediction model. In this paper, we first investigate the impact of latent factors and propose a clustering-based method to eliminate their negative impact on performance prediction. We then develop a robust machine learning-based performance predictor by: i) incorporating the proposed latent factor elimination method into data preprocessing, and ii) adopting a customized domain guided loss function. Extensive experimental results show that our predictor achieves significantly higher prediction accuracy than several other state-of-the-art methods.

Liu, Wuji↗
DOE OSTI2022

Functional Data Analysis for Extracting the Intrinsic Dimensionality of Spectra: Application to Chemical Homogeneity in the Open Cluster M67

High-resolution spectroscopic surveys of the Milky Way have entered the Big Data regime and have opened avenues for solving outstanding questions in Galactic archeology. However, exploiting their full potential is limited by complex systematics, whose characterization has not received much attention in modern spectroscopic analyses. In this work, we present a novel method to disentangle the component of spectral data space intrinsic to the stars from that due to systematics. Using functional principal component analysis on a sample of 18,933 giant spectra from APOGEE, we find that the intrinsic structure above the level of observational uncertainties requires ≈10 functional principal components (FPCs). Our FPCs can reduce the dimensionality of spectra, remove systematics, and impute masked wavelengths, thereby enabling accurate studies of stellar populations. To demonstrate the applicability of our FPCs, we use them to infer stellar parameters and abundances of 28 giants in the open cluster M67. We employ Sequential Neural Likelihood, a simulation-based Bayesian inference method that learns likelihood functions using neural density estimators, to incorporate non-Gaussian effects in spectral likelihoods. By hierarchically combining the inferred abundances, we limit the spread of the following elements in M67: Fe ≲ 0.02 dex; C ≲ 0.03 dex; O, Mg, Si, Ni ≲ 0.04 dex; Ca ≲ 0.05 dex; N, Al ≲ 0.07 dex (at 68% confidence). Our constraints suggest a lack of self-pollution by core-collapse supernovae in M67, which has promising implications for the future of chemical tagging to understand the star formation history and dynamical evolution of the Milky Way.

79 ASTRONOMY AND ASTROPHYSICS↗
NASA NTRS2016

IN13B-1660: Analytics and Visualization Pipelines for Big Data on the NASA Earth Exchange (NEX) and OpenNEX

We are developing capabilities for an integrated petabyte-scale Earth science collaborative analysis and visualization environment. The ultimate goal is to deploy this environment within the NASA Earth Exchange (NEX) and OpenNEX in order to enhance existing science data production pipelines in both high-performance computing (HPC) and cloud environments. Bridging of HPC and cloud is a fairly new concept under active research and this system significantly enhances the ability of the scientific community to accelerate analysis and visualization of Earth science data from NASA missions, model outputs and other sources. We have developed a web-based system that seamlessly interfaces with both high-performance computing (HPC) and cloud environments, providing tools that enable science teams to develop and deploy large-scale analysis, visualization and QA pipelines of both the production process and the data products, and enable sharing results with the community. Our project is developed in several stages each addressing separate challenge - workflow integration, parallel execution in either cloud or HPC environments and big-data analytics or visualization. This work benefits a number of existing and upcoming projects supported by NEX, such as the Web Enabled Landsat Data (WELD), where we are developing a new QA pipeline for the 25PB system.

visualization↗
DOE OSTI2021

X-composer: enabling cross-environments in-situ workflows between HPC and cloud

As large-scale scientific simulations and big data analyses become more popular, it is increasingly more expensive to store huge amounts of raw simulation results to perform post-analysis. To minimize the expensive data I/O, "in-situ" analysis is a promising approach, where data analysis applications analyze the simulation generated data on the fly without storing it first. However, it is challenging to organize, transform, and transport data at scales between two semantically different ecosystems due to the distinct software and hardware difference. To tackle these challenges, we design and implement the X-Composer framework. X-Composer connects cross-ecosystem applications to form an "in-situ" scientific workflow, and provides a unified approach and recipe for supporting such hybrid in-situ workflows on distributed heterogeneous resources. X-Composer reorganizes simulation data as continuous data streams and feeds them seamlessly into the Cloud-based stream processing services to minimize I/O overheads. For evaluation, we use X-Composer to set up and execute a cross-ecosystem workflow, which consists of a parallel Computational Fluid Dynamics simulation running on HPC, and a distributed Dynamic Mode Decomposition analysis application running on Cloud. Our experimental results show that X-Composer can seamlessly couple HPC and Big Data jobs in their own native environments, achieve good scalability, and provide high-fidelity analytics for ongoing simulations in real-time.

Wang, Dali↗