SEARCH · Search NASA

Results for “big data analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8

All sources NASA NTRS DOE OSTI

NASA NTRS2017

Investigating Access Performance of Long Time Series with Restructured Big Model Data

Data sets generated by models are substantially increasing in volume, due to increases in spatial and temporal resolution, and the number of output variables. Many users wish to download subsetted data in preferred data formats and structures, as it is getting increasingly difficult to handle the original full-size data files. For example, application research users such as those involved with wind or solar energy, or extreme weather events are likely only interested in daily or hourly model data at a single point (or for a small area) for a long time period, and prefer to have the data downloaded in a single file. With native model file structures, such as hourly data from NASA Modern-Era Retrospective analysis for Research and Applications Version-2 (MERRA-2), it may take over 10 hours for the extraction of parameters-of-interest at a single point for 30 years. The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) is exploring methods to address this particular user need. One approach is to create value-added data by reconstructing the data files. Taking MERRA-2 data as an example, we have tested converting hourly data from one-day-per-file into different data cubes, such as one-month, or one-year. Performance is compared for reading local data files and accessing data through interoperable services, such as OPeNDAP. Results show that, compared to the original file structure, the new data cubes offer much better performance for accessing long time series. We have noticed that performance is associated with the cube size and structure, the compression method, and how the data are accessed. An optimized data cube structure will not only improve data access, but also may enable better online analysis services

reanalysis↗

DOE OSTI2020

A Survey of Singular Value Decomposition Methods for Distributed Tall/Skinny Data

The Singular Value Decomposition (SVD) is one of the most important matrix factorizations, enjoying a wide variety of applications across numerous application domains. In statistics and data analysis, the common applications of SVD inclue Principal Components Analysis (PCA) and regression. Usually these applications arise on data that has far more rows than columns, so-called "tall/skinny" matrices. In the big data analytics context, this may take the form of hundreds of millions to billions of rows with only a few hundred columns. There is a need, therefore, for fast, accurate, and scalable tall/skinny SVD implementations which can fully utilize modern computing resources. To that end, we present a survey of three different algorithms for computing the SVD for these kinds of tall/skinny data layouts using MPI for communication. We contextualize these with common big data analytics techniques. Finally, we present both CPU and GPU timing results from the Summit supercomputer, and discuss possible alternative approaches.

Schmidt, Drew↗

DOE OSTI2024

Perturbation of soil organic carbon induced by land-use change from primary forest

Abstract The impact of land-use change (LUC) on soil organic carbon (SOC) has been a wide concern of land management policymakers because CO 2 emissions induced by LUC have been the second largest carbon source worldwide. However, due to insufficient data quality and limited biome coverage, a global big picture of the impact of LUC on SOC is still not clear. This study conducted a meta-analysis on 288 independent observations sourced from 62 peer-reviewed papers to provide a global summary of the change in SOC after the conversion of primary forests into other land-use types. The conversion of primary forest to cropland resulted in the most severe SOC loss (−33.2%), followed by conversion into plantation forests (−22.3%) and secondary forests (−19.1%). Nonetheless, SOC increased by 9.1% after a conversion from primary forests into pasture. More SOC loss was found at sites with lower precipitation for primary forests converted to cropland and plantation forests. The SOC loss decreased consistently with increasing mean annual temperature (MAT) for all four types of LUC. Moreover, the loss of SOC tended to worsen over time when primary forests are converted to cropland or plantation forests. In contrast, SOC loss recovered over time following conversion to secondary forests. The gain of SOC gradually increased over time after conversion to pastures. To conclude, the changes in SOC are related not only to the land-use type but also to precipitation, temperature and turn years after LUC. Due to limited data, this study focuses on soil profiles within 30 cm depth, and future research should explore SOC dynamics induced by LUC at greater depths. Overall, cases of SOC loss of approximately 30% following deforestation were very common (except for conversion to pasture), and the results of this study show that the loss of SOC following LUC should be carefully considered and monitored in land management.

Zhang, Zhiyuan (ORCID:0000000223407001)↗

DOE OSTI2023

In-depth analysis on parallel processing patterns for high-performance Dataframes

The Data Science domain has expanded monumentally in both research and industry communities during the past decade, predominantly owing to the Big Data revolution. Artificial Intelligence (AI) and Machine Learning (ML) are bringing more complexities to data engineering applications, which are now integrated into data processing pipelines to process terabytes of data. Typically, a significant amount of time is spent on data preprocessing in these pipelines, and hence improving its efficiency directly impacts the overall pipeline performance. The community has recently embraced the concept of Dataframes as the de-facto data structure for data representation and manipulation. However, the most widely used serial Dataframes today (R, pandas) experience performance limitations while working on even moderately large data sets. We believe that there is plenty of room for improvement by taking a look at this problem from a high-performance computing point of view. In a prior publication, we presented a set of parallel processing patterns for distributed dataframe operators and the reference runtime implementation, Cylon. In this paper, we are expanding on the initial concept by introducing a cost model for evaluating the said patterns. Furthermore, we evaluate the performance of Cylon on the ORNL Summit supercomputer.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2019

NASA GES DISC Giovanni: Current and Future

Giovanni (Geospatial Interactive Online Visualization and Analysis Infrastructure), developed by the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), has established a reputation among NASA users for easy access, analysis, and visualization of NASA Earth science data. Currently, Giovanni supports over 1900 variables in eight disciplinary areas. Like any other enterprise application, Giovanni faces big data challenges, such as servicing increasingly large data volumes and more complex data types, while at the same time addressing the demands of a more diverse user community, e.g., placing requests for long-term time series from multiple spatially and temporally dense data records. I will present how Giovanni has been evolving from an on-premises, monolithic software application towards a cloud-enabled implementation to address these challenges.

Analytics↗

DOE OSTI2025

DESI 2024 VII: cosmological constraints from the full-shape modeling of clustering measurements

We present cosmological results from the measurement of clustering of galaxy, quasar and Lyman-α forest tracers from the first year of observations with the Dark Energy Spectroscopic Instrument (DESI Data Release 1). We adopt the full-shape (FS) modeling of the power spectrum, including the effects of redshift-space distortions, in an analysis which has been thoroughly validated in a series of supporting papers as summarised in [1]. We combine the full-shape information with DESI's DR1 constraints from the baryon acoustic oscillations (BAO) of these tracers. In the flat ΛCDM cosmological model, DESI (FS+BAO), combined with a baryon density prior from Big Bang Nucleosynthesis and a weak prior on the scalar spectral index, determines matter density to Ω m = 0.2962 ± 0.0095, and the amplitude of mass fluctuations to σ 8 = 0.842 ± 0.034. The addition of the cosmic microwave background (CMB) data tightens these constraints to Ω m = 0.3056 ± 0.0049 and σ 8 = 0.8121 ± 0.0053, while further addition of the joint clustering and lensing analysis from the Dark Energy Survey Year-3 (DESY3) data further improves these measurements, and leads to a 0.4% determination of the Hubble constant, H 0 = (68.40 ± 0.27) km s -1 Mpc -1 . In models with a time-varying dark energy equation of state parametrised by w 0 and w a , combinations of DESI (FS+BAO) with CMB and type Ia supernovae continue to show the preference, previously found in the DESI DR1 BAO analysis, for w 0 > -1 and w a < 0 with similar levels of significance. DESI data, in combination with the CMB, improve the upper limits on the sum of the neutrino masses relative to the case when only the DR1 BAO was available, giving ∑m ν < 0.071 eV at 95% confidence. We finally constrain deviations from general relativity represented by two modified gravity parameters. DESI (FS+BAO) data alone measure the parameter that controls the clustering of massive particles, μ 0 = 0.11 +0.45 -0.54 , in agreement with the zero value predicted by general relativity. The combination of DESI with the CMB and the clustering and lensing analysis from DESY3 constrains both modified-gravity parameters, giving μ 0 = 0.04 ± 0.22 and Σ 0 = 0.044 ± 0.047, again in agreement with general relativity.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2022

Machine Learning Using a Simple Feature for Detecting Multiple Types of Events From PMU Data

This paper describes simple and efficient machine learning (ML) methods for efficiently detecting multiple types of power system events captured by PMUs scarcely placed in a large power grid. It uses a single feature from each PMU based on a rectangle area enclosing the event in a given data window. This single feature is sufficient to enable commonly used ML models to detect different types of events quickly and accurately. The feature is used by five ML models on four different data-window sizes. The results indicated a tradeoff between the execution speed and detection accuracy in variety of data-window size choices. Here, the proposed method is insensitive to most data quality issues typical for data from field PMUs, and thus it does not require major data cleansing efforts prior to feature extraction.

Big data↗

NASA NTRS1978

Measurement and analysis of magnetic field variation during a class 2b flare

A digital analysis of high-time-resolution videomagnetograms taken during a class 2b flare that occurred at 60 deg east is reported. The data were obtained at the Big Bear Observatory and calibrated by a Mt. Wilson magnetogram. Changes of weak magnetic fields (less than 100 G) with an amplitude of 30% to 100% have been detected over 55% of the optical flare region, apparently taking place at the initial phase of the flare. Statistical considerations suggest a real flare association with most of these changes. H-alpha observations show that large changes took place over the footpoints of heavily inclined structures like penumbral fibrils, while smaller changes took place over the plage region. An apparent polarity reversal was found at the feet of erupted fibrils. Based on force-free field calculations these changes can be reasonably explained as a transformation of the current-carrying fields to potential fields which produced large changes in the field-line inclination and rotation.

Tanaka, K.↗

NASA NTRS1998

Emission Lines and the High Energy Continuum

Quasars show many striking relationships between line and continuum radiation whose origins remain a mystery. FeII, [OIII], Hbeta, and HeII emission line properties correlate with high energy continuum properties such as the relative strength of X-ray emission, and X-ray continuum slope. At the same time, the shape of the high energy continuum may vary with luminosity. An important tool for studying global properties of Quasi Stellar Objects (QSOs) is the co-addition of data for samples of QSOS. We use this to show that X-ray bright (XB) QSOs show stronger emission lines in general, but particularly from the narrow line region. The difference in the [OIII]/Hbeta ratio is particularly striking, and even more so when blended FeII emission is properly subtracted. Weaker narrow forbidden lines ([OII] and NeV) are enhanced by factors of 2 to 3 in both UV and optical XB composite spectra. The physical origin of these diverse and interrelated correlations has yet to be determined. Unfortunately, many physically informative trends intrinsic to QSOs may be masked by dispersion in the data due to either low signal-to-noise or variability. An important tool for studying global properties of QSOs is the co-addition of data for samples of QSOS. We use this to show that X-ray bright (XB) QSOs show stronger emission lines in general, but particularly from the narrow line region. The difference in the [OIII]/Hbeta ratio is particularly striking, and even more so when blended Fell emission is properly subtracted. Weaker narrow forbidden lines ([OII] and NeV) are enhanced by factors of 2 to 3 in both UV and optical XB composite spectra. We describe a large-scale effort now underway to probe these effects in large samples, using both data and analysis as homogeneous as possible. Using an HST FOS Atlas of QSO spectra, with primary comparison to ROSAT PSPC spectral constraints, we will model the Big Blue Bump, its relationship to luminosity and QSO type, and we will analyze and contrast line emission and UV/X-ray continuum properties. Absorption of the continuum near the broad emission line region may play a profound role, which we will be able to constrain by direct analysis of observed UV/X-ray spectral absorption.

Green, Paul↗

DOE OSTI2021

Fast Extraction and Characterization of Fundamental Frequency Events from a Large PMU Dataset using Big Data Analytics

A novel method for fast extraction of fundamental frequency events (FFE) based on measurements of frequency and rate of change of frequency by Phasor Measurement Units (PMU) is introduced. The method is designed to work with exceptionally large historical PMU datasets. Statistical analysis was used to extract the features and train Random Forest and Catboost classifiers. The method is capable of fast extraction of FFE from a historical dataset containing measurements from hundreds of PMUs captured over multiple years. The reported accuracy of the best algorithm for classification expressed as Area Under the receiver operating Characteristic curve reaches 0.98, which was obtained in out-of-sample evaluations on 109 system-wide events over 2 years observed at 43 PMUs. Then Minimum Volume Enclosing Ellipsoid Algorithm was used to further analyze the events. 93.72% events were correctly characterized, where average duration of the event as seen by the PMU was 9.93 sec.

Baembitov, Rashid↗

NASA NTRS2014

MERRA Analytic Services: Meeting the Big Data Challenges of Climate Science Through Cloud-enabled Climate Analytics-as-a-service

Climate science is a Big Data domain that is experiencing unprecedented growth. In our efforts to address the Big Data challenges of climate science, we are moving toward a notion of Climate Analytics-as-a-Service (CAaaS). We focus on analytics, because it is the knowledge gained from our interactions with Big Data that ultimately produce societal benefits. We focus on CAaaS because we believe it provides a useful way of thinking about the problem: a specialization of the concept of business process-as-a-service, which is an evolving extension of IaaS, PaaS, and SaaS enabled by Cloud Computing. Within this framework, Cloud Computing plays an important role; however, we it see it as only one element in a constellation of capabilities that are essential to delivering climate analytics as a service. These elements are essential because in the aggregate they lead to generativity, a capacity for self-assembly that we feel is the key to solving many of the Big Data challenges in this domain. MERRA Analytic Services (MERRAAS) is an example of cloud-enabled CAaaS built on this principle. MERRAAS enables MapReduce analytics over NASAs Modern-Era Retrospective Analysis for Research and Applications (MERRA) data collection. The MERRA reanalysis integrates observational data with numerical models to produce a global temporally and spatially consistent synthesis of 26 key climate variables. It represents a type of data product that is of growing importance to scientists doing climate change research and a wide range of decision support applications. MERRAAS brings together the following generative elements in a full, end-to-end demonstration of CAaaS capabilities: (1) high-performance, data proximal analytics, (2) scalable data management, (3) software appliance virtualization, (4) adaptive analytics, and (5) a domain-harmonized API. The effectiveness of MERRAAS has been demonstrated in several applications. In our experience, Cloud Computing lowers the barriers and risk to organizational change, fosters innovation and experimentation, facilitates technology transfer, and provides the agility required to meet our customers' increasing and changing needs. Cloud Computing is providing a new tier in the data services stack that helps connect earthbound, enterprise-level data and computational resources to new customers and new mobility-driven applications and modes of work. For climate science, Cloud Computing's capacity to engage communities in the construction of new capabilies is perhaps the most important link between Cloud Computing and Big Data.

Data Analytics↗

NASA NTRSDate not supplied

NASA’s Prototype Spectral Water Inversion Processor and Emulator (SWIPE): Towards Global Coastal and Inland Water Quality and Algal Biodiversity Monitoring

Degradation of Earth’s inland water resources due to anthropogenic perturbations and climate anomalies at both local and global scales continues to place human health at substantial risk. There is now a growing necessity to develop pragmatic approaches that allow timely and effective extrapolation of local processes, to spatially resolved global products, and to promote operational and sustainable resource policy management. This presentation will provide updates on NASA’s prototype open-source aquatic modeling platform, Spectral Water Inversion Processor and Emulator (SWIPE), which is a comprehensive, multi-faceted modeling platform for both forward and inverse modeling of diverse aquatic ecosystems from the benthos to top-of-atmosphere (TOA). SWIPE provides a cohesive application which leverages recent advancements in particle modeling, Big Data analytics, and machine learning to develop a high-fidelity synthetic training ground for sensitivity studies and algorithm development for multispectral or upcoming hyperspectral missions. Some of the prominent features of SWIPE to be discussed include: 1. Advanced hyperspectral modeling of globally diverse algal and non-algal particles using a novel two-layer coated sphere scattering model and radiative transfer modeling, 2. Massive, highly detailed synthetic spectral libraries of Analysis-Ready-Data (ARD) which include spectral libraries of particle microphysics, water biogeophysical and optical properties, as well as surface and TOA reflectances at 1 nm resolution, 3. An ensemble of pre-built analytic, machine learning, and deep learning inversion algorithms for various water quality and biodiversity related retrieval parameters and uncertainty quantification, 4. Sensor-agnostic water quality inversion at wide ranging spatial and spectral resolutions including a codebase for seamless application in the Google Earth Engine and NASA Earth Exchange (NEX) for planetary scale analysis. SWIPE will be a fully open-source platform based in python with comprehensive documentation, tutorials, and options for distributed computing on high performance computing clusters or on single, local machines. Further, we will discuss how we envision SWIPE contributing towards a global analysis of coastal and inland water quality dynamics.

top-of-atmosphere (TOA)↗

DOE OSTI2020

Database Management Module Framework for Dynamic Contingency Analysis and Visualization

The Dynamic Contingency Analysis Tool (DCAT) is being developed to mimic the response of generation dynamics, protection, and load components to extreme contingencies in power systems. Currently, DCAT generates a huge data set, with the analysis results stored in multiple subfolders using a great many files in different formats (power flow cases, time series, figures, spreadsheets). To help power system engineers thoroughly understand and analyze system behavior under many scenarios and contingencies, we developed a database management module (DBM) that enables future integration of DCAT with industry-grade Big Data analytical and interactive visualization platforms. In this paper, we demonstrate the DBM framework with interactive analytics and visualization for DCAT on the synthetic South Carolina 500-bus power system model using hurricane contingencies that capture possibilities for an extreme contingency. DCAT’s advanced data management and interactive visualization can help planning engineers to visually explore and analyze system behavior, all in one view.

Cascading, dynamic simulations, database managemen↗

NASA NTRS2016

An Integrated Gate Turnaround Management Concept Leveraging Big Data/Analytics for NAS Performance Improvements

The Integrated Gate Turnaround Management (IGTM) concept was developed to improve the gate turnaround performance at the airport by leveraging relevant historical data to support optimization of airport gate operations, which include: taxi to the gate, gate services, push back, taxi to the runway, and takeoff, based on available resources, constraints, and uncertainties. By analyzing events of gate operations, primary performance dependent attributes of these events were identified for the historical data analysis such that performance models can be developed based on uncertainties to support descriptive, predictive, and prescriptive functions. A system architecture was developed to examine system requirements in support of such a concept. An IGTM prototype was developed to demonstrate the concept using a distributed network and collaborative decision tools for stakeholders to meet on time pushback performance under uncertainties.

Big Data and Net-enabled ATM↗

DOE OSTI2020

Leading magnetic fusion energy science into the big-and-fast data lane

To crack the code of unlimited clean energy from magnetic fusion, researchers collaborate among numerous facilities that are scattered around the globe. Probing hot fusion plasmas with sophisticated diagnostics, they routinely generate fast streams of high-dimensional time-series data. We present the adaptation of a production code-base for a plasma diagnostic to the big-and-fast data lane. The new code-base is used to stream measurements, made in Korea, into a Top-15 computing facility in the US for remote analysis. We discuss software and design choices made for this new HPC-enabled scientific workflow.

Kube, Ralph↗

NASA NTRS2016

Achieving Fast Operational Intelligence in NASA's Deep Space Network Through Complex Event Processing

NASA’s Deep Space Network (DSN) is a complex, global project, in which the expertise of human operators remain crucial for its successful operation. To find ways to save costs in operations and to improve its services, a number of modernization efforts are underway in the DSN. One such effort is a research and technology development task at the Jet Propulsion Laboratory that is investigating the use of complex event processing (CEP) for intelligent assessment of situations, trend analysis, and advanced automation. The technology leverages the significant business intelligence (BI) and data science advancements made in the enterprise industries over the last several years. The open source big data processing engine Apache SparkTM and the high-throughput, distributed messaging system Apache Kafka form the core of the DSN Complex Event Processing (DCEP) framework. This paper discusses the system engineering perspective of why achieving efficient, lower-cost operations in the DSN is a challenging problem, how the DCEP system handles the use cases that help realize intelligent operations, and how this solution fits into the overall model of the planned DSN Follow-the- Sun Operations (FtSO).

Choi, Joshua S.↗

NASA NTRS1986

AIS spectra of desert shrub canopies

Airborne Imaging Spectrometer (AIS) data were collected 30 August 1985 from a desert shrub community in central Oregon. Spectra from artificial targets placed on the test site and from bare soil, big sagebrush (Artemesia tridentata wyomingensis), silver sagebrush (Artemesia cana bolander), and exposed volcanic rocks were studied. Spectral data from grating position 3 (tree mode) were selected from 25 ground positions for analysis by Principal Factor Analysis (PFA). In this grating position, as many as six factors were identified as significant in contributing to spectral structure. Channels 74 through 84 (tree mode) best characterized between-class differences. Other channels were identified as nondiscriminating and as associated with such errors as excessive atmospheric absorption and grating positin changes. The test site was relatively simple with the two species (A. tridentata and A. cana) representing nearly 95% of biomass and with only two mineral backgrounds, a montmorillonitic soil and volcanic rocks. If, as in this study, six factors of spectral structure can be extracted from a single grating position from data acquired over a simple vegetation community, then AIS data must be considered rich in information-gathering potential.

Murray, R.↗

NASA NTRS2001

Estimating Expressed Temperature and Fractional Area of Hot Lava at the Kilauea Vent with AVIRIS Spectral Measurements

Imaging spectroscopy offers a framework based in physics and chemistry for scientific investigation of a wide range of phenomena of interest in the Earth environment. In the scientific discipline of volcanology knowledge of lava temperature and distribution at the surface provides insight into the volcano status and subsurface processes. A remote sensing strategy to measure surface lava temperatures and distribution would support volcanology research. Hot targets such as molten lava emit spectral radiance as a function of temperature. A figure shows a series of Planck functions calculated radiance spectra for hot targets at different temperatures. A maximum Lambertian solar reflected radiance spectrum is shown as well. While similar in form, each hot target spectrum has a unique spectral shape and is distinct from the solar reflected radiance spectrum. Based on this temperature-dependent signature, imaging spectroscopy provides an innovative approach for the remote-sensing-based measurement of lava temperature. A natural site for investigation of the measurement of lava temperature is the Big Island of Hawaii where molten lava from the Kilauea vent is present at the surface. In the past, Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) data sets have been used for the analysis of hot volcanic targets and hot burning fires. The research presented here builds upon and extends this earlier work. The year 2000 Hawaii AVIRIS data set has been analyzed to derive lava temperatures taking into account factors of fractional fill, solar reflected radiance, and atmospheric attenuation of the surface emitted radiance. The measurements, analyses, and current results for this research are presented here.

Green, Robert O.↗