Search NASA⌕ Search

SEARCH · Search NASA

Results for “large data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 289 records · Page 16

Arithmetic Data Cube as a Data Intensive Benchmark

Data movement across computational grids and across memory hierarchy of individual grid machines is known to be a limiting factor for application involving large data sets. In this paper we introduce the Data Cube Operator on an Arithmetic Data Set which we call Arithmetic Data Cube (ADC). We propose to use the ADC to benchmark grid capabilities to handle large distributed data sets. The ADC stresses all levels of grid memory by producing 2d views of an Arithmetic Data Set of d-tuples described by a small number of parameters. We control data intensity of the ADC by controlling the sizes of the views through choice of the tuple parameters.

Frumkin, Michael A.↗

Extending CF Conventions to Enhance Data FAIRness for Atmospheric Composition Observations

The Hierarchical Data Format (HDF) and Network Common Data Form (NetCDF) are data file formats created to aid users in the creation or use of scientific data. These file formats are useful for handling large data volumes and hosting extensive metadata as global, group, or variable attributes and are popular with the modeling community. HDF and NetCDF files are widely used with atmospheric remote sensing data and have been used to support measurements from numerous field campaigns, from satellite to aircraft or ground and mobile based measurements. The files from airborne field studies, however, vary greatly in terms of the file structure and the amount and content of their metadata. Information relevant to the file that can be useful to the user such as the data producer, location where data was taken, variable descriptions, or information about the instrument might not be included in the file. Recently, the Measurements of Aerosols, Clouds, and their Interactions for Earth System Models (MACIE) group started a grassroots effort to develop a CF-based template for the HDF and NetCDF files for field studies, with the aim of making the data products more interoperable and usable. This template seeks to make the files more compliant to Climate and Forecast (CF) metadata conventions and to standardize the file structure and the global and variable attributes. The template would help to ensure that HDF and NetCDF files contain adequate metadata to better support their use for research, e.g., the modeling community, and to enhance the usability and interoperability of data for research communities at large. The draft template has been applied to recent field studies for various instruments and their merge files in support of the Atmosphere Observing System (AOS) project. The details of the revised template are to be presented, as well as examples of the implementation of these requirements for merge files and lidar observation data files and issues revealed during the implementation process.

Sean Leavor↗

Design Principles for Effective Knowledge Discovery from Big Data

Big data phenomenon refers to the practice of collection and processing of very large data sets and associated systems and algorithms used to analyze these massive datasets. Architectures for big data usually range across multiple machines and clusters, and they commonly consist of multiple special purpose sub-systems. Coupled with the knowledge discovery process, big data movement offers many unique opportunities for organizations to benefit (with respect to new insights, business optimizations, etc.). However, due to the difficulty of analyzing such large datasets, big data presents unique systems engineering and architectural challenges. In this paper, we present three sys- tem design principles that can inform organizations on effective analytic and data collection processes, system organization, and data dissemination practices. The principles presented derive from our own research and development experiences with big data problems from various federal agencies, and we illustrate each principle with our own experiences and recommendations.

Architecture↗

Encounter-Based Simulation Architecture for Detect-And-Avoid Modeling

This paper presents an encounter-based simulation architecture developed at NASA to facilitate flexible and efficient Detect and Avoid modeling in parametric or tradespace studies on large data sets. The basic premise of this tool is that large-scale input data can be reduced to a set of `canonical encounters' and that using the reduced data in simulations does not lead to loss of fidelity. A canonical encounter is specified as ownship and intruder flight portions potentially resulting in a loss of well clear along with a set of properties that characterize the encounter. The advantages of using canonical encounters include faster simulations, reduced memory footprint, ability to select encounters based on user-specified criteria, shared encounters across multiple teams, peer-reviewed encounters, and a better understanding of the input data set, to name a few.

MOPS↗

Encounter-Based Simulation Architecture for Detect and Avoid Modeling

This paper presents an encounter-based simulation architecture developed at NASA to facilitate flexible and efficient Detect and Avoid modeling in parametric or tradespace studies on large data sets. The basic premise of this tool is that large-scale input data can be reduced to a set of `canonical encounters' and that using the reduced data in simulations does not lead to loss of fidelity. A canonical encounter is specified as ownship and intruder flight portions potentially resulting in a loss of well clear along with a set of properties that characterize the encounter. The advantages of using canonical encounters include faster simulations, reduced memory footprint, ability to select encounters based on user-specified criteria, shared encounters across multiple teams, peer-reviewed encounters, and a better understanding of the input data set, to name a few.

MOPS↗

Nanoscale Cluster Detection in Massive Atom Probe Tomography Data

Recent technological advances in atom probe tomography (APT) have led to unprecedented data acquisition capabilities that routinely generate data sets containing hundreds of millions of atoms. Detecting nanoscale clusters of different atom types present in these enormous amounts of data and analyzing their spatial correlations with one another are fundamental to understanding the structural properties of the material from which the data is derived. Extant algorithms for nanoscale cluster detection do not scale to large data sets. Here, a scalable, CUDA-based implementation of an autocorrelation algorithm is presented. It isolates spatial correlations amongst atomic clusters present in massive APT data sets in linear time using a linear amount of storage. Correctness of the algorithm is demonstrated using large synthetically generated data with known spatial distributions. Benefits and limitations of using GPU-acceleration for autocorrelation-based APT data analyses are presented with supporting performance results on data sets with up to billions of atoms. To our knowledge, this is the first nanoscale cluster detection algorithm that scales to massive APT data sets and executes on commodity hardware.

Seal, Sudip K↗

Advance Network Reservation and Provisioning for Science

We are witnessing a new era that offers new opportunities to conduct scientific research with the help of recent advancements in computational and storage technologies. Computational intensive science spans multiple scientific domains, such as particle physics, climate modeling, and bio-informatics simulations. These large-scale applications necessitate collaborators to access very large data sets resulting from simulations performed in geographically distributed institutions. Furthermore, often scientific experimental facilities generate massive data sets that need to be transferred to validate the simulation data in remote collaborating sites. A major component needed to support these needs is the communication infrastructure which enables high performance visualization, large volume data analysis, and also provides access to computational resources. In order to provide high-speed on-demand data access between collaborating institutions, national governments support next generation research networks such as Internet 2 and ESnet (Energy Sciences Network). Delivering network-as-a-service that provides predictable performance, efficient resource utilization and better coordination between compute and storage resources is highly desirable. In this paper, we study network provisioning and advanced bandwidth reservation in ESnet for on-demand high performance data transfers. We present a novel approach for path finding in time-dependent transport networks with bandwidth guarantees. We plan to improve the current ESnet advance network reservation system, OSCARS [3], by presenting to the clients, the possible reservation options and alternatives for earliest completion time and shortest transfer duration. The Energy Sciences Network (ESnet) provides high bandwidth connections between research laboratories and academic institutions for data sharing and video/voice communication. The ESnet On-Demand Secure Circuits and Advance Reservation System (OSCARS) establishes guaranteed bandwidth of secure virtual circuits at a certain time, for a certain bandwidth and length of time. Though OSCARS operates within the ESnet, it also supplies end-to-end provisioning between multiple autonomous network domains. OSCARS gets reservation requests through a standard web service interface, and conducts a Quality-of-service (QoS) path for bandwidth guarantees. Multi-protocol Label Switching (MPLS) and the Resource Reservation Protocol (RSVP) enable to create a virtual circuit using Label Switched Paths (LSP's). It contains three main components: a reservation manager, a bandwidth scheduler, and a path setup subsystem. The bandwidth scheduler needs to have information about the current and future states of the network topology in order to accomplish end-to-end bandwidth guaranteed paths.

97 MATHEMATICS AND COMPUTING↗

Parallel variable selection for effective performance prediction

Large data analysis problems often involve a large number of variables, and the corresponding analysis algorithms may examine all variable combinations to find the optimal solution. For example, to model the time required to complete a scientific workflow, we need to consider the impact of dozens of parameters. To reduce the model building time and reduce the likelihood of overfitting, we look to variable selection methods to identify the critical variables for the performance model. In this work, we create a combination of variable selection and performance prediction methods that is as effective as the exhaustive search approach when the exhaustive search could be completed in a reasonable amount of time. To handle the cases where the exhaustive search is too time consuming, we develop the parallelized variable selection algorithm. Additionally, we develop a parallel grouping mechanism that further reduces the variable selection time by 70%.As a case study, we exercise the variable selection technique with the performance measurement data from the Palomar Transient Factory (PTF) workflow. The application scientists have determined that about 50 variables and parameters are important to the performance of the workflows. Our tests show that the Sequential Backward Selection algorithm is able to approximate the optimal subset relatively quickly. By reducing the number of variables used to build the model from 50 to 4, we are able to maintain the prediction quality while reducing the model building time by a factor of 6. Using the parallelization and grouping techniques we developed in this work, the variable selection process was reduced from over 18 hours to 15 minutes while ending up with the same variable subset.

Wang, J↗

Crack detection in fuel cell electrodes using a spatial filtering technique for overcoming noisy backgrounds

Image processing is a powerful tool that allows for rapid and automated data parsing in settings that occupy large variable spaces and require large data sets. Feature detection on difficultly discerned backgrounds is a subset of image processing that facilitates the extraction of quantitative metrics from otherwise subjective data. Crack detection and quantification is an important capability in polymer electrolyte membrane fuel cell quality control, failure analysis, and optimization. This work presents a technique to perform crack detection and quantification which overcomes challenges faced by commonly used image segmentation techniques. We demonstrate the use of a geometrically filtered noise‐level detection technique to select a binary threshold value from which we then quantify how cracked a sample is. Furthermore, we demonstrate the accuracy of our technique using programmatically generated test images of known crack amounts and their performance on real‐world fuel cell catalyst layer samples.

30 DIRECT ENERGY CONVERSION↗

Methods and Models for Analyzing Methane Sorption Capacity of Coal Based on Its Physicochemical Characteristics

The authors study the influence of physicochemical parameters on methane adsorption capacity of coal and offer the analytical method for the methane adsorption capacity for three-phased condition of methane. It is found that in the depth interval to 300 m below surface, methane adsorption capacity measured in lab can exceed natural gas content of coal obtained from geological exploration data by 30%, and the change in the thermodynamic condition of coal–methane system brings irreversible physicochemical consequences in terms of the altered ratios of physical states of the main components. There is no linear connection between natural gas content of a coal bed and its methane adsorption capacity with respect to occurrence depth. The application of Big Data in treatment and interpretation of large data flows is described. The theoretical data predicted using the proposed method and the experimental data on methane content of Kuzbass coal agree.

01 COAL, LIGNITE, AND PEAT↗

Vehicle Powertrain Simulation Accuracy for Various Drive Cycle Frequencies and Upsampling Techniques

As connected and automated vehicle technologies emerge and proliferate, lower frequency vehicle trajectory data is becoming more widely available. In some cases, entire fleets are streaming position, speed, and telemetry at sample rates of less than 10 seconds. This presents opportunities to apply powertrain simulators such as the National Renewable Energy Laboratory's Future Automotive Systems Technology Simulator to model how advanced powertrain technologies would perform in the real world. However, connected vehicle data tends to be available at lower temporal frequencies than the 1-10 Hz trajectories that have typically been used for powertrain simulation. Higher frequency data, typically used for simulation, is costly to collect and store and therefore is often limited in density and geography. This paper explores the suitability of lower frequency, high availability, connected vehicle data for detailed powertrain simulation. A large data set of 1 Hz trajectories is used to quantify the accuracy loss when simulating energy consumption for conventional, hybrid, and battery electric powertrains using less than 1 Hz data. Techniques to upsample lower frequency drive cycle data in order to increase accuracy are also explored. Median energy consumption errors when simulating energy consumption for a 1/10 Hz trajectory are found to be 3-6% when compared to 1 Hz trajectories. Applying upsampling and interpolation techniques are shown to reduce the simulation errors by roughly 50%. The findings in this work can guide connected vehicle data collection specifications and processing techniques applied when using collected data for powertrain simulation.

ADVANCED PROPULSION SYSTEMS↗

A Remotely Operated Multiple Array Acoustic Range (ROMAAR) and its application for the measurement of airplane flyover noise footprints

The ROMAAR now in operation at NASA will allow direct measurement and display of aircraft noise in several measurement units during takeoff, landing, and flyby operations. This information, in addition to its application in terms of ground noise footprints, will also permit determination of the statistical variation of footprints or contours due to the atmosphere or aircraft operational parameters, and a measure of the impact of various noise reduction techniques and hardware on ground noise footprints. The methods, techniques, and equipment developed for the ROMAAR concept are applicable to CTOL, STOL, General Aviation, and VTOL aircraft. ROMAAR represents a unique combination of state of the art digital and analog noise recording methods, computer-controlled digital communications methods, radar-tracking facilities, quick-look weather capabilities, and a large data handling facility complemented by a large capacity curve fitting and plotting routine. The ROMAAR is set apart from the standard airport noise monitoring system by having the unique features mentioned above plus the fact that at present as many as 38 separate (but simultaneous) noise measurements can be made for each aircraft overflight.

Hilton, D. A.↗

A wind tunnel database using RIM

Engineering data base development which has become increasingly widespread to industry with the availability of data management systems is examined. A large data base was developed for wind tunnel data and related model test information, using RIM as the data base manager. The arrangement of the wind tunnel data into the proper schema for the most efficient database utilization is discussed. The FORTRAN interface program of RIM is used extensively in the loading phases of the data base and by the users. Several examples to illustrate how the Wind Tunnel Data base might be searched for specific data items and test information using RIM are presented.

Wray, W. O., Jr.↗

Access and visualization using clusters and other parallel computers

JPL's Parallel Applications Technologies Group has been exploring the issues of data access and visualization of very large data sets over the past 10 years. this work has used a number of types of parallel computers, and today includes the use of commodity clusters. This talk will highlight some of the applications and tools we have developed, including how they use parallel computing resources, and specifically how we are using modern clusters.

parallel computers clusters↗

Access and visualization using clusters and other parallel computers

JPL's Parallel Applications Technologies Group has been exploring the issues of data access and visualization of very large data sets over the past 10 or so years. this work has used a number of types of parallel computers, and today includes the use of commodity clusters. This talk will highlight some of the applications and tools we have developed, including how they use parallel computing resources, and specifically how we are using modern clusters. Our applications focus on NASA's needs; thus our data sets are usually related to Earth and Space Science, including data delivered from instruments in space, and data produced by telescopes on the ground.

data sets↗

Application of Machine Learning to Rotorcraft Health Monitoring

Machine learning is a powerful tool for data exploration and model building with large data sets. This project aimed to use machine learning techniques to explore the inherent structure of data from rotorcraft gear tests, relationships between features and damage states, and to build a system for predicting gear health for future rotorcraft transmission applications. Classical machine learning techniques are difficult, if not irresponsible to apply to time series data because many make the assumption of independence between samples. To overcome this, Hidden Markov Models were used to create a binary classifier for identifying scuffing transitions and Recurrent Neural Networks were used to leverage long distance relationships in predicting discrete damage states. When combined in a workflow, where the binary classifier acted as a filter for the fatigue monitor, the system was able to demonstrate accuracy in damage state prediction and scuffing identification. The time dependent nature of the data restricted data exploration to collecting and analyzing data from the model selection process. The limited amount of available data was unable to give useful information, and the division of training and testing sets tended to heavily influence the scores of the models across combinations of features and hyper-parameters. This work built a framework for tracking scuffing and fatigue on streaming data and demonstrates that machine learning has much to offer rotorcraft health monitoring by using Bayesian learning and deep learning methods to capture the time dependent nature of the data. Suggested future work is to implement the framework developed in this project using a larger variety of data sets to test the generalization capabilities of the models and allow for data exploration.

machine learning↗

NASA GES DISC Giovanni: Current and Future

Giovanni (Geospatial Interactive Online Visualization and Analysis Infrastructure), developed by the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), has established a reputation among NASA users for easy access, analysis, and visualization of NASA Earth science data. Currently, Giovanni supports over 1900 variables in eight disciplinary areas. Like any other enterprise application, Giovanni faces big data challenges, such as servicing increasingly large data volumes and more complex data types, while at the same time addressing the demands of a more diverse user community, e.g., placing requests for long-term time series from multiple spatially and temporally dense data records. I will present how Giovanni has been evolving from an on-premises, monolithic software application towards a cloud-enabled implementation to address these challenges.

Analytics↗

On the impact of large angle CMB polarization data on cosmological parameters

We study the impact of the large-angle CMB polarization datasets publicly released by the WMAP and Planck satellites on the estimation of cosmological parameters of the ΛCDM model. To complement large-angle polarization, we consider the high resolution (or "high-ℓ") CMB datasets from either WMAP or Planck as well as CMB lensing as traced by Planck's measured four point correlation function. In the case of WMAP, we compute the large-angle polarization likelihood starting over from low resolution frequency maps and their covariance matrices, and perform our own foreground mitigation technique, which includes as a possible alternative Planck 353 GHz data to trace polarized dust. We find that the latter choice induces a downward shift in the optical depth τ, roughly of order 2σ, robust to the choice of the complementary high resolution dataset. When the Planck 353 GHz is consistently used to minimize polarized dust emission, WMAP and Planck 70 GHz large-angle polarization data are in remarkable agreement: by combining them we find τ = 0.066 +0.012-0.013, again very stable against the particular choice for high-ℓ data. We find that the amplitude of primordial fluctuations As, notoriously degenerate with τ, is the parameter second most affected by the assumptions on polarized dust removal, but the other parameters are also affected, typically between 0.5 and 1σ. In particular, cleaning dust with Planck's 353 GHz data imposes a 1σ downward shift in the value of the Hubble constant H0, significantly contributing to the tension reported between CMB based and direct measurements of the present expansion rate. On the other hand, we find that the appearance of the so-called low ℓ anomaly, a well-known tension between the high- and low-resolution CMB anisotropy amplitude, is not significantly affected by the details of large-angle polarization, or by the particular high-ℓ dataset employed.

79 ASTRONOMY AND ASTROPHYSICS↗