SEARCH · Search NASA

Results for “Large data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

All sources NASA NTRS DOE OSTI

NASA NTRS1998

A Review of the New AVIRIS Data Processing System

The processing of AVIRIS data - from the Metrum Very Large Data Store (VLDS) flight tape to delivered data products - has traditionally been performed in essentially the same way, from the beginning of the AVIRIS project up to and including the 1996 flight season. Starting with the 1997 flight season, a drastically different paradigm has been used for the processing of AVIRIS data. This change was made possible by the recent development of and related availability of affordable data storage devices.

Aronsson, Mikael↗

NASA NTRS2003

Proposed Space Flight Experiment Hardware

The primary thrust for this plan is to develop design tools and fundamental understanding that are timely and consistent with the goal of the various exploration initiatives. The plan will utilize ISS facilities, such as the Fluids Integrated Rack (FIR) and the Microgravity Science Glovebox (MSG). A preliminary flow schematic of Two-Phase Flow Facility (T(phi)FFy) which would utilize FIR is shown in Figure 3. MSG can be utilized to use the Boiling eXperiment Facility (BXF) and Contact Line Dynamics Experiment (CLiDE) Facility. The T(phi)FFy system would have multiple test sections whereby different configurations of heat exchangers could be used to study boiling and condensation phenomena. The test sections would be instrumented for pressure drop, void fraction, heat fluxes, temperatures, high-speed imaging and other diagnostics. Besides a high-speed data acquisition system with a large data storage capability, telemetry could be used to update control and test parameters and download limited amounts of data. In addition, there would be multiple accumulators that could be used to investigate system stability and fluid management issues. The system could accommodate adiabatic tests through either the space station nitrogen supply or have an experiment-specific compressor to pressurize a sufficient amount of air or other non-condensable gas for reuse as the supply bottle is depleted.

Source record↗

NASA NTRS2004

Data Management as a Cluster Middleware Centerpiece

Through earth and space modeling and the ongoing launches of satellites to gather data, NASA has become one of the largest producers of data in the world. These large data sets necessitated the creation of a Data Management System (DMS) to assist both the users and the administrators of the data. Halcyon Systems Inc. was contracted by the NASA Center for Computational Sciences (NCCS) to produce a Data Management System. The prototype of the DMS was produced by Halcyon Systems Inc. (Halcyon) for the Global Modeling and Assimilation Office (GMAO). The system, which was implemented and deployed within a relatively short period of time, has proven to be highly reliable and deployable. Following the prototype deployment, Halcyon was contacted by the NCCS to produce a production DMS version for their user community. The system is composed of several existing open source or government-sponsored components such as the San Diego Supercomputer Center s (SDSC) Storage Resource Broker (SRB), the Distributed Oceanographic Data System (DODS), and other components. Since Data Management is one of the foremost problems in cluster computing, the final package not only extends its capabilities as a Data Management System, but also to a cluster management system. This Cluster/Data Management System (CDMS) can be envisioned as the integration of existing packages.

Zero, Jose↗

NASA NTRS2013

Downscaling Satellite Precipitation with Emphasis on Extremes: A Variational 1-Norm Regularization in the Derivative Domain

The increasing availability of precipitation observations from space, e.g., from the Tropical Rainfall Measuring Mission (TRMM) and the forthcoming Global Precipitation Measuring (GPM) Mission, has fueled renewed interest in developing frameworks for downscaling and multi-sensor data fusion that can handle large data sets in computationally efficient ways while optimally reproducing desired properties of the underlying rainfall fields. Of special interest is the reproduction of extreme precipitation intensities and gradients, as these are directly relevant to hazard prediction. In this paper, we present a new formalism for downscaling satellite precipitation observations, which explicitly allows for the preservation of some key geometrical and statistical properties of spatial precipitation. These include sharp intensity gradients (due to high-intensity regions embedded within lower-intensity areas), coherent spatial structures (due to regions of slowly varying rainfall),and thicker-than-Gaussian tails of precipitation gradients and intensities. Specifically, we pose the downscaling problem as a discrete inverse problem and solve it via a regularized variational approach (variational downscaling) where the regularization term is selected to impose the desired smoothness in the solution while allowing for some steep gradients(called 1-norm or total variation regularization). We demonstrate the duality between this geometrically inspired solution and its Bayesian statistical interpretation, which is equivalent to assuming a Laplace prior distribution for the precipitation intensities in the derivative (wavelet) space. When the observation operator is not known, we discuss the effect of its misspecification and explore a previously proposed dictionary-based sparse inverse downscaling methodology to indirectly learn the observation operator from a database of coincidental high- and low-resolution observations. The proposed method and ideas are illustrated in case studies featuring the downscaling of a hurricane precipitation field.

Hurricanes↗

NASA NTRS2003

Towards Automating Spacecraft Attitude Sensor Calibration

With a view towards reducing cost and complexity for spacecraft early mission support at the NASA Goddard Space Flight Center (GSFC), efforts are being made to automate the attitude sensor calibration process. This paper addresses one of the major components needed by such a system. The beneficiaries of an improved calibration process are missions that demand moderate to high precision attitude knowledge or that need to perform accurate attitude slews. Improved slew accuracy reduces the time needed for re-acquisition of fine-pointing after each attitude maneuver, Rapid target acquisition can be very important for astronomical targeting or for off-nadir surface feature targeting by Earth-oriented spacecraft. The normal sequence of on-orbit calibration starts with alignment calibration of the star trackers and possibly the Sun sensor. Their relative alignment needs to be determined using a sufficiently large data set so their fields of view are adequately sampled. Next, the inertial reference unit (IRU) is calibrated for corrections to its alignment and scale factors. The IRU biases are estimated continuously by the onboard attitude control system, but the IRU alignment and scale factors are usually determined on the ground using a batch-processing method on a data set that includes several slews sufficient to give full observability of all the IRU calibration parameters. Finally, magnetometer biases, alignment, and its coupling to the magnetic torquers are determined in order io improve momentum management and occasionally for use in the attitude determination system. The detailed approach used for automating calibrations will depend on whether the automated system resides on the ground or on the spacecraft with an ultimate goal of autonomous calibration. Current efforts focus on a ground-based system driving subsystems that could run either on the ground or onboard. The distinction is that onboard calibration should process the data sequentially rather than in a single large batch since onboard computer data storage is limited. Very good batch- processing calibration utilities have been developed and used extensively at NASA/GSFC for mission support but no sequential calibration utilities are available. To meet this need, this paper presents the mathematical description of a sequential IRU calibration system. The system has been tested using flight data from the Rossi X-ray Timing Explorer (RXTE) during a series of attitude slews. The paper also discusses the current state of the overall automated system and describes plans for adding sequential alignment calibration and other additions that will reduce the amount of analyst time and input.

Sedlak, Joseph↗

NASA NTRS2015

A Data Exploration Tool for Large Sets of Spectra

We present an exploration tool for very large spectrum data sets such as the SDSS (Sloan Digital Sky Survey), LAMOST (Large Sky Area Multi-Object Fiber Spectroscopic Telescope), and 4MOST (4-meter Multi-Object Spectroscopic Telescope) data sets. The tool works in two stages: the first uses batch processing and the second runs interactively. The latter employs the NASA hyperwall, a configuration of 128 workstation displays (8 by 16 array) controlled by a parallelized software suite running on NASA's Pleiades supercomputer. The stellar subset of the Sloan Digital Sky Survey, DR10, was chosen to show how the our tool may be used. In stage one, SDSS files for 569,740 stars are processed through our data pipeline. The pipeline fits each spectrum using an iterative continuum algorithm, distinguishing emission from absorption and handling molecular absorption bands correctly. It then measures 1659 discrete atomic and molecular spectral features that were carefully preselected based on their likelihood of being visible at some spectral type. The depths relative to the local continuum at each feature wavelength are determined for each spectrum: these depths, the local S/N (signal to noise ratio) level, and DR10-supplied variables such as magnitudes, colors, positions, and radial velocities are the basic measured quantities used on the hyperwall. In stage two, each hyperwall panel is used to display a 2-D scatter plot showing the depth of feature A vs the depth of feature B for all of the stars. A and B change from panel to panel. The relationships between the various (A,B) strengths and any distinctive clustering are immediately apparent when examining and inter-comparing the different panels on the hyperwall. The interactive software allows the user to select the stars in any interesting region of any 2-D plot on the hyperwall, immediately rendering the same stars on all the other 2-D plots in a unique color. The process may be repeated multiple times, each selection displaying a distinctive color on all the plots. At any time, the spectra of the selected stars may be examined in detail on a connected workstation display. We illustrate how our approach allows us to quickly isolate and examine such interesting stellar subsets as EMP (Extremely Metal‐Poor) stars, CV (Cataclymic Variable) stars and C (Carbon)-rich stars.

Data Exploration↗

NASA NTRS2004

Benchmarking Memory Performance with the Data Cube Operator

Data movement across a computer memory hierarchy and across computational grids is known to be a limiting factor for applications processing large data sets. We use the Data Cube Operator on an Arithmetic Data Set, called ADC, to benchmark capabilities of computers and of computational grids to handle large distributed data sets. We present a prototype implementation of a parallel algorithm for computation of the operatol: The algorithm follows a known approach for computing views from the smallest parent. The ADC stresses all levels of grid memory and storage by producing some of 2d views of an Arithmetic Data Set of d-tuples described by a small number of integers. We control data intensity of the ADC by selecting the tuple parameters, the sizes of the views, and the number of realized views. Benchmarking results of memory performance of a number of computer architectures and of a small computational grid are presented.

Frumkin, Michael A.↗

NASA NTRS2001

Field Model: An Object-Oriented Data Model for Fields

We present an extensible, object-oriented data model designed for field data entitled Field Model (FM). FM objects can represent a wide variety of fields, including fields of arbitrary dimension and node type. FM can also handle time-series data. FM achieves generality through carefully selected topological primitives and through an implementation that leverages the potential of templated C++. FM supports fields where the nodes values are paired with any cell type. Thus FM can represent data where the field nodes are paired with the vertices ("vertex-centered" data), fields where the nodes are paired with the D-dimensional cells in R(sup D) (often called "cell-centered" data), as well as fields where nodes are paired with edges or other cell types. FM is designed to effectively handle very large data sets; in particular FM employs a demand-driven evaluation strategy that works especially well with large field data. Finally, the interfaces developed for FM have the potential to effectively abstract field data based on adaptive meshes. We present initial results with a triangular adaptive grid in R(sup 2) and discuss how the same design abstractions would work equally well with other adaptive-grid variations, including meshes in R(sup 3).

Moran, Patrick J.↗

NASA NTRS2003

Arithmetic Data Cube as a Data Intensive Benchmark

Data movement across computational grids and across memory hierarchy of individual grid machines is known to be a limiting factor for application involving large data sets. In this paper we introduce the Data Cube Operator on an Arithmetic Data Set which we call Arithmetic Data Cube (ADC). We propose to use the ADC to benchmark grid capabilities to handle large distributed data sets. The ADC stresses all levels of grid memory by producing 2d views of an Arithmetic Data Set of d-tuples described by a small number of parameters. We control data intensity of the ADC by controlling the sizes of the views through choice of the tuple parameters.

Frumkin, Michael A.↗

NASA NTRSDate not supplied

Extending CF Conventions to Enhance Data FAIRness for Atmospheric Composition Observations

The Hierarchical Data Format (HDF) and Network Common Data Form (NetCDF) are data file formats created to aid users in the creation or use of scientific data. These file formats are useful for handling large data volumes and hosting extensive metadata as global, group, or variable attributes and are popular with the modeling community. HDF and NetCDF files are widely used with atmospheric remote sensing data and have been used to support measurements from numerous field campaigns, from satellite to aircraft or ground and mobile based measurements. The files from airborne field studies, however, vary greatly in terms of the file structure and the amount and content of their metadata. Information relevant to the file that can be useful to the user such as the data producer, location where data was taken, variable descriptions, or information about the instrument might not be included in the file. Recently, the Measurements of Aerosols, Clouds, and their Interactions for Earth System Models (MACIE) group started a grassroots effort to develop a CF-based template for the HDF and NetCDF files for field studies, with the aim of making the data products more interoperable and usable. This template seeks to make the files more compliant to Climate and Forecast (CF) metadata conventions and to standardize the file structure and the global and variable attributes. The template would help to ensure that HDF and NetCDF files contain adequate metadata to better support their use for research, e.g., the modeling community, and to enhance the usability and interoperability of data for research communities at large. The draft template has been applied to recent field studies for various instruments and their merge files in support of the Atmosphere Observing System (AOS) project. The details of the revised template are to be presented, as well as examples of the implementation of these requirements for merge files and lidar observation data files and issues revealed during the implementation process.

Sean Leavor↗

NASA NTRS2019

Encounter-Based Simulation Architecture for Detect-And-Avoid Modeling

This paper presents an encounter-based simulation architecture developed at NASA to facilitate flexible and efficient Detect and Avoid modeling in parametric or tradespace studies on large data sets. The basic premise of this tool is that large-scale input data can be reduced to a set of `canonical encounters' and that using the reduced data in simulations does not lead to loss of fidelity. A canonical encounter is specified as ownship and intruder flight portions potentially resulting in a loss of well clear along with a set of properties that characterize the encounter. The advantages of using canonical encounters include faster simulations, reduced memory footprint, ability to select encounters based on user-specified criteria, shared encounters across multiple teams, peer-reviewed encounters, and a better understanding of the input data set, to name a few.

MOPS↗

NASA NTRS2019

Encounter-Based Simulation Architecture for Detect and Avoid Modeling

MOPS↗

NASA NTRS1976

A Remotely Operated Multiple Array Acoustic Range (ROMAAR) and its application for the measurement of airplane flyover noise footprints

The ROMAAR now in operation at NASA will allow direct measurement and display of aircraft noise in several measurement units during takeoff, landing, and flyby operations. This information, in addition to its application in terms of ground noise footprints, will also permit determination of the statistical variation of footprints or contours due to the atmosphere or aircraft operational parameters, and a measure of the impact of various noise reduction techniques and hardware on ground noise footprints. The methods, techniques, and equipment developed for the ROMAAR concept are applicable to CTOL, STOL, General Aviation, and VTOL aircraft. ROMAAR represents a unique combination of state of the art digital and analog noise recording methods, computer-controlled digital communications methods, radar-tracking facilities, quick-look weather capabilities, and a large data handling facility complemented by a large capacity curve fitting and plotting routine. The ROMAAR is set apart from the standard airport noise monitoring system by having the unique features mentioned above plus the fact that at present as many as 38 separate (but simultaneous) noise measurements can be made for each aircraft overflight.

Hilton, D. A.↗

NASA NTRS1984

A wind tunnel database using RIM

Engineering data base development which has become increasingly widespread to industry with the availability of data management systems is examined. A large data base was developed for wind tunnel data and related model test information, using RIM as the data base manager. The arrangement of the wind tunnel data into the proper schema for the most efficient database utilization is discussed. The FORTRAN interface program of RIM is used extensively in the loading phases of the data base and by the users. Several examples to illustrate how the Wind Tunnel Data base might be searched for specific data items and test information using RIM are presented.

Wray, W. O., Jr.↗

NASA NTRS2003

Access and visualization using clusters and other parallel computers

JPL's Parallel Applications Technologies Group has been exploring the issues of data access and visualization of very large data sets over the past 10 years. this work has used a number of types of parallel computers, and today includes the use of commodity clusters. This talk will highlight some of the applications and tools we have developed, including how they use parallel computing resources, and specifically how we are using modern clusters.

parallel computers clusters↗

NASA NTRS2003

Access and visualization using clusters and other parallel computers

JPL's Parallel Applications Technologies Group has been exploring the issues of data access and visualization of very large data sets over the past 10 or so years. this work has used a number of types of parallel computers, and today includes the use of commodity clusters. This talk will highlight some of the applications and tools we have developed, including how they use parallel computing resources, and specifically how we are using modern clusters. Our applications focus on NASA's needs; thus our data sets are usually related to Earth and Space Science, including data delivered from instruments in space, and data produced by telescopes on the ground.

data sets↗

NASA NTRS2017

Application of Machine Learning to Rotorcraft Health Monitoring

Machine learning is a powerful tool for data exploration and model building with large data sets. This project aimed to use machine learning techniques to explore the inherent structure of data from rotorcraft gear tests, relationships between features and damage states, and to build a system for predicting gear health for future rotorcraft transmission applications. Classical machine learning techniques are difficult, if not irresponsible to apply to time series data because many make the assumption of independence between samples. To overcome this, Hidden Markov Models were used to create a binary classifier for identifying scuffing transitions and Recurrent Neural Networks were used to leverage long distance relationships in predicting discrete damage states. When combined in a workflow, where the binary classifier acted as a filter for the fatigue monitor, the system was able to demonstrate accuracy in damage state prediction and scuffing identification. The time dependent nature of the data restricted data exploration to collecting and analyzing data from the model selection process. The limited amount of available data was unable to give useful information, and the division of training and testing sets tended to heavily influence the scores of the models across combinations of features and hyper-parameters. This work built a framework for tracking scuffing and fatigue on streaming data and demonstrates that machine learning has much to offer rotorcraft health monitoring by using Bayesian learning and deep learning methods to capture the time dependent nature of the data. Suggested future work is to implement the framework developed in this project using a larger variety of data sets to test the generalization capabilities of the models and allow for data exploration.

machine learning↗

NASA NTRS2019

NASA GES DISC Giovanni: Current and Future

Giovanni (Geospatial Interactive Online Visualization and Analysis Infrastructure), developed by the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), has established a reputation among NASA users for easy access, analysis, and visualization of NASA Earth science data. Currently, Giovanni supports over 1900 variables in eight disciplinary areas. Like any other enterprise application, Giovanni faces big data challenges, such as servicing increasingly large data volumes and more complex data types, while at the same time addressing the demands of a more diverse user community, e.g., placing requests for long-term time series from multiple spatially and temporally dense data records. I will present how Giovanni has been evolving from an on-premises, monolithic software application towards a cloud-enabled implementation to address these challenges.

Analytics↗