Designing for Situation Awareness and Operator Control with Large Data Volumes in NASA's Deep Space Network
No abstract available
Holloway, Alexandra↗
SEARCH · Search NASA
Results for “Large data”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Data processing 2: Advancements in large scale data processing systems for remote sensing
The development of large scale data processing systems for remote sensing is studied by evaluating: (1) the suitability of several sensor types with regard to producing data required for multispectral machine analysis; (2) various types of data preprocessing necessary to prepare such data for analysis; and (3) transfer of machine processing techniques for earth resources data to user community.
Landgrebe, D. A.↗
Eigenanalysis of Space Weather Climate Data: Discovering Structure within Large Multivariate Data Sets
No abstract available
Krause, Linda Habash↗
Validating a large geophysical data set: Experiences with satellite-derived cloud parameters
We are validating the global cloud parameters derived from the satellite-borne HIRS2 and MSU atmospheric sounding instrument measurements, and are using the analysis of these data as one prototype for studying large geophysical data sets in general. The HIRS2/MSU data set contains a total of 40 physical parameters, filling 25 MB/day; raw HIRS2/MSU data are available for a period exceeding 10 years. Validation involves developing a quantitative sense for the physical meaning of the derived parameters over the range of environmental conditions sampled. This is accomplished by comparing the spatial and temporal distributions of the derived quantities with similar measurements made using other techniques, and with model results. The data handling needed for this work is possible only with the help of a suite of interactive graphical and numerical analysis tools. Level 3 (gridded) data is the common form in which large data sets of this type are distributed for scientific analysis. We find that Level 3 data is inadequate for the data comparisons required for validation. Level 2 data (individual measurements in geophysical units) is needed. A sampling problem arises when individual measurements, which are not uniformly distributed in space or time, are used for the comparisons. Standard 'interpolation' methods involve fitting the measurements for each data set to surfaces, which are then compared. We are experimenting with formal criteria for selecting geographical regions, based upon the spatial frequency and variability of measurements, that allow us to quantify the uncertainty due to sampling. As part of this project, we are also dealing with ways to keep track of constraints placed on the output by assumptions made in the computer code. The need to work with Level 2 data introduces a number of other data handling issues, such as accessing data files across machine types, meeting large data storage requirements, accessing other validated data sets, processing speed and throughput for interactive graphical work, and problems relating to graphical interfaces.
Kahn, Ralph↗
Round-Robin approach to data flow optimization
A large data base, circular in structure, was required (for the Voyager Mission to Jupiter/Saturn) with the capability to completely update the data every four hours during high activity periods. The data is stored in key ordered format for retrieval but is not input in key order. Existing access methods for large data bases with rapid data replacement by keys become inefficient as the volume of data being replaced grows. The Round-Robin method was developed to alleviate this problem. The Round-Robin access method allows rapid updating of the data with continuous self cleaning where the oldest data (by key) is deleted and the newest data (by key) is kept regardless of the order of input.
Witt, J.↗
Information Theoretic Approaches to Rapid Discovery of Relationships in Large Climate Data Sets
Mutual information as the asymptotic Bayesian measure of independence is an excellent starting point for investigating the existence of possible relationships among climate-relevant variables in large data sets, As mutual information is a nonlinear function of of its arguments, it is not beholden to the assumption of a linear relationship between the variables in question and can reveal features missed in linear correlation analyses. However, as mutual information is symmetric in its arguments, it only has the ability to reveal the probability that two variables are related. it provides no information as to how they are related; specifically, causal interactions or a relation based on a common cause cannot be detected. For this reason we also investigate the utility of a related quantity called the transfer entropy. The transfer entropy can be written as a difference between mutual informations and has the capability to reveal whether and how the variables are causally related. The application of these information theoretic measures is rested on some familiar examples using data from the International Satellite Cloud Climatology Project (ISCCP) to identify relation between global cloud cover and other variables, including equatorial pacific sea surface temperature (SST), over seasonal and El Nino Southern Oscillation (ENSO) cycles.
Knuth, Kevin H.↗
Stewardship of very large digital data archives
An archive is a permanent store. There are relatively few very large digital data archives in existence. Most business records are expired within five or ten years. Many kinds of business records that do have long lives are embedded in data bases that are continually updated and re-issued cyclically. Also, a great deal of permanent business records are actually archived as microfilm, fiche, or optical disk images - their digital version being an operational convenience rather than an archive. The problems forseen in stewarding the very large digital data archives that will accumulate during the mission of the Earth Observing System (EOS) are addressed. It focuses on the function of shepherding archived digital data into an endless future. Stewardship entails storing and protecting the archive and providing meaningful service to the community of users. The steward will (1) provide against loss due to physical phenomena; (2) assure that data is not lost due to storage technology obsolescence; and (3) maintain data in a current formatting methodology.
Savage, Patric↗
Validating a large geophysical data set - Experiences with satellite-derived cloud parameters
The goal of this study is to validate the global cloud parameters derived from the satellite-borne HIRS2 and MSU atmospheric sounding instrument measurements, and to use the analysis of these data as one prototype for studying large geophysical data sets in general. The HIRS2/MSU data set contains a total of 40 physical parameters, filling 25 MB/day; raw HIRS2/MSU data are available for a period exceeding 10 years. Validation involves developing a quantitative sense for the physical meaning of the derived parameters over the range of environmental conditions sampled. This is accomplished by comparing the spatial and temporal distributions of the derived quantities with similar measurements made using other techniques, and with model results. The need to work with Level 2 (point) data, rather than Level 3 (gridded) data for validation purposes is discussed, and some techniques developed for charting the assumptions made in deriving an algorithm and generating a code to produce geophysical quantities from measured radiances are presented.
Kahn, Ralph↗
Visualization tools for the processing of airglow data from RAIDS
In anticipation of large data sets associated with a number of atmospheric imaging instruments being prepared for long term global coverage, NRL is developing graphical interfaces for all aspects of the program. For the first of these projects, RAIDS (the Remote Atmospheric and Ionospheric Detection System), a graphical approach to data handling, visualization, and analysis is envisioned and will set the stage for the satellites that follow. An overall system of hardware and a set of software 'tools,' that will allow for both the routine handling of all data and the analysis of large data sets assembled by scientists and instrument engineers, are currently being developed. The software for standard processing and visualization of instrument data is independent of computer platform and will allow for easy adaptation from one experiment to another. The processing will produce data sets that have similar characteristics, allowing for easy comparison of data obtained under similar circumstances. The visualization of both the engineering and scientific data is an important part of the system. By creating graphical environments for engineering evaluations and for scientific analysis data sets can be viewed and analyzed rapidly. This rapid analysis of data will contribute towards a greater portion of the RAIDS data being utilized.
Miller, Gordon J.↗
Program Collects And Analyzes Thermoelectric Data
Large gradient seebeck coefficient measurement computer program gathers many channels of data, performs analysis, and plots each set of data simutaneously. User can modify code to alter number of channels of data collected and sampling rate and equations that process data. Written in HP Basic. Program also adaptable to other types of data.
Chmielewski, Art B.↗
An inventory of four-dimensional data sets for the earth sciences
The wide variety of data available to the diligent researcher and the myriad paths to obtaining it are emphasized. This inventory is an attempt to make accessible much of the four-dimensional data available in the world. In this quick-look catalog are listed separate data sets (Data Sets sections), sources of data including centers and large data bases (Sources section), and some data expected to be available in the future (Future Data section). In the Data Sets section, individual data sets are arranged alphabetically by institution, with those archived in the U.S.A. listed first, followed by those found elsewhere. The Sources section includes large data bases, centers, and directories. Sources are arranged alphabetically by country. This section is followed by a Future Data section which is a collection of data sets, experiments, and other future developments of which we are cognizant. A collection of further information and order blanks provided by some of the archiving institutions are presented in the appendix.
Gregory, Terri↗
Comparison of Two Methods for Estimating the Sampling-Related Uncertainty of Satellite Rainfall Averages Based on a Large Radar Data Set
The uncertainty of rainfall estimated from averages of discrete samples collected by a satellite is assessed using a multi-year radar data set covering a large portion of the United States. The sampling-related uncertainty of rainfall estimates is evaluated for all combinations of 100 km, 200 km, and 500 km space domains, 1 day, 5 day, and 30 day rainfall accumulations, and regular sampling time intervals of 1 h, 3 h, 6 h, 8 h, and 12 h. These extensive analyses are combined to characterize the sampling uncertainty as a function of space and time domain, sampling frequency, and rainfall characteristics by means of a simple scaling law. Moreover, it is shown that both parametric and non-parametric statistical techniques of estimating the sampling uncertainty produce comparable results. Sampling uncertainty estimates, however, do depend on the choice of technique for obtaining them. They can also vary considerably from case to case, reflecting the great variability of natural rainfall, and should therefore be expressed in probabilistic terms. Rainfall calibration errors are shown to affect comparison of results obtained by studies based on data from different climate regions and/or observation platforms.
Lau, William K. M.↗
Science and Applications Space Platform (SASP) End-to-End Data System Study
The capability of present technology and the Tracking and Data Relay Satellite System (TDRSS) to accommodate Science and Applications Space Platforms (SASP) payload user's requirements, maximum service to the user through optimization of the SASP Onboard Command and Data Management System, and the ability and availability of new technology to accommodate the evolution of SASP payloads were assessed. Key technology items identified to accommodate payloads on a SASP were onboard storage devices, multiplexers, and onboard data processors. The primary driver is the limited access to TDRSS for single access channels due to sharing with all the low Earth orbit spacecraft plus shuttle. Advantages of onboard data processing include long term storage of processed data until TRDSS is accessible, thus reducing the loss of data, eliminating large data processing tasks at the ground stations, and providing a more timely access to the data.
Crawford, P. R.↗
Volumetric visualization of 3D data
In recent years, there has been a rapid growth in the ability to obtain detailed data on large complex structures in three dimensions. This development occurred first in the medical field, with CAT (computer aided tomography) scans and now magnetic resonance imaging, and in seismological exploration. With the advances in supercomputing and computational fluid dynamics, and in experimental techniques in fluid dynamics, there is now the ability to produce similar large data fields representing 3D structures and phenomena in these disciplines. These developments have produced a situation in which currently there is access to data which is too complex to be understood using the tools available for data reduction and presentation. Researchers in these areas are becoming limited by their ability to visualize and comprehend the 3D systems they are measuring and simulating.
Russell, Gregory↗
Stewardship of very large digital data archives
This paper addresses the problems foreseen by the author in stewarding the very large digital data archives that will accumulate during the mission of the Earth Orbiting Satellite (EOS). It focuses on the function of 'shepherding' archived digital data into an endless future. Stewardship entails a great deal more than storing and protecting the archive. It also includes all aspects of providing meaningful service to the community of users (scientists) who will want to access the data. The complete steward will be required to do the following: (1) provide against loss due to physical phenomena; (2) assure that data is not 'lost' due to storage technology obsolescence; (3) maintain data in a current formatting methodology with the additional requirement of being able to reconstitute the data to its original, as-received format; (4) secure against loss or pollution of data due to accidental, misguided, or willful software intrusion; (5) prevent unauthorized electronic access to the data, including unauthorized placement of data into the archive; (6) index the data in a metadatabase so that all anticipatable queries can be served without searching through the data itself; (7) provide responsive access to the metadatabase; (8) provide appropriately responsive access to the data; (9) incorporate additions and changes to the archive (and to the metadatabase) in a timely way; and (10) deliver only copies of data to clients - retain physical custody of the 'official' data. Items 1 through 4 are discussed in this paper.
Savage, Patric↗
Efficient Implementation of an Optimal Interpolator for Large Spatial Data Sets
Scattered data interpolation is a problem of interest in numerous areas such as electronic imaging, smooth surface modeling, and computational geometry. Our motivation arises from applications in geology and mining, which often involve large scattered data sets and a demand for high accuracy. The method of choice is ordinary kriging. This is because it is a best unbiased estimator. Unfortunately, this interpolant is computationally very expensive to compute exactly. For n scattered data points, computing the value of a single interpolant involves solving a dense linear system of size roughly n x n. This is infeasible for large n. In practice, kriging is solved approximately by local approaches that are based on considering only a relatively small'number of points that lie close to the query point. There are many problems with this local approach, however. The first is that determining the proper neighborhood size is tricky, and is usually solved by ad hoc methods such as selecting a fixed number of nearest neighbors or all the points lying within a fixed radius. Such fixed neighborhood sizes may not work well for all query points, depending on local density of the point distribution. Local methods also suffer from the problem that the resulting interpolant is not continuous. Meyer showed that while kriging produces smooth continues surfaces, it has zero order continuity along its borders. Thus, at interface boundaries where the neighborhood changes, the interpolant behaves discontinuously. Therefore, it is important to consider and solve the global system for each interpolant. However, solving such large dense systems for each query point is impractical. Recently a more principled approach to approximating kriging has been proposed based on a technique called covariance tapering. The problems arise from the fact that the covariance functions that are used in kriging have global support. Our implementations combine, utilize, and enhance a number of different approaches that have been introduced in literature for solving large linear systems for interpolation of scattered data points. For very large systems, exact methods such as Gaussian elimination are impractical since they require 0(n(exp 3)) time and 0(n(exp 2)) storage. As Billings et al. suggested, we use an iterative approach. In particular, we use the SYMMLQ method, for solving the large but sparse ordinary kriging systems that result from tapering. The main technical issue that need to be overcome in our algorithmic solution is that the points' covariance matrix for kriging should be symmetric positive definite. The goal of tapering is to obtain a sparse approximate representation of the covariance matrix while maintaining its positive definiteness. Furrer et al. used tapering to obtain a sparse linear system of the form Ax = b, where A is the tapered symmetric positive definite covariance matrix. Thus, Cholesky factorization could be used to solve their linear systems. They implemented an efficient sparse Cholesky decomposition method. They also showed if these tapers are used for a limited class of covariance models, the solution of the system converges to the solution of the original system. Matrix A in the ordinary kriging system, while symmetric, is not positive definite. Thus, their approach is not applicable to the ordinary kriging system. Therefore, we use tapering only to obtain a sparse linear system. Then, we use SYMMLQ to solve the ordinary kriging system. We show that solving large kriging systems becomes practical via tapering and iterative methods, and results in lower estimation errors compared to traditional local approaches, and significant memory savings compared to the original global system. We also developed a more efficient variant of the sparse SYMMLQ method for large ordinary kriging systems. This approach adaptively finds the correct local neighborhood for each query point in the interpolation process.
Memarsadeghi, Nargess↗
Information Power Grid: Distributed High-Performance Computing and Large-Scale Data Management for Science and Engineering
We use the term "Grid" to refer to distributed, high performance computing and data handling infrastructure that incorporates geographically and organizationally dispersed, heterogeneous resources that are persistent and supported. This infrastructure includes: (1) Tools for constructing collaborative, application oriented Problem Solving Environments / Frameworks (the primary user interfaces for Grids); (2) Programming environments, tools, and services providing various approaches for building applications that use aggregated computing and storage resources, and federated data sources; (3) Comprehensive and consistent set of location independent tools and services for accessing and managing dynamic collections of widely distributed resources: heterogeneous computing systems, storage systems, real-time data sources and instruments, human collaborators, and communications systems; (4) Operational infrastructure including management tools for distributed systems and distributed resources, user services, accounting and auditing, strong and location independent user authentication and authorization, and overall system security services The vision for NASA's Information Power Grid - a computing and data Grid - is that it will provide significant new capabilities to scientists and engineers by facilitating routine construction of information based problem solving environments / frameworks. Such Grids will knit together widely distributed computing, data, instrument, and human resources into just-in-time systems that can address complex and large-scale computing and data analysis problems. Examples of these problems include: (1) Coupled, multidisciplinary simulations too large for single systems (e.g., multi-component NPSS turbomachine simulation); (2) Use of widely distributed, federated data archives (e.g., simultaneous access to metrological, topological, aircraft performance, and flight path scheduling databases supporting a National Air Space Simulation systems}; (3) Coupling large-scale computing and data systems to scientific and engineering instruments (e.g., realtime interaction with experiments through real-time data analysis and interpretation presented to the experimentalist in ways that allow direct interaction with the experiment (instead of just with instrument control); (5) Highly interactive, augmented reality and virtual reality remote collaborations (e.g., Ames / Boeing Remote Help Desk providing field maintenance use of coupled video and NDI to a remote, on-line airframe structures expert who uses this data to index into detailed design databases, and returns 3D internal aircraft geometry to the field); (5) Single computational problems too large for any single system (e.g. the rotocraft reference calculation). Grids also have the potential to provide pools of resources that could be called on in extraordinary / rapid response situations (such as disaster response) because they can provide common interfaces and access mechanisms, standardized management, and uniform user authentication and authorization, for large collections of distributed resources (whether or not they normally function in concert). IPG development and deployment is addressing requirements obtained by analyzing a number of different application areas, in particular from the NASA Aero-Space Technology Enterprise. This analysis has focussed primarily on two types of users: the scientist / design engineer whose primary interest is problem solving (e.g. determining wing aerodynamic characteristics in many different operating environments), and whose primary interface to IPG will be through various sorts of problem solving frameworks. The second type of user is the tool designer: the computational scientists who convert physics and mathematics into code that can simulate the physical world. These are the two primary users of IPG, and they have rather different requirements. The results of the analysis of the needs of these two types of users provides a broad set of requirements that gives rise to a general set of required capabilities. The IPG project is intended to address all of these requirements. In some cases the required computing technology exists, and in some cases it must be researched and developed. The project is using available technology to provide a prototype set of capabilities in a persistent distributed computing testbed. Beyond this, there are required capabilities that are not immediately available, and whose development spans the range from near-term engineering development (one to two years) to much longer term R&D (three to six years). Additional information is contained in the original.
Johnston, William E.↗
Latest COBE results, large-scale data, and predictions of inflation
One of the predictions of the inflationary scenario of cosmology is that the initial spectrum of primordial density fluctuations (PDFs) must have the Harrison-Zeldovich (HZ) form. Here, in order to test the inflationary scenario, predictions of the microwave background radiation (MBR) anisotropies measured by COBE are computed based on large-scale data for the universe and assuming Omega-1 and the HZ spectrum on large scales. It is found that the minimal scale where the spectrum can first enter the HZ regime is found, constraining the power spectrum of the mass distribution to within the bias factor b. This factor is determined and used to predict parameters of the MBR anisotropy field. For the spectrum of PDFs that reaches the HZ regime immediately after the scale accessible to the APM catalog, the numbers on MBR anisotropies are consistent with the COBE detections and thus the standard inflation can indeed be considered a viable theory for the origin of the large-scale structure in the universe.
Kashlinsky, A.↗