Search NASA⌕ Search

SEARCH · Search NASA

Results for “Large data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
NASA NTRS2005

Large Scale Data Mining to Improve Usability of Data: An Intelligent Archive Testbed

Research in certain scientific disciplines - including Earth science, particle physics, and astrophysics - continually faces the challenge that the volume of data needed to perform valid scientific research can at times overwhelm even a sizable research community. The desire to improve utilization of this data gave rise to the Intelligent Archives project, which seeks to make data archives active participants in a knowledge building system capable of discovering events or patterns that represent new information or knowledge. Data mining can automatically discover patterns and events, but it is generally viewed as unsuited for large-scale use in disciplines like Earth science that routinely involve very high data volumes. Dozens of research projects have shown promising uses of data mining in Earth science, but all of these are based on experiments with data subsets of a few gigabytes or less, rather than the terabytes or petabytes typically encountered in operational systems. To bridge this gap, the Intelligent Archives project is establishing a testbed with the goal of demonstrating the use of data mining techniques in an operationally-relevant environment. This paper discusses the goals of the testbed and the design choices surrounding critical issues that arose during testbed implementation.

Ramapriyan, Hampapuram↗
NASA NTRS1991

Data simulation for the Lightning Imaging Sensor (LIS)

This project aims to build a data analysis system that will utilize existing video tape scenes of lightning as viewed from space. The resultant data will be used for the design and development of the Lightning Imaging Sensor (LIS) software and algorithm analysis. The desire for statistically significant metrics implies that a large data set needs to be analyzed. Before 1990 the quality and quantity of video was insufficient to build a usable data set. At this point in time, there is usable data from missions STS-34, STS-32, STS-31, STS-41, STS-37, and STS-39. During the summer of 1990, a manual analysis system was developed to demonstrate that the video analysis is feasible and to identify techniques to deduce information that was not directly available. Because the closed circuit television system used on the space shuttle was intended for documentary TV, the current value of the camera focal length and pointing orientation, which are needed for photoanalysis, are not included in the system data. A large effort was needed to discover ancillary data sources as well as develop indirect methods to estimate the necessary parameters. Any data system coping with full motion video faces an enormous bottleneck produced by the large data production rate and the need to move and store the digitized images. The manual system bypassed the video digitizing bottleneck by using a genlock to superimpose pixel coordinates on full motion video. Because the data set had to be obtained point by point by a human operating a computer mouse, the data output rate was small. The loan and subsequent acquisition of a Abekas digital frame store with a real time digitizer moved the bottleneck from data acquisition to a problem of data transfer and storage. The semi-automated analysis procedure was developed using existing equipment and is described. A fully automated system is described in the hope that the components may come on the market at reasonable prices in the next few years.

Boeck, William L.↗
NASA NTRS1989

Artificial intelligence issues related to automated computing operations

Large data processing installations represent target systems for effective applications of artificial intelligence (AI) constructs. The system organization of a large data processing facility at the NASA Marshall Space Flight Center is presented. The methodology and the issues which are related to AI application to automated operations within a large-scale computing facility are described. Problems to be addressed and initial goals are outlined.

Hornfeck, William A.↗
NASA NTRS2003

A Fast Implementation of the ISOCLUS Algorithm

Unsupervised clustering is a fundamental building block in numerous image processing applications. One of the most popular and widely used clustering schemes for remote sensing applications is the ISOCLUS algorithm, which is based on the ISODATA method. The algorithm is given a set of n data points in d-dimensional space, an integer k indicating the initial number of clusters, and a number of additional parameters. The general goal is to compute the coordinates of a set of cluster centers in d-space, such that those centers minimize the mean squared distance from each data point to its nearest center. This clustering algorithm is similar to another well-known clustering method, called k-means. One significant feature of ISOCLUS over k-means is that the actual number of clusters reported might be fewer or more than the number supplied as part of the input. The algorithm uses different heuristics to determine whether to merge lor split clusters. As ISOCLUS can run very slowly, particularly on large data sets, there has been a growing .interest in the remote sensing community in computing it efficiently. We have developed a faster implementation of the ISOCLUS algorithm. Our improvement is based on a recent acceleration to the k-means algorithm of Kanungo, et al. They showed that, by using a kd-tree data structure for storing the data, it is possible to reduce the running time of k-means. We have adapted this method for the ISOCLUS algorithm, and we show that it is possible to achieve essentially the same results as ISOCLUS on large data sets, but with significantly lower running times. This adaptation involves computing a number of cluster statistics that are needed for ISOCLUS but not for k-means. Both the k-means and ISOCLUS algorithms are based on iterative schemes, in which nearest neighbors are calculated until some convergence criterion is satisfied. Each iteration requires that the nearest center for each data point be computed. Naively, this requires O(kn) time, where k denotes the current number of centers. Traditional techniques for accelerating nearest neighbor searching involve storing the k centers in a data structure. However, because of the iterative nature of the algorithm, this data structure would need to be rebuilt with each new iteration. Our approach is to store the data points in a kd-tree data structure. The assignment of points to nearest neighbors is carried out by a filtering process, which successively eliminates centers that can not possibly be the nearest neighbor for a given region of space. This algorithm is significantly faster, because large groups of data points can be assigned to their nearest center in a single operation. Preliminary results on a number of real Landsat datasets show that our revised ISOCLUS-like scheme runs about twice as fast.

Memarsadeghi, Nargess↗
NASA NTRS1972

Culling of redundant data

The inversion of a large data set, with it errors, is demonstrated and the tradeoff curve, or resolving power calculations, are discussed. Consideration is given to inverse problems of the earth, represented in the idealized form of a single scaler function of a single coordinate. The difference between the functional for the real earth and the functional for some model is formulated. A procedure is given for using the linear relationships between the differences in the data and the differences in the model as side conditions. Through a process of diagonalization a model is created that fits the data. The data are culled by accepting only those relative standard errors that are less than 100 percent.

Gilbert, F.↗
NASA NTRS1997

Precise Point Positioning for the Efficient and Robust Analysis of GPS Data From Large Networks

Networks of dozens to hundreds of permanently operating precision Global Positioning System (GPS) receivers are emerging at spatial scales that range from 10(exp 0) to 10(exp 3) km. To keep the computational burden associated with the analysis of such data economically feasible, one approach is to first determine precise GPS satellite positions and clock corrections from a globally distributed network of GPS receivers. Then, data from the local network are analyzed by estimating receiver specific parameters with receiver-specific data; satellite parameters are held fixed at their values determined in the global solution. This "precise point positioning" allows analysis of data from hundreds to thousands of sites every day with 40 Mflop computers, with results comparable in quality to the simultaneous analysis of all data. The reference frames for the global and network solutions can be free of distortion imposed by erroneous fiducial constraints on any sites.

Zumberge, J. F.↗
NASA NTRS1997

Precise Point Positioning for the Efficient and Robust Analysis of GPS Data from Large Networks

Networks of dozens to hundreds of permanently operating precision Global Positioning System (GPS) receivers are emerging at spatial scales that range from 10(exp 0) to 10(exp 3) km. To keep the computational burden associated with the analysis of such data economically feasible, one approach is to first determine precise GPS satellite positions and clock corrections from a globally distributed network of GPS receivers. Their, data from the local network are analyzed by estimating receiver- specific parameters with receiver-specific data satellite parameters are held fixed at their values determined in the global solution. This "precise point positioning" allows analysis of data from hundreds to thousands of sites every (lay with 40-Mflop computers, with results comparable in quality to the simultaneous analysis of all data. The reference frames for the global and network solutions can be free of distortion imposed by erroneous fiducial constraints on any sites.

Zumberge, J. F.↗
NASA NTRS1972

Reasoning by analogy as an aid to heuristic theorem proving.

When heuristic problem-solving programs are faced with large data bases that contain numbers of facts far in excess of those needed to solve any particular problem, their performance rapidly deteriorates. In this paper, the correspondence between a new unsolved problem and a previously solved analogous problem is computed and invoked to tailor large data bases to manageable sizes. This paper outlines the design of an algorithm for generating and exploiting analogies between theorems posed to a resolution-logic system. These algorithms are believed to be the first computationally feasible development of reasoning by analogy to be applied to heuristic theorem proving.

Kling, R. E.↗
NASA NTRS1992

Constraints on the power spectrum of the primordial density field from large-scale data - Microwave background and predictions of inflation

It is shown here that, by using galaxy catalog correlation data as input, measurements of microwave background radiation (MBR) anisotropies should soon be able to test two of the inflationary scenario's most basic predictions: (1) that the primordial density fluctuations produced were scale-invariant and (2) that the universe is flat. They should also be able to detect anisotropies of large-scale structure formed by gravitational evolution of density fluctuations present at the last scattering epoch. Computations of MBR anisotropies corresponding to the minimum of the large-scale variance of the MBR anisotropy are presented which favor an open universe with P(k) significantly different from the Harrison-Zeldovich spectrum predicted by most inflationary models.

Kashlinsky, A.↗
NASA NTRS1998

Health Monitoring of Composite Material Structures Using a Vibrometry Technique

Non-destructive evaluation (NDE) methods for quantifying and locating damage are essential for inspecting structures to ensure safety and reliability. Transmittance function monitoring is a potentially new NDE technique being tested as a tool to detect, quantify, and locate damage on flexible structures. The technique has a large spatial range that is practical for detecting damage on large composite material structures such as a reusable launch vehicle. The Transmittance Function (TF) theory is based on structural dynamics principles that define how vibration at one point in a structure is related to a force at another point. This relationship is called the Frequency Response Function (FRF). A Transmittance Function (TF) is derived as the ratio of FRFs, and can detect damage because the FRFs change due to damage. If one excitation is used for the testing, the force does not need to be measured to compute the TF. In the damage detection procedure, the structure is subjected to wide-band vibration and TFs are computed between different accelerometers to detect changes in the structure, presumably due to damage. In the first year of the project the TF method was tested on a bolted panel, a curved panel, and beams, all made of fiberglass. It was shown that damage could be detected using low frequency vibration, 250 to 1,250 Hz. The technique is sensitive to damage, but it requires storage of historical or pre-damage TFs for the healthy structure. This would become a large data storage requirement for large structures. Thus one objective for the second year of the project was to eliminate the need to store historical data. The second year report gives details of how storage of historical data was eliminated. Further results of testing panel structures are also given.

Schulz, Mark J.↗
NASA NTRS2020

Advancing Methodologies for Applying Machine Learning and Evaluating Spatiotemporal Models of Fine Particulate Matter (PM 2.5 ) Using Satellite Data Over Large Regions

Reconstructing the distribution of fine particulate matter (PM 2.5 ) in space and time, even far from ground monitoring sites, is an important exposure science contribution to epidemiologic analyses of PM 2.5 health impacts. Flexible statistical methods for prediction have demonstrated the integration of satellite observations with other predictors, yet these algorithms are susceptible to overfitting the spatiotemporal structure of the training datasets. We present a new approach for predicting PM 2.5 using machine-learning methods and evaluating prediction models for the goal of making predictions where they were not previously available. We apply extreme gradient boosting (XGBoost) modeling to predict daily PM 2.5 on a 1 x 1 km 2 resolution for a 13 state region in the Northeastern USA for the years 2000–2015 using satellite-derived aerosol optical depth and implement a recursive feature selection to develop a parsimonious model. We demonstrate excellent predictions of withheld observations but also contrast an RMSE of 3.11 μg/m 3 in our spatial cross-validation withholding nearby sites versus an overfit RMSE of 2.10 μg/m 3 using a more conventional random ten-fold splitting of the dataset. As the field of exposure science moves forward with the use of advanced machine-learning approaches for spatiotemporal modeling of air pollutants, our results show the importance of addressing data leakage in training, overfitting to spatiotemporal structure, and the impact of the predominance of ground monitoring sites in dense urban sub-networks on model evaluation. The strengths of our resultant modeling approach for exposure in epidemiologic studies of PM 2.5 include improved efficiency, parsimony, and interpretability with robust validation while still accommodating complex spatiotemporal relationships.

air pollution↗
NASA NTRS2012

Vehicle Detection for RCTA/ANS (Autonomous Navigation System)

Using a stereo camera pair, imagery is acquired and processed through the JPLV stereo processing pipeline. From this stereo data, large 3D blobs are found. These blobs are then described and classified by their shape to determine which are vehicles and which are not. Prior vehicle detection algorithms are either targeted to specific domains, such as following lead cars, or are intensity- based methods that involve learning typical vehicle appearances from a large corpus of training data. In order to detect vehicles, the JPL Vehicle Detection (JVD) algorithm goes through the following steps: 1. Take as input a left disparity image and left rectified image from JPLV stereo. 2. Project the disparity data onto a two-dimensional Cartesian map. 3. Perform some post-processing of the map built in the previous step in order to clean it up. 4. Take the processed map and find peaks. For each peak, grow it out into a map blob. These map blobs represent large, roughly vehicle-sized objects in the scene. 5. Take these map blobs and reject those that do not meet certain criteria. Build descriptors for the ones that remain. Pass these descriptors onto a classifier, which determines if the blob is a vehicle or not. The probability of detection is the probability that if a vehicle is present in the image, is visible, and un-occluded, then it will be detected by the JVD algorithm. In order to estimate this probability, eight sequences were ground-truthed from the RCTA (Robotics Collaborative Technology Alliances) program, totaling over 4,000 frames with 15 unique vehicles. Since these vehicles were observed at varying ranges, one is able to find the probability of detection as a function of range. At the time of this reporting, the JVD algorithm was tuned to perform best at cars seen from the front, rear, or either side, and perform poorly on vehicles seen from oblique angles.

Brennan, Shane↗
NASA NTRS1975

Up-to-date state of storage techniques used for large numerical data files

Methods for data storage and output in data banks and memory files are discussed along with a survey of equipment available for this. Topics discussed include magnetic tapes, magnetic disks, Terabit magnetic tape memory, Unicon 690 laser memory, IBM 1360 photostore, microfilm recording equipment, holographic recording, film readers, optical character readers, digital data storage techniques, and photographic recording. The individual types of equipment are summarized in tables giving the basic technical parameters.

Chlouba, V.↗
NASA NTRS1976

Stability analysis of the discrete-data Large-Space-Telescope system

This paper considers the application of the discrete describing function to the stability analysis of the Large-Space Telescope (LST) system. An analytical model of the CMG gimbal friction is derived, which is then used to arrive at a closed-form analytical expression for the discrete describing function of the nonlinearity. The analysis is used for the study of the fine-pointing stability of the LST vehicle. Simulation results corroborate the conclusions from the analytical analysis.

Kuo, B. C.↗