Search NASA⌕ Search

SEARCH · Search NASA

Results for “large data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,531 records · Page 85

Computing contingency statistics in parallel : design trade-offs and limiting cases.

Statistical analysis is typically used to reduce the dimensionality of and infer meaning from data. A key challenge of any statistical analysis package aimed at large-scale, distributed data is to address the orthogonal issues of parallel scalability and numerical stability. Many statistical techniques, e.g., descriptive statistics or principal component analysis, are based on moments and co-moments and, using robust online update formulas, can be computed in an embarrassingly parallel manner, amenable to a map-reduce style implementation. In this paper we focus on contingency tables, through which numerous derived statistics such as joint and marginal probability, point-wise mutual information, information entropy, and {chi}{sup 2} independence statistics can be directly obtained. However, contingency tables can become large as data size increases, requiring a correspondingly large amount of communication between processors. This potential increase in communication prevents optimal parallel speedup and is the main difference with moment-based statistics (which we discussed in [1]) where the amount of inter-processor communication is independent of data size. Here we present the design trade-offs which we made to implement the computation of contingency tables in parallel. We also study the parallel speedup and scalability properties of our open source implementation. In particular, we observe optimal speed-up and scalability when the contingency statistics are used in their appropriate context, namely, when the data input is not quasi-diffuse.

99 GENERAL AND MISCELLANEOUS↗

Large Impact Basins on Mercury: Global Distribution, Characteristics, and Modification History from MESSENGER Orbital Data

The formation of large impact basins (diameter D greater than or equal to 300 km) was an important process in the early evolution of Mercury and influenced the planet's topography, stratigraphy, and crustal structure. We catalog and characterize this basin population on Mercury from global observations by the MESSENGER spacecraft, and we use the new data to evaluate basins suggested on the basis of the Mariner 10 flybys. Forty-two certain or probable impact basins are recognized a few additional basins that may have been degraded to the point of ambiguity are plausible on the basis of new data but are classified as uncertain. The spatial density of large basins (D greater than or equal to 500 km) on Mercury is lower than that on the Moon. Morphological characteristics of basins on Mercury suggest that on average they are more degraded than lunar basins. These observations are consistent with more efficient modification, degradation, and obliteration of the largest basins on Mercury than on the Moon. This distinction may be a result of differences in the basin formation process (producing fewer rings), greater relaxation of topography after basin formation (subduing relief), and/or higher rates of volcanism during the period of heavy bombardment on Mercury compared to the Moon (burying basin rings and interiors).

Fassett, Caleb I.↗

Approximate l-fold cross-validation with Least Squares SVM and Kernel Ridge Regression

Kernel methods have difficulties scaling to large modern data sets. The scalability issues are based on computational and memory requirements for working with a large matrix. These requirements have been addressed over the years by using low-rank kernel approximations or by improving the solvers scalability. However, Least Squares Support VectorMachines (LS-SVM), a popular SVM variant, and Kernel Ridge Regression still have several scalability issues. In particular, the O(n^3) computational complexity for solving a single model, and the overall computational complexity associated with tuning hyperparameters are still major problems. We address these problems by introducing an O(n log n) approximate l-fold cross-validation method that uses a multi-level circulant matrix to approximate the kernel. In addition, we prove our algorithm s computational complexity and present empirical runtimes on data sets with approximately 1 million data points. We also validate our approximate method s effectiveness at selecting hyperparameters on real world and standard benchmark data sets. Lastly, we provide experimental results on using a multi-level circulant kernel approximation to solve LS-SVM problems with hyperparameters selected using our method.

Edwards, Richard E↗

Approximate l-Fold Cross-Validation with Least Squares SVM and Kernel Ridge Regression

Kernel methods have difficulties scaling to large modern data sets. The scalability issues are based on computational and memory requirements for working with a large matrix. These requirements have been addressed over the years by using low-rank kernel approximations or by improving the solvers' scalability. However, Least Squares Support Vector Machines (LS-SVM), a popular SVM variant, and Kernel Ridge Regression still have several scalability issues. In particular, the O(n^3) computational complexity for solving a single model, and the overall computational complexity associated with tuning hyper parameters are still major problems. We address these problems by introducing an O(nlog n) approximate l-fold cross-validation method that uses a multi-level circulant matrix to approximate the kernel. In addition, we prove our algorithm's computational complexity and present empirical runtimes on data sets with approximately one million data points. We also validate our approximate method's effectiveness at selecting hyper parameters on real world and standard benchmark data sets. Lastly, we provide experimental results on using a multi level circulant kernel approximation to solve LS-SVM problems with hyper parameters selected using our method.

Computer Science↗

Systems aspects of COBE science data compression

A general approach to compression of diverse data from large scientific projects has been developed and this paper addresses the appropriate system and scientific constraints together with the algorithm development and test strategy. This framework has been implemented for the COsmic Background Explorer spacecraft (COBE) by retrofitting the existing VAS-based data management system with high-performance compression software permitting random access to the data. Algorithms which incorporate scientific knowledge and consume relatively few system resources are preferred over ad hoc methods. COBE exceeded its planned storage by a large and growing factor and the retrieval of data significantly affects the processing, delaying the availability of data for scientific usage and software test. Embedded compression software is planned to make the project tractable by reducing the data storage volume to an acceptable level during normal processing.

Freedman, I.↗

Application-Controlled Demand Paging for Out-of-Core Visualization

In the area of scientific visualization, input data sets are often very large. In visualization of Computational Fluid Dynamics (CFD) in particular, input data sets today can surpass 100 Gbytes, and are expected to scale with the ability of supercomputers to generate them. Some visualization tools already partition large data sets into segments, and load appropriate segments as they are needed. However, this does not remove the problem for two reasons: 1) there are data sets for which even the individual segments are too large for the largest graphics workstations, 2) many practitioners do not have access to workstations with the memory capacity required to load even a segment, especially since the state-of-the-art visualization tools tend to be developed by researchers with much more powerful machines. When the size of the data that must be accessed is larger than the size of memory, some form of virtual memory is simply required. This may be by segmentation, paging, or by paged segments. In this paper we demonstrate that complete reliance on operating system virtual memory for out-of-core visualization leads to poor performance. We then describe a paged segment system that we have implemented, and explore the principles of memory management that can be employed by the application for out-of-core visualization. We show that application control over some of these can significantly improve performance. We show that sparse traversal can be exploited by loading only those data actually required. We show also that application control over data loading can be exploited by 1) loading data from alternative storage format (in particular 3-dimensional data stored in sub-cubes), 2) controlling the page size. Both of these techniques effectively reduce the total memory required by visualization at run-time. We also describe experiments we have done on remote out-of-core visualization (when pages are read by demand from remote disk) whose results are promising.

Cox, Michael↗

End-to-end data movement using MPI-IO over routed terabits infrastructures. In: NDM '13 Proceedings of the Third International Workshop on Network-Aware Data Management, Article No. 9

Scientific discovery is nowadays driven by large-scale simulations running on massively parallel high-performance computing (HPC) systems. These applications each generate a large amount of data, which then needs to be post-processed for example for data mining or visualization. Unfortunately, the computing platform used for post processing might be different from the one on which the data is initially generated, introducing the challenge of moving large amount of data between computing platforms. This is especially challenging when these two platforms are geographically separated since the data needs to be moved between computing facilities. This is even more critical when scientists tightly couple their domain specific applications with a post processing application. The paper presents a solution for the data transfer between MPI applications using a dedicated wide area network (WAN) terabit infrastructure. The proposed solution is based on parallel access to data files and the Message Passing Interface (MPI) over the Common Communication Infrastructure (CCI) for the data transfer over a routed infrastructure. In the context of this research, the Energy Sciences Network (ESnet) of the U.S. Department of Energy (DOE) is targeted for the transfer of data between DOE national laboratories.

Vallée, Geoffroy↗

X-ray digital imaging petrography of lunar mare soils: modal analyses of minerals and glasses

It is essential that accurate modal (i.e., volume) percentages of the various mineral and glass phases in lunar soils be used for addressing and resolving the effects of space weathering upon reflectance spectra, as well as for their calibration such data are also required for evaluating the resource potential of lunar minerals for use at a lunar base. However, these data are largely lacking. Particle-counting information for lunar soils, originally obtained to study formational processes, does not provide these necessary data, including the percentages of minerals locked in multi-phase lithic fragments and fused-soil particles, such as agglutinates. We have developed a technique for modal analyses, sensu stricto, of lunar soils, using digital imaging of X-ray maps obtained with an energy-dispersive spectrometer mounted on an electron microprobe. A suite of nine soils (90 to 150 micrometers size fraction) from the Apollo 11, 12, 15, and 17 mare sites was used for this study. This is the first collection of such modal data on soils from all Apollo mare sites. The abundances of free-mineral fragments in the mare soils are greater for immature and submature soils than for mature soils, largely because of the formation of agglutinitic glass as maturity progresses. In considerations of resource utilization at a lunar base, the best lunar soils to use for mineral beneficiation (i.e., most free-mineral fragments) have maturities near the immature/submature boundary (Is/FeO approximately or = 30), not the mature soils with their complications due to extensive agglutination. The particle data obtained from the nine mare soils confirm the generalizations for lunar soils predicted by L.A. Taylor and D.S. McKay (1992, Lunar Planet Sci. Conf. 23rd, pp. 1411-1412 [Abstract]).

NASA Discipline Exobiology↗

Forest expansion dominates China’s land carbon sink since 1980

Carbon budget accounting relies heavily on Food and Agriculture Organization land-use data reported by governments. Here we develop a new land-use and cover-change database for China, finding that differing historical survey methods biased China’s reported data causing large errors in Food and Agriculture Organization databases. Land ecosystem model simulations driven with the new data reveal a strong carbon sink of 8.9 ± 0.8 Pg carbon from 1980 to 2019 in China, which was not captured in Food and Agriculture Organization data-based estimations due to biased land-use and cover-change signals. The land-use and cover-change in China, characterized by a rapid forest expansion from 1980 to 2019, contributed to nearly 44% of the national terrestrial carbon sink. In contrast, climate changes (22.3%), increasing nitrogen deposition (12.9%), and rising carbon dioxide (8.1%) are less important contributors. This indicates that previous studies have greatly underestimated the impact of land-use and cover-change on the terrestrial carbon balance of China. This study underlines the importance of reliable land-use and cover-change databases in global carbon budget accounting.

54 ENVIRONMENTAL SCIENCES↗

A formal method for identifying distinct states of variability in time-varying sources: SGR A* as an example

Continuously time variable sources are often characterized by their power spectral density and flux distribution. These quantities can undergo dramatic changes over time if the underlying physical processes change. However, some changes can be subtle and not distinguishable using standard statistical approaches. Here, we report a methodology that aims to identify distinct but similar states of time variability. We apply this method to the Galactic supermassive black hole, where 2.2 μm flux is observed from a source associated with Sgr A* and where two distinct states have recently been suggested. Our approach is taken from mathematical finance and works with conditional flux density distributions that depend on the previous flux value. The discrete, unobserved (hidden) state variable is modeled as a stochastic process and the transition probabilities are inferred from the flux density time series. Using the most comprehensive data set to date, in which all Keck and a majority of the publicly available Very Large Telescope data have been merged, we show that Sgr A* is sufficiently described by a single intrinsic state. However, the observed flux densities exhibit two states: noise dominated and source dominated. Our methodology reported here will prove extremely useful to assess the effects of the putative gas cloud G2 that is on its way toward the black hole and might create a new state of variability.

79 ASTRONOMY AND ASTROPHYSICS↗

A Lakehouse Architecture for the Management and Analysis of Heterogeneous Data for Biomedical Research and Mega-biobanks

Data Lakehouse is a new paradigm in data architectures that embodies and integrates already established concepts for the systematic management of disparate, large-scale data – a data lake for heterogeneous data management, use of open standards for high-performance querying, and systematic maintenance of the data "freshness". In addition to being a new concept, the data lakehouse is also still a conceptual construct. Many projects that use the lakehouse require maturing, empirical studies, and specific implementations. In this paper, we present our implementation of the data lakehouse concept in a biomedical research and health data analytics domain, and we discuss the implementation of some unique and novel features such as support for specialized access controls in support of HIPAA regulation and IRB protocols, and support for the FAIR standard.

Begoli, Edmon↗

Effects of compression and random noise on multispectral data.

Two problem areas are often encountered in the use of multispectral scanner data: (1) large quantities of man and machine time are required to analyze and store the volumes of data gathered and (2) the quality of the data is usually reduced by the introduction of unwanted random noise. This paper will present some effects of data compression and random noise on multispectral data, particularly as they apply to pattern recognition and picture quality.

Ready, P. J.↗

Automated Analysis of a Large-Scale Sky Survey: The SKICAT System

We describe the application of decision tree based classification techniques to the development of an automated tool for the reduction of a large scientific data set. The primary benefits of the SKICAT approach are increased data reduction throughput, repeatability, and consistency of classification.

data analysis image databases↗

Physical Science Informatics: Providing Open Science Access to Microheater Array Boiling Experiment Data

The Physical Science Informatics (PSI) system is the next step in this an effort to make NASA sponsored flight data available to the scientific and engineering community, along with the general public. The experimental data, from six overall disciplines, Combustion Science, Fluid Physics, Complex Fluids, Fundamental Physics, and Materials Science, will present some unique challenges. Besides data in textual or numerical format, large portions of both the raw and analyzed data for many of these experiments are digital images and video, requiring large data storage requirements. In addition, the accessible data will include experiment design and engineering data (including applicable drawings), any analytical or numerical models, publications, reports, and patents, and any commercial products developed as a result of the research. This objective of paper includes the following: Present the preliminary layout (Figure 2) of MABE data within the PSI database. Obtain feedback on the layout. Present the procedure to obtain access to this database.

nucleate boiling↗

Parallel Visualization and Analysis with ParaView on a Cray XT4

Scienti c data sets produced by modern supercomputers like ORNL s Cray XT 4, Jaguar, can be extremely large, making visualization and analysis more di cult as moving large resultant data to dedicated analysis systems can be pro- hibitively expensive. We share our continuing work of integrating a parallel visu- alization system, ParaView, on ORNL s Jaguar system and our e orts to enable extreme scale interactive data visualization and analysis. We will discuss porting challenges and present performance numbers.

97 MATHEMATICS AND COMPUTING↗

Automated Classification and Analysis of Non-metallic Inclusion Data Sets

The aim of this study is to utilize principal component analysis (PCA), clustering methods, and correlation analysis to condense and examine large, multivariate data sets produced from automated analysis of non-metallic inclusions. Non-metallic inclusions play a major role in defining the properties of steel and their examination has been greatly aided by automated analysis in scanning electron microscopes equipped with energy dispersive X-ray spectroscopy. The methods were applied to analyze inclusions on two sets of samples: two laboratory-scale samples and four industrial samples from a near-finished 4140 alloy steel components with varying machinability. The laboratory samples had well-defined inclusions chemistries, composed of MgO-Al{sub 2}O{sub 3}-CaO, spinel (MgO-Al{sub 2}O{sub 3}), and calcium aluminate inclusions. The industrial samples contained MnS inclusions as well as (Ca,Mn)S + calcium aluminate oxide inclusions. PCA could be used to reduce inclusion chemistry variables to a 2D plot, which revealed inclusion chemistry groupings in the samples. Clustering methods were used to automatically classify inclusion chemistry measurements into groups, i.e., no user-defined rules were required.

36 MATERIALS SCIENCE↗

Massively parallel processor computer

An apparatus for processing multidimensional data with strong spatial characteristics, such as raw image data, characterized by a large number of parallel data streams in an ordered array is described. It comprises a large number (e.g., 16,384 in a 128 x 128 array) of parallel processing elements operating simultaneously and independently on single bit slices of a corresponding array of incoming data streams under control of a single set of instructions. Each of the processing elements comprises a bidirectional data bus in communication with a register for storing single bit slices together with a random access memory unit and associated circuitry, including a binary counter/shift register device, for performing logical and arithmetical computations on the bit slices, and an I/O unit for interfacing the bidirectional data bus with the data stream source. The massively parallel processor architecture enables very high speed processing of large amounts of ordered parallel data, including spatial translation by shifting or sliding of bits vertically or horizontally to neighboring processing elements.

Fung, L. W.↗

Systematic uncertainties in the measurement of the neutron lifetime using the Lunar Prospector neutron spectrometer

The lifetime of free neutrons measured in the laboratory has a longstanding disparity of ≈9 s. A space-based technique has recently been proposed to independently measure the neutron lifetime using interactions between the galactic cosmic rays and a low atmosphere planetary body. This technique has not produced competitive results yet due to constraints of nonoptimized data that contain large systematic errors. We use data from the neutron spectrometer on-board NASA's Lunar Prospector, and study two large systematics in the measurement of neutron lifetime: the lunar subsurface temperature and the lunar surface composition. We use the HeCd and HeSn neutron spectrometer data when the spacecraft was in a highly elliptical orbit during the orbit insertion period. We report the neutron lifetime using four different models that each have different choices of surface temperature and composition. The 5° [Prettyman et al ., J. Geophys. Res.: Planets 111, 2005JE002656 (2006)] and 2° [Wilson et al., Phys. Rev. C 104, 045501 (2021)] rebinned maps result in 777.6±11.7 s and 739.6±10.8 s, respectively. For the 20° map (Prettyman et al., 2006), constant equatorial and a latitude-dependent temperature model result in 738.6±10.8 s and 767.3±11.2 s, respectively. Increasing the complexities of the models accounting for the systematic effects increase the measured lifetime. However, the reported measurements are not competitive with the laboratory results due to large unaccounted systematics resulting from nonoptimized measurements and modeling assumptions. This work serves as a study of systematic uncertainties for future neutron lifetime measurements using the space-based technique. We estimate the effect on the lifetime from the choice of temperature model to be to be 28.7 ±15.5 s, and choice of compositional map (for 20° and 5° maps) to be 10.3 ±12.2 s.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗