Search NASA⌕ Search

SEARCH · Search NASA

Results for “Large data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

Efficient bulk-loading of gridfiles

This paper considers the problem of bulk-loading large data sets for the gridfile multiattribute indexing technique. We propose a rectilinear partitioning algorithm that heuristically seeks to minimize the size of the gridfile needed to ensure no bucket overflows. Empirical studies on both synthetic data sets and on data sets drawn from computational fluid dynamics applications demonstrate that our algorithm is very efficient, and is able to handle large data sets. In addition, we present an algorithm for bulk-loading data sets too large to fit in main memory. Utilizing a sort of the entire data set it creates a gridfile without incurring any overflows.

Leutenegger, Scott T.↗

Distributed Large Data-Object Environments: End-to-End Performance Analysis of High Speed Distributed Storage Systems in Wide Area ATM Networks

We have developed and deployed a distributed-parallel storage system (DPSS) in several high speed asynchronous transfer mode (ATM) wide area networks (WAN) testbeds to support several different types of data-intensive applications. Architecturally, the DPSS is a network striped disk array, but is fairly unique in that its implementation allows applications complete freedom to determine optimal data layout, replication and/or coding redundancy strategy, security policy, and dynamic reconfiguration. In conjunction with the DPSS, we have developed a 'top-to-bottom, end-to-end' performance monitoring and analysis methodology that has allowed us to characterize all aspects of the DPSS operating in high speed ATM networks. In particular, we have run a variety of performance monitoring experiments involving the DPSS in the MAGIC testbed, which is a large scale, high speed, ATM network and we describe our experience using the monitoring methodology to identify and correct problems that limit the performance of high speed distributed applications. Finally, the DPSS is part of an overall architecture for using high speed, WAN's for enabling the routine, location independent use of large data-objects. Since this is part of the motivation for a distributed storage system, we describe this architecture.

Johnston, William↗

Efficient Implementation of an Optimal Interpolator for Large Spatial Data Sets

Interpolating scattered data points is a problem of wide ranging interest. A number of approaches for interpolation have been proposed both from theoretical domains such as computational geometry and in applications' fields such as geostatistics. Our motivation arises from geological and mining applications. In many instances data can be costly to compute and are available only at nonuniformly scattered positions. Because of the high cost of collecting measurements, high accuracy is required in the interpolants. One of the most popular interpolation methods in this field is called ordinary kriging. It is popular because it is a best linear unbiased estimator. The price for its statistical optimality is that the estimator is computationally very expensive. This is because the value of each interpolant is given by the solution of a large dense linear system. In practice, kriging problems have been solved approximately by restricting the domain to a small local neighborhood of points that lie near the query point. Determining the proper size for this neighborhood is a solved by ad hoc methods, and it has been shown that this approach leads to undesirable discontinuities in the interpolant. Recently a more principled approach to approximating kriging has been proposed based on a technique called covariance tapering. This process achieves its efficiency by replacing the large dense kriging system with a much sparser linear system. This technique has been applied to a restriction of our problem, called simple kriging, which is not unbiased for general data sets. In this paper we generalize these results by showing how to apply covariance tapering to the more general problem of ordinary kriging. Through experimentation we demonstrate the space and time efficiency and accuracy of approximating ordinary kriging through the use of covariance tapering combined with iterative methods for solving large sparse systems. We demonstrate our approach on large data sizes arising both from synthetic sources and from real applications.

Memarsadeghi, Nargess↗

SAR correlation technique - An algorithm for processing data with large range walk

This paper presents an algorithm for synthetic aperture radar (SAR) azimuth correlation with extraneously large range migration effect which can not be accommodated by the existing frequency domain interpolation approach used in current SEASAT SAR processing. A mathematical model is first provided for the SAR point-target response in both the space (or time) and the frequency domain. A simple and efficient processing algorithm derived from the hybrid algorithm is then given. This processing algorithm enables azimuth correlation by two steps. The first step is a secondary range compression to handle the dispersion of the spectra of the azimuth response along range. The second step is the well-known frequency domain range migration correction approach for the azimuth compression. This secondary range compression can be processed simultaneously with range pulse compression. Simulation results provided here indicate that this processing algorithm yields a satisfactory compressed impulse response for SAR data with large range migration.

Jin, M.↗

An application of parameter identification to the oscillatory motion of an airplane at high C(L)

This paper presents an application of a stepwise regression incorporating polynomial splines to oscillatory flight data from a light research airplane operating at near stall angles of attack. It is shown that data from several experiments can be combined into a large data set for analysis and that hysteresis phenomena can be observed in this large data set. Finally, it is postulated from the analysis of the flight data and theoretical calculations that the observed oscillatory motion is the result of a combination of wing stall and wing wake position at the tail.

Batterson, J. G.↗

Information Power Grid: Distributed High-Performance Computing and Large-Scale Data Management for Science and Engineering

The term "Grid" refers to distributed, high performance computing and data handling infrastructure that incorporates geographically and organizationally dispersed, heterogeneous resources that are persistent and supported. The vision for NASN's Information Power Grid - a computing and data Grid - is that it will provide significant new capabilities to scientists and engineers by facilitating routine construction of information based problem solving environments / frameworks that will knit together widely distributed computing, data, instrument, and human resources into just-in-time systems that can address complex and large-scale computing and data analysis problems. IPG development and deployment is addressing requirements obtained by analyzing a number of different application areas, in particular from the NASA Aero-Space Technology Enterprise. This analysis has focussed primarily on two types of users: The scientist / design engineer whose primary interest is problem solving (e.g., determining wing aerodynamic characteristics in many different operating environments), and whose primary interface to IPG will be through various sorts of problem solving frameworks. The second type of user if the tool designer: The computational scientists who convert physics and mathematics into code that can simulate the physical world. These are the two primary users of IPG, and they have rather different requirements. This paper describes the current state of IPG (the operational testbed), the set of capabilities being put into place for the operational prototype IPG, as well as some of the longer term R&D tasks.

Johnston, William E.↗

An expert system approach to astronomical data analysis

Expert systems technology has much to offer to the problem of astronomical data analysis, where large data volumes and sophisticated analysis goals have caused a variety of interesting problems to arise. The construction of a prototype expert system whose target domain is CCD image calibration, is reported. The prototype is designed to be extensible to different and more complex problems in a straighforward way, and to be largely independent of the details of the specific data analysis system which executes the plan it generates.

Johnston, Mark D.↗

The Atacama Cosmology Telescope: Summary of DR4 and DR5 Data Products and Data Access

Two recent large data releases for the Atacama Cosmology Telescope (ACT), called DR4 and DR5, are available for public access. These data include temperature and polarization maps that cover nearly half the sky at arcminute resolution in three frequency bands; lensing maps and component-separated maps covering ~ 2,100 deg 2 of sky; derived power spectra and cosmological likelihoods; a catalog of over 4,000 galaxy clusters; and supporting ancillary products including beam functions and masks. The data and products are described in a suite of ACT papers; here we provide a summary. In order to facilitate ease of access to these data we present a set of Jupyter IPython notebooks developed to introduce users to DR4, DR5, and the tools needed to analyze these data. The data products (excluding simulations) and the set of notebooks are publicly available on the NASA Legacy Archive for Microwave Background Data Analysis (LAMBDA); simulation products are available on the National Energy Research Scientific Computing Center (NERSC).

79 ASTRONOMY AND ASTROPHYSICS↗

The Atacama Cosmology Telescope: Summary of DR4 and DR5 Data Products and Data Access

Two recent large data releases for the Atacama Cosmology Telescope (ACT), called DR4 and DR5, are available for public access. These data include temperature and polarization maps that cover nearly half the sky at arcminute resolution in three frequency bands; lensing maps and component-separated maps covering ∼2100 deg 2 of sky; derived power spectra and cosmological likelihoods; a catalog of over 4000 galaxy clusters; and supporting ancillary products including beam functions and masks. The data and products are described in a suite of ACT papers; here we provide a summary. In order to facilitate ease of access to these data, we present a set of Jupyter IPython notebooks developed to introduce users to DR4, DR5, and the tools needed to analyze these data. The data products (excluding simulations) and the set of notebooks are publicly available on the NASA Legacy Archive for Microwave Background Data Analysis; simulation products are available on the National Energy Research Scientific Computing Center.

Maya Mallaby-Kay↗

Production and Distribution of NASA MODIS Remote Sensing Products

The two Moderate Resolution Imaging Spectroradiometer (MODIS) instruments on-board NASA's Earth Observing System (EOS) Terra and Aqua satellites make key measurements for understanding the Earth's terrestrial ecosystems. Global time-series of terrestrial geophysical parameters have been produced from MODIS/Terra for over 7 years and for MODIS/Aqua for more than 4 1/2 years. These well calibrated instruments, a team of scientists and a large data production, archive and distribution systems have allowed for the development of a new suite of high quality product variables at spatial resolutions as fine as 250m in support of global change research and natural resource applications. This talk describes the MODIS Science team's products, with a focus on the terrestrial (land) products, the data processing approach and the process for monitoring and improving the product quality. The original MODIS science team was formed in 1989. The team's primary role is the development and implementation of the geophysical algorithms. In addition, the team provided feedback on the design and pre-launch testing of the instrument and helped guide the development of the data processing system. The key challenges the science team dealt with before launch were the development of algorithms for a new instrument and provide guidance of the large and complex multi-discipline processing system. Land, Ocean and Atmosphere discipline teams drove the processing system requirements, particularly in the area of the processing loads and volumes needed to daily produce geophysical maps of the Earth at resolutions as fine as 250 m. The processing system had to handle a large number of data products, large data volumes and processing loads, and complex processing requirements. Prior to MODIS, daily global maps from heritage instruments, such as Advanced Very High Resolution Radiometer (AVHRR), were not produced at resolutions finer than 5 km. The processing solution evolved into a combination of processing the lower level (Level 1) products and the higher level discipline specific Land and Atmosphere products in the MODIS Science Investigator Lead Processing System (SIPS), the MODIS Adaptive Processing System (MODAPS), and archive and distribution of the Land products to the user community by two of NASA s EOS Distributed Active Archive Centers (DAACs). Recently, a part of MODAPS, the Level 1 and Atmosphere Archive and Distribution System (LAADS), took over the role of archiving and distributing the Level 1 and Atmosphere products to the user community.

Wolfe, Robert↗

Data processing large quantities of multispectral information

Method is combination of digital and optical techniques. Multispectral data is coded into binary matrix format and then encoded onto photographic film. Film is holographically correlated with spectral signature to generate single-class classification map. Number of maps are optically superimposed to produce full-color, multiclass classification map.

Haskell, R. E.↗

Envision: An interactive system for the management and visualization of large geophysical data sets

Envision is a software project at the University of Illinois and Texas A&M, funded by NASA's Applied Information Systems Research Project. It provides researchers in the geophysical sciences convenient ways to manage, browse, and visualize large observed or model data sets. Envision integrates data management, analysis, and visualization of geophysical data in an interactive environment. It employs commonly used standards in data formats, operating systems, networking, and graphics. It also attempts, wherever possible, to integrate with existing scientific visualization and analysis software. Envision has an easy-to-use graphical interface, distributed process components, and an extensible design. It is a public domain package, freely available to the scientific community.

Searight, K. R.↗

Hydrogen-Based Energy Storage Systems for Large-Scale Data Center Applications

Global demand for data and data access has spurred the rapid growth of the data center industry. To meet demands, data centers must provide uninterrupted service even during the loss of primary power. Service providers seeking ways to eliminate their carbon footprint are increasingly looking to clean and sustainable energy solutions, such as hydrogen technologies, as alternatives to traditional backup generators. In this viewpoint, a survey of the current state of data centers and hydrogen-based technologies is provided along with a discussion of the hydrogen storage and infrastructure requirements needed for large-scale backup power applications at data centers.

08 HYDROGEN↗

Improving Text Classification with Large Language Model-Based Data Augmentation

Large Language Models (LLMs) such as ChatGPT possess advanced capabilities in understanding and generating text. These capabilities enable ChatGPT to create text based on specific instructions, which can serve as augmented data for text classification tasks. Previous studies have approached data augmentation (DA) by either rewriting the existing dataset with ChatGPT or generating entirely new data from scratch. However, it is unclear which method is better without comparing their effectiveness. This study investigates the application of both methods to two datasets: a general-topic dataset (Reuters news data) and a domain-specific dataset (Mitigation dataset). Our findings indicate that: 1. ChatGPT generated new data consistently enhanced model’s classification results for both datasets. 2. Generating new data generally outperforms rewriting existing data, though crafting the prompts carefully is crucial to extract the most valuable information from ChatGPT, particularly for domain-specific data. 3. The augmentation data size affects the effectiveness of DA; however, we observed a plateau after incorporating 10 samples. 4. Combining the rewritten sample with new generated sample can potentially further improve the model’s performance.

97 MATHEMATICS AND COMPUTING↗

Carbon Consequences of Forest Disturbance and Recovery Across the Conterminous United States

Forests of North America are thought to constitute a significant long term sink for atmospheric carbon. The United States Forest Service Forest Inventory and Analysis (FIA) program has developed a large data base of stock changes derived from consecutive estimates of growing stock volume in the US. These data reveal a large and relatively stable increase in forest carbon stocks over the last two decades or more. The mechanisms underlying this national increase in forest stocks may include recovery of forests from past disturbances, net increases in forest area, and growth enhancement driven by climate or fertilization by CO2 and Nitrogen. Here we estimate the forest recovery component of the observed stock changes using FIA data on the age structure of US forests and carbon stocks as a function of age. The latter are used to parameterize forest disturbance and recovery processes in a carbon cycle model. We then apply resulting disturbance/recovery dynamics to landscapes and regions based on the forest age distributions. The analysis centers on 28 representative climate settings spread about forested regions of the conterminous US. We estimate carbon fluxes for each region and propagate uncertainties in calibration data through to the predicted fluxes. The largest recovery-driven carbon sinks are found in the South central, Pacific Northwest, and Pacific Southwest regions, with spatially averaged net ecosystem productivity (NEP) of about 100 g C / square m / a driven by forest age structure. Carbon sinks from recovery in the Northeast and Northern Lake States remain moderate to large owing to the legacy of historical clearing and relatively low modern disturbance rates from harvest and fire. At the continental scale, we find a conterminous U.S. forest NEP of only 0.16 Pg C/a from age structure in 2005, or only 0.047 Pg C/a of forest stock change after accounting for fire emissions and harvest transfers. Recent estimates of NEP derived from inventory stock change, harvest, and fire data show twice the NEP sink we derive from forest age distributions. We discuss possible reasons for the discrepancies including modeling errors and the possibility of climate and/or fertilization (CO2 or N) growth enhancements.

Williams, Christopher A.↗

NanoSIP: NanoSIMS Applications for Microbial Biology

High-resolution imaging with secondary ion mass spectrometry (nanoSIMS) has become a standard method in systems biology and environmental biogeochemistry and is broadly used to decipher ecophysiological traits of environmental microorganisms, metabolic processes in plant and animal tissues, and cross-kingdom symbioses. When combined with stable isotope-labeling—an approach we refer to as nanoSIP—nanoSIMS imaging offers a distinctive means to quantify net assimilation rates and stoichiometry of individual cell-sized particles in both low- and high-complexity environments. While the majority of nanoSIP studies in environmental and microbial biology have focused on nitrogen and carbon metabolism (using 15 N and 13 C tracers), multiple advances have pushed the capabilities of this approach in the past decade. The development of a high-brightness oxygen ion source has enabled high-resolution metal analyses that are easier to perform, allowing quantification of metal distribution in cells and environmental particles. New preparation methods, tools for automated data extraction from large data sets, and analytical approaches that push the limits of sensitivity and spatial resolution have allowed for more robust characterization of populations ranging from marine archaea to fungi and viruses. Further, NanoSIMS studies continue to be enhanced by correlation with orthogonal imaging and ‘omics approaches; when linked to molecular visualization methods, such as in situ hybridization and antibody labeling, these techniques enable in situ function to be linked to microbial identity and gene expression. Here we present an updated description of the primary materials, methods, and calculations used for nanoSIP, with an emphasis on recent advances in nanoSIMS applications, key methodological steps, and potential pitfalls.

07 ISOTOPE AND RADIATION SOURCES↗

Toward equitable environmental exposure modeling through convergence of data, open, and citizen sciences: an example of air pollution exposure modeling amidst increasing wildfire smoke

Exposure modeling is critical in environmental epidemiology and human health but may face challenges (e.g., skewed data, unequal error, context-insensitive validation, and computational demands). Modeling decisions reflect the intended use of the models and the values that modelers prioritize. We aimed to provide a conceptual framework and machine learning (ML) modeling protocols that address these issues. With 500m-gridded hourly PM 2.5 and O 3 levels in Illinois before, during, and after the 2023 Canadian wildfire season as a motivating example, we conducted modeling experiments to evaluate modeling methods, guided by three domains we propose based on theories of science: 1) Data Diversity, leveraging open and citizen science data to enhance inclusivity, parsimony, and representativeness; 2) Equitable Accuracy, ensuring fairly distributed uncertainties across subpopulations; and 3) Sustainable Modeling, balancing accuracy with reducing computational demands to promote accessibility for under-resourced researchers. Here, we found that ML with publicly available data can achieve high accuracy. Depending on methods, performance may vary substantially, even with identical input data. Large but skewed data may reduce performance. Misuse of cross-validation protocols can underestimate prediction error; although we observed R 2 s of ∼98 %, the modeled estimates varied significantly, indicating the need for careful model validation. By using new modeling protocols including representativeness-considered training and validation data and a new loss function, we achieved high agreement between estimates and ground-based measurements (e.g., R 2 = ∼90 % for PM 2.5 ; ∼80 % for O 3 ), equally distributed errors across sociodemographic strata and urban–rural divides, and reduction in computation time—from several weeks or months to a few days.

Exposure assessment↗

Gaussian Process Regression under Computational and Epistemic Misspecification

Gaussian process regression is a classical kernel method for function estimation and data interpolation. In large data applications, computational costs can be reduced using low-rank or sparse approximations of the kernel. This paper investigates the effect of such kernel approximations on the interpolation error. We introduce a unified framework to analyze Gaussian process regression under important classes of computational misspecification: Karhunen-Loève expansions that result in low-rank kernel approximations, multiscale wavelet expansions that induce sparsity in the covariance matrix, and finite element representations that induce sparsity in the precision matrix. Furthermore, our theory also accounts for epistemic misspecification in the choice of kernel parameters.

Gaussian process regression↗