SEARCH · Search NASA

Results for “data storage and indexing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

All sources NASA NTRS DOE OSTI

DOE OSTI2020

Streaming Data Reorganization at Scale with DeltaFS Indexed Massive Directories

We report complex storage stacks providing data compression, indexing, and analytics help leverage the massive amounts of data generated today to derive insights. It is challenging to perform this computation, however, while fully utilizing the underlying storage media. This is because, while storage servers with large core counts are widely available, single-core performance and memory bandwidth per core grow slower than the core count per die. Computational storage offers a promising solution to this problem by utilizing dedicated compute resources along the storage processing path. We present DeltaFS Indexed Massive Directories (IMDs), a new approach to computational storage. DeltaFS IMDs harvest available (i.e., not dedicated) compute, memory, and network resources on the compute nodes of an application to perform computation on data. We demonstrate the efficiency of DeltaFS IMDs by using them to dynamically reorganize the output of a real-world simulation application across 131,072 CPU cores. DeltaFS IMDs speed up reads by 1,740x while only slightly slowing down the writing of data during simulation I/O for in situ data processing.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2019

Data automation at light sources

Rapidly growing data volumes at light sources demand increasingly automated data collection, distribution, and analysis processes, in order to enable new scientific discoveries while not overwhelming finite human capabilities. We present here the case for automating and outsourcing light source science using cloud-hosted data automation and enrichment services, institutional computing resources, and high- performance computing facilities to provide cost-effective, scalable, and reliable implementations of such processes. We discuss three specific services that accomplish these goals for data distribution, automation, and transformation. In the first, Globus cloud-hosted data automation services are used to implement data capture, distribution, and analysis workflows for Advanced Photon Source and Advanced Light Source beamlines, leveraging institutional storage and computing. In the second, such services are combined with cloud-hosted data indexing and institutional storage to create a collaborative data publication, indexing, and discovery service, the Materials Data Facility (MDF), built to support a host of informatics applications in materials science. The third integrates components of the previous two projects with machine learning capabilities provided by the Data and Learning Hub for science (DLHub) to enable on-demand access to machine learning models from light source data capture and analysis workflows, and provides simplified interfaces to train new models on data from sources such as MDF on leadership scale computing resources. We draw conclusions about best practices for building next-generation data automation systems for future light sources.

Blaiszik, Ben↗

DOE OSTI2018

Scalable Data Management, Analysis, and Visualization (SDAV) Institute

Analytics-driven data layout optimizations and compression (NCSU): We have developed solutions that address challenges related to scientific data indexing, compression, and storage layout problems. Moreover, in addition to focusing on mainstream uniform mesh data, we have also addressed Adaptive Mesh Refinement (AMR) data.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

NASA NTRS2017

Use of Schema on Read in Earth Science Data Archives

Traditionally, NASA Earth Science data archives have file-based storage using proprietary data file formats, such as HDF and HDF-EOS, which are optimized to support fast and efficient storage of spaceborne and model data as they are generated. The use of file-based storage essentially imposes an indexing strategy based on data dimensions. In most cases, NASA Earth Science data uses time as the primary index, leading to poor performance in accessing data in spatial dimensions. For example, producing a time series for a single spatial grid cell involves accessing a large number of data files. With exponential growth in data volume due to the ever-increasing spatial and temporal resolution of the data, using file-based archives poses significant performance and cost barriers to data discovery and access. Storing and disseminating data in proprietary data formats imposes an additional access barrier for users outside the mainstream research community. At the NASA Goddard Earth Sciences Data Information Services Center (GES DISC), we have evaluated applying the schema-on-read principle to data access and distribution. We used Apache Parquet to store geospatial data, and have exposed data through Amazon Web Services (AWS) Athena, AWS Simple Storage Service (S3), and Apache Spark. Using the schema-on-read approach allows customization of indexing spatially or temporally to suit the data access pattern. The storage of data in open formats such as Apache Parquet has widespread support in popular programming languages. A wide range of solutions for handling big data lowers the access barrier for all users. This presentation will discuss formats used for data storage, frameworks with This presentation will discuss formats used for data storage, frameworks with support for schema-on-read used for data access, and common use cases covering data usage patterns seen in a geospatial data archive.

cloud applications↗

NASA NTRS2018

Benchmark Comparison of Cloud Analytics Methods Applied to Earth Observations

Earth Observation data are a vital resource for studying long term changes, but the large data volumes can be challenging to analyze. Time series analysis in particular is hampered by the typical thin-time-slice file organization. We examine several potential solutions inspired in large part by the data-parallel methods that have arisen with cloud computing. These solutions include various combinations of data re-organization, spatial indexing, distributed storage and pre-computation that we term "Analytics Optimized Data Stores" (AODS). We find that even simple solutions (such as a data cube) produce more than an order of magnitude improvement; the best provide two to three orders of magnitude improvement. The most performant solutions have tradeoffs in terms of generality or storage footprint, but may nonetheless be useful components in data analytics frameworks where performance is critical.

parallel processing (computers)↗

DOE OSTI2017

Web-based visual data exploration for improved radiological source detection

Radiation detection can provide a reliable means of detecting radiological material. Such capabilities can help to prevent nuclear and/or radiological attacks, but reliable detection in uncontrolled surroundings requires algorithms that account for environmental background radiation. The Berkeley Data Cloud (BDC) facilitates the development of such methods by providing a framework to capture, store, analyze, and share data sets. In the era of big data, both the size and variety of data make it difficult to explore and find data sets of interest and manage the data. Thus, in the context of big data, visualization is critical for checking data consistency and validity, identifying gaps in data coverage, searching for data relevant to an analyst's use cases, and choosing input parameters for analysis. Downloading the data and exploring it on an analyst's desktop using traditional tools are no longer feasible due to the size of the data. This study describes the design and implementation of a visualization system that addresses the problems associated with data exploration within the context of the BDC. Finally, the visualization system is based on a JavaScript front end communicating via REST with a back end web server.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

NASA NTRS1986

Photocurrent Imaging Detects Solar-Module Defects

Raster-scanned laser beam excites photocurrents in thin-film amorphous silicon devices. Solar-cell laser scanner uses two galvanometer-driven mirrors to scan laser-beam spot over surface of module or cell under test. Position signals from scan controllers used to index storage of photocurrent signal data in scan-converter image memory. Stored image displayed on television monitor.

Kim, Q.↗

DOE OSTI2022

Accelerating shared file checkpoint with local burst buffers

A data management system and method for accelerating shared file checkpointing. Written application data is aggregated in an application data file created in a local burst buffer memory at a compute node, and an associated data mapping built index to maintain information related to the offsets into a shared file at which segments of the application data is to be stored in a parallel file system, and where in the buffer those segments are located. The node asynchronously transfers a data file containing the application data and the associated data mapping index to a file server for shared file storage. The data management system and method further accelerates shared file checkpointing in which a shared file, together with a map file that specifies how the shared file is to be distributed, is asynchronously transferred to local burst buffer memories at the nodes to accelerate reading of the shared file.

Gooding, Thomas↗

NASA NTRS1987

Requirements and principles for the implementation and construction of large-scale geographic information systems

This paper provides a brief survey of the history, structure and functions of 'traditional' geographic information systems (GIS), and then suggests a set of requirements that large-scale GIS should satisfy, together with a set of principles for their satisfaction. These principles, which include the systematic application of techniques from several subfields of computer science to the design and implementation of GIS and the integration of techniques from computer vision and image processing into standard GIS technology, are discussed in some detail. In particular, the paper provides a detailed discussion of questions relating to appropriate data models, data structures and computational procedures for the efficient storage, retrieval and analysis of spatially-indexed data.

Smith, Terence R.↗

DOE OSTI2021

MOSIQS: Persistent Memory Object Storage With Metadata Indexing and Querying for Scientific Computing

Scientific applications often require high-bandwidth shared storage to perform joint simulations and collaborative data analytics. Shared memory pools provide a chance to satisfy such needs. Recently, a high-speed network such as Gen-Z utilizing persistent memory (PM) offers an opportunity to create a shared memory pool connected to compute nodes. However, there are several challenges to use scientific applications on the shared memory pool directly such as scalability, failure-atomicity, and lack of scientific metadata-based search and query. In this paper, we propose MOSIQS, a persistent memory object storage framework with metadata indexing and querying for scientific computing. We design MOSIQS based on the key idea that memory objects on PM pool can live beyond the application lifetime and can become the sharing currency for applications and scientists. MOSIQS provides an aggregate memory pool atop an array of persistent memory devices to store and access memory objects to accelerate scientific computing. MOSIQS uses a lightweight persistent memory key-value store to manage the metadata of memory objects, which enables memory object sharing. To facilitate metadata search and query over millions of memory objects resident on memory pool, we introduce Group Split and Merge (GSM), a novel persistent index data structure designed primarily for scientific datasets. GSM splits and merges dynamically to minimize the query search space and maintains low query processing time while overcoming the index storage overhead. MOSIQS is implemented on top of PMDK. We evaluate the proposed approach on many-core server with an array of real PM devices. Experimental results show that MOSIQS gains a 100% write performance improvement and executes multi-attribute queries efficiently with 2.7× less index storage overhead offering significant potential to speed up scientific computing applications.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2017

GUIDE: A Scalable Information Directory Service to Collect, Federate, and Analyze Logs for Operational Insights into a Leadership HPC Facility

In this paper, we describe the GUIDE framework to collect, federate, and analyze log data from the Oak Ridge Leadership Computing Facility's (OLCF), and how we use it derive insights into center operations. We collectsystem logs and extract monitoring data at every level of the various OLCF subsystems, and have developed a suite of pre-processing tools to make the raw data consumable. The cleansed logs are then ingested and federated into a central, scalable data warehouse, Splunk, that offers storage, indexing, querying, and visualization capabilities. We have further developed and deployed a set of analytics tools to analyze these multiple disparate log streams in concert, and derive operational insights. We describe our experience from developing and deploying the GUIDE infrastructure, and deriving valuable insights on the various subsystems, based on two years of operations in the production OLCF environment.

Vazhkudai, Sudharshan↗

NASA NTRS1975

A system for the input and storage of data in the Besm-6 digital computer

Computer programs used for the decoding and storage of large volumes of data on the the BESM-6 computer are described. The following factors are discussed: the programming control language allows the programs to be run as part of a modular programming system used in data processing; data control is executed in a hierarchically built file on magnetic tape with sequential index storage; and the programs are not dependent on the structure of the data.

Schmidt, K.↗

NASA NTRS1998

Isosurface Extraction in Time-Varying Fields Using a Temporal Hierarchical Index Tree

Many high-performance isosurface extraction algorithms have been proposed in the past several years as a result of intensive research efforts. When applying these algorithms to large-scale time-varying fields, the storage overhead incurred from storing the search index often becomes overwhelming. this paper proposes an algorithm for locating isosurface cells in time-varying fields. We devise a new data structure, called Temporal Hierarchical Index Tree, which utilizes the temporal coherence that exists in a time-varying field and adoptively coalesces the cells' extreme values over time; the resulting extreme values are then used to create the isosurface cell search index. For a typical time-varying scalar data set, not only does this temporal hierarchical index tree require much less storage space, but also the amount of I/O required to access the indices from the disk at different time steps is substantially reduced. We illustrate the utility and speed of our algorithm with data from several large-scale time-varying CID simulations. Our algorithm can achieve more than 80% of disk-space savings when compared with the existing techniques, while the isosurface extraction time is nearly optimal.

Shen, Han-Wei↗

NASA NTRS1976

Compilation of fatigue, fatigue-crack propagation, and fracture data for 2024 and 7075 aluminum, Ti-6Al-4V titanium, and 300M steel. Volume 1: Description of data and data storage on magnetic tape. Volume 2: Data tape (7-track magnetic tape)

Fatigue, fatigue-crack-propagation, and fracture data compiled and stored on magnetic tape are documented. Data for 202 and 7075 aluminum alloys, Ti-6Al-4V titanium alloy, and 300M steel are included in the compilation. Approximately 4,500 fatigue, 6,500 fatigue-crack-propagation, and 1,500 fracture data points are stored on magnetic tape. Descriptions of the data, an index to the data on the magnetic tape, information on data storage format on the tape, a listing of all data source references, and abstracts of other pertinent test information from each data source reference are included.

Rice, R. C.↗

NASA NTRS2015

Evaluation of a Model-Based Groundwater Drought Indicator in the Conterminous U.S.

Monitoring groundwater drought using land surface models is a valuable alternative given the current lack of systematic in situ measurements at continental and global scales and the low resolution of current remote sensing based groundwater data. However, uncertainties inherent to land surface models may impede drought detection, and thus should be assessed using independent data sources. In this study, we evaluated a groundwater drought index (GWI) derived from monthly groundwater storage output from the Catchment Land Surface Model (CLSM) using a GWI similarly derived from in situ groundwater observations. Groundwater observations were obtained from unconfined or semi-confined aquifers in eight regions of the central and northeastern U.S. Regional average GWI derived from CLSM exhibited strong correlation with that from observation wells, with correlation coefficients between 0.43 and 0.92. GWI from both in situ data and CLSM was generally better correlated with the Standard Precipitation Index (SPI) at 12 and 24 month timescales than at shorter timescales, but it varied depending on climate conditions. The correlation between CLSM derived GWI and SPI generally decreases with increasing depth to the water table, which in turn depends on both bedrock depth (a CLSM parameter) and mean annual precipitation. The persistence of CLSM derived GWI is spatially varied and again shows a strong influence of depth to groundwater. CLSM derived GWI generally persists longer than GWI derived from in situ data, due at least in part to the inability of coarse model inputs to capture high frequency meteorological variability at local scales. The study also showed that groundwater can have a significant impact on soil moisture persistence where the water table is shallow. Soil moisture persistence was estimated to be longer in the eastern U.S. than in the west, in contrast to previous findings that were based on models that did not represent groundwater. Assimilation of terrestrial water storage data from the Gravity Recovery and Climate Experiment (GRACE) satellite mission improved the correlation between CLSM based regional average GWI and that based on in situ data in six of the eight regions. Practical issues regarding the application of GRACE assimilated groundwater storage for drought detection are discussed. An important conclusion of this study is that model parameters that control the depth to the water table, including bedrock depth, strongly influence the evolution and persistence of simulated groundwater and require careful configuration for drought monitoring.

Evaluation↗

NASA NTRS2014

Evaluation of a Model-Based Groundwater Drought Indicator in the Conterminous U.S.

hydrology↗

DOE OSTI2021

Tree Diversity, Site Index, and Carbon Storage Decrease With Aridity in Douglas-Fir Forests in Western Canada

Forests are important for biodiversity, timber production and carbon accumulation, but these ecosystem services may be impacted by climate change. Field data collected from individual forest types occurring across a climatic gradient can contribute to forecasting these consequences. We examined how changes in temperature, precipitation and aridity affect ecosystem services in 23 mature Douglas-fir (Pseudotsuga menziesii) forests in nine climatic regions across a 900 km gradient in British Columbia, Canada. Using Canadian National Forest Inventory methodology, we assessed richness and diversity of plant functional groups, site index, and above- and below-ground carbon stocks. As aridity increased, ecosystem-level tree species richness declined on average from four to one species, Douglas-fir site index declined from 30 to 15 m, and ecosystem carbon storage decreased from 565 to 222 Mg ha –1 . Tree species richness was positively and herb species richness negatively correlated with carbon storage. Carbon storage by ecosystem compartment was largest in aboveground live tree biomass, declining in the following order: mineral soils > coarse woody debris and dead standing trees > forest floor > small and fine woody debris > understory plants. Mineral soil carbon at depths of 0-15 cm, 15-35 cm, and 35-55 cm increased with increasing mean annual precipitation and decreasing aridity. Our results indicate that as aridity increases and precipitation decreases, tree species richness, site index and carbon storage in existing Douglas-fir forests declines. However, assisted or natural migration of Douglas-fir into more humid regions could be associated with more diverse, productive, carbon-rich forests. This study informs carbon stock vulnerability and provides empirical data essential for carbon stock forecasts.

54 ENVIRONMENTAL SCIENCES↗

NASA NTRS1994

Holographic testing of composite propfans for a cruise missile wind tunnel model

Each of the approximately 90 composite propfan blades constructed for a 55 percent scale cruise missile wind tunnel model were holographically tested to obtain natural frequencies and mode shapes. These data were used not only for quality assurance, but also to select sets of similar blades for each blade row. Presented along with the natural frequency data is a description of a computer-based image processing system developed to supplement the photographic based system for holographic image analysis and storage. The new system is quicker and cheaper, the holograms are indexed better, and several engineers can access the data simultaneously. The only negative effect is a slight reduction in image resolution, which does not influence the end use.

Miller, Christopher J.↗