Search NASA⌕ Search

SEARCH · Search NASA

Results for “data access”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 469 records · Page 26

Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL

Oak Ridge National Laboratory (ORNL) experimental neutron science facilities produce 1.2 TB a day of raw event-based data that is stored using the standard metadata-rich NeXus schema built on top of the HDF5 file format. Performance of several data reduction workflows is largely determined by the amount of time spent on the loading and processing algorithms in Mantid, an open-source data analysis framework used across several neutron sciences facilities around the world. The present work introduces new data management algorithms to address identified input output (I/O) bottlenecks on Mantid. First, we introduce an in-memory binary-tree metadata index that resemble NeXus data access patterns to provide a scalable search and extraction mechanism. Second, data encapsulation in Mantid algorithms is optimally redesigned to reduce the total compute and memory runtime footprint associated with metadata I/O reconstruction tasks. Results from this work show speed ups in wall-clock time on ORNL data reduction workflows, ranging from 11% to 30% depending on the complexity of the targeted instrument-specific data. Nevertheless, we highlight the need for more research to address reduction challenges as experimental data volumes increase.

Godoy, William↗

Designing a graphics processing unit accelerated petaflop capable lattice Boltzmann solver: Read aligned data layouts and asynchronous communication

The lattice Boltzmann method is a well-established numerical approach for complex fluid flow simulations. Recently, general-purpose graphics processing units (GPUs) have become available as high-performance computing resources at large scale. We report on designing and implementing a lattice Boltzmann solver for multi-GPU systems that achieves 1.79 PFLOPS performance on 16,384 GPUs. To achieve this performance, we introduce a GPU compatible version of the so-called bundle data layout and eliminate the halo sites in order to improve data access alignment. Furthermore, we make use of the possibility to overlap data transfer between the host central processing unit and the device GPU with computing on the GPU. As a benchmark case, we simulate flow in porous media and measure both strong and weak scaling performance with the emphasis being on large-scale simulations using realistic input data.

Computer Science↗

Best practice data life cycle approaches for the life sciences

Throughout history, the life sciences have been revolutionised by technological advances; in our era this is manifested by advances in instrumentation for data generation, and consequently researchers now routinely handle large amounts of heterogeneous data in digital formats. The simultaneous transitions towards biology as a data science and towards a ‘life cycle’ view of research data pose new challenges. Researchers face a bewildering landscape of data management requirements, recommendations and regulations, without necessarily being able to access data management training or possessing a clear understanding of practical approaches that can assist in data management in their particular research domain. Here we provide an overview of best practice data life cycle approaches for researchers in the life sciences/bioinformatics space with a particular focus on ‘omics’ datasets and computer-based data processing and analysis. We discuss the different stages of the data life cycle and provide practical suggestions for useful tools and resources to improve data management practices.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Online Analytics for Remedy Support at DOE Environmental Management Sites

Environmental data is important for managing environmental restoration/waste site remediation, planning of monitoring efforts, addressing climate resilience, and engaging with stakeholders and regulators. A major challenge is how to manage the many different types and the large volume of environmental data in a way that allows practitioners and site managers to understand data implications and support decisions. The Suite Of Comprehensive Rapid Analysis Tools for Environmental Sites (SOCRATES, https://www.pnnl.gov/projects/socrates) is a web application that provides data access, visualization, and rapid analytics to help make sense of environmental data, support remedy decisions, and communicate information. Development of SOCRATES has been funded through the DOE Richland Operations Office (RL) to support communication and decision making for the Hanford Site, thus is only tied into Hanford environmental data. However, the capabilities of SOCRATES are more broadly applicable to DOE-EM sites engaged in environmental remediation and management. This report describes the work to develop mechanisms for bringing non-Hanford data into SOCRATES so that other DOE-EM sites could make use of the visualization and analysis capabilities to support communication and decision making related to managing environmental restoration/waste site remediation, optimization/exit strategies for pump-and-treat systems, planning monitoring efforts, addressing climate resilience, and/or engaging with stakeholders and regulators. The background, approach, data transfer formats, examples, and next steps for this new SOCRATES-EM software are described in this report.

54 ENVIRONMENTAL SCIENCES↗

Atomic Data Revisions for Transitions Relevant to Observations of Interstellar, Circumgalactic, and Intergalactic Matter

Measurements of element abundances in galaxies from astrophysical spectroscopy depend sensitively on the atomic data used. With the goal of making the latest atomic data accessible to the community, we present a compilation of selected atomic data for resonant absorption lines at wavelengths longward of 911.753 Å (the H i Lyman limit), for key heavy elements (heavier than atomic number 5) of astrophysical interest. In particular, we focus on the transitions of those ions that have been observed in the Milky Way interstellar medium (ISM), the circumgalactic medium (CGM) of the Milky Way and/or other galaxies, and the intergalactic medium (IGM). We provide wavelengths, oscillator strengths, associated accuracy grades, and references to the oscillator strength determinations. We also attempt to compare and assess the recent oscillator strength determinations. For about 22% of the lines that have updated oscillator strength values, the differences between the former values and the updated ones are ≳0.1 dex. Our compilation will be a useful resource for absorption line studies of the ISM, as well as studies of the CGM and IGM traced by sight lines to quasars and gamma-ray bursts. Studies (including those enabled by future generations of extremely large telescopes) of absorption by galaxies against the light of background galaxies will also benefit from our compilation.

79 ASTRONOMY AND ASTROPHYSICS↗

SDMS: A scientific data management system

SDMS is a data base management system developed specifically to support scientific programming applications. It consists of a data definition program to define the forms of data bases, and FORTRAN-compatible subroutine calls to create and access data within them. Each SDMS data base contains one or more data sets. A data set has the form of a relation. Each column of a data set is defined to be either a key or data element. Key elements must be scalar. Data elements may also be vectors or matrices. The data elements in each row of the relation form an element set. SDMS permits direct storage and retrieval of an element set by specifying the corresponding key element values. To support the scientific environment, SDMS allows the dynamic creation of data bases via subroutine calls. It also allows intermediate or scratch data to be stored in temporary data bases which vanish at job end.

Massena, W. A.↗

NASA's use of McIDAS technology - A data systems tool for meteorological research and applications

The Earth Science and Applications Division of the NASA Marshall Space Flight Center has been chartered to conduct research, and to develop and use space technology to gain a basic understanding of the earth processes with emphasis on atmospheric processes. An integral part of the research and development efforts has been the Man computer Interactive Data Access System (McIDAS). The McIDAS computer system has permitted integration of data from satellites, aircraft remote sensors, ground based meteorological data sources, and modeled atmospheric radiances. The result has been an increase in knowlege of mesoscale atmospheric processes and has enabled researchers to recommend improvements and suggestions for planned future remote sensing instruments.

Goodman, H. Michael↗

A Low-Cost Clustered Archive Approach for Storing Remote Sensing Data

As part of NASA's Earth Observing System (EOS) Data and Information System (EOSDIS), the Moderate Resolution Imaging Spectrometer (MODIS) Data Processing System (MODAPS) is now processing data from two instruments on the EOS flag ship spacecraft Terra and Aqua. Between the two, MODAPS is generating over 1 Terrabyte of data per day and has surpassed 3 Petabytes of total data. The bulk of the data is stored near-line in StorageTek Powerhorn tape jukeboxes. Accessing data that has been moved to tape involves submitting an order, scheduling the tape, and waiting for the data to become available. I am developing a low cost clustered archive that could enable storing a very large amount of data such as the MODIS data described above in an organized fashion on a cluster of commodity hardware using low cost SATA hard drives such that the files are directly available online. The system takes full advantage of Open Source software, using the GNULinux operating system, PostgreSQL relational database and the Apache HTTP Server. This poster session will depict my approach, the interface to the archive, a brief discussion of the internals of the system and some performance numbers from my prototyping. I will also describe various costs and benefits of this approach versus the traditional large tape jukebox approach currently in use in the EOSDIS.

Tilmes, Curt↗

QuakeSim and the Solid Earth Research Virtual Observatory

We are developing simulation and analysis tools in order to develop a solid Earth science framework for understanding and studying active tectonic and earthquake processes. The goal of QuakeSim and its extension, the Solid Earth Research Virtual Observatory (SERVO), is to study the physics of earthquakes using state-of-the-art modeling, data manipulation, and pattern recognition technologies. We are developing clearly defined accessible data formats and code protocols as inputs to simulations, which are adapted to high-performance computers. The solid Earth system is extremely complex and nonlinear resulting in computationally intensive problems with millions of unknowns. With these tools it will be possible to construct the more complex models and simulations necessary to develop hazard assessment systems critical for reducing future losses from major earthquakes. We are using Web (Grid) service technology to demonstrate the assimilation of multiple distributed data sources (a typical data grid problem) into a major parallel high-performance computing earthquake forecasting code. Such a linkage of Geoinformatics with Geocomplexity demonstrates the value of the Solid Earth Research Virtual Observatory (SERVO) Grid concept, and advances Grid technology by building the first real-time large-scale data assimilation grid.

virtual observatory↗

GES DISC Datalist Improves Earth Science Data Discoverability

At American Geophysical Union(AGU) 2016 Fall Meeting, Goddard Earth Sciences Data Information Services Center (GES DISC) unveiled a novel way to access data: Datalist. Currently, datalist is a collection of predefined data variables from one or more archived datasets, curated by our subject matter expert (SME). Our science support team has curated a predefined Hurricane Datalist and received very positive feedback from the user community. Datalist uses the same architecture our new website uses and have the same look and feel as other datasets on our web site. and also provides a one-stop shopping for data, metadata, citation, documentation, visualization and other available services. Since the last AGU Meeting, we have further developed a few new datalists corresponding to the Big Earth Data Initiative (BEDI) Societal Benefit Areas and A-Train data. We now have four datalists: Hurricane, Wind Energy, Greenhouse Gas and A-Train. We have also started working with our User Working Group members to create their favorite datalists and working with other DAAC to explore the possibility to include their products in our datalists that may also lead to a future of potential federated (cross-DAAC) datalists. Since our datalist prototype effort was a success, we are planning to make datalist operational. It's extremely important to have a common metadata model to support datalist, this will also be the foundation of federated datalist. We mapped our datalist metadata model to the unpublished UMM(Universal Metadata Model)-Var (Variable) (June version) and found that the UMM-var together with UMM-C (Collection) and possible UMM-S (Service) will meet our basic requirements. For example: Dataset shortname, and version are already specified in UMM-C, variable name, long name, units, dimensions are all specified in UMM-Var. UMM-Var also facilitates Science Keywords to allow tagging at variable level and Characteristics for optional variable characteristics. Measurements is useful for grouping of the variables and Set is promising to define datalist. And finally, the UMM-Service model to specify the available services for the variable will be very beneficial. In summary, UMM-Var, UMM-C and UMM-S are the basis of federated datalist and the development and deployment of datalist will contribute to the evolution of the UMM.

datalist↗

Directional Drilling Technology for HLW Disposal - Long Term Monitoring in the Borehole - 16079

Central Research Institute of Electric Power Industry (CRIEPI) has been conducting a project for the directional drilling and measurement/logging technologies to survey hydro-geological condition at the stage of the preliminary investigation (PI) of site selection for the deep geological disposal of HLW. 1000 m length borehole which intersects the Omagari fault was drilled by directional drilling technology in order to verify the drilling technology in Hokkaido. Long term monitoring to clarify pore water pressure and water chemistry in the drilled borehole was also developed. We set measurement intervals on a hanging wall side of a main fracture and started the measurement of pore water pressure in February 2014. We assume that long term monitoring for several boreholes will be achieved in vast areas where power supply and/or communication networks are not available at the PI stage. Thus, we have been developing an automatic data acquisition system for long term monitoring, which enables us to access data remotely. The monitoring system, which was set in Horonobe-town in Hokkaido, is an automatic data acquisition system that performs long term monitoring of pore water pressure. This paper provides an overview of the development of monitoring technologies of pore water pressure measurement in the borehole as well as the automatic data acquisition system. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Domain-Specific Type-Safe APIs for Hierarchical Scientific Data with Modern C++

General-purpose library application programming interfaces (APIs) for self-describing hierarchical scientific data storage, such as the HDF5 and NetCDF libraries, are traditionally of runtime nature. Runtime errors for entry existence and data types are typically caught later in the development process of higher-level application-specific APIs. In this paper, we propose exploiting modern C++ metaprogramming features to add compile-time type-safety to improve the interaction with a well-defined metadata-rich scientific schema in domain-specific hierarchical datasets. We tackle two aspects of common use: (i) direct data access, (ii) flexible “in-memory” index models for efficient search and data processing. The proposed APIs use C++17’s template type auto deduction features, C++11’s enum class for type-safety and C-style preprocessor macros for generative templated code. We showcase the pros and cons of our initial work on the standard NeXus schema used for annotating and storing experimental neutron scattering data at several facilities around the world on top of HDF5. Extendable compile-time type-safe APIs are a desirable feature that could be indexed by any modern integrated development environment (IDE). Hence, such APIs can help ease the learning curve for domain scientists using a less error-prone software interaction to enhance the findability of their data without resorting to a domain-specific language (DSL).

Godoy, William↗

Final Report on the Creation of the Wind Integration National Dataset (WIND) Toolkit and API: October 1, 2013 - September 30, 2015

The primary objective of this work was to create a state-of-the-art national wind resource data set and to provide detailed wind plant output data for specific sites based on that data set. Corresponding retrospective wind forecasts were also included at all selected locations. The combined information from these activities was used to create the Wind Integration National Dataset (WIND), and an extraction tool was developed to allow web-based data access.

17 WIND ENERGY↗

RBDMS, FracFocus, and Gateway Initiatives for FY15/16

Award DE-FE-0000880 from the Department of Energy to the Ground Water Protection Council (GWPC) focuses on state and federal priorities in the areas of state RBDMS development, connectivity between state systems and FracFocus.org, and data sharing initiatives across agencies. The overarching goals of these projects are to: upgrade the Risk Based Data Management System (RBDMS) to a web based application; upgrade the RBDMS user interface to make it more intuitive and easier to use increase connectivity between state RBDMS systems and FracFocus; provide outreach to states and agencies through training programs, special meetings, and state by state outreach on all deliverables; provide greater functionality for non-IT users; update and install RBDMS modules in additional state programs (IL, CA, WY, WV); and; help make data access more transparent. The primary objective is to enhance the Risk Based Data Management System (RBDMS) by adding new components relevant to current environmental topics such as hydraulic fracturing, increasing field inspection capabilities, creating linkages between FracFocus and state programs, upgrading eForm capabilities, and analyzing potential for data sharing. The GWPC will work with state agencies to develop an RBDMS module(s) that meets these needs.

02 PETROLEUM↗

Architecture of a large object-oriented database for remotely sensed data

Attention is given to the proposed Intelligent Information Fusion System (IIFS) within the framework of the Intelligent Data Management project at NASA-Goddard. IIFS is to use connectionist architectures to extract high-level attributes from incoming sensor images, and then send those characterizations and their associated ephemeris and ancillary image data to a large object-oriented database which will serve as the master catalog of sensor data. Important issues facing this project include the choice of rapid-access data structures (RADSs) for cataloging images by their high-level characterization, the implementation of efficient spatial data structures for cataloging images by their scene location, the automated population of such a database from a continuous stream of incoming ephemeris and ancillary data, and the translation and optimization of natural-language database queries so that RADSs are employed when appropriate.

Dorfman, Erik↗

Global Precipitation Measurement: Benefits of Partnering with GPM Mission - Report 2

An important goal of the Global Precipitation Measurement (GPM) mission is to maximize participation by non-NASA partners both domestic and international. A consequence of this objective is the provision for NASA to provide sufficient incentives to achieve partner buy-in and commitment to the program. NASA has identified seven specific areas in which substantive incentives will be offered: (1) partners will be offered participation in governance of GPM mission science affairs including definition of data products; (2) partners will be offered use of NASA's TDRSS capability for uplink and downlink of commands and data in regards to partner provided spacecraft; (3) partners will be offered launch support for placing partner provided spacecraft in orbit conditional upon mutually agreeable co-manifest arrangements; (4) partners will be offered direct data access at the NASA-GPM server level rather than through standard data distribution channels; (5) partners will be offered the opportunity to serve as regional data archive and distribution centers for standard GPM data products; and (6) partners will be offered the option to insert their own specialized filtering and extraction software into the GPM data processing stream or to obtain specialized subsets and products over specific areas of interest (7) partners will be offered GPM developed software tools that can be run on their platforms. Each of these incentives, either individually or in combination, represents a significant advantage to partners who may wish to participate in the GPM mission.

Stocker, Erich F.↗

Hydrological Land Surface Data and Services at NASA GES DISC

NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) is one of twelveNASA Earth Observing System (EOS) data centers that process, archive, document, and distributedata from Earth science missions and related projects. The GES DISC hosts a wide range ofremotely-sensed and model data and provides reliable and robust data access and services to usersworldwide. This presentation, focusing on hydrological land surface data, provides a summary tablefor the hydrological data holdings, along with discussions of recent updates to data and data services.

Rui, Hualan↗

Using Big Data Technologies with Earth Science Data in HDF5: HDF5 Scalable Solutions

HDF5 (Hierarchical Data Format 5) is open-source, high-performance software that consists of an abstract data model, library, and fileformat used for storing and managing extremely large and/or complex data collections. NASA Earth Observing System (EOS) Data and Information Systems use HDF5 as an archival format to store remote sensing data from EOS satellites. HDF5 is also used to store other types of Geoscience and Strophysical data, e.g., seismic data and data from Low-Frequency Array (LOFAR) radio telescopes. Data stored in HDF5 has reached tens of petabytes and is growing at an accelerated rate.With the growing amout of HDF5 Earth Science data to analyze and process, scientists need to adopt big data technologies including new storage paradigms such as cloud and object storage. To run models and perform data analysis they also need to utilizied efficient and diverse ways to access data, from high-performance computing's (HPC) Message Passing Interface (MPI) I/O and deep memory hierarchies (DMH) to non-HPC frameworks such as Apache Hadoop, Spark, and Drill. The HDF Group continually works to enable usage of big data technologies in HDF software.

Knox, Larry↗