Search NASA⌕ Search

SEARCH · Search NASA

Results for “data access”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

Monthly Fossil-Fuel CO2 Emissions: Isomass of Emissions Gridded by One Degree Latitude by One Degree Longitude (1950 - 2007) (V. 2010)

The basic data provided in these data files are derived from time series of Global, Regional, and National Fossil-Fuel CO2 Emissions (http://cdiac.ess-dive.lbl.gov/trends/emis/overview_2013.html), the references therein, and the methodology described in Andres et al. (2011). The data accessible here take these tabular, national, mass-emissions data, multiply them by stable carbon isotopic signature (del 13C) as described in Andres et al. (2000), and distribute them spatially on a one degree latitude by one degree longitude grid. The within-country spatial distribution is achieved through a fixed population distribution as reported in Andres et al. (1996) for years prior to 1990 and a variable population distribution for later years (Andres et al. 2016). Note that the mass-emissions data used here are based on fossil-fuel consumption estimates as these are more representative of within country emissions than fossil-fuel production estimates (see http://cdiac.ess-dive.lbl.gov/faq.html#Q10 for a description why emission totals based upon consumption differ from those based upon production). The monthly, isotopic (δ 13C) fossil-fuel CO2 emissions estimates from 1950-2013 provided in this database are derived from time series of global, regional, and national fossil-fuel CO2 emissions (Boden et al. 2016), the references therein, and the methodology described in Andres et al. (2011). The data accessible here take these tabular, national, mass-emissions data, multiply them by stable carbon isotopic signatures (δ 13C) as described in Andres et al. (2000), and distribute them spatially on a one degree latitude by one degree longitude grid. The within-country spatial distribution is achieved through a fixed population distribution as reported in Andres et al. (1996). Note that the mass-emissions data used here are based on fossil-fuel consumption estimates as these are more representative of within country emissions than fossil-fuel production estimates (see http://cdiac.ess-dive.lbl.gov/faq.html#Q10 for a description why emission totals based upon consumption differ from those based upon production).

54 ENVIRONMENTAL SCIENCES↗

Closing the data life cycle: using information management in macrosystems ecology research

An important goal of macrosystems ecology research is to advance understanding of ecological systems at both fine and broad temporal and spatial scales. Our premise in this paper is that such projects require information management that is integrated into projects from their inception. Such efforts will lead to improved communication and sharing of knowledge among diverse project participants, better science outcomes, and more open science. We promote "closing the data life cycle" by publishing well-documented data sets, which allows for re-use of data to answer new and different questions from the ones conceived by the original projects. The practice of documenting and submitting data sets to publicly accessible data repositories ensures that research results and data are accessible to and useable by other researchers, thus fostering open science. Ecologists are often not familiar with the information management tools and requirements to effectively preserve data, however, and receive little institutional or professional incentive to do so. This paper describes recommended steps to these ends, and gives examples from current macrosystem ecology projects of why information management is so critical to ensuring that scientific results can be both reproduced and data shared for future use.

Ruegg, Janine↗

ESS-DIVE Unoccupied Aerial Systems (UAS) Reporting Format v1

Here we present documentation of the ESS-DIVE reporting format for Unoccupied Aerial System (UAS) data and metadata. This reporting format provides guidance to data contributors on how to store data to maximize their discoverability, facilitate their efficient reuse, and add value to individual datasets. For data users, the reporting format will better allow data repositories to optimize data search and extraction, and more readily integrate similar data into harmonized synthesis products. The reporting format provides templates and guidance for the reporting of metadata for UAS experimental campaigns, individual flights, platform and sensor description. To improve data access and discoverability, the reporting format proposes a data description scheme of Levels based on the degree of processing, where Level 0 includes raw data, through to Level 3 being derived data end products. A range of examples of data types for each Level are given, with suggested file naming schemes. The reporting format presented here is intended to form a foundation for future development that will accommodate new UAS technologies and approaches to data access and use in the future. The reporting format documentation is maintained and updated on the ESS-DIVE Community Space GitHub at https://github.com/ess-dive-community/essdive-uas. This data package is the first published version of this reporting format, and comprises a zip file of the complete content of https://github.com/ess-dive-community/essdive-uas v1.0. The zip contains the reporting format description, instructions and variable definitions in GitHub markdown language (*.md) and metadata templates in csv format. The reporting format is designed to be compatible with other ESS-DIVE formats, and it is specifically recommended that this reporting format be used in conjunction with the File-level metadata (FLMD) and comma separated values (csv) reporting formats for submission to the ESS-DIVE repository.

54 ENVIRONMENTAL SCIENCES↗

LSE-319: LSST Science Platform Vision Document

This document defines and describes the "LSST Science Platform," a set of integrated web applications and services deployed at the LSST Data Access Centers (DACs) through which the scientific community will access, visualize, subset, and perform next-to-the-data analysis of the data collected by the Large Synoptic Survey Telescope (LSST). These services can be broken down to three different "Aspects": a web Portal, designed to provide essential data access and visualization services through a simple-to-use website, a Notebook environment, that will provide a Jupyter Notebook-like interface, based on JupyterLab, enabling next-to-the-data analysis, and an extensive set of Web APIs that the users will be able to use to remotely examine the LSST data set using tools they're already familiar with. This document lays out the high-level vision for the aforementioned Aspects and some associated backend services. It is intentionally brief, and meant to generally guide the flow-down of requirements and development of product specifications, prioritization, and plans for the Agile development of the relevant elements of the DM system.

79 ASTRONOMY AND ASTROPHYSICS↗

Applying Service-Oriented Architecture to Archiving Data in Control and Monitoring Systems

Current trends in the architecture of software systems focus our attention on building systems using a set of loosely coupled components, each providing a specific functionality known as service. It is not much different in control and monitoring systems, where a functionally distinct sub-system can be identified and independently designed, implemented, deployed and maintained. One functionality that renders itself perfectly to becoming a service is archiving the history of the system state. The design of such a service and our experience of using it are the topic of this article. The service is built with responsibility segregation in mind, therefore, it provides for reducing data processing on the data viewer side and separation of data access and modification operations. The service architecture and the details concerning its data store design are discussed. An implementation of a service client capable of archiving EPICS process variables (PV) and LabVIEW shared variables is presented. Data access tools, including a browser-based data viewer and a mobile viewer, are also presented.

Nogiec, J. M.↗

High-Fidelity Solar Irradiance Data: Simple Access to State-of-the-Art Information Accelerates Southeast Asia's Clean Energy Economic Transformation

High quality, robust, and reliable renewable energy resource data is foundational to climate-smart decision making, evidence-based policy planning, and clean energy investment mobilization. USAID and NREL, through the Advanced Energy Partnership for Asia, are expanding access to this critical resource data by providing free, high-fidelity solar resource data for Southeast Asia through the RE Data Explorer platform. This brief highlights several ways the Southeast Asia solar resource data has been used for power system planning and project development in the region.

Advanced Energy Partnership for Asia↗

Preparing for long-term data preservation and access in CMS

The data collected by the LHC experiments are unique and present an opportunity and a challenge for a long-term preservation and re-use. The CMS experiment has defined a policy for the data preservation and access to its data and is starting its implementation. This note describes the driving principles of the policy and summarises the actions and activities which are planned in the starting phase of the project.

Lassila-Perini, Kati↗

Scaling pair count to next galaxy surveys

ABSTRACT Counting pairs of galaxies or stars according to their distance is at the core of real-space correlation analyses performed in astrophysics and cosmology. Upcoming galaxy surveys (LSST, Euclid) will measure properties of billions of galaxies challenging our ability to perform such counting in a minute-scale time relevant for the usage of simulations. The problem is only limited by efficient access to the data, hence belongs to the big data category. We use the popular Apache Spark framework to address it and design an efficient high-throughput algorithm to deal with hundreds of millions to billions of input data. To optimize it, we revisit the question of non-hierarchical sphere pixelization based on cube symmetries and develop a new one dubbed the ‘Similar Radius Sphere Pixelization’ (SARSPix) with very close to square pixels. It provides the most adapted indexing over the sphere for all distance-related computations. Using LSST-like fast simulations, we compute autocorrelation functions on tomographic bins containing between a hundred million to one billion data points. In each case, we achieve the construction of a standard pair-distance histogram in about 2 min, using a simple algorithm that is shown to scale, over a moderate number of nodes (16–64). This illustrates the potential of this new techniques in the field of astronomy where data access is becoming the main bottleneck. They can be easily adapted to other use-cases as nearest-neighbours search, catalogue cross-match or cluster finding. The software is publicly available from https://github.com/astrolabsoftware/SparkCorr.

79 ASTRONOMY AND ASTROPHYSICS↗

NVMalloc: Exposing an Aggregate SSD Store as a Memory Partition in Extreme-Scale Machines

DRAM is a precious resource in extreme-scale machines and is increasingly becoming scarce, mainly due to the growing number of cores per node. On future multi-petaflop and exaflop machines, the memory pressure is likely to be so severe that we need to rethink our memory usage models. Fortunately, the advent of non-volatile memory (NVM) offers a unique opportunity in this space. Current NVM offerings possess several desirable properties, such as low cost and power efficiency, but suffer from high latency and lifetime issues. We need rich techniques to be able to use them alongside DRAM. In this paper, we propose a novel approach for exploiting NVM as a secondary memory partition so that applications can explicitly allocate and manipulate memory regions therein. More specifically, we propose an NVMalloc library with a suite of services that enables applications to access a distributed NVM storage system. We have devised ways within NVMalloc so that the storage system, built from compute node-local NVM devices, can be accessed in a byte-addressable fashion using the memory mapped I/O interface. Our approach has the potential to re-energize out-of-core computations on large-scale machines by having applications allocate certain variables through NVMalloc, thereby increasing the overall memory capacity available. Our evaluation on a 128-core cluster shows that NVMalloc enables applications to compute problem sizes larger than the physical memory in a cost-effective manner. It can bring more performance/efficiency gain with increased computation time between NVM memory accesses or increased data access locality. In addition, our results suggest that while NVMalloc enables transparent access to NVM-resident variables, the explicit control it provides is crucial to optimize application performance.

Computer Science↗

Regression for Assessing Contaminant Plume Trends

The Department of Energy’s (DOE) Hanford site has significant dissolved-phase groundwater contaminant plumes, for which remediation activities are ongoing. Assessing concentration trends is critical for making decisions regarding remedy implementation or performance. The SOCRATES single-page web application provides a quality-assured framework for data access and consistent data analytics using standard methods. To provide trend information for remedial decisions, statistical analyses based on ordinary least squares (OLS) are already supported by SOCRATES. The presence of censored data, however, can skew the result of an OLS regression. Other methods, such as the Akritas-Theil-Sen (ATS) or Tobit regressions can be more powerful. These two regression methods are used by Hanford site personnel to analyze groundwater contaminant concentration data, which often include nondetects (censored data). The ATS method uses comparisons of observation-pair ranks and the Tobit method uses a maximum likelihood estimation method. The Tobit method can also accommodate covariates, such as river stage or groundwater level fluctuations. Both ATS and Tobit regressions can be performed with the statistics software R, but to incorporate these methods into SOCRATES, their R code was ported into JavaScript. For quality assurance verification, the new code was tested with hexavalent chromium concentration data from the Hanford 100-K West Area and compared to regression analyses that were previously conducted on this data using R.

100-K Area↗

NVMalloc: Exposing an Aggregate SSD Store as a Memory Partition in Extreme-Scale Machines

DRAM is a precious resource in extreme-scale machines and is increasingly becoming scarce, mainly due to the growing number of cores per node. On future multi-petaflop and exaflop machines, the memory pressure is likely to be so severe that we need to rethink our memory usage models. Fortunately, the advent of non-volatile memory (NVM) offers a unique opportunity in this space. Current NVM offerings possess several desirable properties, such as low cost and power efficiency, but suffer from high latency and lifetime issues. We need rich techniques to be able to use them alongside DRAM. In this talk, we present a novel approach for exploiting NVM as a secondary memory partition so that applications can explicitly allocate and manipulate memory regions therein. More specifically, we propose an NVMalloc library with a suite of services that enables applications to access a distributed NVM storage system. We have devised ways within NVMalloc so that the storage system, built from compute node-local NVM devices, can be accessed in a byte-addressable fashion using the memory mapped I/O interface. Our approach has the potential to re-energize out-of-core computations on largescale machines by having applications allocate certain variables through NVMalloc, thereby increasing the overall memory capacity available. Our evaluation on a 128-core cluster shows that NVMalloc enables applications to compute problem sizes larger than the physical memory in a cost-effective manner. It can bring more performance/efficiency gain with increased computation time between NVM memory accesses or increased data access locality. In addition, our results suggest that while NVMalloc enables transparent access to NVM-resident variables, the explicit control it provides is crucial to optimize application performance.

99 GENERAL AND MISCELLANEOUS↗

Powering America’s Non-Powered Dams, One Byte at a Time

Compared with other hydropower development opportunities, powering underutilized water infrastructure provides an attractive option for expanding clean energy in the 21st century. Research conducted by the U.S. Department of Energy’s (DOE) Oak Ridge National Laboratory (ORNL) targets hydropower development at non-powered dams (NPD) from several angles and is increasingly looking to new and improved data sources to inform decision-making. Sponsored by the DOE Water Power Technologies Office (WPTO), ORNL’s latest efforts build on a decade of work to advance understanding and data access for improved decision-making. Outcomes of the most recent research are geared toward better informing NPD stakeholders across a wide spectrum of interests. Furthermore, these outcomes are paving the way for additional federal investments to power NPDs in the U.S., including investments in innovative new technologies that incorporate ecological and social objectives while achieving cost reductions, commercialization and deployment. Here, this research centers around improving the breadth and accessibility of NPD data. These new digital information tools and associated tools can lead to a better decision-making process and support new hydropower development, helping to power America’s NPDs, one “byte” at a time.

13 HYDRO ENERGY↗

Assessing the Wind Energy Potential in Bangladesh: Enabling Wind Energy Development with Data Products

USAID Bangladesh and the U.S. Department of Energy’s National Renewable Energy Laboratory partnered with the Government of Bangladesh to develop a national wind resource assessment. The assessment used sophisticated resource modeling that was validated by a ground measurement campaign. Results from the project included a long-term, correlated wind data set; validated high-resolution wind resource maps; and publicly available data accessible through the RE Data Explorer (https://www.re-explorer.org/). The Renewable Energy (RE) Data Explorer allows users to access and download the Bangladesh wind resource assessment data and related geographic information system (GIS) data sets and perform customized technical potential analyses. Figure ES-1 is an example wind resource map for Bangladesh that can be created with the RE Data Explorer. This project improved upon existing global data sets by using best-in-class modeling techniques and analysis. Global models are good first steps in predicting wind resources in various parts of the world, but they do not generate the accuracy needed to reduce project risk and stimulate renewable investment. This project improved the quality of modeled wind resource data for Bangladesh. In addition to the modeling effort, the project team used a multi-year, local data-collection campaign to validate the model and further improve the accuracy of the data sets. The measurement campaign consisted of nine meteorological sites representing all geographical regions of the country. Seven meteorological towers and one remote-sensing, sonic detection and ranging unit (deployed at two sites) collected the data. Site-selection criteria focused on geographic diversity and proximity to existing transmission lines. Figure ES-1 shows measurement locations. The measurement campaign spanned June 2014 through December 2017. The results of the Bangladesh wind resource assessment will help Bangladesh overcome significant energy challenges. Its energy sector suffers from power shortages, increasing demand, decreasing domestic natural gas reserves, and inadequate transmission infrastructure. As part of a comprehensive plan to overcome these challenges, Bangladesh has committed that 10% of the total generation capacity will be renewable energy by 2021 (Power Division 2016). High-quality renewable energy resource data and other GIS data, such as those developed in this assessment, are important if Bangladesh wishes to reach its 10% renewable energy capacity target. These data support informed decision making, ranging from policy and investment decisions to reliable power sector planning. Specifically, the Bangladesh wind resource assessment will help reduce technical risk and encourage private-sector interest in the nascent wind market in Bangladesh. In addition, the Government of Bangladesh may use the results of the wind resource assessment to develop well-designed policies that could encourage investment in wind energy. This report provides a comprehensive description of the Bangladesh wind resource assessment, including details on the modeling approach and methods, instrumentation, data quality-control techniques, and resulting data sets.

17 WIND ENERGY↗

ARCHIE: Data Analysis Acceleration with Array Caching in Hierarchical Storage

© 2018 IEEE. Scientific data analysis typically involves reading massive amounts of data that was generated by simulations, experiments, and observations. Performance of reading such large volumes of data from disk-based file systems is often poor because of the slow and mechanical components in the disks. Recent supercomputing systems are adding non-volatile storage layers in a hierarchy to handle the performance gap between fast main memory and slow disk-based storage. Software libraries for managing this hierarchy not only need efficient reading of data but also reduce user-involvement for cross-layer data movement. Furthermore, these libraries need to support array data access patterns into hierarchical storage management as scientific data is often organized in array-based data structures. Existing software typically manage individual storage layers requiring significant manual process in moving data among them. In this paper, we introduce a new array caching in hierarchical storage (ARCHIE) to accelerate array data analysis in a seamless fashion. ARCHIE evaluates array access patterns and prefetches data with array semantics between storage layers. Our evaluation shows that ARCHIE outperforms state-of-the-art file systems, i.e., Lustre and DataWarp, on a production supercomputing system by up to 5.8× in accessing data by scientific analysis applications.

Dong, Bin↗

SU-D-BRD-07: Automatic Patient Data Audit and Plan Quality Check to Support ARIA and Eclipse

Purpose: To ensure patient safety and treatment quality in RT departments that use Varian ARIA and Eclipse, we developed a computer software system and interface functions that allow previously developed electron chart checking (EcCk) methodologies to support these Varian systems. Methods: ARIA and Eclipse store most patient information in its MSSQL database. We studied the contents in the hundreds database tables and identified the data elements used for patient treatment management and treatment planning. Interface functions were developed in both c-sharp and MATLAB to support data access from ARIA and Eclipse servers using SQL queries. These functions and additional data processing functions allowed the existing rules and logics from EcCk to support ARIA and Eclipse. Dose and structure information are important for plan quality check, however they are not stored in the MSSQL database but as files in Varian private formats, and cannot be processed by external programs. We have therefore implemented a service program, which uses the DB Daemon and File Daemon services on ARIA server to automatically and seamlessly retrieve dose and structure data as DICOM files. This service was designed to 1) consistently monitor the data access requests from EcCk programs, 2) translate the requests for ARIA daemon services to obtain dose and structure DICOM files, and 3) monitor the process and return the obtained DICOM files back to EcCk programs for plan quality check purposes. Results: EcCk, which was previously designed to only support MOSAIQ TMS and Pinnacle TPS, can now support Varian ARIA and Eclipse. The new EcCk software has been tested and worked well in physics new start plan check, IMRT plan integrity and plan quality checks. Conclusion: Methods and computer programs have been implemented to allow EcCk to support Varian ARIA and Eclipse systems. This project was supported by a research grant from Varian Medical System.

60 APPLIED LIFE SCIENCES↗

Why don’t we share data?

We are constantly hearing suggestions to make all data gathered in biology experiments available online. This is an appealing idea because most data that we collect from experiments never sees the light of day. A smattering of our data appears in papers, of course, but we all recognize that this is usually a highly selected subset of all that is collected, intended to support the story that is being touted at the moment. If we could somehow make all of our data available to the community, the idea goes, biological progress would be greatly accelerated. Despite the appeal of making all biological data accessible, there are enormous hurdles that currently make it impractical. For one, sharing all data requires that we agree on a set of standards. This is perhaps reasonable for large-scale automated technologies, such as microarrays, but the logistics of converting every western blot, ELISA and protein assay into a structured and accessible data format would be a nightmare -- and probably not worth the effort.

59 BASIC BIOLOGICAL SCIENCES↗

SSD-optimized workload placement with adaptive learning and classification in HPC environments

In recent years, non-volatile memory devices such as SSD drives have emerged as a viable storage solution due to their increasing capacity and decreasing cost. Due to the unique capability and capacity requirements in large scale HPC (High Performance Computing) storage environment, a hybrid configuration (SSD and HDD) may represent one of the most available and balanced solutions considering the cost and performance. Under this setting, effective data placement as well as movement with controlled overhead become a pressing challenge. In this paper, we propose an integrated object placement and movement framework and adaptive learning algorithms to address these issues. Specifically, we present a method that shuffle data objects across storage tiers to optimize the data access performance. The method also integrates an adaptive learning algorithm where realtime classification is employed to predict the popularity of data object accesses, so that they can be placed on, or migrate between SSD or HDD drives in the most efficient manner. We discuss preliminary results based on this approach using a simulator we developed to show that the proposed methods can dynamically adapt storage placements and access pattern as workloads evolve to achieve the best system level performance such as throughput.

Wan, Lipeng↗

The SNS/HFIR Web Portal System for SANS

In a busy world, continuing with the status-quo, to do things the way we are already familiar, often seems to be the most efficient way to conduct our work. We look for the value-add to decide if investing in a new method is worth the effort. How shall we evaluate if we have reached this tipping point for change? For contemporary researchers, understanding the properties of the data is a good starting point. The new generation of neutron scattering instruments being built are higher resolution and produce one or more orders of magnitude larger data than the previous generation of instruments. For instance, we have grown out of being able to perform some important tasks with our laptops – the data are too big and the computations would simply take too long. These large datasets can be problematic as facility users now begin to grapple with many of the same issues faced by more established computing communities. These issues include data access, management, and movement, data format standards, distributed computing, and collaboration among others. The Neutron Science Portal has been architected, designed, and implemented to provide users with an easy-to-use interface for managing and processing data, while also keeping an eye on meeting modern cybersecurity requirements imposed on institutions. The cost of entry for users has been lowered by utilizing a web interface providing access to backend portal resources. Users can browse or search for data which they are allowed to see, data reduction applications can be run without having to load the software, sample activation calculations can be performed for SNS and HFIR beamlines, McStas simulations can be run on TeraGrid and ORNL computers, and advanced analysis applications such as those being produced by the DANSE project can be run. Behind the scenes is a “live cataloging” system which automatically catalogs and archives experiment data via the data management system, and provides proposal team members access to their experiment data. The complexity of data movement and utilizing distributed computing resources has been taken care on behalf of users. Collaboration is facilitated by providing users a read/writeable common area, shared across all experiment team members. To date, these shared areas are the fastest growing data spaces. The portal currently has over 370 registered users, almost 7TB of experiment and user data, approximately 1,000,000 files cataloged, and had almost 10,000 unique visits last year. Future directions for enhancing portal robustness include examining how to mirror data and portal services, better facilitation of collaborations via virtual organizations, enhancing disconnected service via “thick client” applications, and better inter-facility connectivity to support cross-cutting research. The portal has established itself in the SNS/HFIR user community, and the development team strives to continue to improve the quality of features and services provided in order to better serve the community.

data↗