Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Distribution Service”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

Data Access Tools And Services At The Goddard Distributed Active Archive Center (GDAAC)

As one of the largest providers of Earth Science data from the Earth Observing System, GDAAC provides the latest data from the Moderate Resolution Imaging Spectroradiometer (MODIS), Atmospheric Infrared Sounder (AIRS), Solar Radiation and Climate Experiment (SORCE) data products via GDAAC's data pool (50TB of disk cache). In order to make this huge volume of data more accessible to the public and science communities, the GDAAC offers multiple data access tools and services: Open Source Project for Network Data Access Protocol (OPeNDAP), Grid Analysis and Display System (GrADS/DODS) (GDS), Live Access Server (LAS), OpenGlS Web Map Server (WMS) and Near Archive Data Mining (NADM). The objective is to assist users in retrieving electronically a smaller, usable portion of data for further analysis. The OPeNDAP server, formerly known as the Distributed Oceanographic Data System (DODS), allows the user to retrieve data without worrying about the data format. OPeNDAP is capable of server-side subsetting of HDF, HDF-EOS, netCDF, JGOFS, ASCII, DSP, FITS and binary data formats. The GrADS/DODS server is capable of serving the same data formats as OPeNDAP. GDS has an additional feature of server-side analysis. Users can analyze the data on the server there by decreasing the computational load on their client's system. The LAS is a flexible server that allows user to graphically visualize data on the fly, to request different file formats and to compare variables from distributed locations. Users of LAS have options to use other available graphics viewers such as IDL, Matlab or GrADS. WMS is based on the OPeNDAP for serving geospatial information. WMS supports OpenGlS protocol to provide data in GIs-friendly formats for analysis and visualization. NADM is another access to the GDAAC's data pool. NADM gives users the capability to use a browser to upload their C, FORTRAN or IDL algorithms, test the algorithms, and mine data in the data pool. With NADM, the GDAAC provides an environment physically close to the data source. NADM will benefit users with mining or offer data reduction algorithms by reducing large volumes of data before transmission over the network to the user.

Pham, Long↗

Big Data Platforms as a Service: Challenges and Approach

Infrastructure-as-a-Service has revolutionized the manner in which users commission computing infrastructure. Coupled with Big Data platforms (Hadoop, Cassandra), IaaS has democratized the ability to store and process massive datasets. For users that need to customize or create new Big Data stacks, however, readily available solutions do not yet exist. Users must first acquire the necessary cloud computing infrastructure, and manually install the prerequisite software. For complex distributed services this can be a daunting challenge. To address this issue, we argue that distributed services should be viewed as a single application consisting of virtual machines. Users should no longer be concerned about individual machines or their internal organization. To illustrate this concept, we introduce Cloud-Get, a distributed package manager that enables the simple installation of distributed services in a cloud computing environment. Cloud-Get enables users to instantiate and modify distributed services, including Big Data services, using simple commands. Cloud-Get also simplifies creating new distributed services via standardized package definitions.

99 GENERAL AND MISCELLANEOUS↗

Workflows Community Summit 2022: A Roadmap Revolution

Scientific workflows have become integral tools in broad scientific computing use cases. Science discovery is increasingly dependent on workflows to orchestrate large and complex scientific experiments that range from the execution of a cloud-based data preprocessing pipeline to multi-facility instrument-to-edge-to-HPC computational workflows. Given the changing landscape of scientific computing (often referred to as a computing continuum) and the evolving needs of emerging scientific applications, it is paramount that the development of novel scientific workflows and system functionalities seek to increase the efficiency, resilience, and pervasiveness of existing systems and applications. Specifically, the proliferation of machine learning/artificial intelligence (ML/AI) workflows, need for processing large-scale datasets produced by instruments at the edge, intensification of near real-time data processing, support for long-term experiment campaigns, and emergence of quantum computing as an adjunct to HPC, have significantly changed the functional and operational requirements of workflow systems. Workflow systems now need to, for example, support data streams from the edge-to-cloud-to-HPC, enable the management of many small-sized files, allow data reduction while ensuring high accuracy, orchestrate distributed services (workflows, instruments, data movement, provenance, publication, etc.) across computing and user facilities, among others. Further, to accelerate science, it is also necessary that these systems implement specifications/standards and APIs for seamless (horizontal and vertical) integration between systems and applications, as well as enable the publication of workflows and their associated products according to the FAIR principles.

97 MATHEMATICS AND COMPUTING↗

StarView: The object oriented design of the ST DADS user interface

StarView is the user interface being developed for the Hubble Space Telescope Data Archive and Distribution Service (ST DADS). ST DADS is the data archive for HST observations and a relational database catalog describing the archived data. Users will use StarView to query the catalog and select appropriate datasets for study. StarView sends requests for archived datasets to ST DADS which processes the requests and returns the database to the user. StarView is designed to be a powerful and extensible user interface. Unique features include an internal relational database to navigate query results, a form definition language that will work with both CRT and X interfaces, a data definition language that will allow StarView to work with any relational database, and the ability to generate adhoc queries without requiring the user to understand the structure of the ST DADS catalog. Ultimately, StarView will allow the user to refine queries in the local database for improved performance and merge in data from external sources for correlation with other query results. The user will be able to create a query from single or multiple forms, merging the selected attributes into a single query. Arbitrary selection of attributes for querying is supported. The user will be able to select how query results are viewed. A standard form or table-row format may be used. Navigation capabilities are provided to aid the user in viewing query results. Object oriented analysis and design techniques were used in the design of StarView to support the mechanisms and concepts required to implement these features. One such mechanism is the Model-View-Controller (MVC) paradigm. The MVC allows the user to have multiple views of the underlying database, while providing a consistent mechanism for interaction regardless of the view. This approach supports both CRT and X interfaces while providing a common mode of user interaction. Another powerful abstraction is the concept of a Query Model. This concept allows a single query to be built form a single or multiple forms before it is submitted to ST DADS. Supporting this concept is the adhoc query generator which allows the user to select and qualify an indeterminate number attributes from the database. The user does not need any knowledge of how the joins across various tables are to be resolved. The adhoc generator calculates the joins automatically and generates the correct SQL query.

Williams, J. D.↗

Planetary Boundary Layer Height from AIRS, MERRA-2, and GPS Radio Occultation Data Products at NASA GES DISC, and Insights from Their Profiles Intercomparison

The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) provides archive and distribution services for several data products in the Planetary Boundary Layer (PBL) category. As a new variable added to the Atmospheric Infrared Sounder (AIRS) Version 6 support product, the PBL height from AIRS is derived based on the gradients of the retrieved relative humidity profile, and provides the atmospheric pressure at the top of the PBL over the ocean. The GES DISC also hosts the Modern-Era Retrospective analysis for Research and Applications-2 (MERRA-2) product generated by the Goddard Earth Observing System Model Version 5 (GEOS-5) data assimilation system. The PBL height from MERRA-2 is based on the total eddy diffusion coefficient of heat. The monthly PBL height has been made available in the Giovanni system (Giovanni is a Web-based application developed by the GES DISC providing a simple and intuitive way to visualize, analyze, and access vast amounts of Earth science remote sensing data). Recently, the GES DISC began serving the global PBL height climatology product derived from the COSMIC/FORMOSAT-3 and TerraSAR-X Global Positioning System (GPS) radio occultation (RO) measurements from June 2006 to December 2015. In a previous study, we presented the monthly PBL height data from AIRS and MERRA-2 and demonstrated the GES DISC services which support data intercomparison, such as access, plotting, sub-setting, regridding, and generation of a multi-year monthly mean. We also provided intercomparison results, and found that different PBL height definitions contributed to significant differences of PBL height values between AIRS and MERRA-2. In this work, we present the 10-year seasonal climatologies from the AIRS, MERRA-2 and GPS-RO. We also used the cross section and vertical profile services in Giovanni to display and analyze the vertical atmosphere structure over regions where the PBL height derived from the AIRS and MERRA-2 are quite different. The examination of the AIRS and MERRA-2 three-dimensional data found that the relative humidity profiles had larger differences than the temperature profiles. The MERRA-2 gives more details than the AIRS for the vertical distribution of the humidity.

Ding, Feng↗

Managing and Servicing Physical Oceanographic Data at a NASA Distributed Active Archive Center

The NASA Earth Science Data Information Systems Project funds and operates 12 Distributed Active Archive Center(s) (DAAC) throughout the United States. Of these 12 centers, the Physical Oceanography DAAC (PO.DAAC) is committed to providing long term archival, distribution and stewardship for NASA physical oceanographic data, primarily derived from space-born satellite systems, but also including a growing set of recent and future in situ observations from the SPURS-1 and SPURS-2 campaigns. Notable NASA missions supported include: Seasat, TOPEX/Poseidon, NSCAT, QuikSCAT, ISS-RapidScat, Jason-1, Jason-2/OSTM, GRACE, Aquarius, GHRSST, and MODIS. The following interagency and international missions are also supported by PO.DAAC: AVHRR, Coriolis, DMSP, MetOp-A, MetOp-B, Oceansat-2. The PO.DAAC currently holds 525 datasets in public distribution, spanning the following observational parameters: sea surface temperature, sea surface salinity, ocean color, ocean surface currents, ocean surface wind speed, ocean surface wind direction, sea surface height, significant wave height, ocean water mass/thickness, and sea ice age. A hundred of these datasets are available in near-real-time. Datasets are distributed through a variety of open-source access protocols including FTP, OPeNDAP, and THREDDS. FTP will soon be phased out in favor of a recently introduced HTTPS PO.DAAC Drive interface that supports WebDAV and interoperable machine-to-machine communication. OPeNDAP supports remote data/metadata query, subset, and download. THREDDS provides the features of OPeNDAP with the additional feature of temporal aggregation. PO.DAAC also offers proprietary tools and services to further enhance the data discovery, visualization and analysis experience, including but not limited to: State of the Ocean, Web Services (data/metadata discovery and extraction), HiTIDE Level-2 subsetter, Live Access Server (LAS), Webification (w10nsci), and Rich Site Summary (RSS) Datacasting. To assist with provenance of datasets, PO.DAAC has implemented DOIs for the data it distributes so that they can be properly cited. There is a user forum and helpdesk that contains data recipes and via which users can get guidance. In summary, this presentation aims to provide a general overview of PO.DAAC’s web portal and data holdings along with a set of illustrative examples leading prospective data users into the practical utility of its tools and services.

Moroni, David F.↗

NASA ESDS Citizen Science Data Working Group

This document provides guidelines for legal, policy, and ethical issues; standards for citizen science data collection and management; information on ensuring usability of citizen science data and communication regarding its use; and best practices for long-term archival of citizen science data.Section1 contains a detailed discussion of policy, ethical, and legal considerations influencing citizen science data collection. Section 2 considers standards for documentation, including documentation of instrumentation, procedures, and the data itself. It concludes with a discussion of how citizen science data should be attributed. Section 3 provides guidance about how to ensure citizen science data are collected and stored in a useable way. It also considers how NASA and data producers should notify the scientific community, including citizen scientists and the public, about citizen science datasets and the scientific conclusions reached using them. Finally, Section 4 provides detailed information regarding what should be archived from projects using a citizen science approach, including data and code. It provides guidance about archive location, process, and timeframe, as well as information about data access and distribution services provided by NASA that may be relevant to data producers working with citizen scientists.

Citizen Science↗

EDR: An Energy-Aware Runtime Load Distribution System for Data-Intensive Applications in the Cloud

Data centers account for a growing percentage of US power consumption. Energy e_ciency is now a first-class design constraint for the data centers that support cloud services. Service providers must distribute their data e_ciently across multiple data centers. This includes creation of data replicas that provide multiple copies of data for e_cient access. However, selecting replicas to maximize performance while minimizing energy waste is an open problem. State of the art replica selection approaches either do not address energy, lack scalability and/or are vulnerable to crashes due to use of a centralized coordinator. Therefore, we propose, develop and evaluate a simple cost-oriented decentralized replica selection system named EDR, implemented with two distributed optimization algorithms

Cloud software↗

A Discussion of Time Management Concepts and Time Constraint Equations for Multi-Rate Federation Executions

The High Level Architecture (HLA) is a simulation interoperability standard developed by the Simulation Interoperability Standards Organization (SISO) and published as the international standard IEEE 1516-2010 by the Institute for Electrical and Electronics Engineers (IEEE). HLA is a widely used standard for the development and execution of collaborative distributed simulations. HLA provides a number of Management Services to simulation developers: Federation, Declaration, Object, Ownership, Data Distribution, and Time. Of those services, Time Management Services is probably one of the least understood and least used. However, Time Management Services are critical to technical simulations like those created for space systems using the Space Reference Federation Object Model (SpaceFOM). Time Management can be used to insure data coherence and execution repeatability in distributed simulations. When combined with real time execution policies, Time Management is being used to support real time execution of mixed software and hardware in the loop integration, verification, and validation simulations for active space systems development. This paper starts by providing an overview of the HLA Time Management Services. This provides the background to discuss the challenges associated with Time Management and its use, starting with simple common rate frame scheduled simulations, then simple multi-rate simulations, and ending with complex mixed rate simulations. The authors then formulate the significant time constraint relationships between identified frame scheduling parameters. The intent of the paper is to provide a concise discussion of how to use Time Management in both simple cases and in more complex mixed frame rate federation executions.

Simulation Interoperability↗

A Discussion of Time Management Concepts and Time Constraint Equations for Multi-Rate Federation Executions

The High Level Architecture (HLA) is a simulation interoperability standard developed by the Simulation Interoperability Standards Organization (SISO) and published as the international standard IEEE 1516-2010 by the Institute for Electrical and Electronics Engineers (IEEE). HLA is a widely used standard for the development and execution of collaborative distributed simulations. HLA provides a number of Management Services to simulation developers: Federation, Declaration, Object, Ownership, Data Distribution, and Time. Of those services, Time Management Services is probably one of the least understood and least used. However, Time Management Services are critical to technical simulations like those created for space systems using the Space Reference Federation Object Model (SpaceFOM). Time Management can be used to insure data coherence and execution repeatability in distributed simulations. When combined with real time execution policies, Time Management is being used to support real time execution of mixed software and hardware in the loop integration, verification, and validation simulations for active space systems development. This paper starts by providing an overview of the HLA Time Management Services. This provides the background to discuss the challenges associated with Time Management and its use, starting with simple common rate frame scheduled simulations, then simple multi-rate simulations, and ending with complex mixed rate simulations. The authors then formulate the significant time constraint relationships between identified frame scheduling parameters. The intent of the paper is to provide a concise discussion of how to use Time Management in both simple cases and in more complex mixed frame rate federation executions.

Simulation Interoperability↗

Contents of the NASA ocean data system archive, version 11-90

The National Aeronautics and Space Administration (NASA) Ocean Data System (NODS) archive at the Jet Propulsion Laboratory (JPL) includes satellite data sets for the ocean sciences and global-change research to facilitate multidisciplinary use of satellite ocean data. Parameters include sea-surface height, surface-wind vector, sea-surface temperature, atmospheric liquid water, and surface pigment concentration. NODS will become the Data Archive and Distribution Service of the JPL Distributed Active Archive Center for the Earth Observing System Data and Information System (EOSDIS) and will be the United States distribution site for Ocean Topography Experiment (TOPEX)/POSEIDON data and metadata.

Smith, Elizabeth A.↗

Integrating a local database into the StarView distributed user interface

A distributed user interface to the Space Telescope Data Archive and Distribution Service (DADS) known as StarView is being developed. The DADS architecture consists of the data archive as well as a relational database catalog describing the archive. StarView is a client/server system in which the user interface is the front-end client to the DADS catalog and archive servers. Users query the DADS catalog from the StarView interface. Query commands are transmitted via a network and evaluated by the database. The results are returned via the network and are displayed on StarView forms. Based on the results, users decide which data sets to retrieve from the DADS archive. Archive requests are packaged by StarView and sent to DADS, which returns the requested data sets to the users. The advantages of distributed client/server user interfaces over traditional one-machine systems are well known. Since users run software on machines separate from the database, the overall client response time is much faster. Also, since the server is free to process only database requests, the database response time is much faster. Disadvantages inherent in this architecture are slow overall database access time due to the network delays, lack of a 'get previous row' command, and that refinements of a previously issued query must be submitted to the database server, even though the domain of values have already been returned by the previous query. This architecture also does not allow users to cross correlate DADS catalog data with other catalogs. Clearly, a distributed user interface would be more powerful if it overcame these disadvantages. A local database is being integrated into StarView to overcome these disadvantages. When a query is made through a StarView form, which is often composed of fields from multiple tables, it is translated to an SQL query and issued to the DADS catalog. At the same time, a local database table is created to contain the resulting rows of the query. The returned rows are displayed on the form as well as inserted into the local database table. Identical results are produced by reissuing the query to either the DADS catalog or to the local table. Relational databases do not provide a 'get previous row' function because of the inherent complexity of retrieving previous rows of multiple-table joins. However, since this function is easily implemented on a single table, StarView uses the local table to retrieve the previous row. Also, StarView issues subsequent query refinements to the local table instead of the DADS catalog, eliminating the network transmission overhead. Finally, other catalogs can be imported into the local database for cross correlation with local tables. Overall, it is believe that this is a more powerful architecture for distributed, database user interfaces.

Silberberg, D. P.↗

On the Use of Smart Meter Data to Estimate the Voltage Magnitude on the Primary Side of Distribution Service Transformers

This paper develops a novel method to estimate the voltage magnitude on the primary side of distribution service transformers. The proposed method relies exclusively on smart meters, and therefore it is fully data-driven. This is an important feature because electric utilities have detailed models of only the primary network - that is, the network between the distribution substation and the primary side of service transformers that are installed closer to end-customer sites. The network that connects the secondary side of service transformers to end-customer sites, referred to as the secondary network, is simply represented by a lumped load. For each secondary network, the proposed method uses data acquired from only 2 smart meters: the closest and the farthest-in the sense of electrical distance - from the service transformer. As a reference to this feature, the proposed method is named SM2Vp. To our knowledge, this is the first time a method is shown to provide actionable information for realtime operation and control of power distribution grids using only two smart meters per secondary network. This is important because utilities have experienced barriers in managing and using large data sets for real-time operation and control. SM2Vp is primarily intended to provide pseudo-measurements for distribution system state estimation, but it can also be used directly for voltage control schemes. The performance of SM2Vp is demonstrated by numerical simulations carried out on three secondary network synthetic models and by using field data provided by a utility partner serving customers in southwestern California. A maximum relative error of approximately 3.9% or less is observed for the primary voltage magnitude estimates in all numerical experiments.

distribution service transformer↗

On the Use of Smart Meter Data to Estimate the Voltage Magnitude on the Primary Side of Distribution Service Transformers: Preprint

This paper develops a novel method to estimate the voltage magnitude on the primary side of distribution service transformers. The proposed method relies exclusively on smart meters, and therefore it is fully data-driven. This is an important feature because electric utilities have detailed models of only the primary network--that is, the network between the distribution substation and the primary side of service transformers that are installed closer to end-customer sites. The network that connects the secondary side of service transformers to end-customer sites, referred to as the secondary network, is simply represented by a lumped load. For each secondary network, the proposed method uses data acquired from only 2 smart meters: the closest and the farthest--in the sense of electrical distance--from the service transformer. As a reference to this feature, the proposed method is named SM2Vp. To our knowledge, this is the first time a method is shown to provide actionable information for real-time operation and control of power distribution grids using only two smart meters per secondary network. This is important because utilities have experienced barriers in managing and using large data sets for real-time operation and control. SM2Vp is primarily intended to provide pseudo-measurements for distribution system state estimation, but it can also be used directly for voltage control schemes. The performance of SM2Vp is demonstrated by numerical simulations carried out on three secondary network synthetic models and by using field data provided by a utility partner serving customers in southwestern California. A maximum relative error of approximately 3.9% or less is observed for the primary voltage magnitude estimates in all numerical experiments.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

PBL Height from AIRS, GPS RO, and MERRA-2 Products in NASA GES DISC and Their 10 Year Seasonal Mean Intercomparison

Within the planetary boundary layer (PBL), surface forcing response, drag, turbulence, and vertical mixing are important processes and play a more critical role here than in the overlying “free atmosphere”. The PBL Height (PBLH) is an important parameter in climate models, weather forecasts, and air quality prediction. The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) provides data processing, archiving, and distribution services for numerous Earth science products. PBLH is a parameter in three products served by GES DISC, which are from the Atmospheric Infrared Sounder (AIRS), the Global Positioning System (GPS) radio occultation (RO) experiment, and the NASA reanalysis product Modern-Era Retrospective analysis for Research and Applications – 2 (MERRA-2). These products have different spatial and temporal resolutions and coverages, and their PBLH definitions are also different. To better serve the PBL research community, we have summarized the specifications of these products. A ten-year seasonal mean intercomparison is also conducted to provide further guidance to users. The intercomparison results show that MERRA-2 has a much shallower PBL than AIRS and GPS RO. An experimental study indicates the different PBLH definition in MERRA-2 caused smaller values of PBLH. The improvement of the water vapor retrieval in AIRS version 7 over version 6 results in the version 7 PBLH agreeing better with GPS RO and MERRA-2 than version 6, especially near the equator and low latitudes.

Feng Ding↗

Global Change Data Center: Mission, Organization, Major Activities, and 2003 Highlights

Rapid, efficient access to Earth sciences data from satellites and ground validation stations is fundamental to the nation's efforts to understand the effects of global environmental changes and their implications for public policy. It becomes a bigger challenge in the future when data volumes increase from current levels to terabytes per day. Demands on data storage, data access, network throughput, processing power, and database and information management are increased by orders of magnitude, while budgets remain constant and even shrink.The Global Change Data Center's (GCDC) mission is to develop and operate data systems, generate science products, and provide archival and distribution services for Earth science data in support of the U.S. Global Change Program and NASA's Earth Sciences Enterprise. The ultimate product of the GCDC activities is access to data to support research, education, and public policy.

FROM↗