Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Distribution Service”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 253 records · Page 14

Past and Future Operations Concepts of NASA's Earth Science Data and Information System

NASA committed to support the collection and distribution of Earth science data to study global change in the 1990's. A series of Earth science remote sensing satellites, the Earth Observing System (EOS), was to be the centerpiece. The concept for the science data system, the EOS Data and Information System (EOSDIS), created new challenges in the data processing of multiple satellite instrument observations for climate research and in the distribution of global-coverage remote sensor products to a large and growing science research community. EOSDIS was conceived to facilitate easy access to EOS science data for a wide heterogeneous national and international community of users. EOSDIS was to provide a spectrum of services designed for research scientists working on NASA focus areas but open to the general public and international science community. EOSDIS would give researchers tools and assistance in searching, selecting and acquiring data, allowing them to focus on Earth science climate research rather than complex product generation. Goals were to promote exchange of data and research results and expedite development of new geophysical algorithms. The system architecture had to accommodate a diversity of data types, data acquisition and product generation operations, data access requirements and different centers of science discipline expertise. Steps were taken early to make EOSDIS flexible by distributing responsibility for basic services. Many of the system operations concept decisions made in the 90s continued to this day. Once implemented, concepts such as the EOSDIS data model played a critical role developing effective data services, now a hallmark of EOSDIS. In other cases, EOSDIS architecture has evolved to enable more efficient operations, taking advantage of new technology and thereby shifting more resources on data services and less on operating and maintaining infrastructure. In looking to the future, EOSDIS may be able to take advantage of commercial compute environments for infrastructure and further enable large scale climate research. In this presentation, we will discuss key EOSDIS operations concepts from the 1990's, how they were implemented and evolved in the architecture, and look at concepts and architectural challenges for EOSDIS operations utilizing commercial cloud services.

Moses, John F.↗

Asynchronous Message Service Reference Implementation

This software provides a library of middleware functions with a simple application programming interface, enabling implementation of distributed applications in conformance with the CCSDS AMS (Consultative Committee for Space Data Systems Asynchronous Message Service) specification. The AMS service, and its protocols, implement an architectural concept under which the modules of mission systems may be designed as if they were to operate in isolation, each one producing and consuming mission information without explicit awareness of which other modules are currently operating. Communication relationships among such modules are self-configuring; this tends to minimize complexity in the development and operations of modular data systems. A system built on this model is a society of generally autonomous, inter-operating modules that may fluctuate freely over time in response to changing mission objectives, modules functional upgrades, and recovery from individual module failure. The purpose of AMS, then, is to reduce mission cost and risk by providing standard, reusable infrastructure for the exchange of information among data system modules in a manner that is simple to use, highly automated, flexible, robust, scalable, and efficient. The implementation is designed to spawn multiple threads of AMS functionality under the control of an AMS application program. These threads enable all members of an AMS-based, distributed application to discover one another in real time, subscribe to messages on specific topics, and to publish messages on specific topics. The query/reply (client/server) communication model is also supported. Message exchange is optionally subject to encryption (to support confidentiality) and authorization. Fault tolerance measures in the discovery protocol minimize the likelihood of overall application failure due to any single operational error anywhere in the system. The multi-threaded design simplifies processing while enabling application nodes to operate at high speeds; linked lists protected by mutex semaphores and condition variables are used for efficient, inter-thread communication. Applications may use a variety of transport protocols underlying AMS itself, including TCP (Transmission Control Protocol), UDP (User Datagram Protocol), and message queues.

Burleigh, Scott C.↗

Towards a Scalable and Adaptive Application Support Platform for Large-Scale Distributed E-Sciences in High-Performance Network Environments

The advent of large-scale collaborative scientific applications has demonstrated the potential for broad scientific communities to pool globally distributed resources to produce unprecedented data acquisition, movement, and analysis. System resources including supercomputers, data repositories, computing facilities, network infrastructures, storage systems, and display devices have been increasingly deployed at national laboratories and academic institutes. These resources are typically shared by large communities of users over Internet or dedicated networks and hence exhibit an inherent dynamic nature in their availability, accessibility, capacity, and stability. Scientific applications using either experimental facilities or computation-based simulations with various physical, chemical, climatic, and biological models feature diverse scientific workflows as simple as linear pipelines or as complex as a directed acyclic graphs, which must be executed and supported over wide-area networks with massively distributed resources. Application users oftentimes need to manually configure their computing tasks over networks in an ad hoc manner, hence significantly limiting the productivity of scientists and constraining the utilization of resources. The success of these large-scale distributed applications requires a highly adaptive and massively scalable workflow platform that provides automated and optimized computing and networking services. This project is to design and develop a generic Scientific Workflow Automation and Management Platform (SWAMP), which contains a web-based user interface specially tailored for a target application, a set of user libraries, and several easy-to-use computing and networking toolkits for application scientists to conveniently assemble, execute, monitor, and control complex computing workflows in heterogeneous high-performance network environments. SWAMP will enable the automation and management of the entire process of scientific workflows with the convenience of a few mouse clicks while hiding the implementation and technical details from end users. Particularly, we will consider two types of applications with distinct performance requirements: data-centric and service-centric applications. For data-centric applications, the main workflow task involves large-volume data generation, catalog, storage, and movement typically from supercomputers or experimental facilities to a team of geographically distributed users; while for service-centric applications, the main focus of workflow is on data archiving, preprocessing, filtering, synthesis, visualization, and other application-specific analysis. We will conduct a comprehensive comparison of existing workflow systems and choose the best suited one with open-source code, a flexible system structure, and a large user base as the starting point for our development. Based on the chosen system, we will develop and integrate new components including a black box design of computing modules, performance monitoring and prediction, and workflow optimization and reconfiguration, which are missing from existing workflow systems. A modular design for separating specification, execution, and monitoring aspects will be adopted to establish a common generic infrastructure suited for a wide spectrum of science applications. We will further design and develop efficient workflow mapping and scheduling algorithms to optimize the workflow performance in terms of minimum end-to-end delay, maximum frame rate, and highest reliability. We will develop and demonstrate the SWAMP system in a local environment, the grid network, and the 100Gpbs Advanced Network Initiative (ANI) testbed. The demonstration will target scientific applications in climate modeling and high energy physics and the functions to be demonstrated include workflow deployment, execution, steering, and reconfiguration. Throughout the project period, we will work closely with the science communities in the fields of climate modeling and high energy physics including Spallation Neutron Source (SNS) and Large Hadron Collider (LHC) projects to mature the system for production use.

97 MATHEMATICS AND COMPUTING↗

Two Weeks In The Life of Skybridge

This report documents a new large public data set for researchers studying the behavior of large commodity high-performance computing systems. Such large data sets are typically confined within institutions and access to them is limited to institutional partners. We provide it to promote HPC research more widely. The data set provides a two week time series of performance data collected once per minute using the Lightweight Distributed Metric Service from the system Skybridge at Sandia National Laboratories and the corresponding job-level accounting information. General system log information is not provided.

97 MATHEMATICS AND COMPUTING↗

Synthesis and distribution of environmental satellite data

The activities of two National Oceanographic and Atmospheric Administration and two National Aeronautics and Space Administration facilities involved in the synthesis or distribution of space environmental data are reviewed. The data products, user services, and pertinent publications are given. The computer systems that support three of these facilities are discussed and some details of the synthesis procedures are given. Data from the following satellite series are included in the discussion: TIROS, NIMBUS, ESSA, NOAA, SMS/GOES, ATS, and SOLRAD. Orbital position data for HAWKEYE 1, IMP H and J, VELA 5B, PIONEERS 6-9, PIONEER 11, HELIOS 1 and 2, and SOLRAD 11A and 11B in several coordinate systems are discussed.

Vette, J. I.↗

Notes on a storage manager for the Clouds kernel

The Clouds project is research directed towards producing a reliable distributed computing system. The initial goal is to produce a kernel which provides a reliable environment with which a distributed operating system can be built. The Clouds kernal consists of a set of replicated subkernels, each of which runs on a machine in the Clouds system. Each subkernel is responsible for the management of resources on its machine; the subkernal components communicate to provide the cooperation necessary to meld the various machines into one kernel. The implementation of a kernel-level storage manager that supports reliability is documented. The storage manager is a part of each subkernel and maintains the secondary storage residing at each machine in the distributed system. In addition to providing the usual data transfer services, the storage manager ensures that data being stored survives machine and system crashes, and that the secondary storage of a failed machine is recovered (made consistent) automatically when the machine is restarted. Since the storage manager is part of the Clouds kernel, efficiency of operation is also a concern.

Pitts, David V.↗

Implementation of a Campuswide Distributed Mass Storage Service: the Dream Versus Reality

In 1990, a technical team at NASA Lewis Research Center, Cleveland, Ohio, began defining a Mass Storage Service to pro- wide long-term archival storage, short-term storage for very large files, distributed Network File System access, and backup services for critical data dw resides on workstations and personal computers. Because of software availability and budgets, the total service was phased in over dm years. During the process of building the service from the commercial technologies available, our Mass Storage Team refined the original vision and learned from the problems and mistakes that occurred. We also enhanced some technologies to better meet the needs of users and system administrators. This report describes our team's journey from dream to reality, outlines some of the problem areas that still exist, and suggests some solutions.

Prahst, Stephen↗

Data processing pipeline with transaction-oriented data sharing

This paper makes three contributions to the area of modern science data processing systems. First, the paper describes the science data processing pipeline, developed at the Multi-mission Image Processing Lab of JPL, for transforming raw space data into high quality image data and automating the distribution of data using a high-performance file transaction service. File Exchange Interface is the file transaction service developed MIPL. Second, it presents the FEI component architecture in the are of file transaction management, security,a nd file integrity verfication. Finally, the paper presents the federated model for the FEI service to demonstrate how to create a pool of file trasaction services to support load balancing and service fallover, and simplify service management.

science data processing↗

Three Generations of Tracking and Data Relay Satellite (TDRS) Spacecraft

The current Tracking and Data Relay Satellite configuration consists of nine in-orbit satellites (four first generation, three second generation and two third generation satellites) globally distributed in geosynchronous orbit to provide near continuous data relay service to missions like Hubble Space Telescope and the International Space Station. The 1st generation spacecraft were designed by TRW/Northrop Grumman with their launches of the five spacecraft ranging from 1983 through 1995. The 2nd and 3rd generation spacecraft were designed by Boeing with their launches ranging 2000 - 2002 and 2013 - 2017 respectively. TDRS-3 is now 27 years on orbit, continues to be a capable asset for the TDRS constellation. Lack of need for inclination control combined with large fuel reserves and redundancy on critical elements provides spacecraft that operate well past design life, all of which contributes to expanded TDRS constellation support capabilities. All spacecraft generations have issues. Significant issues will be summarized with the focus on the Boeing related problems. Degradations and failures are continually assessed and provide the foundation for yearly updates to spacecraft reliability models, constellation service projections and deorbit plans (in order to meet NASAs mandate of limiting orbital debris). Even when accounting for degradations and failures, the life expectancy for the Boeing delivered 2nd generation TDRS-8, 9 10 TDRS are anticipated to be 25+ years.

TDRS↗

Results of an Automated GPS Tracking System in Support of Topex/Poseidon and GPSMet

A fully automated near real-time GPS tracking system has been developed around JPL's GIPSY/OASIS II software. The system produces <25 cm (3D rms) GPS orbits and one-half nanosecond (15 cm) clock estimates. The process starts automatically when a favorable global distribution of ground data from the IGS network (International GPS Service for Geodynamics) becomes available. Ionospherically corrected phase and pseudorange data are optimally combined to remove satellite and ground receiver clock errors, including selective availability. After the GPS orbits are determined within the data arc, they are then propagated with empirically determined dynamic force models. Real-time <2 meter (3D rms) GPS orbits are always available. As a by-product of this process, other calibration estimates such as station clocks, troposphere estimates, and Earth orientation parameters are also produced. For daily arc fits, the process requires 7-8 hours of CPU time on an HP9000/735 workstation.

GPS↗

Newly Released GPCP Version 3.2 Global Precipitation Datasets at NASA GES DISC

The Global Precipitation Climatology Project (GPCP) is the precipitation component of an internationally coordinated set of (mainly) satellite-based global products dealing with the Earth's water and energy cycles, under the auspices of the Global Water and Energy Experiment (GEWEX) Data and Assessment Panel (GDAP) of the World Climate Research Program. As the follow-on to the GPCP Version 2.X products, GPCP Version 3 (GPCP V3.2) seeks to continue the production of long, homogeneous precipitation record using modern input and calibration datasets. The GPCP V3.2 provides globally complete analyses of surface precipitation on a 0.5°x 0.5° latitude/longitude grid at both monthly and daily intervals, respectively covering 1983 to the present and June 2000 to the present. New data fields have been introduced to better characterize the precipitation, particularly including an estimate of the fraction of the precipitation that is liquid (rain) in both the Monthly and Daily, and a Quality Index for the Monthly. Compared to the operational GPCP V2.3 Monthly, the V3.2 Monthly provides a more reasonable climatology in the Southern Ocean, and increases the global average precipitation by about 4.46%, which is in line with recommendations of recent assessments. However, the two versions have comparable global and regional trends for 1983-2020. Compared to the operational One-Degree Daily (Version 1.3) product, the V3.2 Daily better represents the histogram of precipitation rates, particularly at high values. In this presentation, we will present the latest GPCP V3.2 daily and monthly datasets archived and distributed at NASA Goddard Earth Sciences (GES) Data and Information Services Center (DISC) along with examples from NASA Giovanni.

Precipitation↗

The NASA Merra-2 Reanalysis Products: Data and Tools Used for Aerosol and Air Quality Studies

The NASA Modern-Era Retrospective analysis for Research and Applications Version 2 (MERRA-2) is atmospheric reanalysis data spanning 1980 to present. It has been produced by the NASA Global Modeling and Assimilation Office (GMAO) and is distributed by the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC). MERRA-2 data includes 100 collections of Earth system variables, mainly from the atmospheric model, such as aerosol fields and meteorological fields, radiation fields, and aerosol fields, guided by the assimilation of as many as six million observations every six hours. MERRA-2 has been one of the most popular datasets from NASA and is widely used in interdisciplinary research and applications, with increasing numbers of new users. For example, at least 7000 users accessed MERRA-2 data at GES DISC in the year 2021, ~1000 more users than in the year 2020. In this presentation, we will introduce the MERRA-2 datasets associated with aerosol and air quality studies and use a wildfire case study to demonstrate the data tools developed at GES DISC to analyze and visualize MERRA-2 data, such as Giovanni and the level 3 and level 4 subsetter, and Jupyter Python notebook. We will also update the status of cloud migration of the MERRA-2 data to Amazon Web Services (AWS).

Xiaohua Pan↗

Data Assimilation and Reanalysis

This presentation introduces the Data Assimilation and Reanalysis principle, then details the NASA Modern-Era Retrospective analysis for Research and Applications Version 2 (MERRA-2). The MERRA-2 is atmospheric reanalysis data spanning 1980 to the present. It has been produced by the NASA Global Modeling and Assimilation Office (GMAO) and is distributed by the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC). In this presentation, I will introduce the MERRA-2 datasets associated with aerosol and air quality studies and use case studies to demonstrate the data tools developed at GES DISC to analyze and visualize MERRA-2 data, such as Giovanni and Jupyter Python notebook.

Xiaohua Pan↗

A reliable multicast for XTP

Multicast services needed for current distributed applications on LAN's fall generally into one of three categories: datagram, semi-reliable, and reliable. Transport layer multicast datagrams represent unreliable service in which the transmitting context 'fires and forgets'. XTP executes these semantics when the MULTI and NOERR mode bits are both set. Distributing sensor data and other applications in which application-level error recovery strategies are appropriate benefit from the efficiency in multidestination delivery offered by datagram service. Semi-reliable service refers to multicasting in which the control algorithms of the transport layer--error, flow, and rate control--are used in transferring the multicast distribution to the set of receiving contexts, the multicast group. The multicast defined in XTP provides semi-reliable service. Since, under a semi-reliable service, joining a multicast group means listening on the group address and entails no coordination with other members, a semi-reliable facility can be used for communication between a client and a server group as well as true peer-to-peer group communication. Resource location in a LAN is an important application domain. The term 'semi-reliable' refers to the fact that group membership changes go undetected. No attempt is made to assess the current membership of the group at any time--before, during, or after--the data transfer.

Dempsey, Bert J.↗

Neutron Science TeraGrid Gateway

The unique contributions of the Neutron Science TeraGrid Gateway (NSTG) are the connection of national user facility instrument data sources to the integrated cyberinfrastructure of the National Science FoundationTeraGrid and the development of a neutron science gateway that allows neutron scientists to use TeraGrid resources to analyze their data, including comparison of experiment with simulation. The NSTG is working in close collaboration with the Spallation Neutron Source (SNS) at Oak Ridge as their principal facility partner. The SNS is a next-generation neutron source. It has completed construction at a cost of $1.4 billion and is ramping up operations. The SNS will provide an order of magnitude greater flux than any previous facility in the world and will be available to all of the nation's scientists, independent of funding source, on a peer-reviewed merit basis. With this new capability, the neutron science community is facing orders of magnitude larger data sets and is at a critical point for data analysis and simulation. There is a recognized need for new ways to manage and analyze data to optimize both beam time and scientific output. The TeraGrid is providing new capabilities in the gateway for simulations using McStas and a fitting service on distributed TeraGrid resources to improved turnaround. NSTG staff are also exploring replicating experimental data in archival storage. As part of the SNS partnership, the NSTG provides access to gateway support, cyberinfrastructure outreach, community development, and user support for the neutron science community. This community includes not only SNS staff and users but extends to all the major worldwide neutron scattering centers.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

NASA EOSDIS Data Usage Metrics- Insight and Assessment

NASA's Earth Science Data and Information System (ESDIS) Project collects Earth science data usage metrics on a daily basis through the ESDIS Metrics System (EMS). This includes metrics on distribution of data products, users, data volumes, and number of files, which are key parameters in evaluating system-level performance of any of the Distributed Active Archive Centers (DAACs) encompassed by NASA's Earth Observing System (EOS) Data and Information System (EOSDIS). EOSDIS data usage metrics illustrate the benefits of making NASA data openly available to the public and show a rapid growth in data distribution to a worldwide user community. In fact, each year since 2014 the EOSDIS has distributed over one billion data files of products from EOS satellite, airborne, and in situ observations. An assessment of the long-term trends of data usage metrics and user characterization provides insights into data usability.This study will focus on describing the EMS as a metrics collection tool and will provide a comprehensive analysis of EOSDIS data usage metrics over the last 10 years. This study will also characterize the product distribution metrics by various tools and services, such as Giovanni, the Open-source Project for a Network Data Access Protocol (OPeNDAP), and subsets, to address how these tools/services have extended the usage of data in the EOSDIS collection. Data usage patterns based on discipline and study area will further assist in understanding how EOSDIS data user needs have evolved over time. Results from this study will provide useful information for the DAACs that can help them improve the functionality of their tools and services as well as more efficiently allocate the resources necessary for enhanced access and availability of their data products. Knowledge of these metrics may also benefit user discovery of data in the EOSDIS collection, promote research collaboration, and stimulate new ideas from work and research conducted using specific datasets and data collections.

Kafle, Durga N.↗

An Update on Global Satellite-Based Precipitation Products and Services at NASA GES DISC

The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) is home to major NASA satellite precipitation measurement missions including the Tropical Rainfall Measuring Mission (TRMM) and the Global Precipitation Measurement (GPM) as well as other NASA projects. Accurate and timely available global and regional precipitation products play an important role in research and applications around the world. The GES DISC provides near-real-time, near-global precipitation products (e.g.IMERG) to support a wide variety of interdisciplinary research and operational activities including flood modeling, landslides, crop monitoring and assessment, vector-borne diseases, etc. Climate data record products (e.g.GPCP3) are essential for climate research, assessment, model evaluation and applications. To facilitate data access and exploration, the GES DISC has developed data services such asGiovanni, an online visualization and analysis tool for easy access to over 2000 satellite- and model-based variables. Established in the mid 1980s, the GES DISC also distributes data in other disciplines including hydrology, atmospheric chemistry, atmospheric dynamics, etc. In this presentation, we will present an update on global precipitation products and data services including the new IMERG V06B suite, the latest version of GPCP (Version 3) and value-added data subsetting services (L34RS, L2S).

Liu, Zhong↗

Data Information for Global Change Studies: NASA's Distributed Active Archive Centers and Cooperating Data Centers

The Earth Observing System (EOS) is an integral part of the National Aeronautics and Space Administration's (NASA's) Earth Science Enterprise (ESE). ESE is a long-term global change research program designed to improve our understanding of the Earth's interrelated processes involving the atmosphere, oceans, land surfaces, and polar regions. Data from EOS instruments and other Earth science measurement systems are useful in understanding the causes and processes of global climate change and the consequences of human activities. The EOS Data and Information System (EOSDIS) provides a structure for data management and user services for products derived from EOS satellite instruments and other NASA Earth science data. Within the EOSDIS framework, the Distributed Active Archive Centers (DAACs) have been established to provide expertise in one or more Earth science disciplines. The DAACs and cooperating data centers provide data and information services to support the global change research community. Much of the development of the DAACs has been in anticipation of the enormous amount of data expected from EOS instruments to be launched within the next two decades. Terra, the EOS flagship launched in December 1999, is the first of a series of EOS satellites to carry several instruments with multispectral capabilities. Some data products from these instruments are now available from several of the DAACs. These and other data products can be ordered through the EOS Data Gateway (EDG) and DAAC-specific online ordering systems.

Source record↗