Search NASA⌕ Search

SEARCH · Search NASA

Results for “DISTRIBUTED DATA PROCESSING”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 217 records · Page 12
All sourcesNASA NTRSDOE OSTI
NASA NTRS1995

Parallel volume ray-casting for unstructured-grid data on distributed-memory architectures

As computing technology continues to advance, computational modeling of scientific and engineering problems produces data of increasing complexity: large in size and unstructured in shape. Volume visualization of such data is a challenging problem. This paper proposes a distributed parallel solution that makes ray-casting volume rendering of unstructured-grid data practical. Both the data and the rendering process are distributed among processors. At each processor, ray-casting of local data is performed independent of the other processors. The global image composing processes, which require inter-processor communication, are overlapped with the local ray-casting processes to achieve maximum parallel efficiency. This algorithm differs from previous ones in four ways: it is completely distributed, less view-dependent, reasonably scalable, and flexible. Without using dynamic load balancing, test results on the Intel Paragon using from two to 128 processors show, on average, about 60% parallel efficiency.

Ma, Kwan-Liu↗
NASA NTRS2018

Advanced Analytics and Big Earth Data

NASA's Earth Science Data Systems process, archive and distribute petabytes of Earth Observation data to a variety of end users. These end users will face dramatically increased data size in the near future, bringing about new challenges and opportunities in analyzing those data. One area of particular ferment currently is Machine Learning. Many Machine Learning methods are black boxes, limiting direct insight into the data's properties. However, they can be used for a variety of data enhancement purposes, such as parameter retrieval, data fusion and image classification and segmentation. The Earth Observing System Data and Information System is also evolving to host large data volumes in the cloud, enabling data proximal analysis. As part of this effort, an Analytics framework is being developed to support and enhance user analysis of the data. By using standards based services in the framework, diverse user communities can be served, while also allowing inter-system collaboration in the analysis process.

Cloud Computing↗
DOE OSTI2019

Data management and processing workflow for the Materials Physics and Engineering group beamlines at the Advanced Photon Source

The ability to store, organize, process and distribute experimental data effectively, efficiently and securely is particularly important for large user facilities like the Advanced Photon Source. In this article, the deployment of the APS Data Management System (DM) at the 1-ID and 6-BM beamlines of the APS is described. These two beamlines support a wide range of experimental techniques and generate data at relatively high rates, making them ideal candidates to illustrate the deployment and customization of the DM system and its tools. Using several usage examples at these beamlines, various capabilities of the DM system are described.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
NASA NTRS1996

(abstract) The Evolving Spaceborne Radar Data Support to Earth Science and Operations at the Alaska SAR Facility

The Alaska SAR Facility (ASF) has been receiving, processing, archiving, and distributing data for Earth scientists and operations since it began receiving data in 1991. Four radar satellites are now being handled. Recent developments have served to increase the level of services of ASF to the Earth science community considerably. These developments are discussed.

mapping radar interferometry Alaska topography ant↗
DOE OSTI2016

Accelerating Science Impact through Big Data Workflow Management and Supercomputing

The Large Hadron Collider (LHC), operating at the international CERN Laboratory in Geneva, Switzerland, is leading Big Data driven scientific explorations. ATLAS, one of the largest collaborations ever assembled in the the history of science, is at the forefront of research at the LHC. To address an unprecedented multi-petabyte data processing challenge, the ATLAS experiment is relying on a heterogeneous distributed computational infrastructure. To manage the workflow for all data processing on hundreds of data centers the PanDA (Production and Distributed Analysis)Workload Management System is used. An ambitious program to expand PanDA to all available computing resources, including opportunistic use of commercial and academic clouds and Leadership Computing Facilities (LCF), is realizing within BigPanDA and megaPanDA projects. These projects are now exploring how PanDA might be used for managing computing jobs that run on supercomputers including OLCF’s Titan and NRC-KI HPC2. The main idea is to reuse, as much as possible, existing components of the PanDA system that are already deployed on the LHC Grid for analysis of physics data. The next generation of PanDA will allow many data-intensive sciences employing a variety of computing platforms to benefit from ATLAS experience and proven tools in highly scalable processing.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2012

Distributed error and alarm processing in the CMS data acquisition system

The error and alarm system for the data acquisition of the Compact Muon Solenoid (CMS) at CERN was successfully used for the physics runs at Large Hadron Collider (LHC) during first three years of activities. Error and alarm processing entails the notification, collection, storing and visualization of all exceptional conditions occurring in the highly distributed CMS online system using a uniform scheme. Alerts and reports are shown on-line by web application facilities that map them to graphical models of the system as defined by the user. A persistency service keeps a history of all exceptions occurred, allowing subsequent retrieval of user defined time windows of events for later playback or analysis. This paper describes the architecture and the technologies used and deals with operational aspects during the first years of LHC operation. In particular we focus on performance, stability, and integration with the CMS sub-detectors.

Bauer, G.↗
NASA NTRS1975

The Large Area Crop Inventory Experiment (LACIE). Part 3: A systematic approach to the practical application of remote-sensing technology

The application of remote sensing technology by the U.S. Department of Agriculture (USDA) is examined. The activities of the USDA Remote-Sensing User Requirement Task Force which include cataloging USDA requirements for earth resources data, determining those requirements that would return maximum benefits by using remote sensing technology and developing a plan for acquiring, processing, analyzing, and distributing data to satisfy those requirements are described. Emphasis is placed on the large area crop inventory experiment and its relationship to the task force.

Murphy, J. D.↗
DOE OSTI2021

Sensor enabled data-driven predictive analytics for modeling and control with high penetration of DERs in distribution systems

The electric power grid is undergoing a tremendous transformation due to the increasing penetration of renewable energy resources beginning with wind and more recently with the distributed energy resources (DERs) such as solar and battery storage. DERs have dramatically changed the role of the distribution systems in the overall power grid, and they are expected to contribute a significant portion of power generation in the future. If current trends for DERs continue, system operation and control will need to change dramatically for improved grid reliability and resiliency. As renewable resources increase in penetration, new and challenging operational, planning, and design problems are expected to emerge. Some of the key challenges that arise in the planning and operation of the future grid are: 1) Quantifying the impact of high DER penetration in distribution systems on bulk grid behavior over multiple time scales. 2) Identifying whether a particular DER configuration/settings have a large impact on the overall grid behavior. These challenges can be addressed in an offline manner using detailed T&D grid models and they can also be addressed in an online manner using sensor measurements. In particular, the advancement and planned growth in sensor technology in power grid over various voltage levels provide us with a unique opportunity to tackle these challenges from a data analytic perspective without needing detailed T&D grid models. A few questions that naturally arise when addressing the challenges from DERs using sensor data are: 1) How can we use limited sensor measurements to monitor & control voltage stability and small signal stability of the bulk system? 2) How can we ensure that the developed data analytic methods are robust to data availability and quality issues? 3) How can we compute the developed analytics in a scalable manner using streaming measurements? In this project, we addressed the aforementioned challenges arising from DERs and answered the questions raised above on how to effectively use the sensor measurements to enhance the reliability and performance of the electric grid. Thus, the overarching goal of this project is to develop effective reduced/representative system models from data that make the computational complexity sufficiently manageable so as to be useful to simulate, analyze, and even control complex non-linear power systems dynamics with large penetrations of DERs. In order to achieve the objective, the project team established a four-fold technical approach 1) Formulated a combined transmission-distribution co-simulation framework for data generation and validation, 2) Derived reduced/representative models of power systems based on data-driven methods for efficient computation and appropriate representation of system behavior, 3) Developed data driven characterization of power system behavior based on transfer operator theory, machine learning and optimization for model estimation, 4) Incorporated a scalable data management and processing architecture using distributed Kafka streaming applications that coordinate input data streams to the developed data analytics. The key accomplishments of the project are: 1) Development of a scalable multi-timescale T&D co-simulation framework (both for steady state and for dynamic co-simulation) using commercial solvers (PSSE and GridLAB-D). The steady-state T&D co-simulation interface is shared with our industry partner (PJM). 2) A structured reduced order dynamic model of distribution systems that can represent partial motor stalling along with a systematic procedure to derive the model parameters. 3) A PMU based online method to monitor, localize and mitigate fault-induced delayed voltage recovery using DER reactive support and load control in distribution systems. 4) Development of linear operator based robust methodologies for dynamic state estimation, uncertainty quantification, system identification and trajectory prediction for power system dynamics. 5) An adaptive damping control for utilizing wind energy resources to provide oscillation damping and system stability. 6) Implementation of Kafka-based framework for efficient processing of streaming data using Linux-based local virtual environment.

DER integration↗
DOE OSTI2019

Data automation at light sources

Rapidly growing data volumes at light sources demand increasingly automated data collection, distribution, and analysis processes, in order to enable new scientific discoveries while not overwhelming finite human capabilities. We present here the case for automating and outsourcing light source science using cloud-hosted data automation and enrichment services, institutional computing resources, and high- performance computing facilities to provide cost-effective, scalable, and reliable implementations of such processes. We discuss three specific services that accomplish these goals for data distribution, automation, and transformation. In the first, Globus cloud-hosted data automation services are used to implement data capture, distribution, and analysis workflows for Advanced Photon Source and Advanced Light Source beamlines, leveraging institutional storage and computing. In the second, such services are combined with cloud-hosted data indexing and institutional storage to create a collaborative data publication, indexing, and discovery service, the Materials Data Facility (MDF), built to support a host of informatics applications in materials science. The third integrates components of the previous two projects with machine learning capabilities provided by the Data and Learning Hub for science (DLHub) to enable on-demand access to machine learning models from light source data capture and analysis workflows, and provides simplified interfaces to train new models on data from sources such as MDF on leadership scale computing resources. We draw conclusions about best practices for building next-generation data automation systems for future light sources.

Blaiszik, Ben↗
NASA NTRS1979

Seasat low-rate data system

The Seasat low-rate data system, an end-to-end data-processing and data-distribution system for the four low-rate sensors (radar altimeter, Seasat-A scatterometer system, scanning multichannel microwave radiometer, and visible and infrared radiometer) carried aboard the satellite, is discussed. The function of the distributed, nonreal-time, magnetic-tape system is to apply necessary calibrations, corrections, and conversions to yield geophysically meaningful products from raw telemetry data. The algorithms developed for processing data from the different sensors are described, together with the data catalogs compiled.

Brown, J. W.↗
NASA NTRS1992

The NASA Electronic Still Camera system

The Electronic Still Camera (ESC) project has been implemented in its first form for space shuttle operations. The system is composed of three components: (1) a hand-held battery-operated fully digital and programmable high-resolution camera; (2) a laptop-computer-based playback/downlink unit for onboard image processing; and (3) a ground station capable of receiving shuttle data, processing images, producing hardcopies, and distributing the data and hardcopies to end users. The advantages of the ESC system over film and satellites are discussed, and system specifications are summarized.

Holland, S. D.↗
NASA NTRS2020

The XRISM Science Data Center: Optimizing the Scientific Return from a Unique X-ray Observatory

The X-Ray Imaging and Spectroscopy Mission, XRISM, is currently scheduled to launch in 2022 with the objective of building on the brief, but significant, successes of the ASTRO-H (Hitomi) mission in solving outstanding astrophysical questions using high resolution X-ray spectroscopy. The XRISM Science Operations Team (SOT) consists of the JAXA-led Science Operations Center (SOC) and NASA-led Science Data Center (SDC), which work together to optimize the scientific output from the Resolve high-resolution spectrometer and the Xtend wide-field imager through planning and scheduling of observations, processing and distribution of data, development and distribution of software tools and the calibration database (CaldB), support of ground and in-flight calibration, and support of XRISM users in their scientific investigations of the energetic universe. Here, we summarize the roles and responsibilities of the SDC and its current status and future plans. The Resolve instrument poses particular challenges due to its unprecedented combination of high spectral resolution and throughput, broad spectral coverage, and relatively small field-of-view and large pixel-size. We highlight those challenges and how they are being met.

XRISM↗
NASA NTRS2014

Earth Science Informatics Comes of Age

The volume and complexity of Earth science data have steadily increased, placing ever-greater demands on researchers, software developers and data managers tasked with handling such data. Additional demands arise from requirements being levied by funding agencies and governments to better manage, preserve and provide open access to data. Fortunately, over the past 10-15 years significant advances in information technology, such as increased processing power, advanced programming languages, more sophisticated and practical standards, and near-ubiquitous internet access have made the jobs of those acquiring, processing, distributing and archiving data easier. These advances have also led to an increasing number of individuals entering the field of informatics as it applies to Geoscience and Remote Sensing. Informatics is the science and technology of applying computers and computational methods to the systematic analysis, management, interchange, and representation of data, information, and knowledge. Informatics also encompasses the use of computers and computational methods to support decisionmaking and other applications for societal benefits.

Jodha, Siri↗
NASA NTRS1993

A data distributed parallel algorithm for ray-traced volume rendering

This paper presents a divide-and-conquer ray-traced volume rendering algorithm and a parallel image compositing method, along with their implementation and performance on the Connection Machine CM-5, and networked workstations. This algorithm distributes both the data and the computations to individual processing units to achieve fast, high-quality rendering of high-resolution data. The volume data, once distributed, is left intact. The processing nodes perform local ray tracing of their subvolume concurrently. No communication between processing units is needed during this locally ray-tracing process. A subimage is generated by each processing unit and the final image is obtained by compositing subimages in the proper order, which can be determined a priori. Test results on both the CM-5 and a group of networked workstations demonstrate the practicality of our rendering algorithm and compositing method.

Ma, Kwan-Liu↗
NASA NTRS1983

TOPEX ground data system

The TOPEX Project is a proposed oceanographic mission to measure the topography of the sea surface for a period of three years. This mission is sponsored by the National Aeronautics and Space Administration and managed by the Jet Propulsion Laboratory. Measurements of topography are used to study ocean currents, tides, bathymetry and the oceanic geoid. Several of the primary goals of this mission are to process and verify the altimetric data, and distribute them within days to the science investigators. This paper describes the TOPEX end-to-end ground data system. In addition to controlling the TOPEX satellite, the ground data system has been designed to minimize the time from data acquisition to science processing and data distribution. A centralized design supports the favorable response time of the system and also allows for operational efficiencies. Networking of real time and non-real time elements of the data system provides for more effective data processing.

Rosell, S. N.↗
NASA NTRS2003

A Distributed Simulation Software System for Multi-Spacecraft Missions

The paper will provide an overview of the web-based distributed simulation software system developed for end-to-end, multi-spacecraft mission design, analysis, and test at the NASA Goddard Space Flight Center (GSFC). This software system was developed for an internal research and development (IR&D) activity at GSFC called the Distributed Space Systems (DSS) Distributed Synthesis Environment (DSE). The long-term goal of the DSS-DSE is to integrate existing GSFC stand-alone test beds, models, and simulation systems to create a "hands on", end-to-end simulation environment for mission design, trade studies and simulations. The short-term goal of the DSE was therefore to develop the system architecture, and then to prototype the core software simulation capability based on a distributed computing approach, with demonstrations of some key capabilities by the end of Fiscal Year 2002 (FY02). To achieve the DSS-DSE IR&D objective, the team adopted a reference model and mission upon which FY02 capabilities were developed. The software was prototyped according to the reference model, and demonstrations were conducted for the reference mission to validate interfaces, concepts, etc. The reference model, illustrated in Fig. 1, included both space and ground elements, with functional capabilities such as spacecraft dynamics and control, science data collection, space-to-space and space-to-ground communications, mission operations, science operations, and data processing, archival and distribution addressed.

Burns, Richard↗
NASA NTRS2003

An Internet Protocol-Based Software System for Real-Time, Closed-Loop, Multi-Spacecraft Mission Simulation Applications

The paper will provide an overview of the web-based distributed simulation software system developed for end-to-end, multi-spacecraft mission design, analysis, and test at the NASA Goddard Space Flight Center (GSFC). This software system was developed for an internal research and development (IR&D) activity at GSFC called the Distributed Space Systems (DSS) Distributed Synthesis Environment (DSE). The long-term goal of the DSS-DSE is to integrate existing GSFC stand-alone test beds, models, and simulation systems to create a "hands on", end-to-end simulation environment for mission design, trade studies and simulations. The short-term goal of the DSE was therefore to develop the system architecture, and then to prototype the core software simulation capability based on a distributed computing approach, with demonstrations of some key capabilities by the end of Fiscal Year 2002 (FY02). To achieve the DSS-DSE IR&D objective, the team adopted a reference model and mission upon which FY02 capabilities were developed. The software was prototyped according to the reference model, and demonstrations were conducted for the reference mission to validate interfaces, concepts, etc. The reference model, illustrated in Fig. 1, included both space and ground elements, with functional capabilities such as spacecraft dynamics and control, science data collection, space-to-space and space-to-ground communications, mission operations, science operations, and data processing, archival and distribution addressed.

Davis, George↗