Search NASA⌕ Search

SEARCH · Search NASA

Results for “distributed systems, data management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 307 records · Page 17

Omics.pnl.gov: A Portal for the Distribution and Sharing of Multi-Disciplinary Pan-Omics Information

The data production of scientific studies is growing at a nearly exponential rate, leading to challenges in disseminating primary experimental results for peer review and public access, while simultaneously providing information that enables reproducing the studies and/or analyzing the results in a proper context. Recent mandates from various public funding agencies are requiring data release plans be included as a project goal. This requirement is coupled with an increased need for transparency in complex research, as evidenced by the data release policies of respected peer-reviewed journals. This combination of good scientific citizenship and funding requirements has brought the data distribution issue to the domain of scientific information management researchers. Herein, we present a web site designed to address these goals, the Biological MS Data and Software Distribution Center. This site leverages vast amounts of pre-existing experimental metadata stored in our purpose-built data management system to streamline the process of making these data available.

99 GENERAL AND MISCELLANEOUS↗

iDDS: intelligent distributed dispatch and scheduling for workflow orchestration

The intelligent distributed dispatch and scheduling (iDDS) service is a versatile workflow orchestration system designed for large-scale, distributed scientific computing. iDDS extends traditional workload and data management by integrating data-aware execution, conditional logic, and programmable workflows, enabling automation of complex and dynamic processing pipelines. Originally developed for the ATLAS experiment at the large hadron collider, iDDS has evolved into an experiment-agnostic platform that supports both template-driven workflows and a Function-as-a-Task model for Python-based orchestration. This paper presents the architecture and core components of iDDS, highlighting its scalability, modular message-driven design, and integration with systems such as PanDA and Rucio. We demonstrate its versatility through real-world use cases: fine-grained tape resource optimization for ATLAS, orchestration of large Directed Acyclic Graph (DAG) workflows for the Rubin Observatory, distributed hyperparameter optimization for machine learning applications, active learning for physics analyses, and AI-assisted detector design at the electron–ion collider. By unifying workload scheduling, data movement, and adaptive decision-making, iDDS reduces operational overhead and enables reproducible, high-throughput workflows across heterogeneous infrastructures. We conclude with current challenges and future directions, including interactive, cloud-native, and serverless workflow support.

97 MATHEMATICS AND COMPUTING↗

NASA Tech Briefs, January 2012

Contents of this issue are: (1) Energy-Based Tetrahedron Sensor for High-Temperature, High-Pressure Environments (2) Handheld Universal Diagnostic Sensor (3) Large-Area Vacuum Ultraviolet Sensors (4) Fiber Bragg Grating Sensor System for Monitoring Smart Composite Aerospace Structures (5) Health-Enabled Smart Sensor Fusion Technology (6) Extended-Range Passive RFID and Sensor Tags (7) Hybrid Collaborative Learning for Classification and Clustering in Sensor Networks (8) Self-Healing, Inflatable, Rigidizable Shelter (9) Improvements in Cold-Plate Fabrication (10) Technique for Radiometer and Antenna Array Calibration - TRAAC (11) Real-Time Cognitive Computing Architecture for Data Fusion in a Dynamic Environment (12) Programmable Digital Controller (13) Use of CCSDS Packets Over SpaceWire to Control Hardware (14) Key Decision Record Creation and Approval Module (15) Enhanced Graphics for Extended Scale Range (16) Debris Examination Using Ballistic and Radar Integrated Software (17) Data Distribution System (DDS) and Solar Dynamic Observatory Ground Station (SDOGS) (18) Integration Manager (19) Eclipse-Free-Time Assessment Tool for IRIS (20) Automated and Manual Rocket Crater Measurement Software (21) MATLAB Stability and Control Toolbox Trim and Static Stability Module (22) Patched Conic Trajectory Code (23) Ring Image Analyzer (24) SureTrak Probability of Impact Display (25) Implementation of a Non-Metallic Barrier in an Electric Motor (26) Multi-Mission Radioisotope Thermoelectric Generator Heat Exchangers for the Mars Science Laboratory Rover (27) Uniform Dust Distributor for Testing Radiative Emittance of Dust-Coated Surfaces (28) MicroProbe Small Unmanned Aerial System (29) Highly Stable and Active Catalyst for Sabatier Reactions (30) Better Proton-Conducting Polymers for Fuel-Cell Membranes (31) CCD Camera Lens Interface for Real-Time Theodolite Alignment (32) Peregrine 100-km Sounding Rocket Project (33) SOFIA Closed- and Open-Door Aerodynamic Analyses (34) Sonic Thermometer for High-Altitude Balloons (35) Near-Infrared Photon-Counting Camera for High-Sensitivity Observations (36) Integrated Optics Achromatic Nuller for Stellar Interferometry (37) High-Speed Digital Interferometry (38) Ultra-Miniature Lidar Scanner for Launch Range Data Collection (39) Shape and Color Features for Object Recognition Search (40) Explanation Capabilities for Behavior-Based Robot Control (41) A DNA-Inspired Encryption Methodology for Secure, Mobile Ad Hoc Networks (42) Quality Control Method for a Micro-Nano-Channel Microfabricated Device (43) Corner-Cube Retroreflector Instrument for Advanced Lunar Laser Ranging (44) Electrospray Collection of Lunar Dust (45) Fabrication of a Kilopixel Array of Superconducting Microcalorimeters with Microstripline Wiring Spacecraft Attitude Tracking and Maneuver Using Combined Magnetic Actuators (46) Coherent Detector for Near-Angle Scattering and Polarization Characterization of Telescope Mirror Coatings

Source record↗

Performance Analysis of Data Processing in Distributed File Systems with Near Data Processing

In the era of big data, the escalating volume and velocity of data generation pose significant challenges in data processing. Traditional systems like Spark and Hadoop manage the increasing amount and velocity of data by improving data placement and processing speeds. However, they face inherent limitations due to the essential data movement required for processing. In this paper, we explore the Skyhook framework, a novel extension of the Ceph distributed system, which significantly reduces the need for data movement. We present an extensive case study using the Skyhook framework, applying it with the TPC-H and K-means clustering algorithms. More specifically, we leverage the TPC-H benchmark to distinguish between CPU-intensive and I/O-intensive tasks. We explore the integration of K-means clustering into SQL, coupled with a near-data processing system to offload the computational burden of the K-means clustering algorithm to storage nodes. We conduct a comprehensive performance evaluation of distributed data processing applications across three processing approaches: traditional layout (baseline), optimized layout, and near-data processing. Additionally, we introduce the use of the FIO tool to simulate real-world system workloads, enabling the measurement of performance metrics such as average latency and CPU utilization. Our research is a significant advance in understanding how to optimize data processing systems to meet the demands of the modern data landscape.

Hou, Shiyue↗

DataFed: Towards Reproducible Research via Federated Data Management

The increasingly collaborative, globalized nature of scientific research combined with the need to share data and the explosion in data volumes present an urgent need for a scientific data management system (SDMS). An SDMS presents a logical and holistic view of data that greatly simplifies and empowers data organization, curation, searching, sharing, dissemination, etc. We present DataFed - a lightweight, distributed SDMS that spans a federation of storage systems within a loosely-coupled network of scientific facilities. Unlike existing SDMS offerings, DataFed uses high-performance and scalable user management and data transfer technologies that simplify deployment, maintenance, and expansion of DataFed. DataFed provides web-based and command-line interfaces to manage data and integrate with complex scientific workflows. DataFed represents a step towards reproducible scientific research by enabling reliable staging of the correct data at the desired environment.

Stansberry, Dale↗

UPS: Unified PMU-Data Storage System to Enhance T+D PMU Data Usability

The emerging distribution-level phasor measurement unit (D-PMU) is expected to play an important role in enhancing distribution system observability and situational-awareness. It is a demanding yet challenging task to develop advanced D-PMU data management and analytics tools to improve D-PMU data usability and further promote D-PMU projects deployment. This paper focuses on D-PMU data processing and storage. It presents a brief review of existing D-PMU data storage systems and points out their limitations on high-performance, flexibility, and scalability. To overcome the limitations, a unified PMU-data storage system (UPS) is proposed. Specifically, a unified I/O interface between storage servers and computing jobs is developed to effectively reduce the overhead of managing various computing jobs and data analytics over multiple storage infrastructures; and PMUCache with PMUCache partitioning and PMUCache replacement algorithms are designed to support in-situ data processing and shared distributed data storage and further serve the computing jobs/queries with different Quality of Service (QoS) requirements. Through a series of experiments, it is demonstrated that UPS achieves high performance on fast and big data processing and storage, and efficiently increases the flexibility and scalability of PMU data management systems.

42 ENGINEERING↗

Space station - Technology development

The NASA manned space station program's systems technology effort involves the development of novel techniques that will reduce the scope of tasks neeeded for design, development, testing and evaluation of the hardware. Operations technology efforts encompass analyses that will define those techniques best able to improve the efficiency and reduce the costs of space station functions. The technology objective for data management calls for a fault-tolerant, distributed, expandable and adaptable, as well as repairable and user-friendly, flight data management system that employs state-of-the-art hardware and software. The space station's power system includes the largest element, a 'solar blanket', and the heaviest component, the batteries, of all the subsystems. A thermal management system for the power system is of paramount importance. Attention is also given to the exacting demands of attitude control and stabilization and a regenerative life support system of the requisite capacity and reliability.

Carlisle, R. F.↗

Biological and Environmental Research Network Requirements

The Energy Sciences Network (ESnet) is the primary provider of network connectivity for the U.S. Department of Energy (DOE) Office of Science (SC), the single largest supporter of basic research in the physical sciences in the United States. In support of SC programs, ESnet regularly updates and refreshes its understanding of the networking requirements of the instruments, facilities, scientists, and science programs that it serves. This focus has helped ESnet be a highly successful enabler of scientific discovery for over 25 years. In November 2012, ESnet and the Office of Biological and Environmental Research (BER) of the DOE SC organized a review to characterize the networking requirements of the programs funded by the BER program office. Several key findings resulted from the review. Among them: 1) The scale of data sets available to science collaborations continues to increase exponentially. This has broad impact, both on the network and on the computational and storage systems connected to the network. 2) Many science collaborations require assistance to cope with the systems and network engineering challenges inherent in managing the rapid growth in data scale. 3) Several science domains operate distributed facilities that rely on high-performance networking for success. Key examples illustrated in this report include the Earth System Grid Federation (ESGF) and the Systems Biology Knowledgebase (KBase). This report expands on these points, and addresses others as well. The report contains a findings section as well as the text of the case studies discussed at the review.

09 BIOMASS FUELS↗

Evolution of CMS Workload Management Towards Multicore Job Support

The successful exploitation of multicore processor architectures is a key element of the LHC distributed computing system in the coming era of the LHC Run 2. High-pileup complex-collision events represent a challenge for the traditional sequential programming in terms of memory and processing time budget. The CMS data production and processing framework is introducing the parallel execution of the reconstruction and simulation algorithms to overcome these limitations. CMS plans to execute multicore jobs while still supporting singlecore processing for other tasks difficult to parallelize, such as user analysis. The CMS strategy for job management thus aims at integrating single and multicore job scheduling across the Grid. This is accomplished by employing multicore pilots with internal dynamic partitioning of the allocated resources, capable of running payloads of various core counts simultaneously. An extensive test programme has been conducted to enable multicore scheduling with the various local batch systems available at CMS sites, with the focus on the Tier-0 and Tier-1s, responsible during 2015 of the prompt data reconstruction. Scale tests have been run to analyse the performance of this scheduling strategy and ensure an efficient use of the distributed resources. This paper presents the evolution of the CMS job management and resource provisioning systems in order to support this hybrid scheduling model, as well as its deployment and performance tests, which will enable CMS to transition to a multicore production model for the second LHC run.

Perez-Calero Yzquierdo, A.↗

Recommendations for a service framework to access astronomical archives

There are a large number of astronomical archives and catalogs on-line for network access, with many different user interfaces and features. Some systems are moving towards distributed access, supplying users with client software for their home sites which connects to servers at the archive site. Many of the issues involved in defining a standard framework of services that archive/catalog suppliers can use to achieve a basic level of interoperability are described. Such a framework would simplify the development of client and server programs to access the wide variety of astronomical archive systems. The primary services that are supplied by current systems include: catalog browsing, dataset retrieval, name resolution, and data analysis. The following issues (and probably more) need to be considered in establishing a standard set of client/server interfaces and protocols: Archive Access - dataset retrieval, delivery, file formats, data browsing, analysis, etc.; Catalog Access - database management systems, query languages, data formats, synchronous/asynchronous mode of operation, etc.; Interoperability - transaction/message protocols, distributed processing mechanisms (DCE, ONC/SunRPC, etc), networking protocols, etc.; Security - user registration, authorization/authentication mechanisms, etc.; Service Directory - service registration, lookup, port/task mapping, parameters, etc.; Software - public vs proprietary, client/server software, standard interfaces to client/server functions, software distribution, operating system portability, data portability, etc. Several archive/catalog groups, notably the Astrophysics Data System (ADS), are already working in many of these areas. In the process of developing StarView, which is the user interface to the Space Telescope Data Archive and Distribution Service (ST-DADS), these issues and the work of others were analyzed. A framework of standard interfaces for accessing services on any archive system which would benefit archive user and supplier alike is proposed.

Travisano, J. J.↗

A Framework to Survey the Energy Efficiency of Installed Motor Systems

While motors are ubiquitous throughout the globe, there is insufficient data to properly assess their level of energy efficiency across regional boundaries. Furthermore, many of the existing data sets focus on motor efficiency and neglect the connected drive and system. Without a comprehensive survey of the installed motor system base, a baseline energy efficiency of a country or region’s motor systems cannot be developed. The lack of data impedes government agencies, utilities, manufacturers, distributers, and energy managers when identifying where to invest resources to capture potential energy savings, creating programs aimed at reducing electrical energy consumption, or quantifying the impacts of such programs. This paper will outline a data collection framework for use when conducting a survey under a variety of execution models to characterize motor system energy efficiency within a country or region. The framework is intended to standardize the data collected ensuring consistency across independently conducted surveys. Consistency allows for the surveys to be leveraged against each other enabling comparisons to motor system energy efficiencies from other regions. In creating the framework, an analysis of various motor driven systems, including compressed air, pumping, and fan systems, was conducted and relevant parameters characterizing the efficiency of these systems were identified. A database using the framework will enable policymakers and industry to better assess the improvement potential of their installed motor system base particularly with respect to other regions, assisting in efforts to promote improvements to the energy efficiency of motor driven systems.

Rao, Prakash↗

Network interface unit design options performance analysis

An analysis is presented of three design options for the Space Station Freedom (SSF) onboard Data Management System (DMS) Network Interface Unit (NIU). The NIU provides the interface from the Fiber Distributed Data Interface (FDDI) local area network (LAN) to the DMS processing elements. The FDDI LAN provides the primary means for command and control and low and medium rate telemetry data transfers on board the SSF. The results of this analysis provide the basis for the implementation of the NIU.

Miller, Frank W.↗

Evolution of Information Management at the GSFC Earth Sciences (GES) Data and Information Services Center (DISC): 2006-2007

Increasingly sophisticated National Aeronautics and Space Administration (NASA) Earth science missions have driven their associated data and data management systems from providing simple point-to-point archiving and retrieval to performing user-responsive distributed multisensor information extraction. To fully maximize the use of remote-sensor-generated Earth science data, NASA recognized the need for data systems that provide data access and manipulation capabilities responsive to research brought forth by advancing scientific analysis and the need to maximize the use and usability of the data. The decision by NASA to purposely evolve the Earth Observing System Data and Information System (EOSDIS) at the Goddard Space Flight Center (GSFC) Earth Sciences (GES) Data and Information Services Center (DISC) and other information management facilities was timely and appropriate. The GES DISC evolution was focused on replacing the EOSDIS Core System (ECS) by reusing the In-house developed disk-based Simple, Scalable, Script-based Science Product Archive (S4PA) data management system and migrating data to the disk archives. Transition was completed in December 2007

Kempler, Steven↗

Catalyzing deep decarbonization with federated battery diagnosis and prognosis for better data management in energy storage systems

Industrial data analytics methods play a central role in improving energy storage performance and efficiency, impacting the future of electrified transportation and renewable electricity generation. However, significant challenges hinder the large-scale deployment of batteries. Conventional methods rely on centralized collection and processing of fleet-level data, leading to database size issues and privacy concerns due to potential data breaches. To enable scalable deployment of battery management systems, this article proposes a federated battery diagnosis and prognosis model, which distributes the processing of battery standard current-voltage-time-usage data in a privacy-preserving manner. Instead of transferring the raw data, this approach communicates only the locally processed parameters, thus reducing communication load and preserving data confidentiality. The federated model offers a paradigm shift in battery health management through privacy-preserving distributed methods for battery data processing and lifetime prediction, ensuring the reliable and sustainable deployment of lithium-ion batteries in a rapidly evolving world.

asset health management↗

Distributed science operations for JPL planetary missions

Advances in spacecraft, flight instruments, and ground systems provide an impetus and an opportunity for scientific investigation teams to take direct control of their instruments' operations and data collection while at the same time, providing a cost effective and flexible approach in support of increasingly complex science missions. Operations of science instruments have generally been integrated into planetary flight and ground systems at a very detailed level. That approach has been successful, but the cost of incorporating instrument expertise into the central mission operations system has been high. This paper discusses an approach to simplify planetary science operations by distributing instrument computing and data management tasks from the central mission operations system to each investigator's home center of observational expertise. Some early results of this operations concept will be presented based on the Mars Observer (MO) Project experience and Cassini Project plans.

Benson, Richard D.↗

Space station data management system - A common GSE test interface for systems testing and verification

This paper examines the fundamental problems and goals associated with test, verification, and flight-certification of man-rated distributed data systems. First, a summary of the characteristics of modern computer systems that affect the testing process is provided. Then, verification requirements are expressed in terms of an overall test philosophy for distributed computer systems. This test philosophy stems from previous experience that was gained with centralized systems (Apollo and the Space Shuttle), and deals directly with the new problems that verification of distributed systems may present. Finally, a description of potential hardware and software tools to help solve these problems is provided.

Martinez, Pedro A.↗

Towards G2G: Systems of Technology Database Systems

We present an approach and methodology for developing Government-to-Government (G2G) Systems of Technology Database Systems. G2G will deliver technologies for distributed and remote integration of technology data for internal use in analysis and planning as well as for external communications. G2G enables NASA managers, engineers, operational teams and information systems to "compose" technology roadmaps and plans by selecting, combining, extending, specializing and modifying components of technology database systems. G2G will interoperate information and knowledge that is distributed across organizational entities involved that is ideal for NASA future Exploration Enterprise. Key contributions of the G2G system will include the creation of an integrated approach to sustain effective management of technology investments that supports the ability of various technology database systems to be independently managed. The integration technology will comply with emerging open standards. Applications can thus be customized for local needs while enabling an integrated management of technology approach that serves the global needs of NASA. The G2G capabilities will use NASA s breakthrough in database "composition" and integration technology, will use and advance emerging open standards, and will use commercial information technologies to enable effective System of Technology Database systems.

Maluf, David A.↗

Payload carrier systems for conducting sortie mode science

The capabilities and characteristics of the payload carriers developed to provide structural and operational interfaces between the Space Shuttle and the various types of experiments designed to operate in the sortie mode are discussed. The Spacelab is a flexible laboratory system composed of interchangeable elements that can be put together in eight different combinations of pallets and pressurized modules, and provides considerable standard services to users in such areas as equipment installation, power distribution, thermal control, command and data management, software, pointing systems and crew participation. A modular three-axis pointing control system designated the Annular Suspension and Pointing System, is being developed to provide additional pointing capabilities to those payloads that require capabilities not provided by the Spacelab instrument pointing system. Two engineering models of the Spacelab pallet have been designated Orbital Flight Test Pallets which, together with a special experiment support structure, are intended for initial and operational payloads that do not constitute a complete Spacelab mission. The simplest and smallest payload carriers are the Getaway Special cans, intended for small, self-contained, self-sufficient payloads, and the orbiter middeck lockers. In this way, most of the user requirements for Shuttle sortie missions identified to date can be fulfilled.

Jean, O. C.↗