Search NASA⌕ Search

SEARCH · Search NASA

Results for “data transfer”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 577 records · Page 32

Improving the Effectiveness of Burst Buffers for Big Data Processing in HPC Systems with Eley

Here, burst buffer is an effective solution for reducing the data transfer time and the I/O interference in HPC systems. Extending Burst Buffers (BBs) to handle Big Data applications is challenging because BBs must account for the large data inputs of Big Data applications and the Quality-of-Service (QoS) of HPC applications which are considered as first-class citizens in HPC systems. Existing BBs focus on only intermediate data of Big Data applications and incur a high performance degradation of both Big Data and HPC applications. We present Eley , a burst buffer solution that helps to accelerate the performance of Big Data applications while guaranteeing the QoS of HPC applications. To achieve this goal, Eley embraces interference-aware prefetching technique that makes reading data input faster while introducing low interference for HPC applications. Evaluations using a wide range of Big Data and HPC applications demonstrate that Eley improves the performance of Big Data applications by up to 30% compared to existing BBs while maintaining the QoS of HPC applications.

97 MATHEMATICS AND COMPUTING↗

Federated learning for 2D synchrotron x-ray diffractometry: a cross-institutional approach for phase quantification of Ti–6Al–4V alloy

High-energy Two dimensional (2D) synchrotron x-ray diffractometry provides important insights into the atomistic structure and phase evolution of materials, yet traditional analysis methods remain complex, knowledge-intensive, and computationally demanding. Deep-learning models offer a powerful alternative for automating their analysis. Institutions that hold these datasets may be unwilling to share their data due to privacy and security policies, as well as the challenges associated with large-scale data transfer. As a result, models trained on local datasets often perform well only on their own data but exhibit bias and poor generalization across different instruments or facilities. To overcome these limitations, we explore federated learning (FL) for 2D synchrotron diffractograms, enabling collaborative model training without exchanging raw data. In this study, 2D synchrotron diffractograms of Ti–6Al–4V alloy collected from two independent facilities are used to train convolutional neural networks for predicting the β-phase volume fraction. Experimental results show that federated global models significantly outperform locally trained models in terms of generalization and achieve accuracy comparable to centralized trained models. These findings demonstrate the potential of FL to enable secure, cross-institutional collaboration and enhance the scalability of deep-learning-based materials characterization.

36 MATERIALS SCIENCE↗

Virtual Neuron: A Neuromorphic Approach for Encoding Numbers

Neuromorphic computers perform computations by emulating the human brain and are expected to be indispensable for energy-efficient computing in the future. They are primarily used in spiking neural network-based machine learning applications. However, neuromorphic computers are unable to preprocess data for these applications. Currently, data is preprocessed on a CPU or a GPU-this incurs a significant cost of transferring data from the CPU/GPU to the neuromorphic processor and vice versa. This cost can be avoided if preprocessing is done on the neuromorphic processor. To efficiently preprocess data on a neuromorphic processor, we first need an efficient mechanism for encoding data that can lend itself to all general-purpose preprocessing operations. Current encoding approaches have limited applicability and may not be suitable for all preprocessing operations. In this paper, we present the virtual neuron as a mechanism for encoding integers and rational numbers on neuromorphic processors. We evaluate the performance of the virtual neuron on physical and simulated neuromorphic hardware and show that it can perform an addition operation using 23 nJ of energy on average using a mixed-signal, memristor-based neuromorphic processor. The virtual neuron encoding approach is the first step in preprocessing data on a neuromorphic processor.

Date, Prasanna↗

Streaming Data from Experimental Facilities to Supercomputers for Real-Time Data Processing

In this paper we demonstrate direct data streaming from instruments and detectors at a large-scale experimental facility to a supercomputer for real-time data processing and feedback. Streaming data to supercomputers introduces the potential for novel scientific applications and workflow models, including the ability to provide real-time feedback from very large datasets during an experiment and the integration of real-time ML training and inference at scale. We discuss a successful demonstration for real-time processing of data from the Advanced Photon Source (APS) on the Polaris supercomputer using an EPICS-based streaming framework. We describe the capabilities of the streaming framework itself, and outline the architecture that allows us to process experimentally derived data on a supercomputer without file-based data transfers. We present throughput measurements that are indicative of system performance capable of sustaining the expected data production rates of the facility, as well as discuss some outstanding challenges and our future directions.

real-time processing↗

Pan Air Geometry Management System (PAGMS): A data-base management system for PAN AIR geometry data

A data-base management system called PAGMS was developed to facilitate the data transfer in applications computer programs that create, modify, plot or otherwise manipulate PAN AIR type geometry data in preparation for input to the PAN AIR system of computer programs. PAGMS is composed of a series of FORTRAN callable subroutines which can be accessed directly from applications programs. Currently only a NOS version of PAGMS has been developed.

Hall, J. F.↗

Advanced Utilization of the Payload Executive Processor (PEP) Ethernet Port to Support JSL Based Payloads

Increased interest by Payload Developers in utilization of the Joint Station LAN (Local Area Network) (JSL) for command and data transactions has driven the investigation of providing bi-directional Ethernet communication with PEP (Payload Executive Processor). Ethernet-only payload developers are interested in taking advantage of the services provided by PEP. Enabling Ethernet communications within PEP requires more than just turning on the hardware. PEP contains no Operating System (OS) or Ethernet stack so it does not inherit any built in functionality for Ethernet support. Both hardware drivers and application software must be developed from scratch. In the absence of an OS (Operating System) the design must address security issues in terms of access and data transferred, i.e. blocking unauthorized users and preventing denial of service. Developing a design to process Ethernet data within the existing real-time constraints of PEP requires a few compromises: data rates, supported protocols, custom packet format and limited client connections to name a few. Integration of the light weight internet protocol (LwIP). Ethernet stack provides a customizable solution for the embedded real-time environment. As utilization of the JSL increases, the successful implementation of an Ethernet interface enhances the utilization of PEP, and provides a new path for future Ethernet-only payloads to gain access to around the clock command and data services.

Guyette, Greg↗

A scalable framework for efficient coupling of thermal and microstructural simulations in additive manufacturing

Predicting microstructure evolution in metal additive manufacturing (AM) is important for process optimization, but spatiotemporal scale disparities between thermal transport and microstructure evolution create significant challenges for efficient data transfer between simulation codes. To address this, we present Stork, a scalable framework for coupling thermal and microstructural simulations. Stork uses a sparse data representation to identify and store active solidification sub-volumes, enabling highly parallel quad-linear interpolation from coarse thermal grids to fine microstructure grids without large intermediate storage. We demonstrate the framework by coupling the semi-analytic heat transfer code 3DThesis with the time-parallel cellular automata code Toucan. This approach achieves over two orders of magnitude reduction in data generation time and file size compared to prior workflows. Numerical studies show that quad-linear interpolation preserves grain morphology and crystallographic texture in laser powder bed fusion (LPBF) simulations for coarsening ratios up to 16. Overall, Stork provides a scalable pathway for high-throughput, component-scale AM simulations on modern high-performance computing systems.

36 MATERIALS SCIENCE↗

High Energy Electromagnetic Particle Transportation on the GPU

We present massively parallel high energy electromagnetic particle transportation through a finely segmented detector on a Graphics Processing Unit (GPU). Simulating events of energetic particle decay in a general-purpose high energy physics (HEP) detector requires intensive computing resources, due to the complexity of the geometry as well as physics processes applied to particles copiously produced by primary collisions and secondary interactions. The recent advent of hardware architectures of many-core or accelerated processors provides the variety of concurrent programming models applicable not only for the high performance parallel computing, but also for the conventional computing intensive application such as the HEP detector simulation. The components of our prototype are a transportation process under a non-uniform magnetic field, geometry navigation with a set of solid shapes and materials, electromagnetic physics processes for electrons and photons, and an interface to a framework that dispatches bundles of tracks in a highly vectorized manner optimizing for spatial locality and throughput. Core algorithms and methods are excerpted from the Geant4 toolkit, and are modified and optimized for the GPU application. Program kernels written in C/C++ are designed to be compatible with CUDA and OpenCL and with the aim to be generic enough for easy porting to future programming models and hardware architectures. To improve throughput by overlapping data transfers with kernel execution, multiple CUDA streams are used. Issues with floating point accuracy, random numbers generation, data structure, kernel divergences and register spills are also considered. Performance evaluation for the relative speedup compared to the corresponding sequential execution on CPU is presented as well.

Canal, P.↗

Resource Management for Processing Wide Area Data Streams on Supercomputers

Modern scientific instruments generate enormous amount of data. Typically, the data collected from the instruments are stored in one or more files that are then moved to a distant supercomputer for processing. The final results are sent back to the user. In order to make effective use of the time on expensive instruments, experimenters want to process the data as they are generated. They want to stream the data from instruments' memory directly to a supercomputer's memory for analysis. Since the compute nodes in a supercomputer are not connected directly to the wide area network, the data streams need to be passed through intermediate gateway nodes. As opposed to the best effort file transfers, data streaming applications require resources at a specific time for a specific period. In this paper, we present a system model for enabling data streaming through gateway nodes and an algorithm to efficiently allocate gateway node resources along with compute nodes. We evaluate the algorithm using real-world traces on the Chameleon Cloud. The results show that our system can schedule compute and gateway resources efficiently for streaming analysis.

data streaming↗

E4D_RT

Time-lapse ERT imaging for monitoring both natural and engineered subsurface processes has advanced rapidly over the past 15-20 years. However, imaging results generally required a significant amount of manual and computational effort, and therefore were not available while the process was occurring. Although the value of real-time imaging was recognized, several obstacles prevented it's implementation. E4D_RT addresses these obstacles by 1) providing specialized algorithms that negate the need for user intervention, thereby automating the time-lapse data processing steps, 2) linking field data collection systems with parallel supercomputing systems via wireless data transfer link, and 3) addressing the computational burdens by utilizing distributed memory supercomputing resources, thereby enabling rapid data processing and imaging results.

Source record↗

BULKI-Store v0.3.2

BULKI-Store is a distributed object storage system optimized for high-performance computing environments. Built with a Rust core and Python bindings, it efficiently manages scientific and machine learning datasets across HPC clusters. The system employs a client-server architecture with MPI integration, enabling seamless scaling on supercomputers like Perlmutter. BULKI-Store's object-oriented approach provides intuitive data organization with rich metadata support, contrasting with traditional file-based solutions. Key optimizations include selective checkpoint loading, unified checkpoint files, and object chunking for large data transfers. For machine learning workloads, BULKI-Store offers advantages through fine-grained access patterns, dynamic data sharing between training instances, and reduced memory pressure. Memory management features include strategic Python GC calls, minimized data copies, and batch processing capabilities. The system leverages Rayon's thread pool for asynchronous data prefetching and supports multiple CPU architectures (ARM64, x86, AMD, RISC-V). By combining performance optimizations with developer-friendly APIs, BULKI-Store addresses the complex data management challenges of modern HPC applications while maintaining compatibility across heterogeneous computing environments.

Zhang, Wei [Lawrence Berkeley National Laboratory ↗

A 128K-bit CCD buffer memory system

A prototype system was implemented to demonstrate that CCD's can be applied advantageously to the problem of low power digital storage and particularly to the problem of interfacing widely varying data rates. 8K-bit CCD shift register memories were used to construct a feasibility model 128K-bit buffer memory system. Peak power dissipation during a data transfer is less than 7 W., while idle power is approximately 5.4 W. The system features automatic data input synchronization with the recirculating CCD memory block start address. Descriptions are provided of both the buffer memory system and a custom tester that was used to exercise the memory. The testing procedures and testing results are discussed. Suggestions are provided for further development with regards to the utilization of advanced versions of CCD memory devices to both simplified and expanded memory system applications.

Siemens, K. H.↗

A wide bandwidth CCD buffer memory system

A prototype system was implemented to demonstrate that CCD's can be applied advantageously to the problem of low power digital storage and particularly to the problem of interfacing widely varying data rates. CCD shift register memories (8K bit) were used to construct a feasibility model 128 K-bit buffer memory system. Serial data that can have rates between 150 kHz and 4.0 MHz can be stored in 4K-bit, randomly-accessible memory blocks. Peak power dissipation during a data transfer is less than 7 W, while idle power is approximately 5.4 W. The system features automatic data input synchronization with the recirculating CCD memory block start address. System expansion to accommodate parallel inputs or a greater number of memory blocks can be performed in a modular fashion. Since the control logic does not increase proportionally to increase in memory capacity, the power requirements per bit of storage can be reduced significantly in a larger system.

Siemens, K.↗

A proposed group management scheme for XTP multicast

The purpose of a group management scheme is to enable its associated transfer layer protocol to be responsive to user determined reliability requirements for multicasting. Group management (GM) must assist the client process in coordinating multicast group membership, allow the user to express the subset of the multicast group that a particular multicast distribution must reach in order to be successful (reliable), and provide the transfer layer protocol with the group membership information necessary to guarantee delivery to this subset. GM provides services and mechanisms that respond to the need of the client process or process level management protocols to coordinate, modify, and determine attributes of the multicast group, especially membership. XTP GM provides a link between process groups and their multicast groups by maintaining a group membership database that identifies members in a name space understood by the underlying transfer layer protocol. Other attributes of the multicast group useful to both the client process and the data transfer protocol may be stored in the database. Examples include the relative dispersion, most recent update, and default delivery parameters of a group.

Dempsey, Bert J.↗

Influence of tube-entrance configuration on average heat-transfer coefficients and friction factors for air flowing in an Inconel tube

A heat-transfer investigation was conducted with air flowing through an electrically heated Inconel tube having either a long-approach or a right-angle-edge entrance, an inside diameter of 0.402 inch, and a length of 24 inches over a range of Reynolds numbers up to 375,000 and average inside-tube-wall temperatures up to 2000 degrees R. Good correlation of heat-transfer data was obtained for both entrances, which substantiates work previously reported. A fair correlation of friction data was obtained for both entrances. The entrance configuration had little effect on the average heat-transfer and friction coefficients.

HEAT-TRANSFER THEORY AND EXPERIMENT↗

Network and User-Perceived Performance of Web Page Retrievals

The development of the HTTP protocol has been driven by the need to improve the network performance of the protocol by allowing the efficient retrieval of multiple parts of a web page without the need for multiple simultaneous TCP connections between a client and a server. We suggest that the retrieval of multiple page elements sequentially over a single TCP connection may result in a degradation of the perceived performance experienced by the user. We attempt to quantify this perceived degradation through the use of a model which combines a web retrieval simulation and an analytical model of TCP operation. Starting with the current HTTP/l.1 specification, we first suggest a client@side heuristic to improve the perceived transfer performance. We show that the perceived speed of the page retrieval can be increased without sacrificing data transfer efficiency. We then propose a new client/server extension to the HTTP/l.1 protocol to allow for the interleaving of page element retrievals. We finally address the issue of the display of advertisements on web pages, and in particular suggest a number of mechanisms which can make efficient use of IP multicast to send advertisements to a number of clients within the same network.

Kruse, Hans↗

Systems Development, Datamining and Knowledge Discovery Abstract

This summer, 2020, during my NASA internship I worked with my mentor, Ali Shaykhian, as well as a group of four other interns: Javel Gramling, Janelisse Morales, Tristian Running Crane, and Zulmarie Jiménez. Our research and projects all differ but work together to solve datamining unstructured data into an easy to read format. Turning unstructured data into something easier to follow is important for helping quickly pull data out of larger documents, so that one doesn’t have to go through multiple pages to find certain data points. By being able to structure data pulled from a document, it can be usedto collect data from mass amounts of forms and arrange it in an easy to glance at table instead of multiple forms. I chose to focus mainly on creating form templates with both Microsoft Word and Excel and getting used to the types of data that can be collected; as well as learning where both programs differed. After I was familiar with what could be gathered, I worked towards taking data collected by a Word form and importing it into an Excel spreadsheet. By being able to transfer data from a Word document to an Excel document, there is an added layer of functionality to the datamining. Moving data around between Excel sheets isn’t that complex of a process, but when you try to import from a Word document a lot of formatting and readability can be lost. The purpose of my research is to reduce that loss by using Visual Basic scripts to clean and arrange imported data.

Makayla Amber Renfro↗

Open-Source Data Engineering at NASA: CCMC's Approach to Managing Petabyte-Scale Heliophysics Data

The Community Coordinated Modeling Center (CCMC) at NASA Goddard Space Flight Center (GSFC) leads heliophysics research by providing open access to numerous models and their outputs. Our resources are available on-demand and continuously updated with real-time data, covering sun-earth interactions across multiple domains. These domains include coronal, heliosphere, inner and global magnetosphere, ionosphere, thermosphere, and lower atmosphere interactions. Operating in a hybrid environment, CCMC utilizes both self-owned hardware and Amazon Web Services (AWS) cloud infrastructure. Managing petabytes of data across multiple locations necessitates robust data engineering solutions. To address this challenge, CCMC has adopted industry-standard and open-source tools. We use Apache Airflow as our primary data engineering platform, Python for scripting and data processing, and GitLab for version control and CI/CD. Additionally, we employ Kubernetes for containerized services, Grafana and Prometheus for metrics and monitoring, and Terraform and Puppet for reproducible infrastructure as code. This presentation will discuss lessons learned from our data engineering experiences, platforms evaluated but found unsuitable for our scientific data requirements, and specific techniques developed to enhance data transfer speed and reliability. By using these technologies effectively, CCMC continues to advance heliophysics research through efficient data management and open-access modeling.

space weather↗