Search NASA⌕ Search

SEARCH · Search NASA

Results for “File Transfer Service”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

FTS3: Data Movement Service in containers deployed in OKD [Slides]

The File Transfer Service (FTS3) is a data movement service developed at CERN which is used to distribute the majority of the Large Hadron Collider's data across the Worldwide LHC Computing Grid (WLCG) infrastructure. At Fermilab, we have deployed a couple of FTS3 instances for Intensity Frontier experiments (e.g. DUNE) to transfer data in America and Europe, using a container-based strategy. During this talk, we are going to present the two different configurations currently running at Fermilab, comparing and contrasting a Docker-based OKD deployment against a traditional RPM-based deployment and giving an overview of the possible issues encountered. In addition, we discuss our method of certificate management and maintenance utilizing Kubernetes cronjobs.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

FTS3: Data Movement Service in containers deployed in OKD

The File Transfer Service (FTS3) is a data movement service developed at CERN which is used to distribute the majority of the Large Hadron Collider's data across the Worldwide LHC Computing Grid (WLCG) infrastructure. At Fermilab, we have deployed FTS3 instances for Intensity Frontier experiments (e.g. DUNE) to transfer data in America and Europe, using a container-based strategy. In this article we summarize our experience building docker images based on work from the SLATE project (slateci.io) and deployed in OKD, the community distribution of Red Hat OpenShift. Additionally, we discuss our method of certificate management and maintenance utilizing Kubernetes CronJobs. Finally, we also report on the two different configurations currently running at Fermilab, comparing and contrasting a Docker-based OKD deployment against a traditional RPM-based deployment.

Lobato Pardavila, Lorena↗
DOE OSTI2021

FTS3: Data Movement Service in containers deployed in OKD [Slides]

The File Transfer Service (FTS3) is a data movement service developed at CERN which is used to distribute the majority of the Large Hadron Collider's data across the Worldwide LHC Computing Grid (WLCG) infrastructure. At Fermilab, we have deployed FTS3 instances for Intensity Frontier experiments (e.g. DUNE) to transfer data in America and Europe, using a container-based strategy. In this article we summarize our experience building docker images based on work from the SLATE project (slateci.io) and deployed in OKD, the community distribution of Red Hat OpenShift. Additionally, we discuss our method of certificate management and maintenance utilizing Kubernetes CronJobs. Finally, we also report on the two different configurations currently running at Fermilab, comparing and contrasting a Docker-based OKD deployment against a traditional RPM-based deployment.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1987

Performance analysis of an optimal file transfer protocol for integrated mobile satellite services

A file-transfer protocol based on the selective-repeat automatic-retransmission-request (ARQ) scheme has been developed for the Mobile Satellite Experiment (MSAT-X). In this protocol, each file is divided into packets, which are error-detected and acknowledge individually. A file can be transferred in either the open- or closed-end connection. In the open-end connection, the file is transmitted continuously according to the selective-repeat scheme until all packets are positively acknowledged; in the closed-end connection, each retransmission will initiate its own connection request. The relationship between the packet size and the average end-to-end delay in the mobile fading environment is investigated. The communications channel uses trellis-coded modulation with symbol interleaving, packet replication, and error detection. The packet error performance in such a channel is evaluated by simulation, and the optimal packet size for various operating conditions is derived, using the simulated results. It is shown that for a symbol-energy-to-noise ratio of 12 dB, each file should be divided into packets of length between 256 bits and 512 bits (excluding the overhead) to minimize the average end-to-end delay.

Wang, Charles C.↗
DOE OSTI2025

Toward a persistent event-streaming system for high-performance computing applications

High-performance computing (HPC) applications have traditionally relied on parallel file systems and file transfer services to manage data movement and storage. Alternative approaches have been proposed that use direct communications between application components, trading persistence and fault tolerance for speed. Event-driven architectures, as popularized in enterprise contexts, present a compelling middle ground, avoiding the performance cost and API constraints of parallel file systems while retaining persistence and offering impedance matching between application components. However, adapting streaming frameworks to HPC workloads requires addressing challenges unique to HPC systems. This paper investigates the potential for a streaming framework designed for HPC infrastructures and use cases. We introduce Mofka, a persistent event-streaming framework designed specifically for HPC environments. Mofka combines the capabilities of a traditional streaming service with optimizations tailored to the HPC context, such as support for massively multicore nodes, efficient scaling for large producer-consumer workflows, RDMA-enabled high-performance network communications, specialized network fabrics with multiple links per node, and efficient handling of large scientific data payloads. Built using the Mochi suite of HPC data service components, Mofka provides a lightweight, modular, and high-performance solution for persistent streaming in HPC systems. We present the architecture of Mofka and evaluate its performance against Kafka and Redpanda using benchmarks on diverse platforms, including Argonne's Polaris and Oak Ridge's Frontier supercomputers, showing up to 8× improvement in throughput in some scenarios. We then demonstrate its utility in several real-world applications: a tomographic reconstruction pipeline, a workflow for the discovery of metal-organic frameworks for carbon capture, and the instrumentation of Dask workflows for provenance tracking and performance analysis.

HPC↗
DOE OSTI2026

Operational Evolution of FTS3: A DevOps Driven Approach to Elastic Operations

The File Transfer Service (FTS3) is a distributed data movement service developed at CERN and widely used to transfer data across the Worldwide LHC Computing Grid (WLCG). At Fermilab, FTS3 supports data transfers for multiple experiments, including Intensity Frontier experiments such as DUNE, enabling reliable data movement between WebDAV endpoints in Europe and the Americas.​ At CHEP 2021, we reported on the initial containerized deployment of FTS3 on OKD, the community Kubernetes distribution of Red Hat OpenShift. In this work, we present the subsequent evolution of this deployment, focusing on new operational capabilities introduced to improve scalability, robustness, and long-term maintainability.​ We describe the adoption of more secure and reproducible container build workflows, the integration of DevOps-driven operational practices, and enhancements in monitoring and automation. A key new result is the introduction of horizontal scaling and elastic resource management, allowing FTS3 components to dynamically adapt to workload variations while maintaining service reliability. We also discuss improvements in fault tolerance and operational procedures derived from production experience.​ Finally, we summarize lessons learned from operating FTS3 as a Kubernetes-native service and outline how these developments have improved the resilience and efficiency of data movement operations at Fermilab.

Munoz Flores, Victor Leopoldo [Fermilab]↗
NASA NTRS2004

ViDI: Virtual Diagnostics Interface: Unified File Format and Web Services as Applied to Seamless Data Transfer - Volume 2

The desire to revolutionize the aircraft design cycle from its currently lethargic pace to a fast turn-around operation enabling the optimization of non-traditional configurations is a critical challenge facing the aeronautics industry. In response, a large scale effort is underway to not only advance the state of the art in wind tunnel testing, computational modeling, and information technology, but to unify these often disparate elements into a cohesive design resource. This paper will address Seamless Data Transfer, the critical central nervous system that will enable a wide variety of varied components to work together.

Fleming, Gary A.↗
NASA NTRS1990

Signalling design and architecture for a proposed mobile satellite network

In a frequency-division/demand-assigned multiple-access (FD/DAMA) architecture, each mobile subscriber must make a connection request to the Network Management Center before transmission for either open-end or closed-end services. Open-end services are for voice calls and long file transfer and are processed on a blocked-call-cleared basis. Closed-end services are for transmitting burst data and are processed on a first-come first-served basis. This paper presents the signalling design and architecture for non-voice services of an FD/DAMA mobile satellite network. The connection requests are made through the recently proposed multiple channel collision resolution scheme which provides a significantly higher throughput than the traditional slotted ALOHA scheme. For non-voice services, it is well known that retransmissions are necessary to ensure the delivery of a message in its entirety from the source to destination. Retransmission protocols for open-end and closed-end data transfer are investigated. The signal structure for the proposed network is derived from X-25 standards with appropriate modifications. The packet types and their usages are described in this paper.

Yan, T.-Y.↗
NASA NTRS1988

A framework for implementing data services in multi-service mobile satellite systems

Mobile satellite systems being planned for introduction in the early 1990s are expected to be invariably of the multi-service type. Mobile Telephone Service (MTS), Mobile Radio Service (MRS), and Mobile Data Service (MDS) are the major classifications used to categorize the many user applications to be supported. The MTS and MRS services encompass circuit-switched voice communication applications, and may be efficiently implemented using a centralized Demand-Assigned Multiple Access (DAMA) scheme. Applications under the MDS category are, on the other hand, message-oriented and expected to vary widely in characteristics; from simplex mode short messaging applications to long duration, full-duplex interactive data communication and large file transfer applications. For some applications under this service category, the conventional circuit-based DAMA scheme may prove highly inefficient due to the long time required to set up and establish communication links relative to the actual message transmission time. It is proposed that by defining a set of basic bearer services to be supported in MDS and optimizing their transmission and access schemes independent of the MTS and MRS services, the MDS applications can be more efficiently integrated into the multi-service design of mobile satellite systems.

Ali, Mohammed O.↗
DOE OSTI2022

DYnamic and Asynchronous Data Streamliner

DYAD aims to help sharing data files between producer and consumer job elements, especially between co-scheduled jobs or within an ensemble. DYAD provides the service by two components: a FLUX module and a I/O wraper set. DYAD transparently synchronizes file I/O between producer and consumer, and transfers data from the producer location to the consumer location managed by the service. Users only need to use the file path that is under the directory managed by the service.

Ahn, DongH↗
NASA NTRS2008

Mars Interoperability 2008-2015: Options for Relay Orbiter Support to Mars Bound Assets

The current relay orbiter infrastructure at Mars, in support of the user assets presently at Mars and those scheduled for the 2008 to 2015 time frame, only have a need to store-and-forward the returned data collected by an asset to Earth as a single non-prioritized data file. In the forward direction, the relay orbiters are only required to relay the forward link data, i.e., command sequences, software and configuration information assembled on the ground, to the Asset. There are currently no requirements for status/control messages or files to be sent in the return direction from an Asset to an application located on the Relay, nor are there requirements in the forward direction to send status/control messages or files originating on a Relay to an Asset. In addition there are currently no networking requirements at Mars where data originally sent by an Asset is to be delivered to another user asset. However, we foresee the day when standard services for 1) on-board file prioritization of user asset data by a Relay, 2) control/status message transfer between assets and a Relay, and 3) networking between assets i.e., Asset to Asset message/file transfer via a Relay will be required. Each of these services will require the Relay to be capable of understanding the data type and routing needs of the user asset data and be capable of processing it for these purposes. This paper will focus on the innovative enhancements required to the existing communications infrastructure at Mars to enable these future services.

File Transfer↗
NASA NTRS1994

The Space Communications Protocol Standards Program

In the fall of 1992 NASA and the Department of Defense chartered a technical team to explore the possibility of developing a common set of space data communications standards for potential dual-use across the U.S. national space mission support infrastructure. The team focused on the data communications needs of those activities associated with on-lined control of civil and military aircraft. A two-pronged approach was adopted: a top-down survey of representative civil and military space data communications requirements was conducted; and a bottom-up analysis of available standard data communications protocols was performed. A striking intersection of civil and military space mission requirements emerged, and an equally striking consensus on the approach towards joint civil and military space protocol development was reached. The team concluded that wide segments of the U.S. civil and military space communities have common needs for: (1) an efficient file transfer protocol; (2) various flavors of underlying data transport service; (3) an optional data protection mechanism to assure end-to-end security of message exchange; and (4) an efficient internetworking protocol. These recommendations led to initiating a program to develop a suite of protocols based on these findings. This paper describes the current status of this program.

Jeffries, Alan↗
NASA NTRS2004

Automated Euler and Navier-Stokes Database Generation for a Glide-Back Booster

The past two decades have seen a sustained increase in the use of high fidelity Computational Fluid Dynamics (CFD) in basic research, aircraft design, and the analysis of post-design issues. As the fidelity of a CFD method increases, the number of cases that can be readily and affordably computed greatly diminishes. However, computer speeds now exceed 2 GHz, hundreds of processors are currently available and more affordable, and advances in parallel CFD algorithms scale more readily with large numbers of processors. All of these factors make it feasible to compute thousands of high fidelity cases. However, there still remains the overwhelming task of monitoring the solution process. This paper presents an approach to automate the CFD solution process. A new software tool, AeroDB, is used to compute thousands of Euler and Navier-Stokes solutions for a 2nd generation glide-back booster in one week. The solution process exploits a common job-submission grid environment, the NASA Information Power Grid (IPG), using 13 computers located at 4 different geographical sites. Process automation and web-based access to a MySql database greatly reduces the user workload, removing much of the tedium and tendency for user input errors. The AeroDB framework is shown. The user submits/deletes jobs, monitors AeroDB's progress, and retrieves data and plots via a web portal. Once a job is in the database, a job launcher uses an IPG resource broker to decide which computers are best suited to run the job. Job/code requirements, the number of CPUs free on a remote system, and queue lengths are some of the parameters the broker takes into account. The Globus software provides secure services for user authentication, remote shell execution, and secure file transfers over an open network. AeroDB automatically decides when a job is completed. Currently, the Cart3D unstructured flow solver is used for the Euler equations, and the Overflow structured overset flow solver is used for the Navier-Stokes equations. Other codes can be readily included into the AeroDB framework.

Chaderjian, Neal M.↗
NASA NTRS2006

Lessons Learned from Daily Uplink Operations during the Deep Impact Mission

The daily preparation of uplink products (commands and files) for Deep Impact was as problematic as the final encounter images were spectacular. The operations team was faced with many challenges during the six-month mission to comet Tempel One of the biggest difficulties was that the Deep Impact Flyby and Impactor vehicles necessitated a high volume of uplink products while also utilizing a new uplink file transfer capability. The Jet Propulsion Laboratory (JPL) Multi-Mission Ground Systems and Services (MGSS) Mission Planning and Sequence Team (MPST) had the responsibility of preparing the uplink products for use on the two spacecraft. These responsibilities included processing nearly 15,000 flight products, modeling the states of the spacecraft during all activities for subsystem review, and ensuring that the proper commands and files were uplinked to the spacecraft. To guarantee this transpired and the health and safety of the two spacecraft were not jeopardized several new ground scripts and procedures were developed while the Deep Impact Flyby and Impactor spacecraft were en route to their encounter with Tempel-1. These scripts underwent several adaptations throughout the entire mission up until three days before the separation of the Flyby and Impactor vehicles. The problems presented by Deep Impact's daily operations and the development of scripts and procedures to ease those challenges resulted in several valuable lessons learned. These lessons are now being integrated into the design of current and future MGSS missions at JPL.

uplink↗
DOE OSTI2024

Verifying Cyber Implementation Best Practices With Malcolm

Network traffic analysis can reveal a lot about what's right or wrong with a network's cybersecurity footing. Using Malcolm, a powerful open-source network traffic analysis tool suite for network security monitoring, cyber analysts and asset owners can validate cybersecurity best practices and uncover red flags in network configuration, including: proper network segmentation east-west (cross-segment) and north-south traffic unsecure or outdated network protocols authentication using clear text credentials rogue devices and services unexpected protocols (e.g., IPv6, DNS, DHCP, update checks, etc.) suspicious file transfers

99 GENERAL AND MISCELLANEOUS↗
NASA NTRS2006

Asynchronous Message Service for Deep Space Mission Operations

While the CCSDS (Consultative Committee for Space Data Systems) File Delivery Protocol (CFDP) provides internationally standardized file transfer functionality that can offer significant benefits for deep space mission operations, not all spacecraft communication requirements are necessarily best met by file transfer. In particular, continuous event-driven asynchronous message exchange may also be useful for communications with, among, and aboard spacecraft. CCSDS has therefore undertaken the development of a new Asynchronous Message Service (AMS) standard, designed to provide common functionality over a wide variety of underlying transport services, ranging from shared memory message queues to CCSDS telemetry systems. The present paper discusses the design concepts of AMS, their applicability to deep space mission operations problems, and the results of preliminary performance testing obtained from exercise of a prototype implementation.

asynchronous message exchanges↗
NASA NTRS2012

Mars Express Forward Link Capabilities for the Mars Relay Operations Service (MaROS)

This software provides a new capability for landed Mars assets to perform forward link relay through the Mars Express (MEX) European Union orbital spacecraft. It solves the problem of standardizing the relay interface between lander missions and MEX. The Mars Operations Relay Service (MaROS) is intended as a central point for relay planning and post-pass analysis for all Mars landed and orbital assets. Through the first two phases of implementation, MaROS supports relay coordination through the Odyssey orbiter and the Mars Reconnaissance Orbiter (MRO). With this new software, MaROS now fully integrates the Mars Express spacecraft into the relay picture. This new software generates and manages a new set of file formats that allows for relay request to MEX for forward and return link relay, including the parameters specific to MEX. Existing MEX relay planning interactions were performed via email exchanges and point-to-point file transfers. By integrating MEX into MaROS, all transactions are managed by a centralized service for tracking and analysis. Additionally, all lander missions have a single, shared interface with MEX and do not have to integrate on a mission-by mission basis. Relay is a critical element of Mars lander data management. Landed assets depend largely upon orbital relay for data delivery, which can be impacted by the availability and health of each orbiter in the network. At any time, an issue may occur to prevent relay. For this reason, it is imperative that all possible orbital assets be integrated into the overall relay picture.

Allard, Daniel A.↗
NASA NTRS2013

Multi-User Space Link Extension (SLE) System

The Multi-User Space (MUS) Link Extension system, a software and data system, provides Space Link Extension (SLE) users with three space data transfer services in timely, complete, and offline modes as applicable according to standards defined by the Consultative Committee for Space Data Systems (CCSDS). MUS radically reduces the schedule, cost, and risk of implementing a new SLE user system, minimizes operating costs with a lights-out approach to SLE, and is designed to require no sustaining engineering expense during its lifetime unless changes in the CCSDS SLE standards, combined with new provider implementations, force changes. No software modification to MUS needs to be made to support a new mission. Any systems engineer with Linux experience can begin testing SLE user service instances with MUS starting from a personal computer (PC) within five days. For flight operators, MUS provides a familiar-looking Web page for entering SLE configuration data received from SLE. Operators can also use the Web page to back up a space mission's entire set of up to approximately 500 SLE service instances in less than five seconds, or to restore or transfer from another system the same amount of data from a MUS backup file in about the same amount of time. Missions operate each MUS SLE service instance independently by sending it MUS directives, which are legible, plain ASCII strings. MUS directives are usually (but not necessarily) sent through a TCP-IP (Transmission Control Protocol Internet Protocol) socket from a MOC (Mission Operations Center) or POCC (Payload Operations Control Center) system, under scripted control, during "lights-out" spacecraft operation. MUS permits the flight operations team to configure independently each of its data interfaces; not only commands and telemetry, but also MUS status messages to the MOC. Interfaces can use single- or multiple-client TCP/IP server sockets, TCP/IP client sockets, temporary disk files, the system log, or standard in, standard out, or standard error as applicable. By defining MUS templates in ASCII, the flight operations team can include any MUS system variable in telemetry or command headers or footers, and/or in status messages. Data fields can be arranged within messages in different sequences, according to the mission s needs. The only constraints imposed are on the format of MUS directive strings, and some bare minimum logical requirements that must be met in order for MUS to read the mission control center's spacecraft command inputs. The MUS system imposes no limits or constraints on the numbers and combinations of missions and SLE service instances that it will support simultaneously. At any time, flight operators may add, change, delete, bind, connect, or disconnect.

Perkins, Toby↗