SEARCH · Search NASA

Results for “storage and I/O”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10

All sources NASA NTRS DOE OSTI

DOE OSTI2021

Columnar data analysis with ATLAS analysis formats

Future analysis of ATLAS data will involve new small-sized analysis formats to cope with the increased storage needs. The smallest of these, named DAOD_PHYSLITE, has calibrations already applied to allow fast downstream analysis and avoid the need for further analysis-specific intermediate formats. This allows for application of the "columnar analysis" paradigm where operations are applied on a per-array instead of a per-event basis. We will present methods to read the data into memory, using Uproot, and also discuss I/O aspects of columnar data and alternatives to the ROOT data format. Furthermore, we will show a representation of the event data model using the Awkward Array package and present proof of concept for a simple analysis application.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1997

A Parallel Pipelined Renderer for the Time-Varying Volume Data

This paper presents a strategy for efficiently rendering time-varying volume data sets on a distributed-memory parallel computer. Time-varying volume data take large storage space and visualizing them requires reading large files continuously or periodically throughout the course of the visualization process. Instead of using all the processors to collectively render one volume at a time, a pipelined rendering process is formed by partitioning processors into groups to render multiple volumes concurrently. In this way, the overall rendering time may be greatly reduced because the pipelined rendering tasks are overlapped with the I/O required to load each volume into a group of processors; moreover, parallelization overhead may be reduced as a result of partitioning the processors. We modify an existing parallel volume renderer to exploit various levels of rendering parallelism and to study how the partitioning of processors may lead to optimal rendering performance. Two factors which are important to the overall execution time are re-source utilization efficiency and pipeline startup latency. The optimal partitioning configuration is the one that balances these two factors. Tests on Intel Paragon computers show that in general optimal partitionings do exist for a given rendering task and result in 40-50% saving in overall rendering time.

Chiueh, Tzi-Cker↗

NASA NTRS1990

High-performance optical disk mass storage for aerospace imaging systems

The specifications and techniques related to the development of the NASA Spaceflight Optical Disk Recorder (SODR) are presented. The system is being developed for the temporary storage of high-resolution sensor data that will be collected at high data rates by future space programs. Typical spaceflight recording requirements are given for the Space Station Freedom, the Earth Observing System, and some planetary missions. SODR module characteristics are listed, emphasizing the two independent optical disk recorders and the random access to 5 gigabytes of data on each side of the disks. The modular design includes two I/O ports providing 150 Mbits/sec user data rate each, and multimodule reconfigurable architecture permitting the synchronization of up to 16 modules with a maximum data rate of 4.8 gigabits/sec.

Ravner, Stephen M.↗

NASA NTRS1982

On the inversion of block tridiagonals without storage constraints

A strategy was developed to permit trade-offs between the number of floating point operations required and the storage requirements for the solution of certain difference problems, such as block tridiagonal systems of equations. This is done by recomputing some intermediate results instead of storing them. Reducing the storage to the square root of the current requirement roughly doubles the number of computations. Reducing the storage more than this tends to make the number of computations prohibitively large. In theory, if m is the order of each sub-matrix in the block tridiagonal matrix, one can solve any linear system with only 5m(2) + 1 temporary storage cells. In many cases m is a constant and quite small. For example, in solving a factored form of the three-dimensional Navier-Stokes equations, the size m of the block tridiagonals is 5. This method lends itself to efficient use on computers with parallel processing or vector processing architectures. On these computers the larger number of floating point operations is more than offset by the decrease in I/O and the increased percentage of vector operations made possible by this algorithm.

Merriam, M. L.↗

NASA NTRS1977

Programmable data communications controller requirements

The design requirements for a Programmable Data Communications Controller (PDCC) that reduces the difficulties in attaching data terminal equipment to a computer are presented. The PDCC is an interface between the computer I/O channel and the bit serial communication lines. Each communication line is supported by a communication port that handles all line control functions and performs most terminal control functions. The port is fabricated on a printed circuit board that plugs into a card chassis, mating with a connector that is joined to all other card stations by a data bus. Ports are individually programmable; each includes a microprocessor, a programmable read-only memory for instruction storage, and a random access memory for data storage.

Source record↗

NASA NTRS1990

Twin-page storage management for rapid transaction-undo recovery

This paper presents and evaluates a new twin-page disk-storage management scheme for rapid database transaction-undo recovery. In contrast to previous twin-page schemes, the present approach uses static page mapping and allows dirty pages in the main memory to be written, at any instant, onto disk without the requirement of undo logging. No explicit undo is required when a transaction is aborted. Transaction undo is implicitly performed by not subsequently fetching from disk the invalid pages updated by the aborted transaction. Performance in terms of disk I/O and CPU overhead for transaction-undo recovery is analyzed and compared with a previous approach TWIST. It is shown that the scheme achieves rapid transaction-undo recovery without degrading average system performance for various workloads, and that the scheme is well suited for applications with a large number of updates and frequent transaction aborts.

Wu, Kun-Lung↗

DOE OSTI2024

Framework for custom event sample augmentations for ATLAS analysis data

For HEP event processing, data is typically stored in column-wise synchronized containers, such as most prominently ROOT’s TTree, which have been used for several decades to store by now over 1 exabyte. These containers can combine row-wise association capabilities needed by most HEP event processing frameworks (e.g. Athena for ATLAS) with column-wise storage, which typically results in better compression and more efficient support for many analysis use-cases. One disadvantage is that these containers, TTree in the HEP use-case, require to contain the same attributes for each entry/row (representing events), which can make extending the list of attributes very costly in storage, even if those are only required for a small subsample of events. Since the initial design, the ATLAS software framework features powerful navigational infrastructure to allow storing custom data extensions for subsamples of events in separate, but synchronized containers. This allows adding event augmentations to ATLAS standard data products (such as DAOD-PHYS or PHYSLITE) avoiding duplication of those core data products, while limiting their size increase. For this functionality, the framework does not rely on any associations made by the I/O technology (i.e. ROOT), however it supports TTree friends and builds the associated index to allow for analysis outside of the ATLAS framework. A prototype based on the Long-Lived Particle search is implemented and preliminary results with this prototype will be presented. At this point, augmented data are stored within the same file as the core data. Storing them in separate files will be investigated in future, as this could provide more flexibility, e.g. certain sites may only want a subset of several augmentations or augmentations can be archived to tape once their analysis is complete.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2020

UNITY: Unified Memory and Storage Space

UNITY is a 36-month project focused on providing design and evaluate a new distributed storage paradigm that unifies the traditionally distinct application views of memory- and file-based data storage into a single scalable and resilient environment. The project is a collaboration among Oak Ridge National Laboratory (ORNL, Lead institution), Los Alamos National Labs (LANL), and Georgia Tech (GT). The main contributions of the GT team have been around development of low level systems software for best leveraging the capabilities of new types of persistent memory technologies, and for development of methods for intelligent data management across memory/storage substrates with heterogeneous components. GT contributed the Phoenix library for optimized checkpoint/restart for HPC I/O for systems with non-volatile memory (NVM), the NVStream library for NVM-specialized streaming I/O for HPC workflows, the CoMerge, Mnemo and Kleio solutions for intelligent data management on NVM-based systems. These contributions result in significant improvements in both application performance and system efficiency.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1997

International Ultraviolet Explorer Final Archive

CSC processed IUE images through the Final Archive Data Processing System. Raw images were obtained from both NDADS and the IUEGTC optical disk platters for processing on the Alpha cluster, and from the IUEGTC optical disk platters for DECstation processing. Input parameters were obtained from the IUE database. Backup tapes of data to send to VILSPA were routinely made on the Alpha cluster. IPC handled more than 263 requests for priority NEWSIPS processing during the contract. Staff members also answered various questions and requests for information and sent copies of IUE documents to requesters. CSC implemented new processing capabilities into the NEWSIPS processing systems as they became available. In addition, steps were taken to improve efficiency and throughput whenever possible. The node TORTE was reconfigured as the I/O server for Alpha processing in May. The number of Alpha nodes used for the NEWSIPS processing queue was increased to a maximum of six in measured fashion in order to understand the dependence of throughput on the number of nodes and to be able to recognize when a point of diminishing returns was reached. With Project approval, generation of the VD FITS files was dropped in July. This action not only saved processing time but, even more significantly, also reduced the archive storage media requirements, and the time required to perform the archiving, drastically. The throughput of images verified through CDIVS and processed through NEWSIPS for the contract period is summarized below. The number of images of a given dispersion type and camera that were processed in any given month reflects several factors, including the availability of the required NEWSIPS software system, the availability of the corresponding required calibrations (e.g., the LWR high-dispersion ripple correction and absolute calibration), and the occurrence of reprocessing efforts such as that conducted to incorporate the updated SWP sensitivity-degradation correction in May.

Source record↗

NASA NTRS1988

PC-CUBE: A Personal Computer Based Hypercube

PC-CUBE is an ensemble of IBM PCs or close compatibles connected in the hypercube topology with ordinary computer cables. Communication occurs at the rate of 115.2 K-band via the RS-232 serial links. Available for PC-CUBE is the Crystalline Operating System III (CrOS III), Mercury Operating System, CUBIX and PLOTIX which are parallel I/O and graphics libraries. A CrOS performance monitor was developed to facilitate the measurement of communication and computation time of a program and their effects on performance. Also available are CXLISP, a parallel version of the XLISP interpreter; GRAFIX, some graphics routines for the EGA and CGA; and a general execution profiler for determining execution time spent by program subroutines. PC-CUBE provides a programming environment similar to all hypercube systems running CrOS III, Mercury and CUBIX. In addition, every node (personal computer) has its own graphics display monitor and storage devices. These allow data to be displayed or stored at every processor, which has much instructional value and enables easier debugging of applications. Some application programs which are taken from the book Solving Problems on Concurrent Processors (Fox 88) were implemented with graphics enhancement on PC-CUBE. The applications range from solving the Mandelbrot set, Laplace equation, wave equation, long range force interaction, to WaTor, an ecological simulation.

Ho, Alex↗

DOE OSTI2020

NCSP Analytical Methods Subtask 3, AMPX Development and Maintenance, and NCSP Nuclear Data Subtask 6, SAMMY Modernization

The modernization of SAMMY continued with the removal of all container array access for the main program. The container array was used due to the computer limitations at the time of the initial writing of SAMMY, but it prevented the use of some modern code testing options. Additionally, the authors continued to make the code more modular by shifting the storage of the energy grid and the covariance information to C++ storage, thus allowing for better unit testing and the elimination of scratch files. For AMPX, the authors continued to implement the new Evaluated Nuclear Data File format to allow AMPX to read new evaluations as they become available.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2021

Feasibility of using distributed chemical sensing for CO2 leakage monitoring

The feasibility study focused on the development of new fiber for distributed chemical sensing (DCS) that will allow direct detection of CO 2 leakages in the environment. This is particularly important for monitoring well integrity for carbon capture and storage (CCS), to provide early warning for an incoming well failure and potential CO 2 leaking through it. We proposed using optical spectroscopy in optical fiber for direct detection of CO 2 . The main approach is based on Raman interrogation within gas-filled Holey Fibers (HoFs), so that the location and concentration of the gases would be provided simultaneously via backscattering. Additionally, Infrared (IR) Absorption Spectroscopy could also be used, and the architecture would be more discrete since interleaving with standard solid core fibers and Fiber Bragg Grating (FBG) sections is required to enable reflection to the I/O controls. The possible Raman length or the IR numbers of sections would be defined based on signal to noise ratio. The optical spectroscopy methodology would overcome current roadblocks to CCS, as fiber optics will allow for CO 2 (and other gases) detection in wells with direct in-situ measurements of concentration along with other important parameters such as temperature and pressure as the baseline of environment background.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2022

Understanding the Impact of Data Staging for Coupled Scientific Workflows

We report the rate of data generated by cutting-edge experimental science facilities and large-scale simulations enabled by current high-performance computing (HPC) systems has continued to grow at a far greater pace than the development of the network and storage capabilities on which these systems rely. To cope with this challenge, scientist are moving toward the creation of autonomous experiments and HPC simulations using machine learning. However, efficiently moving, storing, and processing large amounts of data away from the point of origin presents an incredible challenge. In-memory computing, in situ analysis, data staging, and data streaming are recognized viable alternatives to traditional file-based methods for transferring data between coupled workflows. However, the performance trade-offs and limitations for these methods are not fully understood when used in HPC applications. This article presents a comprehensive performance assessment of the current solutions for data staging when applied to applications that are not necessary I/O intensive which makes them not ideal candidates for these methods. Our study is based on experiments running at scale on Oak Ridge National Laboratory's Summit supercomputer using applications and simulations that cover typical computational motifs and patterns. We investigated the usability and cost/benefit trade-offs of staging algorithms for HPC applications under different scenarios and highlight opportunities for optimizing the dataflow between coupled simulation workflows.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1992

Data storage and retrieval system abstract

The STX mass storage system design is intended for environments requiring high speed access to large volumes of data (terabyte and greater). Prior to commitment to a product design plan, STX conducted an exhaustive study of the commercially available off-the-shelf hardware and software. STX also conducted research into the area of emerging technologies in networks and storage media so that the design could easily accommodate new interfaces and peripherals as they came on the market. All the selected system elements were brought together in a demo suite sponsored jointly by STX and ALLIANT where the system elements were evaluated based on actual operation using a client-server mirror image configuration. Testing was conducted to assess the various component overheads and results were compared against vendor data claims. The resultant system, while adequate to meet our capacity requirements, fell short of transfer speed expectations. A product team lead by STX was assembled and chartered with solving the bottleneck issues. Optimization efforts yielded a 60 percent improvement in throughput performance. The ALLIANT computer platform provided the I/O flexibility needed to accommodate a multitude of peripheral interfaces including the following: up to twelve 25MB/s VME I/O channels; up to five HiPPI I/O full duplex channels; IPI-s, SCSI, SMD, and RAID disk array support; standard networking software support for TCP/IP, NFS, and FTP; open architecture based on standard RISC processors; and V.4/POSIX-based operating system (Concentrix). All components including the software are modular in design and can be reconfigured as needs and system uses change. Users can begin with a small system and add modules as needed in the field. Most add-ons can be accomplished seamlessly without revision, recompilation or re-linking of software.

Matheson, Barbara↗

NASA NTRS1993

Rapid transaction-undo recovery using twin-page storage management

This paper presents a twin-page storage method, which is an alternative to the TWIST (twin slot) approach by Reuter, for rapid transaction-undo recovery. In contrast to TWIST, our twin-page approach allows dirty pages in the buffer to be written at any instant onto disk without the requirement of undo logging, and, when a transaction is aborted, no explicit undo is required. As a result, all locks accumulated by the aborted transaction can be released earlier, allowing other transactions waiting for the locks to proceed. Through maintenance of aborted transaction identifiers, invalid pages written by the aborted transaction coexist with other valid pages and are guaranteed not be accessed by subsequent transactions. Instead of an explicit undo, most of the invalid pages are overwritten by subsequent normal updates. Performance in terms of disk I/O and CPU overhead for transaction-undo recovery is analyzed and compared with TWIST. It is shown that our scheme is particularly suited for applications where there are a large number of updates written onto disk when transactions are aborted, and there are frequent aborts. The approach, however, is not as applicable to environments where transactions are typically short or rarely aborted, or most updates are not written onto disk before a commitment.

Wu, Kun-Lung↗

NASA NTRS2012

A Spacecraft Housekeeping System-on-Chip in a Radiation Hardened Structured ASIC

Housekeeping systems are essential to health monitoring of spacecraft and instruments. Typically, sensors are distributed across various sub-systems and data is collected using components such as analog-to-digital converters, analog multiplexers and amplifiers. In most cases programmable devices are used to implement the data acquisition control and storage, and the interface to higher level systems. Such discrete implementations require additional size, weight, power and interconnect complexity versus an integrated circuit solution, as well as the qualification of multiple parts. Although commercial devices are readily available, they are not suitable for space applications due the radiation tolerance and qualification requirements. The Housekeeping System-o n-A-Chip (HKSOC) is a low power, radiation hardened integrated solution suitable for spacecraft and instrument control and data collection. A prototype has been designed and includes a wide variety of functions including a 16-channel analog front-end for driving and reading sensors, analog-to-digital and digital-to-analog converters, on-chip temperature sensor, power supply current sense circuits, general purpose comparators and amplifiers, a 32-bit processor, digital I/O, pulse-width modulation (PWM) generators, timers and I2C master and slave serial interfaces. In addition, the device can operate in a bypass mode where the processor is disabled and external logic is used to control the analog and mixed signal functions. The device is suitable for stand-alone or distributed systems where multiple chips can be deployed across different sub-systems as intelligent nodes with computing and processing capabilities.

Suarez, George↗

NASA NTRS1993

Storage media pipelining: Making good use of fine-grained media

This paper proposes a new high-performance paradigm for accessing removable media such as tapes and especially magneto-optical disks. In high-performance computing the striping of data across multiple devices is a common means of improving data transfer rates. Striping has been used very successfully for fixed magnetic disks improving overall system reliability as well as throughput. It has also been proposed as a solution for providing improved bandwidth for tape and magneto-optical subsystems. However, striping of removable media has shortcomings, particularly in the areas of latency to data and restricted system configurations, and is suitable primarily for very large I/Os. We propose that for fine-grained media, an alternative access method, media pipelining, may be used to provide high bandwidth for large requests while retaining the flexibility to support concurrent small requests and different system configurations. Its principal drawback is high buffering requirements in the host computer or file server. This paper discusses the possible organization of such a system including the hardware conditions under which it may be effective, and the flexibility of configuration. Its expected performance is discussed under varying workloads including large single I/O's and numerous smaller ones. Finally, a specific system incorporating a high-transfer-rate magneto-optical disk drive and autochanger is discussed.

Vanmeter, Rodney↗

DOE OSTI2024

MLEC-Sim: A Simulator for Evaluating Multi-Level Erasure Coding

We present MLEC-Sim, a sophisticated simulator for Multi-Level Erasure Coding (MLEC), developed in approximately 13 KLOC. The simulator is engineered to analyze the impact of various system configurations and erasure coding policies on system durability and network overhead. It supports a comprehensive range of parameters including disk capacity, disk I/O bandwidth, failure rates, network bandwidth, and system scale, accommodating various erasure coding approaches such as Single-Level Erasure Coding (SLEC), Multi-Level Erasure Coding (MLEC), and Local Reconstruction Codes (LRC). MLEC-Sim provides support for multiple chunk placement policies, including clustered parity and declustered parity, and encompasses a variety of repair methods like Repair-ALL, Repair-FCO, Repair-HYB, and Repair-MIN. It is capable of simulating disk failures through a variety of means, including distribution-based or trace-based mechanisms, and can handle complex multi-level (de)clustered placements and repair processes. A key feature of MLEC-Sim is its adoption of the splitting simulation method for evaluating system durabilities at extremely high levels, which are challenging to assess with traditional simulation approaches. This feature allows for a detailed evaluation of system resilience under a range of conditions, aiding in the selection of appropriate erasure coding solutions for enhancing system durability. MLEC-Sim contributes to the field of data storage and reliability by providing a tool for the detailed evaluation of the durability and efficiency of erasure coding configurations, intended for use by researchers and practitioners in the design and optimization of storage systems.

Wang, Meng↗