SEARCH · Search NASA

Results for “Parallel file systems”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13

All sources NASA NTRS DOE OSTI

NASA NTRS2019

Leveraging STARE for Co-aligned Data Locality with netCDF and Python MPI

We have leveraged STARE indexing to package partitioned data chunks from diverse datasets into netCDF files, distributed them on a cluster of 16 lightweight nodes with their placements spatiotemporally co-aligned, and demonstrated a few integrative analyses using netCDF parallel I/O and Python MPI, with single-user performance and scalability comparable to, or even better than, that of a parallel array database management system (ADBMS) such as SciDB. However, records of the node location and STARE index ranges for each data chunk, similar to the chunk maps of SciDB, must be maintained and consulted by the I/O and analysis code for coordinating the analytic operations in parallel, in order to achieve the good performance and scalability.

Kwo-Sen Kuo↗

DOE OSTI2018

Crystal structure of lubiprostone Polymorph B, C 20 H 32 F 2 O 5

The crystal structure of lubiprostone has been refined using synchrotron X-ray powder diffraction data, and optimized using density functional techniques. Lubiprostone crystallizes in space group P 1 (#1) with a = 9.02025(2), b = 10.72121(2), c = 12.32817(4) Å, α = 78.5566(2), β = 69.6858(2), γ = 77.3292(2)°, V = 1081.069(3) Å 3 , and Z = 2. The two independent molecules occur in an extended conformation, aligned approximately along the c -axis. The hydrophobic side chains are adjacent to each other, resulting in layers parallel to the ac plane. The two carboxylic acid groups form an eight-membered ring, resulting in dimers of the two independent molecules. Each hydroxyl group acts as a hydrogen bond donor to the ketone of the fused ring system. The powder pattern is included in the Powder Diffraction File™ as entry 00-066-1622.

Kaduk, James A. (ORCID:000000016610541X)↗

NASA NTRS1997

Visualization of Unsteady Computational Fluid Dynamics

The current compute environment that most researchers are using for the calculation of 3D unsteady Computational Fluid Dynamic (CFD) results is a super-computer class machine. The Massively Parallel Processors (MPP's) such as the 160 node IBM SP2 at NAS and clusters of workstations acting as a single MPP (like NAS's SGI Power-Challenge array and the J90 cluster) provide the required computation bandwidth for CFD calculations of transient problems. If we follow the traditional computational analysis steps for CFD (and we wish to construct an interactive visualizer) we need to be aware of the following: (1) Disk space requirements. A single snap-shot must contain at least the values (primitive variables) stored at the appropriate locations within the mesh. For most simple 3D Euler solvers that means 5 floating point words. Navier-Stokes solutions with turbulence models may contain 7 state-variables. (2) Disk speed vs. Computational speeds. The time required to read the complete solution of a saved time frame from disk is now longer than the compute time for a set number of iterations from an explicit solver. Depending, on the hardware and solver an iteration of an implicit code may also take less time than reading the solution from disk. If one examines the performance improvements in the last decade or two, it is easy to see that depending on disk performance (vs. CPU improvement) may not be the best method for enhancing interactivity. (3) Cluster and Parallel Machine I/O problems. Disk access time is much worse within current parallel machines and cluster of workstations that are acting in concert to solve a single problem. In this case we are not trying to read the volume of data, but are running the solver and the solver outputs the solution. These traditional network interfaces must be used for the file system. (4) Numerics of particle traces. Most visualization tools can work upon a single snap shot of the data but some visualization tools for transient problems require dealing with time.

Haimes, Robert↗

DOE OSTI2016

TU-AB-BRC-07: Efficiency of An IAEA Phase-Space Source for a Low Energy X-Ray Tube Using Egs++

Purpose: To extend the capability of the EGSnrc C++ class library (egs++) to write and read IAEA phase-space files as a particle source, and to assess the relative efficiency gain in dose calculation using an IAEA phase-space source for modelling a miniature low energy x-ray source. Methods: We created a new ausgab object to score particles exiting a user-defined geometry and write them to an IAEA phase-space file. A new particle source was created to read from IAEA phase-space data. With these tools, a phase-space file was generated for particles exiting a miniature 50 kVp x-ray tube (The INTRABEAM System, Carl Zeiss). The phase-space source was validated by comparing calculated PDDs with a full electron source simulation of the INTRABEAM. The dose calculation efficiency gain of the phase-space source was determined relative to the full simulation. The efficiency gain as a function of i) depth in water, and ii) job parallelization was investigated. Results: The phase-space and electron source PDDs were found to agree to 0.5% RMS, comparable to statistical uncertainties. The use of a phase-space source for the INTRABEAM led to a relative efficiency gain of greater than 20 over the full electron source simulation, with an increase of up to a factor of 196. The efficiency gain was found to decrease with depth in water, due to the influence of scattering. Job parallelization (across 2 to 256 cores) was not found to have any detrimental effect on efficiency gain. Conclusion: A set of tools has been developed for writing and reading IAEA phase-space files, which can be used with any egs++ user code. For simulation of a low energy x-ray tube, the use of a phase-space source was found to increase the relative dose calculation efficiency by factor of up to 196. The authors acknowledge partial support by the CREATE Medical Physics Research Training Network grant of the Natural Sciences and Engineering Research Council (Grant No. 432290).

60 APPLIED LIFE SCIENCES↗

NASA NTRS2004

Parallel Performance Characterization of Columbia

Using a collection of benchmark problems of increasing levels of realism and computational effort, we will characterize the strengths and limitations of the 10,240 processor Columbia system to deliver supercomputing value to application scientists. Scientists need to be able to determine if and how they can utilize Columbia to carry extreme workloads, either in terms of ultra-large applications that cannot be run otherwise (capability), or in terms of very large ensembles of medium-scale applications to populate response matrices (capacity). We select existing application benchmarks that scale from a small number of processors to the entire machine, and that highlight different issues in running supercomputing-calss applicaions, such as the various types of memory access, file I/O, inter- and intra-node communications and parallelization paradigms. http://www.nas.nasa.gov/Software/NPB/

Biswas, Rupak↗

NASA NTRS1985

A software toolbox for robotics

A system of programs was developed to simulate the concurrent command/response interaction between a parallel jaw end effector and the LISP program controlling it. The overall structure of the simulation system was described in a paper submitted to the IEEE Conference on Automation and Robotics. A user's guide for the system was written. A line numbering program on the VAX (Pascal), a program for aiding in file transfer from the VAX to an LSI11 over the RTNET (FORTRAN), and a file scanning program (a crude SCAN) for the LSI11 (FORTRAN) were also developed.

Bynum, W. L.↗

DOE OSTI2015

Optimized Data Transfers Based on the OpenCL Event Management Mechanism

In standard OpenCL programming, hosts are supposed to control their compute devices. Since compute devices are dedicated to kernel computation, only hosts can execute several kinds of data transfers such as internode communication and file access. These data transfers require one host to simultaneously play two or more roles due to the need for collaboration between the host and devices. The codes for such data transfers are likely to be system-specific, resulting in low portability. Here, this article proposes an OpenCL extension that incorporates such data transfers into the OpenCL event management mechanism. Unlike the current OpenCL standard, the main thread running on the host is not blocked to serialize dependent operations. Hence, an application can easily use the opportunities to overlap parallel activities of hosts and compute devices. In addition, the implementation details of data transfers are hidden behind the extension, and application programmers can use the optimized data transfers without any tricky programming techniques. The evaluation results show that the proposed extension can use the optimized data transfer implementation and thereby increase the sustained data transfer performance by about 18% for a real application accessing a big data file.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2022

Automatic variable selection in ecological niche modeling: A case study using Cassin’s Sparrow (Peucaea cassinii)

MERRA/Max provides a feature selection approach to dimensionality reduction that enables direct use of global climate model outputs in ecological niche modeling. The system accomplishes this reduction through a Monte Carlo optimization in which many independent MaxEnt runs, operating on a species occurrence file and a small set of randomly selected variables in a large collection of variables, converge on an estimate of the top contributing predictors in the larger collection. These top predictors can be viewed as potential candidates in the variable selection step of the ecological niche modeling process. MERRA/Max’s Monte Carlo algorithm operates on files stored in the underlying filesystem, making it scalable to large data sets. Its software components can run as parallel processes in a high-performance cloud computing environment to yield near real-time performance. In tests using Cassin’s Sparrow (Peucaea cassinii) as the target species, MERRA/Max selected a set of predictors from Worldclim’s Bioclim collection of 19 environmental variables that have been shown to be important determinants of the species’ bioclimatic niche. It also selected biologically and ecologically plausible predictors from a more diverse set of 86 environmental variables derived from NASA’s Modern-Era Retrospective Analysis for Research and Applications Version 2 (MERRA-2) reanalysis, an output product of the Goddard Earth Observing System Version 5 (GEOS-5) modeling system. We believe these results point to a technological approach that could expand the use global climate model outputs in ecological niche modeling, foster exploratory experimentation with otherwise difficult-to-use climate data sets, streamline the modeling process, and, eventually, enable automated bioclimatic modeling as a practical, readily accessible, low-cost, commercial cloud service.

John L. Schnase↗

NASA NTRS2011

Parallel Wavefront Analysis for a 4D Interferometer

This software provides a programming interface for automating data collection with a PhaseCam interferometer from 4D Technology, and distributing the image-processing algorithm across a cluster of general-purpose computers. Multiple instances of 4Sight (4D Technology s proprietary software) run on a networked cluster of computers. Each connects to a single server (the controller) and waits for instructions. The controller directs the interferometer to several images, then assigns each image to a different computer for processing. When the image processing is finished, the server directs one of the computers to collate and combine the processed images, saving the resulting measurement in a file on a disk. The available software captures approximately 100 images and analyzes them immediately. This software separates the capture and analysis processes, so that analysis can be done at a different time and faster by running the algorithm in parallel across several processors. The PhaseCam family of interferometers can measure an optical system in milliseconds, but it takes many seconds to process the data so that it is usable. In characterizing an adaptive optics system, like the next generation of astronomical observatories, thousands of measurements are required, and the processing time quickly becomes excessive. A programming interface distributes data processing for a PhaseCam interferometer across a Windows computing cluster. A scriptable controller program coordinates data acquisition from the interferometer, storage on networked hard disks, and parallel processing. Idle time of the interferometer is minimized. This architecture is implemented in Python and JavaScript, and may be altered to fit a customer s needs.

Rao, Shanti R.↗

NASA NTRS2017

Archive Management of NASA Earth Observation Data to Support Cloud Analysis

NASA collects, processes and distributes petabytes of Earth Observation (EO) data from satellites, aircraft, in situ instruments and model output, with an order of magnitude increase expected by 2024. Cloud-based web object storage (WOS) of these data can simplify the execution of such an increase. More importantly, it can also facilitate user analysis of those volumes by making the data available to the massively parallel computing power in the cloud. However, storing EO data in cloud WOS has a ripple effect throughout the NASA archive system with unexpected challenges and opportunities. One challenge is modifying data servicing software (such as Web Coverage Service servers) to access and subset data that are no longer on a directly accessible file system, but rather in cloud WOS. Opportunities include refactoring of the archive software to a cloud-native architecture; virtualizing data products by computing on demand; and reorganizing data to be more analysis-friendly.

Lynnes, Christopher↗

NASA NTRS2017

Archive Management of NASA Earth Observation Data to Support Cloud Analysis

Lynnes, Christopher↗

DOE OSTI2022

MPLBM-UT: Multiphase LBM library for permeable media analysis

MPLBM-UT is a specialized lattice-Boltzmann library that makes running single- and two-phase flow simulations in porous media accessible to everyone. We provide a suite of tools to pre-process computational domains for simulation, to set up custom boundary conditions, to run simulations, to post-process simulation outputs, and to visualize simulation results and data. All of these tools are easily accessible to users through the mplbm_utils Python package included in and automatically installed with MPLBM-UT. The high-performance, highly parallel library Palabos is used as the solver backend. MPLBM-UT is easily deployed in a variety of systems, from laptops to supercomputer clusters. MPLBM-UT also features multiple examples and benchmark templates that allow for fast prototyping of different porous media problems. We also provide an interface for reading in different file types and downloading domains from the Digital Rocks Portal to perform simulations.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1986

Mountain building processes in the Central Andes

False color composite images of the Thematic Mapper (TM) bands 5, 4, and 2 were examined to make visual interpretations of geological features. The use of the roam mode of image display with the International Imaging Systems (IIS) System 600 image processing package running on the IIS Model 75 was very useful. Several areas in which good comparisons with ground data existed, were examined in detail. Parallel to the visual approach, image processing methods are being developed which allow the complete use of the seven TM bands. The data was organized into easily accessible files and a visual cataloging of the quads (quarter TM scenes) with preliminary registration with the best available charts for the region. The catalog has proved to be a valuable tool for the rapid scanning of quads for a specific investigation. Integration of the data into a complete approach to the problems of uplift, deformation, and magnetism in relation to the Nazca-South American plate interaction is at an initial stage.

Bloom, A. L.↗

NASA NTRS2008

Orchestrator Telemetry Processing Pipeline

Orchestrator is a software application infrastructure for telemetry monitoring, logging, processing, and distribution. The architecture has been applied to support operations of a variety of planetary rovers. Built in Java with the Eclipse Rich Client Platform, Orchestrator can run on most commonly used operating systems. The pipeline supports configurable parallel processing that can significantly reduce the time needed to process a large volume of data products. Processors in the pipeline implement a simple Java interface and declare their required input from upstream processors. Orchestrator is programmatically constructed by specifying a list of Java processor classes that are initiated at runtime to form the pipeline. Input dependencies are checked at runtime. Fault tolerance can be configured to attempt continuation of processing in the event of an error or failed input dependency if possible, or to abort further processing when an error is detected. This innovation also provides support for Java Message Service broadcasts of telemetry objects to clients and provides a file system and relational database logging of telemetry. Orchestrator supports remote monitoring and control of the pipeline using browser-based JMX controls and provides several integration paths for pre-compiled legacy data processors. At the time of this reporting, the Orchestrator architecture has been used by four NASA customers to build telemetry pipelines to support field operations. Example applications include high-volume stereo image capture and processing, simultaneous data monitoring and logging from multiple vehicles. Example telemetry processors used in field test operations support include vehicle position, attitude, articulation, GPS location, power, and stereo images.

Powell, Mark↗

DOE OSTI2022

Accelerating Scientific Workflows on HPC Platforms with In Situ Processing

Scientific workflows drive most modern large-scale science breakthroughs by allowing scientists to define their computations as a set of jobs executed in a given order based on their data dependencies. Workflow management systems (WMSs) have become key to automating scientific workflows-executing computational jobs and orchestrating data transfers between those jobs running on complex high-performance computing (HPC) platforms. Traditionally, WMSs use files to communicate between jobs: a job writes out files that are read by other jobs. However, HPC machines face a growing gap between their storage and compute capabilities. To address that concern, the scientific community has adopted a new approach called in situ, which bypasses costly parallel filesystem I/O operations with faster in-memory or in-network communications. When using in situ approaches, communication and computations can be interleaved. In this work, we leverage the Decaf in situ dataflow framework to accelerate task-based scientific workflows managed by the Pegasus WMS, by replacing file communications with faster MPI messaging. We propose a new execution engine that uses Decaf to manage communications within a sub-workflow (i.e., set of jobs) to optimize inter-job communications. We consider two workflows in this study: (i) a synthetic workflow that benchmarks and compares file- and MPI-based communication; and (ii) a realistic bioinformatics workflow that computes mu-tational overlaps in the human genome. Experiments show that in situ communication can improve the bioinformatics workflow execution time by 22% to 30% compared with file communication. Our results motivate further opportunities and challenges for bridging traditional WMSs with in situ frameworks.

Decaf↗

NASA NTRS1995

Visualization and Tracking of Parallel CFD Simulations

We describe a system for interactive visualization and tracking of a 3-D unsteady computational fluid dynamics (CFD) simulation on a parallel computer. CM/AVS, a distributed, parallel implementation of a visualization environment (AVS) runs on the CM-5 parallel supercomputer. A CFD solver is run as a CM/AVS module on the CM-5. Data communication between the solver, other parallel visualization modules, and a graphics workstation, which is running AVS, are handled by CM/AVS. Partitioning of the visualization task, between CM-5 and the workstation, can be done interactively in the visual programming environment provided by AVS. Flow solver parameters can also be altered by programmable interactive widgets. This system partially removes the requirement of storing large solution files at frequent time steps, a characteristic of the traditional 'simulate (yields) store (yields) visualize' post-processing approach.

Vaziri, Arsi↗

DOE OSTI2014

MO-G-BRE-01: A Real-Time Virtual Delivery System for Photon Radiotherapy Delivery Monitoring

Purpose: Treatment delivery monitoring is important for radiotherapy, which enables catching dosimetric error at the earliest possible opportunity. This project develops a virtual delivery system to monitor the dose delivery process of photon radiotherapy in real-time using GPU-based Monte Carlo (MC) method. Methods: The simulation process consists of 3 parallel CPU threads. A thread T1 is responsible for communication with a linac, which acquires a set of linac status parameters, e.g. gantry angles, MLC configurations, and beam MUs every 20 ms. Since linac vendors currently do not offer interface to acquire data in real time, we mimic this process by fetching information from a linac dynalog file at the set frequency. Instantaneous beam fluence map (FM) is calculated. A FM buffer is also created in T1 and the instantaneous FM is accumulated to it. This process continues, until a ready signal is received from thread T2 on which an inhouse developed MC dose engine executes on GPU. At that moment, the accumulated FM is transferred to T2 for dose calculations, and the FM buffer in T1 is cleared. Once the calculation finishes, the resulting 3D dose distribution is directed to thread T3, which displays it in three orthogonal planes overlaid on the CT image for treatment monitoring. This process continues to monitor the 3D dose distribution in real-time. Results: An IMRT and a VMAT cases used in our patient-specific QA are studied. Maximum dose differences between our system and treatment planning system are 0.98% and 1.58% for the two cases, respectively. The average time per MC calculation is 0.1sec with <2% relative uncertainty. The update frequency of ∼10Hz is considered as real time. Conclusion: By embedding a GPU-based MC code in a novel data/work flow, it is possible to achieve real-time MC dose calculations to monitor delivery process.

60 APPLIED LIFE SCIENCES↗

NASA NTRS2011

Cable Tester Box

Cables are very important electrical devices that carry power and signals across multiple instruments. Any fault in a cable can easily result in a catastrophic outcome. Therefore, verifying that all cables are built to spec is a very important part of Electrical Integration Procedures. Currently, there are two methods used in lab for verifying cable connectivity. (1) Using a Break-Out Box and an ohmmeter this method is time-consuming but effective for custom cables and (2) Commercial Automated Cable Tester Boxes this method is fast, but to test custom cables often requires pre-programmed configuration files, and cables used on spacecraft are often uniquely designed for specific purposes. The idea is to develop a semi-automatic continuity tester that reduces human effort in cable testing, speeds up the electrical integration process, and ensures system safety. The JPL-Cable Tester Box is developed to check every single possible electrical connection in a cable in parallel. This system indicates connectivity through LED (light emitting diode) circuits. Users can choose to test any pin/shell (test node) with a single push of a button, and any other nodes that are shorted to the test node, even if they are in the same connector, will light up with the test node. The JPL-Cable Tester Boxes offers the following advantages: 1. Easy to use: The architecture is simple enough that it only takes 5 minutes for anyone to learn how operate the Cable Tester Box. No pre-programming and calibration are required, since this box only checks continuity. 2. Fast: The cable tester box checks all the possible electrical connections in parallel at a push of a button. If a cable normally takes half an hour to test, using the Cable Tester Box will improve the speed to as little as 60 seconds to complete. 3. Versatile: Multiple cable tester boxes can be used together. As long as all the boxes share the same electrical potential, any number of connectors can be tested together.

Lee, Jason H.↗