SEARCH · Search NASA
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Computer program for investigating computer and operating system turn-around for problems employing multidimensional arrays
We have implemented a flexible User Defined Operator (UDO) for labeling connected components of a binary mask expressed as an array in SciDB, a parallel distributed database management system based on the array data model. This UDO is able to process very large multidimensional arrays by exploiting SciDB's memory management mechanism that efficiently manipulates arrays whose memory requirements far exceed available physical memory. The UDO takes as primary inputs a binary mask array and a binary stencil array that specifies the connectivity of a given cell to its neighbors. The UDO returns an array of the same shape as the input mask array with each foreground cell containing the label of the component it belongs to. By default, dimensions are treated as non-periodic, but the UDO also accepts optional input parameters to specify periodicity in any of the array dimensions. The UDO requires four stages to completely label connected components. In the first stage, labels are computed for each subarray or chunk of the mask array in parallel across SciDB instances using the weighted quick union (WQU) with half-path compression algorithm. In the second stage, labels around chunk boundaries from the first stage are stored in a temporary SciDB array that is then replicated across all SciDB instances. Equivalences are resolved by again applying the WQU algorithm to these boundary labels. In the third stage, relabeling is done for each chunk using the resolved equivalences. In the fourth stage, the resolved labels, which so far are "flattened" coordinates of the original binary mask array, are renamed with sequential integers for legibility. The UDO is demonstrated on a 3-D mask of O(1011) elements, with O(108) foreground cells and O(106) connected components. The operator completes in 19 minutes using 84 SciDB instances.
Generating local addresses and communication sets is an important issue in distributed-memory implementations of data-parallel languages such as High Performance Fortran. We show that for an array A affinely aligned to a template that is distributed across p processors with a cyclic(k) distribution, and a computation involving the regular section A, the local memory access sequence for any processor is characterized by a finite state machine of at most k states. We present fast algorithms for computing the essential information about these state machines, and extend the framework to handle multidimensional arrays. We also show how to generate communication sets using the state machine approach. Performance results show that this solution requires very little runtime overhead and acceptable preprocessing time.
Generating local addresses and communication sets is an important issue in distributed-memory implementations of data-parallel languages such as High Performance FORTRAN. We show that, for an array A affinely aligned to a template that is distributed across p processors with a cyclic(k) distribution and a computation involving the regular section A(l:h:s), the local memory access sequence for any processor is characterized by a finite state machine of at most k states. We present fast algorithms for computing the essential information about these state machines, and extend the framework to handle multidimensional arrays. We also show how to generate communication sets using the state machine approach. Performance results show that this solution requires very little run-time overhead and acceptable preprocessing time.
A novel algorithm, called the lambda test, is presented for an efficient and accurate data dependence analysis of multidimensional array references. It extends the numerical methods to allow all dimensions of array references to be tested simultaneously. Hence, it combines the efficiency and the accuracy of the both approaches. This algorithm has been implemented in PARAFRASE, a FORTRAN program parallelization restructurer developed at the University of Illinois at Urbana-Champaign. Some experimental results are presented to show its effectiveness.
The Paradigm Compiler implements a new approach to compiling programs written in high level languages for execution on highly parallel computers. The general approach is to identify the principal data structures constructed by the program and to map these structures onto the processing elements of the target machine. The mapping is chosen to maximize performance as determined through compile time global analysis of the source program. The source language is Sisal, a functional language designed for scientific computations, and the target language is Paris, the published low level interface to the Connection Machine. The data structures considered are multidimensional arrays whose dimensions are known at compile time. Computations that build such arrays usually offer opportunities for highly parallel execution; they are data parallel. The Connection Machine is an attractive target for these computations, and the parallel for construct of the Sisal language is a convenient high level notation for data parallel algorithms. The principles and organization of the Paradigm Compiler are discussed.
The National Space Science Data Center's Common Data Formate (CDF) is capable of storing many types of data such as scalar data items, vectors, and multidimensional arrays of bytes, integers, or floating point values. However, regardless of the dimensionality and data type, the data break down into a sequence of bytes that can be fed into a data compression function to reduce the amount of data without losing data integrity and thus remaining fully reconstructible. Because of the diversity of data types and high performance speed requirements, a general-purpose, fast, simple data compression algorithm is required to incorporate data compression into CDF. The questions to ask are how to evaluate and compare compression algorithms, and what compression algorithm meets all requirements. The object of this paper is to address these questions and determine the most appropriate compression algorithm to use within the CDF data management package that would be applicable to other software packages with similar data compression needs.
Gridded geospatial remote sensing (satellite) data has traditionally been stored in file-based multidimensional arrays to preserve the locality of data. Measurements from locations that are physically next to each other on earth remain next to each other in the arrays. Maintaining this locality is useful when running calculations like reprojection, but unnecessary for many other calculations. This talk will go through a real world example of a tool redesign at the Goddard Earth Sciences Data and Information Services Center (GES DISC), showing the advantages of using the data frame model for calculating summary statistics, where measurement proximity is unimportant.
Continuous atmospheric data analysis is an important factor for space launch vehicle design and operations. The balloon quality control tool was developed by NASA’s Marshall Space Flight Center (MSFC) Natural Environments Branch (NEB) for monitoring quality control processes and verifying the automated flags created on the balloon data sets analyzed. The data sets currently analyzed are comprised of high-resolution and low-resolution balloon data from NASA Kennedy Space Center (KSC), co-located on the United States Air Force’s Eastern range (ER) at the Cape Canaveral Air Force Station. The NEB was tasked to perform a quality assessment of these data sets and needed a tool to confirm the quality control (QC) flags produced from an automated process and add additional QC flags if necessary. This Graphical User Interface (GUI) was developed to visualize all of the data from these balloon sets, display any flags from the automated QC process, and add additional flags to variables if necessary. The GUI was developed in Python 3.6 utilizing different packages available such as pandas for data analysis and manipulation, NumPy for high-performance multidimensional array and tools to compute with and manipulate arrays, Matplotlib for plotting data and Tkinter to build the GUI.
Continuous atmospheric data analysis is an important factor for space launch vehicle design and operations. The balloon quality control tool was developed by NASA’s Marshall Space Flight Center (MSFC) Natural Environments Branch (NEB) for monitoring quality control processes and verifying the automated flags created on the balloon data sets analyzed. The data sets currently analyzed are comprised of high-resolution and low-resolution balloon data from NASA Kennedy Space Center (KSC), co-located on the United States Air Force’s Eastern range (ER) at the Cape Canaveral Air Force Station. The NEB was tasked to perform a quality assessment of these data sets and needed a tool to confirm the quality control (QC) flags produced from an automated process and add additional QC flags if necessary. This Graphical User Interface (GUI) was developed to visualize all of the data from these balloon sets, display any flags from the automated QC process, and add additional flags to variables if necessary. The GUI was developed in Python 3.6 utilizing different packages available such as pandas for data analysis and manipulation, NumPy for high-performance multidimensional array and tools to compute with and manipulate arrays, Matplotlib for plotting data and Tkinter to build the GUI.
We investigate the impact of data placement for two Big Data technologies, Spark and SciDB, with a use case from Earth Science where data arrays are multidimensional. Simultaneously, this investigation provides an opportunity to evaluate the performance of the technologies involved. Two datastores, HDFS and Cassandra, are used with Spark for our comparison. It is found that Spark with Cassandra performs better than with HDFS, but SciDB performs better yet than Spark with either datastore. The investigation also underscores the value of having data aligned for the most common analysis scenarios in advance on a shared nothing architecture. Otherwise, repartitioning needs to be carried out on the fly, degrading overall performance.
We investigate the problem of evaluating FORTRAN 90 style array expressions on massively parallel distributed-memory machines. On such machines, an elementwise operation can be performed in constant time for arrays whose corresponding elements are in the same processor. If the arrays are not aligned in this manner, the cost of aligning them is part of the cost of evaluating the expression. The choice of where to perform the operation then affects this cost. We present algorithms based on dynamic programming to solve this problem efficiently for a wide variety of interconnection schemes, including multidimensional grids and rings, hypercubes, and fat-trees. We also consider expressions containing operations that change the shape of the arrays, and show that our approach extends naturally to handle this case.
The PARADIGM compiler project provides an automated means to parallelize programs, written in a serial programming model, for efficient execution on distributed-memory multicomputers. .A previous implementation of the compiler based on the PTD representation allowed symbolic array sizes, affine loop bounds and array subscripts, and variable number of processors, provided that arrays were single or multi-dimensionally block distributed. The techniques presented here extend the compiler to also accept multidimensional cyclic and block-cyclic distributions within a uniform symbolic framework. These extensions demand more sophisticated symbolic manipulation capabilities. A novel aspect of our approach is to meet this demand by interfacing PARADIGM with a powerful off-the-shelf symbolic package, Mathematica. This paper describes some of the Mathematica routines that performs various transformations, shows how they are invoked and used by the compiler to overcome the new challenges, and presents experimental results for code involving cyclic and block-cyclic arrays as evidence of the feasibility of the approach.
This paper presents the hyperwall, a visualization cluster that uses coordinated visualizations for interactive exploration of multidimensional data and simulations. The system strongly leverages the human eye-brain system with a generous 7x7 array offlat panel LCD screens powered by a beowulf clustel: With each screen backed by a workstation class PC, graphic and compute intensive applications can be applied to a broad range of data. Navigational tools are presented that allow for investigation of high dimensional spaces.
Topics covered include: Protein Sensors Based on Optical Ring Resonators; Phase Sensor for Aligning a Segmented Telescope Mirror; Control Software for Advanced Video Guidance Sensor; Generating Control Commands From Gestures Sensed by EMG; Multiple-Flat-Panel System Displays Multidimensional Data; 3D X-Ray Luggage-Screening System; Probe Station and Near-Field Scanner for Testing Antennas; Photodetector Arrays for Multicolor Visible/Infrared Imaging; Semiconductor Bolometers Give Background-Limited Performance; Multichannel X-Band Dielectric-Resonator Oscillator; Automatic Alignment of Displacement-Measuring Interferometer; Earth Observing System Data Gateway; Power User Interface; Mercury Shopping Cart Interface; Cassini Archive Tracking System; Architecture Adaptive Computing Environment; Computing Fault Displacements from Surface Deformations; Oxygen-Permeable, Hydrophobic Membranes of Silanized alpha-Al2O3; SiC Composite Turbine Vanes; Retaining Device for the Interior Structure of a Spacecraft Payload; Tool for Torquing Circular Electrical-Connector Collars; System for Continuous Deaeration of Hydraulic Oil; Solar-Powered Cooler and Heater for an Automobile Interior; Improved Oxygen-Beam Texturing of Glucose-Monitoring Optics; Tool for Two Types of Friction Stir Welding; Stationary Apparatus Would Apply Forces of Walking to Feet; Instrument Would Detect and Collect Biological Aerosols; Boundary Condition for Modeling Semiconductor Nanostructures; Miniature Distillation Column for Producing LOX From Air; Even Illumination from Fiber-Optic-Coupled Laser Diodes; Optically Driven Deformable Mirrors; Algorithm for Automated Detection of Edges of Clouds; Exploiting Quantum Resonance to Solve Combinatorial Problems; Hybrid Terrain Database; On Release of Microbe-Laden Particles from Mars Landers; A Concept for Run-Time Support of the Chapel Language; Thermoelectric Inhomogeneities in (Ag(sub 1-y)SbTe2)(sub x)(PbTe)(sub 1-x); and Spacecraft Escape Capsule.
An apparatus for processing multidimensional data with strong spatial characteristics, such as raw image data, characterized by a large number of parallel data streams in an ordered array is described. It comprises a large number (e.g., 16,384 in a 128 x 128 array) of parallel processing elements operating simultaneously and independently on single bit slices of a corresponding array of incoming data streams under control of a single set of instructions. Each of the processing elements comprises a bidirectional data bus in communication with a register for storing single bit slices together with a random access memory unit and associated circuitry, including a binary counter/shift register device, for performing logical and arithmetical computations on the bit slices, and an I/O unit for interfacing the bidirectional data bus with the data stream source. The massively parallel processor architecture enables very high speed processing of large amounts of ordered parallel data, including spatial translation by shifting or sliding of bits vertically or horizontally to neighboring processing elements.
The world coordinate system (WCS) for dispersion calibrated spectra used in the IRAF/NOAO spectroscopy packages is described. In particular, the image header keywords which define the coordinates in an 'image' pixel array. These keywords appear both as part of the IRAF image structure and map directly to FITS format. The types of spectra include multidimensional images with spectral dispersion as one axis, one dimensional images, and a special 'multispec' format consisting of multiple, semi-independent, one dimensional spectra in two or three dimensional images. The types of coordinate systems include multidimensional linear coordinates with individual keywords and one dimensional linear and nonlinear coordinates for the 'multispec' images stored in IRAF WCS attributes. The nonlinear world coordinate systems include polynomial, spline, sampled table, and look-up table function representations.
The processing of synthetic aperture radar (SAR) signals using the massively parallel processor (MPP) is discussed. The fast Fourier transform convolution procedures employed in the algorithms are described. The MPP architecture comprises an array unit (ARU) which processes arrays of data; an array control unit which controls the operation of the ARU and performs scalar arithmetic; a program and data management unit which controls the flow of data; and a unique staging memory (SM) which buffers and permutes data. The ARU contains a 128 by 128 array of bit-serial processing elements (PE). Two-by-four surarrays of PE's are packaged in a custom VLSI HCMOS chip. The staging memory is a large multidimensional-access memory which buffers and permutes data flowing with the system. Efficient SAR processing is achieved via ARU communication paths and SM data manipulation. Real time processing capability can be realized via a multiple ARU, multiple SM configuration.