Search NASA⌕ Search

SEARCH · Search NASA

Results for “data parallel algorithms”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 937 records · Page 52

Distributed caching for processing raw arrays

As applications continue to generate multi-dimensional data at exponentially increasing rates, fast analytics to extract meaningful results is becoming extremely important. The database community has developed array databases that alleviate this problem through a series of techniques. In-situ mechanisms provide direct access to raw data in the original format---without loading and partitioning. Parallel processing scales to the largest datasets. In-memory caching reduces latency when the same data are accessed across a workload of queries. However, we are not aware of any work on distributed caching of multi-dimensional raw arrays. In this paper, we introduce a distributed framework for cost-based caching of multi-dimensional arrays in native format. Given a set of files that contain portions of an array and an online query workload, the framework computes an effective caching plan in two stages. First, the plan identifies the cells to be cached locally from each of the input files by continuously refining an evolving R-tree index. In the second stage, an optimal assignment of cells to nodes that collocates dependent cells in order to minimize the overall data transfer is determined. We design cache eviction and placement heuristic algorithms that consider the historical query workload. A thorough experimental evaluation over two real datasets in three file formats confirms the superiority -- by as much as two orders of magnitude -- of the proposed framework over existing techniques in terms of cache overhead and workload execution time.

Zhao, Weijie↗

Performance analysis of a large-grain dataflow scheduling paradigm

A paradigm for scheduling computations on a network of multiprocessors using large-grain data flow scheduling at run time is described and analyzed. The computations to be scheduled must follow a static flow graph, while the schedule itself will be dynamic (i.e., determined at run time). Many applications characterized by static flow exist, and they include real-time control and digital signal processing. With the advent of computer-aided software engineering (CASE) tools for capturing software designs in dataflow-like structures, macro-dataflow scheduling becomes increasingly attractive, if not necessary. For parallel implementations, using the macro-dataflow method allows the scheduling to be insulated from the application designer and enables the maximum utilization of available resources. Further, by allowing multitasking, processor utilizations can approach 100 percent while they maintain maximum speedup. Extensive simulation studies are performed on 4-, 8-, and 16-processor architectures that reflect the effects of communication delays, scheduling delays, algorithm class, and multitasking on performance and speedup gains.

Young, Steven D.↗

Automated Euler and Navier-Stokes Database Generation for a Glide-Back Booster

The past two decades have seen a sustained increase in the use of high fidelity Computational Fluid Dynamics (CFD) in basic research, aircraft design, and the analysis of post-design issues. As the fidelity of a CFD method increases, the number of cases that can be readily and affordably computed greatly diminishes. However, computer speeds now exceed 2 GHz, hundreds of processors are currently available and more affordable, and advances in parallel CFD algorithms scale more readily with large numbers of processors. All of these factors make it feasible to compute thousands of high fidelity cases. However, there still remains the overwhelming task of monitoring the solution process. This paper presents an approach to automate the CFD solution process. A new software tool, AeroDB, is used to compute thousands of Euler and Navier-Stokes solutions for a 2nd generation glide-back booster in one week. The solution process exploits a common job-submission grid environment, the NASA Information Power Grid (IPG), using 13 computers located at 4 different geographical sites. Process automation and web-based access to a MySql database greatly reduces the user workload, removing much of the tedium and tendency for user input errors. The AeroDB framework is shown. The user submits/deletes jobs, monitors AeroDB's progress, and retrieves data and plots via a web portal. Once a job is in the database, a job launcher uses an IPG resource broker to decide which computers are best suited to run the job. Job/code requirements, the number of CPUs free on a remote system, and queue lengths are some of the parameters the broker takes into account. The Globus software provides secure services for user authentication, remote shell execution, and secure file transfers over an open network. AeroDB automatically decides when a job is completed. Currently, the Cart3D unstructured flow solver is used for the Euler equations, and the Overflow structured overset flow solver is used for the Navier-Stokes equations. Other codes can be readily included into the AeroDB framework.

Chaderjian, Neal M.↗

Airborne Visible/Infrared Imaging Spectrometer 3 (AVIRIS-3)

The Airborne Visible/Infrared Imaging Spectrometer 3 (AVIRIS-3) is the third of the NASA AVIRIS spectrometer series and is being developed in parallel with the Compact Wide-swath Imaging Spectrometer II (CWIS-II) for the University of Zurich, Switzerland. The core spectrometer of AVIRIS-3 is a copy of the optically fast, F/1.8 Dyson imaging spectrometer used by the Earth Surface Mineral Dust Source Investigation (EMIT) that is in development and scheduled for launch to the International Space Station (ISS) in 2022. AVIRIS-3 is intended to provide state-of-the-art imaging spectroscopy measurements for NASA science and application through the next decade and beyond. AVIRIS-3 uses the EMIT spectrometer design interfaced with a scaled two mirror telescope enclosed in a portable vacuum vessel to enable measurements from airborne platforms ranging from a Twin Otter to a business jet or a NASA ER-2. AVIRIS-3 is a cryogenic instrument with advanced system control and real-time onboard spectroscopic data processing algorithms evolved from AVIRIS-NG. The spectral range of AVIRIS-3 is 380 to 2500 nm with 7.4 nm sampling. The radiometric range is from 0 to max terrestrial Lambertian radiance with higher signal-to-noise ratio performance than AVIRIS-Classic or AVIRIS-Next Generation. The spatial field-of-view is 39.5 degrees with 0.56 milliradian sampling. This paper describes the design and development of AVIRIS-3 and presents its characteristics in comparison to the previous generation imaging spectrometers.

Helmlinger, Mark↗

Autoencoders on FPGAs for real-time, unsupervised new physics detection at 40 MHz at the Large Hadron Collider

In this paper, we show how to adapt and deploy anomaly detection algorithms based on deep autoencoders, for the unsupervised detection of new physics signatures in the extremely challenging environment of a real-time event selection system at the Large Hadron Collider (LHC). We demonstrate that new physics signatures can be enhanced by three orders of magnitude, while staying within the strict latency and resource constraints of a typical LHC event filtering system. This would allow for collecting datasets potentially enriched with high-purity contributions from new physics processes. Through per-layer, highly parallel implementations of network layers, support for autoencoder-specific losses on FPGAs and latent space based inference, we demonstrate that anomaly detection can be performed in as little as $80\,$ns using less than 3% of the logic resources in the Xilinx Virtex VU9P FPGA. Opening the way to real-life applications of this idea during the next data-taking campaign of the LHC.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Extending OpenSHMEM with Aggregation Support for Improved Message Rate Performance

OpenSHMEM is a highly efficient one-sided communication API that implements the PGAS parallel programming model, and is known for its low latency communication operations that can be mapped efficiently to RDMA capabilities of network interconnects. However, applications that use OpenSHMEM can be sensitive to point-to-point message rates, as many-to-many communication patterns can generate large amounts of small messages which tend to overwhelm network hardware that has predominantly been optimised for bandwidth over message rate. Additionally, many important emerging classes of problems such as data analytics are similarly troublesome for the irregular access patterns they employ. Message aggregation strategies have been proven to significantly enhance network performance, but their implementation often involves complex restructuring of user code, making them unwieldy. This paper shows how to combine the best qualities of message aggregation within the communication model of OpenSHMEM such that applications with small and irregular access patterns can improve network performance while maintaining their algorithmic simplicity. We do this by providing a path to a message aggregation framework called conveyors through a minimally intrusive OpenSHMEM extension introducing aggregation contexts that fit more naturally to the OpenSHMEM atomics, gets, and puts model. We test these extensions using four of the bale 3.0 applications which contain essential many-to-many access patterns to show how they can produce performance improvements of up to 65×.

Welch, Aaron↗

Fiber Bragg Grating Sensor System for Monitoring Smart Composite Aerospace Structures

Lightweight, electromagnetic interference (EMI) immune, fiber-optic, sensor- based structural health monitoring (SHM) will play an increasing role in aerospace structures ranging from aircraft wings to jet engine vanes. Fiber Bragg Grating (FBG) sensors for SHM include advanced signal processing, system and damage identification, and location and quantification algorithms. Potentially, the solution could be developed into an autonomous onboard system to inspect and perform non-destructive evaluation and SHM. A novel method has been developed to massively multiplex FBG sensors, supported by a parallel processing interrogator, which enables high sampling rates combined with highly distributed sensing (up to 96 sensors per system). The interrogation system comprises several subsystems. A broadband optical source subsystem (BOSS) and routing and interface module (RIM) send light from the interrogation system to a composite embedded FBG sensor matrix, which returns measurand-dependent wavelengths back to the interrogation system for measurement with subpicometer resolution. In particular, the returned wavelengths are channeled by the RIM to a photonic signal processing subsystem based on powerful optical chips, then passed through an optoelectronic interface to an analog post-detection electronics subsystem, digital post-detection electronics subsystem, and finally via a data interface to a computer. A range of composite structures has been fabricated with FBGs embedded. Stress tensile, bending, and dynamic strain tests were performed. The experimental work proved that the FBG sensors have a good level of accuracy in measuring the static response of the tested composite coupons (down to submicrostrain levels), the capability to detect and monitor dynamic loads, and the ability to detect defects in composites by a variety of methods including monitoring the decay time under different dynamic loading conditions. In addition to quasi-static and dynamic load monitoring, the system can capture acoustic emission events that can be a prelude to structural failure, as well as piezoactuator-induced ultrasonic Lamb-waves-based techniques as a basis for damage detection.

Moslehi, Behzad↗

An Approach to Solving Enclosure Radiation Problems in A Multi-Physics Context

Thermal protection system analysis of complex features or damage sites can sometimes require modeling of high temperature enclosures. Implementing efficient and accurate view-factor algorithms required to model such problems is complex. The current work leverages the Non-equilibrium Radiation (NERO) software, which solves the radiation transport equation in a finite-volume scheme, to alleviating challenges often faced with view-factor calculations. By assuming heat transfer occurs only between grey bodies and that the medium is non-participating, computational cost of the method is significantly reduced. The enclosure physics are modeled through emitting and reflecting boundary conditions in NERO. The emitted radiative flux is dependent on the wall temperature which is a solution to the material response, obtained from Icarus, in this context. The Ares framework manages the time-advancement and exchange of the necessary data between the solvers. The surface energy balance is modified to account for the enclosure terms within the material response boundary condition. The methodology was verified against analytical solutions including radiating parallel plates, a hollow cylinder (shown in Fig. 1), and a hemisphere. Application of the methodology to inform the design of components of the Dragonfly system will be shown.

Ablation↗

Parallel-Processing Software for Creating Mosaic Images

A computer program implements parallel processing for nearly real-time creation of panoramic mosaics of images of terrain acquired by video cameras on an exploratory robotic vehicle (e.g., a Mars rover). Because the original images are typically acquired at various camera positions and orientations, it is necessary to warp the images into the reference frame of the mosaic before stitching them together to create the mosaic. [Also see "Parallel-Processing Software for Correlating Stereo Images," Software Supplement to NASA Tech Briefs, Vol. 31, No. 9 (September 2007) page 26.] The warping algorithm in this computer program reflects the considerations that (1) for every pixel in the desired final mosaic, a good corresponding point must be found in one or more of the original images and (2) for this purpose, one needs a good mathematical model of the cameras and a good correlation of individual pixels with respect to their positions in three dimensions. The desired mosaic is divided into slices, each of which is assigned to one of a number of central processing units (CPUs) operating simultaneously. The results from the CPUs are gathered and placed into the final mosaic. The time taken to create the mosaic depends upon the number of CPUs, the speed of each CPU, and whether a local or a remote data-staging mechanism is used.

Klimeck, Gerhard↗

Population Count on Intel® CPU, GPU, and FPGA

Population count is a primitive used in many applications. Commodity processors have dedicated instructions for achieving high-performance population count. Motivated by the productivity of high-level synthesis and the importance of population count, in this paper we investigated the OpenCL implementations of population count algorithms, and evaluated their performance and resource utilizations on an FPGA. Based on the results, we select the most efficient implementation. Then we derived a reduction pattern from a representative application of population count. We parallelized the reduction with atomic functions, and optimized it with vectorized memory accesses, tree reduction, and compute-unit duplication. We evaluated the performance of the reduction kernel on an Intel (R) Xeon (R) CPU and an Intel (R) IrisT Pro integrated GPU, and an FPGA card that features an Intel (R) Arria (R) 10 FPGA. When DRAM memory bandwidth is comparable on the three computing platforms, the FPGA can achieve the highest kernel performance for large workload. On the other hand, we described performance bottlenecks on the FPGA. To make FPGAs more competitive in raw performance compared to high-performant CPU and GPU platforms, it is important to increase external memory bandwidth, minimize data movement between a host and a device, and reduce OpenCL runtime overhead on an FPGA.

heterogeneous computing↗

AI ATAC 1: An Evaluation of Prominent Commercial Malware Detectors

This work presents an evaluation of six prominent commercial endpoint malware detectors, a network malware detector, and a file-conviction algorithm from a cyber technology vendor. The evaluation was administered as the first of the Artificial I ntelligence Applications t o Autonomous Cybersecurity (AI ATAC) prize challenges, funded by / completed in service of the US Navy. The experiment employed 100K files (50/50% benign/malicious) with a stratified distribution of file types, including ~1K zero-day program executables (increasing experiment size two orders of magnitude over previous work). We present an evaluation process of delivering a file to a fresh virtual machine donning the detection technology, waiting 90s to allow static detection, then executing the file and waiting another period for dynamic detection; this allows greater fidelity in the observational data than previous experiments, in particular, resource and time-to-detection statistics. To execute all 800K trials (100K files × 8 tools), a software framework is designed to choreograph the experiment into an automated, time-synced, and reproducible workflow with substantial parallelization. Software with base classes for this framework are provided. A cost-benefit model was configured to integrate the tools’ detection statistics into a comparable quantity by simulating costs of use. This provides a ranking methodology for cyber competitions and a lens for reasoning about the varied statistical results. The results provide insights on state of commercial malware detection.

Bridges, Robert↗

Data Reduction Pipeline for the CHARIS Integral-Field Spectrograph I: Detector Readout Calibration and Data Cube Extraction

We present the data reduction pipeline for CHARIS, a high-contrast integral-field spectrograph for the Subaru Telescope. The pipeline constructs a ramp from the raw reads using the measured nonlinear pixel response and reconstructs the data cube using one of three extraction algorithms: aperture photometry, optimal extraction, or chi-squared fitting. We measure and apply both a detector flatfield and a lenslet flatfield and reconstruct the wavelength- and position-dependent lenslet point-spread function (PSF) from images taken with a tunable laser. We use these measured PSFs to implement a chi-squared-based extraction of the data cube, with typical residuals of approximately 5 percent due to imperfect models of the under-sampled lenslet PSFs. The full two-dimensional residual of the chi-squared extraction allows us to model and remove correlated read noise, dramatically improving CHARIS's performance. The chi-squared extraction produces a data cube that has been deconvolved with the line-spread function and never performs any interpolations of either the data or the individual lenslet spectra. The extracted data cube also includes uncertainties for each spatial and spectral measurement. CHARIS's software is parallelized, written in Python and Cython, and freely available on github with a separate documentation page. Astrometric and spectrophotometric calibrations of the data cubes and PSF subtraction will be treated in a forthcoming paper.

Brandt, Timothy D.↗

Electrical Capacitance Volume Tomography with High-Contrast Dielectrics

The Electrical Capacitance Volume Tomography (ECVT) system has been designed to complement the tools created to sense the presence of water in nonconductive spacecraft materials, by helping to not only find the approximate location of moisture but also its quantity and depth. The ECVT system has been created for use with a new image reconstruction algorithm capable of imaging high-contrast dielectric distributions. Rather than relying solely on mutual capacitance readings as is done in traditional electrical capacitance tomography applications, this method reconstructs high-resolution images using only the self-capacitance measurements. The image reconstruction method assumes that the material under inspection consists of a binary dielectric distribution, with either a high relative dielectric value representing the water or a low dielectric value for the background material. By constraining the unknown dielectric material to one of two values, the inverse math problem that must be solved to generate the image is no longer ill-determined. The image resolution becomes limited only by the accuracy and resolution of the measurement circuitry. Images were reconstructed using this method with both synthetic and real data acquired using an aluminum structure inserted at different positions within the sensing region. The cuboid geometry of the system has two parallel planes of 16 conductors arranged in a 4 4 pattern. The electrode geometry consists of parallel planes of copper conductors, connected through custom-built switch electronics, to a commercially available capacitance to digital converter. The figure shows two 4 4 arrays of electrodes milled from square sections of copper-clad circuit-board material and mounted on two pieces of glass-filled plastic backing, which were cut to approximately square shapes, 10 cm on a side. Each electrode is placed on 2.0-cm centers. The parallel arrays were mounted with the electrode arrays approximately 3 cm apart. The open ends were surrounded by a metal guard to reduce the sensitivity of the electrodes to outside interference and to help maintain the spacing between the arrays. Other uses for this innovation potentially include quantifying the amount of commodity remaining in the fuel and oxidizer tanks while on-orbit without having to fire spacecraft engines. Another orbit application is moisture sensing in plant-growth experiments because microgravity causes moisture in soil to distribute itself in unusual ways. At the moment, the hardware and image reconstruction technique may only be of interest to people involved in nondestructive evaluation. The reconstructed image takes almost a full week to reproduce with existing computer power. However, because computer power and speeds follows Moore s Law, execution times are likely to become acceptable within the next five to eight years. The code was written in Mathematica for dedicated use with the ECVT system. In its present form, it is not suitable to be used directly as a consumer product. However, the code could be likely improved by rewriting it in a compiled language such as C or Fortran.

Nurge, Mark↗

Implementing Connected Component Labeling as a User Defined Operator for SciDB

We have implemented a flexible User Defined Operator (UDO) for labeling connected components of a binary mask expressed as an array in SciDB, a parallel distributed database management system based on the array data model. This UDO is able to process very large multidimensional arrays by exploiting SciDB's memory management mechanism that efficiently manipulates arrays whose memory requirements far exceed available physical memory. The UDO takes as primary inputs a binary mask array and a binary stencil array that specifies the connectivity of a given cell to its neighbors. The UDO returns an array of the same shape as the input mask array with each foreground cell containing the label of the component it belongs to. By default, dimensions are treated as non-periodic, but the UDO also accepts optional input parameters to specify periodicity in any of the array dimensions. The UDO requires four stages to completely label connected components. In the first stage, labels are computed for each subarray or chunk of the mask array in parallel across SciDB instances using the weighted quick union (WQU) with half-path compression algorithm. In the second stage, labels around chunk boundaries from the first stage are stored in a temporary SciDB array that is then replicated across all SciDB instances. Equivalences are resolved by again applying the WQU algorithm to these boundary labels. In the third stage, relabeling is done for each chunk using the resolved equivalences. In the fourth stage, the resolved labels, which so far are "flattened" coordinates of the original binary mask array, are renamed with sequential integers for legibility. The UDO is demonstrated on a 3-D mask of O(1011) elements, with O(108) foreground cells and O(106) connected components. The operator completes in 19 minutes using 84 SciDB instances.

UDO↗

Energy-efficient mapping of large-scale workflows under deadline constraints in big data computing systems

Large-scale workflows for big data analytics have become a main consumer of energy in data centers where moldable parallel computing models such as MapReduce are widely applied to meet high computational demands with time-varying computing resources. The granularity of task partitioning in each moldable job of such big data workflows has a significant impact on energy efficiency, which remains largely unexplored. In this paper, we analyze the properties of moldable jobs and formulate a workflow mapping problem to minimize the dynamic energy consumption of a given workflow request under a deadline constraint in big data systems. Since this problem is strongly NP-hard, we design a fully polynomial-time approximation scheme (FPTAS) for a special case with a pipeline-structured workflow on a homogeneous cluster and a heuristic for the generalized problem with an arbitrary workflow on a heterogeneous cluster. The performance superiority of the proposed solution in terms of dynamic energy saving and deadline missing rate is illustrated by extensive simulation results in comparison with existing algorithms, and further validated by real-life workflow implementation and experimental results in Hadoop/YARN systems.

97 MATHEMATICS AND COMPUTING↗

Integrated fluorescence light microscopy-guided cryo-focused ion beam-milling for in situ montage cryo-ET

Cryogenic-electron tomography (cryo-ET) permits the in situ visualization of biological macromolecules at the molecular level. Owing to the variable thickness of cells, tissues and organisms, frozen specimens may need to be thinned by cryo-focused ion beam (FIB) milling to produce thin (<500 nm) cryo-lamellae suitable for cryo-ET. Locating regions of interest remains a challenge because untargeted milling can lead to inadvertent ablation and removal of regions of interest. Correlative light and electron microscopy, combined with cryo-FIB milling, can guide the identification of labeled targets in the cellular milieu. Multiple transfers between cryo-imaging instruments, cumbersome correlation algorithms, limited accuracy and low throughput have hindered the routine adoption of cryo-FIB milling within a multimodal correlative workflow for in situ structural biology. Here, in this study, we present a workflow for 3D correlative cryo-fluorescence light microscopy-FIB-ET that streamlines fluorescence light microscopy-guided FIB milling, improving throughput while preserving both structural and contextual information. The complete integration of hardware and software described here minimizes sample contamination from cross-platform exchanges and greatly enhances the efficiency of 3D targeting in cryo-milling. We then describe procedures for implementing montage parallel array cryo-ET (MPACT), which can be easily adapted to any modern life-science transmission electron microscope. MPACT supports high-throughput cryo-ET acquisitions (10 tilt series in 1.5 h) for structure determination and comprehensive contextual understanding of macromolecules within their native surroundings. A complete session from sample preparation to MPACT data processing takes 5−7 d for an individual experienced in both cryo-EM and cryo-FIB milling.

Yang, Jie E. [Univ. of Wisconsin, Madison, WI (Uni↗

The G2 erosion model: An algorithm for month-time step assessments

Highlights: • G2 is a soil erosion model for developing monthly erosion maps at regional scale. • G2 enhances the spatio-temporal variability of cover management and erosivity factors. • The V-factor models the impact of vegetation cover, land use and imperviousness layer. • G2 uses data from ESDAC, LUCAS, COPERNICUS, CORINE, SENTINEL and EU-DEM databases. • G2 has been applied in Strymonas Catchment, Crete, Cyprus and Albania catchments. A detailed description of the G2 erosion model is presented, in order to support potential users. G2 is a complete, quantitative algorithm for mapping soil loss and sediment yield rates on month-time intervals. G2 has been designed to run in a GIS environment, taking input from geodatabases available by European or other international institutions. G2 adopts fundamental equations from the Revised Universal Soil Loss Equation (RUSLE) and the Erosion Potential Method (EPM), especially for rainfall erosivity, soil erodibility, and sediment delivery ratio. However, it has developed its own equations and matrices for the vegetation cover and management factor and the effect of landscape alterations on erosion. Provision of month-time step assessments is expected to improve understanding of erosion processes, especially in relation to land uses and climate change. In parallel, G2 has full potential to decision-making support with standardised maps on a regular basis. Geospatial layers of rainfall erosivity, soil erodibility, and terrain influence, recently developed by the Joint Research Centre (JRC) on a European or global scale, will further facilitate applications of G2.

54 ENVIRONMENTAL SCIENCES↗

Quantum annealing algorithms for Boolean tensor networks

Abstract Quantum annealers manufactured by D-Wave Systems, Inc., are computational devices capable of finding high-quality heuristic solutions of NP-hard problems. In this contribution, we explore the potential and effectiveness of such quantum annealers for computing Boolean tensor networks. Tensors offer a natural way to model high-dimensional data commonplace in many scientific fields, and representing a binary tensor as a Boolean tensor network is the task of expressing a tensor containing categorical (i.e., $$\{0, 1\}$$ { 0 , 1 } ) values as a product of low dimensional binary tensors. A Boolean tensor network is computed by Boolean tensor decomposition, and it is usually not exact. The aim of such decomposition is to minimize the given distance measure between the high-dimensional input tensor and the product of lower-dimensional (usually three-dimensional) tensors and matrices representing the tensor network. In this paper, we introduce and analyze three general algorithms for Boolean tensor networks: Tucker, Tensor Train, and Hierarchical Tucker networks. The computation of a Boolean tensor network is reduced to a sequence of Boolean matrix factorizations, which we show can be expressed as a quadratic unconstrained binary optimization problem suitable for solving on a quantum annealer. By using a novel method we introduce called parallel quantum annealing, we demonstrate that Boolean tensor’s with up to millions of elements can be decomposed efficiently using a DWave 2000Q quantum annealer.

97 MATHEMATICS AND COMPUTING↗