Reeber v2.0
Reeber is a library for shared- and distributed-memory parallel computation of merge trees
SEARCH · Search NASA
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Reeber is a library for shared- and distributed-memory parallel computation of merge trees
Abstract Modern cosmological simulations have reached the trillion-element scale, rendering data storage and subsequent analysis formidable tasks. To address this circumstance, we present a new MPI-parallel approach for analysis of simulation data while the simulation runs, as an alternative to the traditional workflow consisting of periodically saving large data sets to disk for subsequent ‘offline’ analysis. We demonstrate this approach in the compressible gasdynamics/ N -body code Nyx, a hybrid $\mbox{MPI}+\mbox{OpenMP}$ MPI + OpenMP code based on the BoxLib framework, used for large-scale cosmological simulations. We have enabled on-the-fly workflows in two different ways: one is a straightforward approach consisting of all MPI processes periodically halting the main simulation and analyzing each component of data that they own (‘ in situ ’). The other consists of partitioning processes into disjoint MPI groups, with one performing the simulation and periodically sending data to the other ‘sidecar’ group, which post-processes it while the simulation continues (‘in-transit’). The two groups execute their tasks asynchronously, stopping only to synchronize when a new set of simulation data needs to be analyzed. For both the in situ and in-transit approaches, we experiment with two different analysis suites with distinct performance behavior: one which finds dark matter halos in the simulation using merge trees to calculate the mass contained within iso-density contours, and another which calculates probability distribution functions and power spectra of various fields in the simulation. Both are common analysis tasks for cosmology, and both result in summary statistics significantly smaller than the original data set. We study the behavior of each type of analysis in each workflow in order to determine the optimal configuration for the different data analysis algorithms.
This study presents a principled empirical evaluation of image storage systems for training deep neural networks. We employ the Caffe deep learning framework to train neural network models for three different data sets, MNIST, CIFAR-10, and ImageNet. While training the models, we evaluate five different options to retrieve training image data: (1) PNG-formatted image files on local file system; (2) pushing pixel arrays from image files into a single HDF5 file on local file system; (3) in-memory arrays to hold the pixel arrays in Python and C++; (4) loading the training data into LevelDB, a log-structured merge tree based key-value storage; and (5) loading the training data into LMDB, a B+tree based key-value storage. The experimental results quantitatively highlight the disadvantage of using normal image files on local file systems to train deep neural networks and demonstrate reliable performance with key-value storage based storage systems. When training a model on the ImageNet dataset, the image file option was more than 17 times slower than the key-value storage option. Along with measurements on training time, this study provides in-depth analysis on the cause of performance advantages/disadvantages of each back-end to train deep neural networks. We envision the provided measurements and analysis will shed light on the optimal way to architect systems for training neural networks in a scalable manner.
The process of contourization is presented which converts a raster image into a set of plateaux or contours. These contours can be grouped into a hierarchical structure, defining total spatial inclusion, called a contour tree. A contour coder has been developed which fully describes these contours in a compact and efficient manner and is the basis for an image compression method. Simplification of the contour tree has been undertaken by merging contour tree nodes thus lowering the contour tree's entropy. This can be exploited by the contour coder to increase the image compression ratio. By applying general and simple rules derived from physiological experiments on the human vision system, lossy image compression can be achieved which minimizes noticeable artifacts in the simplified image.
As data sets grow to exascale, automated data analysis and visualisation are increasingly important, to intermediate human understanding and to reduce demands on disk storage via in situ analysis. Trends in architecture of high performance computing systems necessitate analysis algorithms to make effective use of combinations of massively multicore and distributed systems. One of the principal analytic tools is the contour tree, which analyses relationships between contours to identify features of more than local importance. Unfortunately, the predominant algorithms for computing the contour tree are explicitly serial, and founded on serial metaphors, which has limited the scalability of this form of analysis. While there is some work on distributed contour tree computation, and separately on hybrid GPU-CPU computation, there is no efficient algorithm with strong formal guarantees on performance allied with fast practical performance. Here in this paper, we report the first shared SMP algorithm for fully parallel contour tree computation, withfor-mal guarantees of O(lgnlgt) parallel steps and O(n lgn) work, and implementations with up to 10x parallel speed up in OpenMP and up to 50x speed up in NVIDIA Thrust.
We discuss and illustrate the properties of several parton-shower algorithms available in Pythia and Vincia, in the context of Higgs production via vector boson fusion (VBF). In particular, the distinctive colour topology of VBF processes allows to define observables sensitive to the coherent radiation pattern of additional jets. We study a set of such observables, using the Vincia sector-antenna shower as our main reference, and contrast it to Pythia's transverse-momentum-ordered DGLAP shower as well as Pythia's dipole-improved shower. We then investigate the robustness of these predictions as successive levels of higher-order perturbative matrix elements are incorporated, including next-to-leading-order matched and tree-level merged calculations, using Powheg Box and Sherpa respectively to generate the hard events.
STAT is a light weight debugging tool that gathers and merges stack traces from all of the processes in a parallel application. STAT uses the MRNet tree based overlay network to broadcast commands from the tool front-end to the STAT daemons and for the front-end to gather the traces from the STAT daemons. As the traces propagate through the MRNet network tree, they are merged across all tasks to fonn a single call prefix tree. The call prefix tree can be examined to identify tuks with similar function call patterns and to delineate a small set of equivalence classes. A representative task from each of these classes can then be fed into a full feature debugger like TotalView for root cause analysis.
STAT is a light weight debugging tool that gathers and merges stack traces from all of the processes in a parallel application. STAT uses the MRNet tree based overlay network to broadcast commands from the tool front-end to the STAT daemons and for the front-end to gather the traces from the STAT daemons. As the traces propagate through the MRNet network tree, they are merged across all tasks to form a similar function call patterns and to delineate a small set of equivalence classes. A representative task from each of these classes can then be fed into a full feature debugger like TolalView for root cause analysis.
In the algorithm presented here, the ME+PS approach to merge samples of tree-level matrix elements into inclusive event samples is combined with the POWHEG method, which includes exact next-to-leading order matrix elements in the parton shower. The advantages of the method are discussed and the quality of its implementation in SHERPA is exemplified by results for e{sup +}e{sup -} annihilation into hadrons at LEP, for deep-inelastic lepton-nucleon scattering at HERA, for Drell-Yan lepton-pair production at the Tevatron and for W{sup +}W{sup -}-production at LHC energies. The simulation of hard QCD radiation in parton-shower Monte Carlos has seen tremendous progress over the last years. It was largely stimulated by the need for more precise predictions at LHC energies where the large available phase space allows additional hard QCD radiation alongside known Standard Model processes or even signals from new physics. Two types of algorithms have been developed, which allow to improve upon the soft-collinear approximations made in the parton shower, such that hard radiation is simulated according to exact matrix elements. In the ME+PS approach [1] higher-order tree-level matrix elements for different final-state jet multiplicity are merged with each other and with subsequent parton shower emissions to generate an inclusive sample. Such a prescription is invaluable for analyses which are sensitive to final states with a large jet multiplicity. The only remaining deficiency of such tree-level calculations is the large uncertainty stemming from scale variations. The POWHEG method [2] solves this problem for the lowest multiplicity subprocess by combining full NLO matrix elements with the parton shower. While this leads to NLO accuracy in the inclusive cross section and the exact radiation pattern for the first emission, it fails to describe higher-order emissions with improved accuracy. Thus it is not sufficient if final states with high jet multiplicities are considered. With the complementary advantages of these two approaches, the question arises naturally whether it would be possible to combine them into an even more powerful one. Such a combined algorithm was independently developed in [5] and [6]. Here a summary of the algorithm is given and predictions from corresponding Monte-Carlo predictions are presented.
Herein this article presents a new type of hybrid model for Bayesian optimization (BO) adept at managing mixed variables, encompassing both quantitative (continuous and integer) and qualitative (categorical) types. Our proposed new hybrid models (named hybridM) merge the Monte Carlo Tree Search structure (MCTS) for categorical variables with Gaussian Processes (GP) for continuous ones. hybridM leverages the upper confidence bound tree search (UCTS) for MCTS strategy, showcasing the tree architecture’s integration into Bayesian optimization. Our innovations, including dynamic online kernel selection in the surrogate modeling phase and a unique UCTS search strategy, position our hybrid models as an advancement in mixed-variable surrogate models. Numerical experiments underscore the superiority of hybrid models, highlighting their potential in Bayesian optimization.
Identification of high cost modules has been viewed as one mechanism to improve overall system reliability, since such modules tend to produce more than their share of problems. A decision tree model was used to identify such modules. In this current paper, a previously developed axiomatic model of program complexity is merged with the previously developed decision tree process for an improvement in the ability to identify such modules. This improvement was tested using data from the NASA Software Engineering Laboratory.
Using a hydrodynamics plus N -body simulation of galaxy cluster formation within a large volume and mock Chandra X-ray observations, we study the form and evolution of the intrinsic scatter about the best-fit X-ray temperature-mass relation for clusters. We investigate the physical origin of the scatter by correlating it with quantities that are closely related to clusters' formation and merging histories. We also examine the distribution of the scatter for merging and non-merging populations, identified using halo merger trees derived from the simulation as well as X-ray substructure measures. We find a strong correlation between the scatter in the M-T X relation and the halo concentration, in the sense that more concentrated clusters tend to be cooler than clusters with similar masses. No bias is found between the merging and relaxed clusters, but merging clusters generally have greater scatter, which is related to the properties of the distribution of halo concentrations. We also detect a signature of non-lognormality in the distribution of scatter for our simulated clusters both at z = 0 and at z = 1. A detailed comparison of merging clusters identified by substructure measures and by halo merger trees is given in the discussion. Finally we conclude that, when cooling-related effects are neglected, the variation in halo concentrations is a more important factor for driving the intrinsic scatter in the M-T X relation, while departures from hydrostatic equilibrium due to cluster mergers have a minor effect.
Bounding the best achievable error probability for binary classification problems is relevant to many applications including machine learning, signal processing, and information theory. Many bounds on the Bayes binary classification error rate depend on information divergences between the pair of class distributions. Recently, the Henze–Penrose (HP) divergence has been proposed for bounding classification error probability. We consider the problem of empirically estimating the HP-divergence from random samples. We derive a bound on the convergence rate for the Friedman–Rafsky (FR) estimator of the HP-divergence, which is related to a multivariate runs statistic for testing between two distributions. The FR estimator is derived from a multicolored Euclidean minimal spanning tree (MST) that spans the merged samples. We obtain a concentration inequality for the Friedman–Rafsky estimator of the Henze–Penrose divergence. We validate our results experimentally and illustrate their application to real datasets.
As data sets grow to exascale, automated data analysis and visualization are increasingly important, to intermediate human understanding and to reduce demands on disk storage via in situ analysis. Trends in architecture of high performance computing systems necessitate analysis algorithms to make effective use of combinations of massively multicore and distributed systems. One of the principal analytic tools is the contour tree, which analyses relationships between contours to identify features of more than local importance. Unfortunately, the predominant algorithms for computing the contour tree are explicitly serial, and founded on serial metaphors, which has limited the scalability of this form of analysis. While there is some work on distributed contour tree computation, and separately on hybrid GPU-CPU computation, there is no efficient algorithm with strong formal guarantees on performance allied with fast practical performance. Furthermore, we report the first shared SMP algorithm for fully parallel contour tree computation, with formal guarantees of O(lg V lg t) parallel steps and O(V lg V) work for data with V samples and t contour tree supernodes, and implementations with more than 30× parallel speed up on both CPU using TBB and GPU using Thrust and up 70× speed up compared to the serial sweep and merge algorithm.
The classification of large dimensional data sets arising from the merging of remote sensing data with more traditional forms of ancillary data is considered. Decision tree classification, a popular approach to the problem, is characterized by the property that samples are subjected to a sequence of decision rules before they are assigned to a unique class. An automated technique for effective decision tree design which relies only on apriori statistics is presented. This procedure utilizes a set of two dimensional canonical transforms and Bayes table look-up decision rules. An optimal design at each node is derived based on the associated decision table. A procedure for computing the global probability of correct classfication is also provided. An example is given in which class statistics obtained from an actual LANDSAT scene are used as input to the program. The resulting decision tree design has an associated probability of correct classification of .76 compared to the theoretically optimum .79 probability of correct classification associated with a full dimensional Bayes classifier. Recommendations for future research are included.
Mapping of boreal forest's type, biomass, and other structural parameters are critical for understanding of the boreal forest's significance in the carbon cycle, its response to and impact on global climate change. We believe the nature of the forest structure information available from MISR and GLAS can be used to help identify forest type, age class, and estimate above ground biomass levels beyond that now possible with MODIS alone. The ground measurements will be used to develop relationships between remote sensing observables and forest characteristics and provide new information for understanding forest changes with respect to environmental change. Lidar is a laser altimeter that determines the distance from the instrument to the physical surface by measuring the time elapsed between the pulse emission and the reflected return. Other studies have shown that the returned signal may identify multiple returns originating from trees, building and other objects and permits the calculation of their height. Studies using field data have shown that lidar data can provide estimates of structural parameters such as biomass, stand volume and leaf area index and allows remarkable differentiation between primary and secondary forest. NASA's IceSAT Geoscience Laser Altimeter System (GLAS) was launched in January 2003 and collected data during February and September of that year. This study used data acquired over our study sites in central Siberia to examine the GLAS signal as a source of forest height and other structural characteristics. The purpose of our Siberia project is to improve forest cover maps and produce above-ground biomass maps of the boreal forest in Northern Eurasia from MODIS by incorporating structural information inherent in the Terra MISR and ICESAT Geoscience Laser Altimeter System (GLAS) instruments. A number of forest cover classifications exist for the boreal forest. We believe the limiting factor in these products is the lack of structural information, particularly in the vertical dimension. The emphasis of this project is to improve upon satellite maps of boreal forest structure parameters (i.e. height and biomass) using temporal, multi-angle, and vertical profile information of GLAS data. The existing and near future lidar data is useful for demonstrating these techniques and pursuing current estimates. Future lidar missions may be several years in the future, so we will work other new data sets that may aide in biomass estimates such as ALOS PALSAR We will continue this work to produce an accurate map of current above ground forest phytomass/carbon storage possible for the study area. We plan to develop, test, and integrate remote sensing methods for extracting forest canopy structure measures. We are compiling our field measurements and will compare them with the remote sensing methods where possible. We also be able to produce a realistic error bound on the remotely sensed carbon estimates.