Search NASA⌕ Search

SEARCH · Search NASA

Results for “DISTRIBUTED DATA PROCESSING”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,081 records · Page 60

Efficient Delaunay Tessellation through K-D Tree Decomposition

Delaunay tessellations are fundamental data structures in computational geometry. They are important in data analysis, where they can represent the geometry of a point set or approximate its density. The algorithms for computing these tessellations at scale perform poorly when the input data is unbalanced. We investigate the use of k-d trees to evenly distribute points among processes and compare two strategies for picking split points between domain regions. Because resulting point distributions no longer satisfy the assumptions of existing parallel Delaunay algorithms, we develop a new parallel algorithm that adapts to its input and prove its correctness. We evaluate the new algorithm using two late-stage cosmology datasets. The new running times are up to 50 times faster using k-d tree compared with regular grid decomposition. Moreover, in the unbalanced data sets, decomposing the domain into a k-d tree is up to five times faster than decomposing it into a regular grid.

Morozov, Dmitriy↗

Data processing pipeline with transaction-oriented data sharing

This paper makes three contributions to the area of modern science data processing systems. First, the paper describes the science data processing pipeline, developed at the Multi-mission Image Processing Lab of JPL, for transforming raw space data into high quality image data and automating the distribution of data using a high-performance file transaction service. File Exchange Interface is the file transaction service developed MIPL. Second, it presents the FEI component architecture in the are of file transaction management, security,a nd file integrity verfication. Finally, the paper presents the federated model for the FEI service to demonstrate how to create a pool of file trasaction services to support load balancing and service fallover, and simplify service management.

science data processing↗

Distributed Data-Flow for In-Situ Visualization and Analysis at Petascale

We conducted a feasibility study to research modifications to data-flow architectures to enable data-flow to be distributed across multiple machines automatically. Distributed data-flow is a crucial technology to ensure that tools like the VisIt visualization application can provide in-situ data analysis and post-processing for simulations on peta-scale machines. We modified a version of VisIt to study load-balancing trade-offs between light-weight kernel compute environments and dedicated post-processing cluster nodes. Our research focused on memory overheads for contouring operations, which involves variable amounts of generated geometry on each node and computation of normal vectors for all generated vertices. Each compute node independently decided whether to send data to dedicated post-processing nodes at each stage of pipeline execution, depending on available memory. We instrumented the code to allow user settable available memory amounts to test extremely low-overhead compute environments. We performed initial testing of this prototype distributed streaming framework, but did not have time to perform scaling studies at and beyond 1000 compute-nodes.

97 MATHEMATICS AND COMPUTING↗

Determination of Organic Partitioning Coefficients in Water-Supercritical CO 2 Systems by Simultaneous in Situ UV and Near-Infrared Spectroscopies

CO2 injected into depleted oil or gas reservoirs for long-term storage has the potential to mobilize organic compounds and distribute them between sediments and reservoir brines. Understanding this process is important when considering health and environmental risks, but little quantitative data currently exists on the partitioning of organics between supercritical CO2 and water. In this work, a high-pressure, in situ measurement capability was developed to assess the distribution of organics between CO2 and water at conditions relevant to deep underground storage of CO2. The apparatus consists of a titanium reactor with quartz windows, near-infrared and UV spectroscopic detectors, and switching valves that facilitate quantitative injection of organic reagents into the pressurized reactor. To demonstrate the utility of the system, partitioning coefficients were determined for benzene in water/supercritical CO2 over the range 35-65 °C and approximately 25-150 bar. Density changes in the CO2 phase with increasing pressure were shown to have dramatic impacts on benzene's partitioning behavior. Our partitioning coefficients were approximately 5-15 times lower than values previously determined by ex situ techniques that are prone to sampling losses. The in situ methodology reported here could be applied to quantify the distribution behavior of a wide range of organic compounds that may be present in geologic CO2 storage scenarios.

Environmental Molecular Sciences Laboratory↗

Using Pilot Jobs and CernVM File System for Simplified Use of Containers and Software Distribution

High Energy Physics (HEP) experiments entail an abundance of computing resources, i.e. sites, to run simulations and analyses by processing data. This requirement is fulfilled by local batch farms, grid sites, private/commercial clouds, and supercomputing centers via High Throughput Computing (HTC). The growing needs of such experiments and resources being prone to trends of heterogeneity make it difficult for physicists to handle these resources directly. Additionally, HEP collaborations heavily rely on data and software releases, typically in the order of tens of gigabytes, while conducting simulations and analyses. Hence, aspects of scalability, reliability, and maintenance become crucial with regards to the distribution of the necessary data and software stack. The GlideinWMS [4] framework helps with the resource management problem by using pilot jobs, aka Glideins, to provision reliable elastic virtual clusters. Glideins are submitted to unreliable heterogeneous resources which are validated and customized by the Glideins to make the worker nodes available for end-user job execution. On the other hand, the CernVM File System (CernVM-FS or CVMFS) [1] helps with data distribution. It is a write-once, read-everywhere filesystem used to deploy scientific software to thousands of nodes on a worldwide distributed computing infrastructure. CVMFS is based on the Hyper Text Transfer Protocol and has been widely used within the particle physics community for (1) distributing experiment software and data such as calibrations, and (2) facilitating containerization by efficiently hosting container images along with providing containerization software, especially Singularity [3] GlideinWMS relies on CVMFS installed locally on the computing resources to satisfy the experiments' software needs. This requires system administrators' effort to install and maintain CVMFS at the sites and limits the use of sites, especially HPC resources, that do not have CVMFS installed. This poster presents a solution, taking advantage of Glideins to provide CVMFS at most sites without the need for a local installation. Doing so expands the pool of resources available for HEP experiments and reduces the effort of system administrators for current resources. Additionally, the proposed solution allows GlideinWMS to also start Singularity [3], a containerization software that can run unprivileged, on sites where neither CVMFS nor Singularity are available, including HPC sites. The benefits provided by this solution are: (1) lower overhead for site administrators in that they have less software to install, (2) an expanded pool of resources that run user jobs with easy access to software and data provided by CVMFS, thus making life easier for the scientists, and (3) improved flexibility to use HPC resources by enabling GlideinWMS pilot jobs to support HPC sites.

Urs, Namratha↗

Decon2LS: An Open-Source Software Package for Automated Processing and Visualization of High Resolution Mass Spectrometry Data

Data generated from liquid chromatography coupled to high-resolution mass spectrometry (LC-MS)-based studies of a biological sample can contain large amounts of biologically significant information in the form of proteins, peptides, and metabolites. Interpreting these data uses the mass spectra to infer masses and abundances of biomolecules injected into the instrument. Because of the inherent complexity of mass spectral patterns produced by these biomolecules, the analysis is significantly enhanced by using visualization capabilities to inspect and confirm results. In this paper we describe Decon2LS, an open-source software package for automated processing and visualization of high-resolution MS data. Drawing extensively on algorithms developed over the last ten years for ICR2LS, Decon2LS packages the algorithms as a rich set of modular, reusable processing classes for performing diverse functions such as raw data readers, routine peak finding, theoretical isotope distribution modelling, and deisotoping. Because the source code is openly available, these functionalities can now be used to build derivative applications in relatively fast manner. In addition, Decon2LS provides an extensive set of visualization tools, such as high performance chart controls.

59 BASIC BIOLOGICAL SCIENCES↗

Combinatorial appraisal of transition states for in situ pair distribution function analysis

In situ total scattering measurements are increasingly utilized to follow atomic and nanoscale structural details of phase transitions and other transient processes in materials. This contribution presents an automated method and associated tool set to analyze series of diffraction and pair distribution function data with a linear combination of end-member states. It is demonstrated that the combinatorial appraisal of transition states (CATS) software tracks phase changes, relative phase fractions and length scales of interest in experimental data series. Here it is further demonstrated, using a series of local structure data simulations, that the misfit of such a model can reveal details of phase aggregation and growth related to the pair distribution function's sensitivity to interphase correlations. CATS may be applied to quantitative evaluation of many transient processes, including amorphous-to-crystalline phase transitions, the evolution of solid-solution behaviors, the precipitation and growth of aggregates, and other atomic to nanoscale details of crystallization and phase transformation phenomena.

36 MATERIALS SCIENCE↗

Framework for Processing Citizens Science Data for Applications to NASA Earth Science Missions

Citizen science (or crowdsourcing) has drawn much high-level recent and ongoing interest and support. It is poised to be applied, beyond the by-now fairly familiar use of, e.g., Twitter for natural hazards monitoring, to science research, such as augmenting the validation of NASA earth science mission data. This interest and support is seen in the 2014 National Plan for Civil Earth Observations, the 2015 White House forum on citizen science and crowdsourcing, the ongoing Senate Bill 2013 (Crowdsourcing and Citizen Science Act of 2015), the recent (August 2016) Open Geospatial Consortium (OGC) call for public participation in its newly-established Citizen Science Domain Working Group, and NASA's initiation of a new Citizen Science for Earth Systems Program (along with its first citizen science-focused solicitation for proposals). Over the past several years, we have been exploring the feasibility of extracting from the Twitter data stream useful information for application to NASA precipitation research, with both "passive" and "active" participation by the twitterers. The Twitter database, which recently passed its tenth anniversary, is potentially a rich source of real-time and historical global information for science applications. The time-varying set of "precipitation" tweets can be thought of as an organic network of rain gauges, potentially providing a widespread view of precipitation occurrence. The validation of satellite precipitation estimates is challenging, because many regions lack data or access to data, especially outside of the U.S. and in remote and developing areas. Mining the Twitter stream could augment these validation programs and, potentially, help tune existing algorithms. Our ongoing work, though exploratory, has resulted in key components for processing and managing tweets, including the capabilities to filter the Twitter stream in real time, to extract location information, to filter for exact phrases, and to plot tweet distributions. The key step is to process the "precipitation" tweets to be compatible with satellite-retrieved precipitation data. These key components for processing and managing "precipitation" tweets (and additional ones to be developed) are not limited to precipitation, nor are they limited to the Twitter social medium. Indeed, to maximize the value of our work for NASA earth science programs, these components should be generalized and be part of an overall framework for processing citizen science data for science research. In this paper, we outline such a framework.

earth science satellite data↗

Correlated prompt fission data in transport simulations

Detailed information on the fission process can be inferred from the observation, modeling and theoretical understanding of prompt fission neutron and γ-ray observables. Beyond simple average quantities, the study of distributions and correlations in prompt data, e.g., multiplicity-dependent neutron and γ-ray spectra, angular distributions of the emitted particles, n -n, n - γ, and γ - γ correlations, can place stringent constraints on fission models and parameters that would otherwise be free to be tuned separately to represent individual fission observables. The FREYA and CGMF codes have been developed to follow the sequential emissions of prompt neutrons and γ rays from the initial excited fission fragments produced right after scission. Both codes implement Monte Carlo techniques to sample initial fission fragment configurations in mass, charge and kinetic energy and sample probabilities of neutron and γ emission at each stage of the decay. This approach naturally leads to using simple but powerful statistical techniques to infer distributions and correlations among many observables and model parameters. The comparison of model calculations with experimental data provides a rich arena for testing various nuclear physics models such as those related to the nuclear structure and level densities of neutron-rich nuclei, the γ-ray strength functions of dipole and quadrupole transitions, the mechanism for dividing the excitation energy between the two nascent fragments near scission, and the mechanisms behind the production of angular momentum in the fragments, etc. Beyond the obvious interest from a fundamental physics point of view, such studies are also important for addressing data needs in various nuclear applications. The inclusion of the FREYA and CGMF codes into the MCNP6.2 and MCNPX - PoliMi transport codes, for instance, provides a new and powerful tool to simulate correlated fission events in neutron transport calculations important in nonproliferation, safeguards, nuclear energy, and defense programs. Here, this review provides an overview of the topic, starting from theoretical considerations of the fission process, with a focus on correlated signatures. It then explores the status of experimental correlated fission data and current efforts to address some of the known shortcomings. Numerical simulations employing the FREYA and CGMF codes are compared to experimental data for a wide range of correlated fission quantities. The inclusion of those codes into the MCNP6.2 and MCNPX - PoliMi transport codes is described and discussed in the context of relevant applications. The accuracy of the model predictions and their sensitivity to model assumptions and input parameters are discussed. Lastly, a series of important experimental and theoretical questions that remain unanswered are presented, suggesting a renewed effort to address these shortcomings.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Machine Learning Potentials with the Iterative Boltzmann Inversion: Training to Experiment

Methodologies for training machine learning potentials (MLPs) with quantum-mechanical simulation data have recently seen tremendous progress. Experimental data have a very different character than simulated data, and most MLP training procedures cannot be easily adapted to incorporate both types of data into the training process. Here, we investigate a training procedure based on iterative Boltzmann inversion that produces a pair potential correction to an existing MLP using equilibrium radial distribution function data. By applying these corrections to an MLP for pure aluminum based on density functional theory, we observe that the resulting model largely addresses previous overstructuring in the melt phase. Interestingly, the corrected MLP also exhibits improved performance in predicting experimental diffusion constants, which are not included in the training procedure. The presented method does not require autodifferentiating through a molecular dynamics solver and does not make assumptions about the MLP architecture. Our results suggest a practical framework for incorporating experimental data into machine learning models to improve the accuracy of molecular dynamics simulations.

36 MATERIALS SCIENCE↗

Apple Strength Issues

Strength of the apple parts has been noticed to decrease, especially those installed by the new induction heating system since the LEP campaign started. Fig. 1 shows the ultimate tensile strength (UTS), yield strength (YS), and elongation of the installed or installation-simulated apples on various systems. One can clearly see the mean values of UTS and YS of the post-LEP parts decreased by about 8 ksi and 6 ksi respectively from those of the pre-LEP parts. The slight increase in elongation seen in Fig.1 can be understood from the weak inverse relationship between the strength and elongation in metals. Fig.2 shows the weak correlation between the YS and elongation of the parts listed in Fig. 1. Strength data listed in Figure 1 were re-plotted as histograms in Figs. 3 and 4. Figs. 3a and 4a show histograms of all UTS and YS data. Figs. 3b and 4b shows histograms of pre-LEP data and Figs. 3c and 4c of post-LEP data. Data on statistical scatter of tensile strengths have been rarely published by material suppliers. Instead, only the minimum 'guaranteed' strength data are typically presented. An example of strength distribution of aluminum 7075-T6 sheet material, listed in Fig. 5, show that its scatter width of both UTS and YS for a single sheet can be about 6 ksi and for multi-lot scatter can be as large as 11 ksi even though the sheets have been produced through well-controlled manufacturing process. By approximating the histograms shown in Figs. 3 and 4 by a Gaussian or similar type of distribution curves, one can plausibly see the strength reductions in the later or more recent apples. The pre-LEP data in Figs. 3b and 4b show wider scatter than the post-LEP data in Figs. 3c and 4c and seem to follow the binomial distribution of strength indicating that the apples might have been made from two different lots of material, either from two different vendors or from two different melts of perhaps slightly different chemical composition by a single vendor. The post-LEP apples seem to have been from a single batch of material. The pre-LEP apples of the weak strength and the post-LEP apples with even weaker strength could have been made of the same batch of material, and the small strength differential might be due to the difference in the induction heating system. If the pre-LEP apples with the lower strength and the post LEP apples are made from the same batch of material, their combined scatter of strength data would be wider and can be understood as a result of the additional processing steps of stress relief and induction heating as discussed.

36 MATERIALS SCIENCE↗

A Semantic Search Capability for a Grid Model Repository

The operation of electric grids is becoming more challenging due to continued increases in wind and solar generation, the proliferation of decentralization generation resources, and cyber security threats. To support the needed research, new large scale realistic public grid models have been developed and placed in a repository for access by researchers. To assist researchers in easily finding grid models that are suitable for their specific research needs, an approach to searching numerous large grid models was evaluated, prototyped and developed. Here, the research identified multiple possible approaches to using graph database techniques and a natural language processing for the search algorithm. Results of the prototype, the final approach for production repository and the search algorithm formulation are described.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Factors Affecting the Evolution of Hurricane Erin and the Distributions of Hydrometeors: Role of Microphysical Processes

Fine-resolution simulations of Hurricane Erin 2001 are conducted using the Penn State University/National Center for Atmospheric Research mesoscale model version 3.5 to investigate the role of thermodynamic, boundary layer and microphysical processes in Erin's growth and maintenance, and their effects on the horizontal and vertical distributions of hydrometeors. Through comparison against radar, radiometer, and dropsonde data collected during the Convection and Moisture Experiment 4, it is seen that realistic simulations of Erin are obtained provided that fine resolution simulations with detailed representations of physical processes are conducted. The principle findings of the study are as follows: 1) a new iterative condensation scheme, which limits the unphysical increase of equivalent potential temperature associated with most condensation schemes, increases the horizontal size of the hurricane, decreases its maximum rainfall rate, reduces its intensity, and makes its eye more moist; 2) in general, microphysical parameterization schemes with more categories of hydrometeors produce more intense hurricanes, larger hydrometeor mixing ratios, and more intense updrafts and downdrafts; 3) the choice of coefficients describing hydrometeor fall velocities has as big of an impact on the hurricane simulations as does choice of microphysical parameterization scheme with no clear relationship between fall velocity and hurricane intensity; and 4) in order for a tropical cyclone to adequately intensify, an advanced boundary layer scheme (e.g., Burk-Thompson scheme) must be used to represent boundary layer processes. The impacts of varying simulations on the horizontal and vertical distributions of different categories of hydrometeor species, on equivalent potential temperature, and on storm updrafts and downdrafts are examined to determine how the release of latent heat feedbacks upon the structure of Erin. In general, all simulations tend to overpredict precipitation rate and hydrometeor mixing ratios. The ramifications of these findings for quantitative precipitation forecasts (QPFs) of tropical cyclones are discussed.

McFarquhar, Greg M.↗

W{gamma} Production and Limits on Anomalous WW{gamma} Couplings in pp Collisions at {radical}(s)=1.96 TeV

We measure the cross section and the difference in rapidities between photons and charged leptons for inclusive W({yields}l{nu})+{gamma} production in e{gamma} and {mu}{gamma} final states. Using data corresponding to an integrated luminosity of 4.2 fb{sup -1} collected with the D0 detector at the Fermilab Tevatron Collider, the measured cross section times branching fraction for the process pp{yields}W{gamma}+X{yields}l{nu}{gamma}+X and the distribution of the charge-signed photon-lepton rapidity difference are found to be in agreement with the standard model. These results provide the most stringent limits on anomalous WW{gamma} couplings for data from hadron colliders: -0.4<{Delta}{kappa}{sub {gamma}<}0.4 and -0.08<{lambda}{sub {gamma}<}0.07 at the 95% C.L.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Evaluating the Performance of Single and Double Moment Microphysics Schemes During a Synoptic-Scale Snowfall Event

Increases in computing resources have allowed for the utilization of high-resolution weather forecast models capable of resolving cloud microphysical and precipitation processes among varying numbers of hydrometeor categories. Several microphysics schemes are currently available within the Weather Research and Forecasting (WRF) model, ranging from single-moment predictions of precipitation content to double-moment predictions that include a prediction of particle number concentrations. Each scheme incorporates several assumptions related to the size distribution, shape, and fall speed relationships of ice crystals in order to simulate cold-cloud processes and resulting precipitation. Field campaign data offer a means of evaluating the assumptions present within each scheme. The Canadian CloudSat/CALIPSO Validation Project (C3VP) represented collaboration among the CloudSat, CALIPSO, and NASA Global Precipitation Measurement mission communities, to observe cold season precipitation processes relevant to forecast model evaluation and the eventual development of satellite retrievals of cloud properties and precipitation rates. During the C3VP campaign, widespread snowfall occurred on 22 January 2007, sampled by aircraft and surface instrumentation that provided particle size distributions, ice water content, and fall speed estimations along with traditional surface measurements of temperature and precipitation. In this study, four single-moment and two double-moment microphysics schemes were utilized to generate hypothetical WRF forecasts of the event, with C3VP data used in evaluation of their varying assumptions. Schemes that incorporate flexibility in size distribution parameters and density assumptions are shown to be preferable to fixed constants, and that a double-moment representation of the snow category may be beneficial when representing the effects of aggregation. These results may guide forecast centers in optimal configurations of their forecast models for winter weather and identify best practices present within these various schemes.

Molthan, Andrew L.↗

Hail Size Distribution Mapping

A 3-D weather radar visualization software program was developed and implemented as part of an experimental Launch Pad 39 Hail Monitor System. 3DRadPlot, a radar plotting program, is one of several software modules that form building blocks of the hail data processing and analysis system (the complete software processing system under development). The spatial and temporal mapping algorithms were originally developed through research at the University of Central Florida, funded by NASA s Tropical Rainfall Measurement Mission (TRMM), where the goal was to merge National Weather Service (NWS) Next-Generation Weather Radar (NEXRAD) volume reflectivity data with drop size distribution data acquired from a cluster of raindrop disdrometers. In this current work, we adapted these algorithms to process data from a cluster of hail disdrometers positioned around Launch Pads 39A or 39B, along with the corresponding NWS radar data. Radar data from all NWS NEXRAD sites is archived at the National Climatic Data Center (NCDC). That data can be readily accessed at . 3DRadPlot plots Level III reflectivity data at four scan elevations (this software is available at Open Channel Software, ). By using spatial and temporal interpolation/extrapolation based on hydrometeor fall dynamics, we can merge the hail disdrometer array data coupled with local Weather Surveillance Radar-1988, Doppler (WSR-88D) radial velocity and reflectivity data into a 4-D (3-D space and time) picture of hail size distributions. Hail flux maps can then be generated and used for damage prediction and assessment over specific surfaces corresponding to structures within the disdrometer array volume. Immediately following a hail storm, specific damage areas and degree of damage can be identified for inspection crews.

Source record↗

Impact of Salinity on Ground Ice Distribution Across an Arctic Coastal Polygonal Tundra Environment

The heterogeneous distribution of ground ice in the Arctic is a key driver of uneven ground subsidence as permafrost thaws, significantly impacting infrastructure and surface/subsurface hydrology. These topographic and hydrological changes contribute to major uncertainties in energy and carbon fluxes and storage in a warming Arctic. This study aims to improve our understanding of the controls on ground ice and organic matter distribution within the top 3 m of permafrost in coastal polygonal tundra near Utqiagvik, Alaska. To this end, we apply a neural network approach to bulk density distributions derived from nondestructive X-ray tomography of soil cores, trained with laboratory analyses, to improve the resolution and spatial coverage of estimates of dry bulk density, ice content, and organic matter content. In addition, we use capacitively coupled geophysical imaging to map soil electrical conductivity and salinity variations. The results show that sedimentary deposits from ocean transgressions, along with subsequent ice wedge polygon geomorphological processes, jointly influence the distribution of ice content at various scales. The impact of the latter decreases with depth, whereas the influence of salinity and sedimentary history increases. Although the controls on the distribution of soil organic matter content (g/cm 3 ) remain unclear, the pronounced heterogeneity in bulk density strongly influences its calculation from laboratory mass fraction measurements (g/g). From a methodological perspective, the interdependencies among soil components and the need for increased data coverage underscore the value of high-resolution density measurements, such as using X-ray tomography. Overall, this study emphasizes the importance of considering salinity constraints on ice content distribution in coastal permafrost regions. The results are expected to aid in the development of data products and process representations in geomorphological and ecosystem models.

Dafflon, Baptiste [Lawrence Berkeley National Labo↗