Search NASA⌕ Search

SEARCH · Search NASA

Results for “distributed databases”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

End-to-end online performance data capture and analysis for scientific workflows

With the increased prevalence of employing workflows for scientific computing and a push towards exascale computing, it has become paramount that we are able to analyze characteristics of scientific applications to better understand their impact on the underlying infrastructure and vice-versa. Such analysis can help drive the design, development, and optimization of these next generation systems and solutions. Here, we present the architecture, integrated with existing well-established and newly developed tools, to collect online performance statistics of workflow executions from various, heterogeneous sources and publish them in a distributed database (Elasticsearch). Using this architecture, we are able to correlate online workflow performance data, with data from the underlying infrastructure, and present them in a useful and intuitive way via an online dashboard. We have validated our approach by executing two classes of real-world workflows, both under normal and anomalous conditions. The first is an I/O-intensive genome analysis workflow; the second, a CPU- and memory-intensive material science workflow. Based on the data collected in Elasticsearch, we are able to demonstrate that we can correctly identify anomalies that we injected. The resulting end-to-end data collection of workflow performance data is an important resource of training data for automated machine learning analysis.

97 MATHEMATICS AND COMPUTING↗

The Open High Throughput Computing Content Delivery Network

LHC experiments make extensive use of web proxy caches, especially for software distribution via the CernVM File System and for conditions data via the Frontier Distributed Database Caching system. Since many jobs read the same data, cache hit rates are high and hence most of the traffic flows efficiently over Local Area Networks. However, it is not always possible to have local web caches, particularly for opportunistic cases whereexperiments have little control over site services. The Open High Throughput Computing (HTC) Content Delivery Network (CDN), openhtc.io, aims to address this by using web proxy caches from a commercial CDN provider. Cloudflare provides a simple interface for registering DNS aliases of any web server and does reverse proxy web caching on those aliases. The openhtc.io domain is hosted on Cloudflare's free tier CDN which has no bandwidth limit and makes use of data centers throughout the world, so the average performance for clients is much improved compared to reading from CERN or a Tier 1. The load on WLCG servers is also significantly reduced. WLCG Web Proxy Auto Discovery is used to select local web caches when they are available and otherwise select openhtc.io caching. This paper describes the Open HTC CDN in detail and provides initial results from its use for LHC@Home and USCMS opportunistic computing.

97 MATHEMATICS AND COMPUTING↗

Web Proxy Auto Discovery for the WLCG

All four of the LHC experiments depend on web proxies (that is, squids) at each grid site to support software distribution by the CernVM FileSystem (CVMFS). CMS and ATLAS also use web proxies for conditions data distributed through the Frontier Distributed Database caching system. ATLAS & CMS each have their own methods for their grid jobs to find out which web proxies to use for Frontier at each site, and CVMFS has a third method. Those diverse methods limit usability and flexibility, particularly for opportunistic use cases, where an experiment’s jobs are run at sites that do not primarily support that experiment. This paper describes a new Worldwide LHC Computing Grid (WLCG) system for discovering the addresses of web proxies. The system is based on an internet standard called Web Proxy Auto Discovery (WPAD). WPAD is in turn based on another standard called Proxy Auto Configuration (PAC). Both the Frontier and CVMFS clients support this standard. The input into the WLCG system comes from squids registered in the ATLAS Grid Information System (AGIS) and CMS SITECONF files, cross-checked with squids registered by sites in the Grid Configuration Database (GOCDB) and the OSG Information Management (OIM) system, and combined with some exceptions manually configured by people from ATLAS and CMS who operate WLCG Squid monitoring. WPAD servers at CERN respond to http requests from grid nodes all over the world with a PAC file that lists available web proxies, based on IP addresses matched from a database that contains the IP address ranges registered to organizations. Large grid sites are encouraged to supply their own WPAD web servers for more flexibility, to avoid being affected by short term long distance network outages, and to offload the WLCG WPAD servers at CERN. The CERN WPAD servers additionally support requests from jobs running at non-grid sites (particularly for LHC@Home) which it directs to the nearest publicly accessible web proxy servers. Furthermore, the responses to those requests are geographically ordered based on a separate database that maps IP addresses to longitude and latitude.

97 MATHEMATICS AND COMPUTING↗

Touring DNS Open Houses for Trends and Configurations

DNS is a critical component of the Internet. It maps domain names to IP addresses and serves as a distributed database for various other applications, including mail, Web, and spam filtering. This paper examines DNS zones in the Internet for diversity, adoption rates of new technologies, and prevalence of configuration issues. To gather data, we sweep 60% of the Internet's domains in June - August 2007 for zone transfers. 6.6% of them allow us to transfer their complete information. Surprisingly, this includes a large fraction of the domains deploying DNSSEC. We find that DNS zones vary significantly in size and some span many ASes. Also, while anti-spam technologies appear to be getting deployed, the adoption rates of DNSSEC and IPv6 continue to be low. Finally, we also find that carelessness in handing DNS records can lead to reduced availability of name servers, email, and Web servers. This also undermines anti-spam efforts and the efforts to shut down phishing sites or to contain malware infections.

99 GENERAL AND MISCELLANEOUS↗

Datashare

Datashare facilitates communication and data sharing within local networks in potentially dangerous situations such as an explosive ordnance disposal. During such events, there is a need to transmit information rapidly around the incident area. It is a distributed database that does not require an internet connection for operation. In addition, Datashare interfaces with XTK and other software applications, allowing for seamless integration and data management. Datashare supports video calls over the network, enabling real-time communication among users. This software serves to organize, package, and share between responders on location and export data to those off location. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Eldridge, Bryce [Sandia National Lab. (SNL-CA), Li↗

Visualization and Data Analysis for High-Performance Computing

This is a set of slides from a guest lecture for a class at the University of Texas, El Paso on visualization and data analysis for high-performance computing. The topics covered are the following: trends in high-performance computing; scientific visualization, such as OpenGL, ray tracing and volume rendering, VTK, and ParaView; data science at scale, such as in-situ visualization, image databases, distributed memory parallelism, shared memory parallelism, VTK-m, "big data", and then an analysis example.

97 MATHEMATICS AND COMPUTING↗

Evaluation of regional isoprene emission factors and modeled fluxes in California

Accurately modeled biogenic volatile organic compound (BVOC) emissions are a vital input to atmospheric chemistry simulations of ozone and particle formation. BVOC emission models rely on basal emission factor (BEF) distribution maps based on emission measurements and vegetation land-cover data but these critical input components of the models as well as model simulations lack validation by regional scale measurements. We directly assess isoprene emission-factor distribution databases for BVOC emission models by deriving BEFs from direct airborne eddy covariance (AEC) fluxes (Misztal et al., 2014) scaled to the surface and normalized by the activity factor of the Guenther et al. (2006) algorithm. The available airborne BEF data from approx. 10 000 km of flight tracks over California were averaged spatially over 48 defined ecological zones called ecoregions. Consistently, BEFs used by three different emission models were averaged over the same ecoregions for quantitative evaluation. Ecoregion-averaged BEFs from the most current land cover used by the Model of Emissions of Gases and Aerosols from Nature (MEGAN) v.2.1 resulted in the best agreement among the tested land covers and agreed within 10 % with BEFs inferred from measurement. However, the correlation was sensitive to a few discrepancies (either overestimation or underestimation) in those ecoregions where land-cover BEFs are less accurate or less representative for the flight track. The two other land covers demonstrated similar agreement (within 30 % of measurements) for total average BEF across all tested ecoregions but there were a larger number of specific ecoregions that had poor agreement with the observations. Independently, we performed evaluation of the new California Air Resources Board (CARB) hybrid model by directly comparing its simulated isoprene area emissions averaged for the same flight times and flux footprints as actual measured area emissions. The model simulation and the observed surface area emissions agreed on average within 20 %. We demonstrate that the choice of model land-cover input data has the most critical influence on model-measurement agreement and the uncertainty in meteorology inputs has a lesser impact at scales relevant to regional air quality modeling.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

AISL Development Toolkit

AISLDT is a library of utility functions supporting other AISL software. Code provides various utility functions for Common Lisp, including an object-oriented database, distributed objects, logic query engine, web content management, chart drawing, packet sniffing, text processing, and various data structures.

Source record↗

WLCG Web Proxy Auto Discovery for Dynamically Created Web Proxies

The WLCG Web Proxy Auto Discovery (WPAD) service provides a convenient mechanism for jobs running anywhere on the WLCG to dynamically discover web proxy cache servers that are nearby. The web proxy caches are general purpose for a number of different http applications, but different applications have different usage characteristics and not all proxy caches are engineered to work with the heaviest loads. For this reason, the initial sources of information for WLCG WPAD were the static configurations that ATLAS and CMS maintain for the Conditions data that they read through the Frontier Distributed Database system, which is the most demanding popular WLCG application for web proxy caches. That works well for use at traditional statically defined WLCG sites, but now that usage of commercial clouds is increasing, there is also a need for web proxy caches to dynamically register themselves as they are created. A package called Shoal had already been created to manage dynamically created web proxy caches. This paper describes the integration of the Shoal package into the WLCG WPAD system, such that both statically and dynamically created web proxy caches can be located from a single source. It also describes other improvements to the WLCG WPAD system since the last CHEP publication.

Dykstra, Dave↗

The Effect of Spectral Resolution of Light Sources on Photopic and a-opic Quantities

There is a growing interest in including light source spectrum in advanced lighting software tools and simulations. Given that traditional lighting software tools have been used primarily for calculating photometric quantities, their simplifying assumptions may not be suitable for calculating other a-opic quantities. Commonly used simulation tools use three values to represent the three primary colors (red, green, and blue), but others have expanded the number of bands (i.e., the spectral resolution) to 9, 27, or 81 in an attempt to more accurately represent variations in light source spectrum and downstream spectrally-derived quantities. It remains unclear, however, to what extent spectral resolution affects calculated quantities. To address this gap, a numerical analysis was completed using a large spectral power distribution database (n = 1,302 light sources). Calculated illuminance, a-opic irradiance, luminous efficacy of radiation (LER), melanopic efficacy of radiation (MER), and melanopic to photopic (M/P) ratio were compared for spectral resolutions of 3 and 9 bands compared to a baseline of 81 bands. Across all examined lighting quantities, considerable errors—a mean absolute percent error of 18.4%—were found when using 3 band calculations. These were reduced to 4% for 9 band calculations. The errors varied by metric and light source type. The results suggest that the use of 9 bands can more accurately characterize performance of different light source types across a range of metrics compared to using 3 bands.

Abboushi, Belal K.↗

A study on the criticality safety for the direct disposal of used nuclear fuel in Japan: Application of burnup credit to the criticality safety evaluation for the disposal canister - 14049

Japan has recently started the technical development of the direct disposal of used nuclear fuel (UNF) to prepare various disposal options. Unlike the disposal of radioactive waste from UNF reprocessing, criticality safety is important here because of the presence of a certain amount of fissile nuclides in UNF. This paper provides an outline of the research to be addressed in this field and the relevant studies in Japan. It especially includes the first result of a criticality safety evaluation of a disposal canister model adopting burnup credit. In this evaluation, it was assumed that a steel disposal canister loaded with fuel assemblies was surrounded by a clay buffer material, and the fuel assemblies and the disposal canister were intact. Conditions representative of discharged pressurized-water reactor (PWR) fuel were applied in this study - initial U-235 enrichment of 4.5 wt% and burnup of 45 GWd/t. The isotopic composition of the burnt fuel was evaluated using SWAT3.1 code. In the depletion calculation, conservative parameters were applied within the typical PWR operation range. The effective neutron multiplication factor (k{sub eff}) was then calculated using a continuous-energy Monte Carlo code MVP-2.0, adopting the JENDL-4.0 nuclear data library. Uncertainty in k{sub eff} due to depletion calculation errors was evaluated using the correction factors of isotopic composition, which were obtained via multiple comparisons between the calculated and measured isotopic concentrations of irradiated fuels. In addition, the effects of axial burnup profile and horizontal burnup gradient on k{sub eff} were evaluated using the open burnup distribution database. The k{sub eff} including the uncertainties and conservatism was found to be below 0.95 for both actinide-only and actinide and fission product considerations when the fuel assemblies and the disposal canister were assumed to be intact. (authors)

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Distributed Wind Project Database

The PNNL research team continually collects cost, incentive, generation, and customer data from turbine manufacturers, operations and maintenance providers, state and federal agencies, and other stakeholders for distributed wind projects installed in the United States. These data are critical for identifying trends, opportunities for growth, and prioritizing investments for both WETO and industry stakeholders. Making this information available allows interested parties to better understand distributed wind market trends and characteristics.

17 WIND ENERGY↗

Density-based similarity measures for content based search

We consider the query by multiple example problem where the goal is to identify database samples whose content is similar to a coUection of query samples. To assess the similarity we use a relative content density which quantifies the relative concentration of the query distribution to the database distribution. If the database distribution is a mixture of the query distribution and a background distribution then it can be shown that database samples whose relative content density is greater than a particular threshold {rho} are more likely to have been generated by the query distribution than the background distribution. We describe an algorithm for predicting samples with relative content density greater than {rho} that is computationally efficient and possesses strong performance guarantees. We also show empirical results for applications in computer network monitoring and image segmentation.

97 MATHEMATICS AND COMPUTING↗

CWS4DB: A Customizable Web Service for Efficient Access to Distributed Nuclear Physics Relational Databases

Tech-X proposes to develop a system providing an efficient access for the High-Energy and Nuclear Physics data stored in distributed heterogeneous relational databases. The system will consist of a generic Web Service, customized client APIs, and tools that facilitate generation of custom client APIs. The access to the data will be mediated by mechanisms that provide data caching and adaptive scheduling of the query plans.

99 GENERAL AND MISCELLANEOUS↗