Search NASA⌕ Search

SEARCH · Search NASA

Results for “Bloom Filter”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Bloom filter de-identifier

A bloom filter is a binary data structure that is designed to be memory efficient and able to determine whether an element is present in a set. The bloom filter is a probabilistic data structure. It can tell you whether an element, or set of elements, is likely to exist within a set. It can also tell you whether an element, or set of elements, does not exist within a set. So false positives are probable, but false negatives are not. Our novel approach utilizes the bloom filter to determine how many bigrams (two letter sequence) each value has in common. The resulting metric approximates the relative similarity of two text values.

Bleeker, Amelia↗

Bloom filters for molecules

Abstract Ultra-large chemical libraries are reaching 10s to 100s of billions of molecules. A challenge for these libraries is to efficiently check if a proposed molecule is present. Here we propose and study Bloom filters for testing if a molecule is present in a set using either string or fingerprint representations. Bloom filters are small enough to hold billions of molecules in just a few GB of memory and check membership in sub milliseconds. We found string representations can have a false positive rate below 1% and require significantly less storage than using fingerprints. Canonical SMILES with Bloom filters with the simple FNV (Fowler-Noll-Voll) hashing function provide fast and accurate membership tests with small memory requirements. We provide a general implementation and specific filters for detecting if a molecule is purchasable, patented, or a natural product according to existing databases at https://github.com/whitead/molbloom .

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Jacobian sparsity detection using Bloom filters

Determining Jacobian sparsity structure is an important step in the efficient computation of sparse Jacobians. We introduce a new method for determining Jacobian sparsity patterns by combining bit vector probing with Bloom filters. In conclusion, we further refine Bloom filter probing by combining it with hierarchical probing to yield a highly effective strategy for Jacobian sparsity pattern determination.

Bloom filter↗

Developing Bloom Filters for Web Archives’ Holdings (Final Project Report)

The main goal of the project was to develop a framework for web archives to create Bloom filters based on their holdings of archived web resources. A Bloom filter (BF) is a data structure that, in our scenario, contains hash values of all (or a subset of) URLs, of which an archive has one or more archival copies. Two main use cases fall in scope for this project and are supported by a BF implementation: 1) Sharing of an archive’s holdings (URLs) and 2) Querying the holdings of one or more archives. Since URL strings are hashed before ingested into the BF, the index of an archive is not shared in plain text when the BF is shared with trusted parties. On the other hand, a BF implementation does allow queries for a URL to confirm if an archive indeed has one or more archival copies of that URL. This collaborative project between Los Alamos National Laboratory (LANL) and the Croatian Web Archive (HAW), from the National and University Library in Zagreb (NSK), developed by NSK and University of Zagreb University Computing Center SRCE), aimed at developing software to create BFs, evaluate the scalability of the approach, pilot a search service based on BFs, and design a framework for archives to share their filters with trusted parties. In the remainder of this document, we will report on the work completed with respect to the individual deliverables, outline aspects of future work, and conclude with our recommendations for the use of BFs for the web archiving community.

97 MATHEMATICS AND COMPUTING↗

Bloom Filter framework for Web Archives

The software provides a framework to build a fast look up layer that reflects the holdings of an archive or database and operates between search/discovery system and disk storage. The software utilizes Bloom filter (BF) data structure. The discovery service powered by the Bloom filter layer comes with a very high level of accuracy yet space-saving, since BF compresses data to fit into RAM.

Balakireva, Lyudmila↗

Singleton Sieving: Overcoming the Memory/Speed Trade-Off in Exascale k-mer Analysis

Traditional filter data structures, such as Bloom filters, do not offer necessary features that modern high-performance data analytics applications need in order to efficiently perform complex data analysis tasks. For example, MetaHipMer, a de novo metagenome assembler, can use filters to weed out singleton k-mers and reduce memory usage by 30%-70%. However, the filter needs the ability to associate values with k-mers in order to perform the analysis in a single communication pass. Bloom filters do not support value associations and cause the application to perform an extra communication pass, thereby increasing the run time. Therefore, MetaHipMer faces a trade off between memory and speed due to the limited capabilities of traditional filters. In this paper, we overcome the memory and speed trade off in MetaHipMer by integrating a GPU-based feature-rich filter, the Two-Choice filter (TCF), in the MetaHipMer pipeline. The TCF uses key-value association to approximately store k-mers with extensions. This allows MetaHipMer to perform k-mer analysis on the GPUs in a single communication pass. Our empirical analysis shows a 50% reduction in memory usage in k-mer analysis on each node in MetaHipMer without any effect on the overall run time or assembly quality. The memory reduction in turn results in a 43% reduction in the number of nodes required to assemble datasets and enables MetaHipMer to scale to much larger datasets.

McCoy, Hunter↗

Improved Distributed-memory Triangle Counting by Exploiting the Graph Structure

Graphs are ubiquitous in modeling complex systems and representing interactions between entities to uncover structural information of the domain. Traditionally, graph analytics workloads are challenging to efficiently scale (both strong and weak cases) on distributed memory due to the irregular memory-access driven nature (with little or no computations) of the methods. The structure of graphs and their relative distribution over the processing elements poses another level of complexity, making it difficult to attain sustainable scalability across platforms. In this paper, we discuss enhancements to TriC, a distributed-memory implementation of graph triangle counting using Message Passing Interface (MPI), which was featured in the 2020 Graph Challenge competition. We have made some incremental enhancements to TriC, primarily adopting a user-defined buffering strategy to overcome the startup problem for large graphs (by fixing the memory for intermediate data), and experimenting with probabilistic data structures such as bloom filter to improve the query response time for assessing edge existence, at the expense of increasing the overall false positive rate. These adjustments have led to a modest improvements in most cases, as compared to the previous version.

Graph Analytics, HPC↗

An optical filtering system for remote sensing of phytoplankton and suspended sediment

Aerial photography revealing blue-green Anacystis phytoplankton blooms in the Potomac River is presented. Data scanned were near-IR, green, and yellow nadir radiance; concurrent ground measurements are correlated with these data, including salinity, Secchi disk depth, chlorophyll a concentration, and river depth. In nonbloomed areas, R (radiance) is proportional to the reciprocal of Secchi depth for the green and yellow filters; in bloomed areas, R is constant for the green filter and random for the near-IR and yellow filters.

Bressette, W. E.↗

Development of a narrow stopband filter for spectroscopic fast ion deuterium-alpha measurements

A new technique to attenuate the unshifted deuterium Balmer-alpha (D-alpha) emission is developed and tested for the fast ion D-alpha (FIDA) diagnostic. The unshifted D-alpha emission, at λ = 656.1 nm, is around three orders of magnitude higher than the desired FIDA emission. Blocking the strong emission feature is essential to prevent blooming and light smearing on the CCD chip and scattered light contamination. The new method is a notch filter approach that utilizes the reflection from ultra-narrow bandpass filters to block the saturating signal before it enters the spectrometer. Collimated light from the fibers is reflected off the filter at a 15° angle of incidence. Measurements show that a center wavelength transmission of 0.006 and a blocking full width at half maximum of ~ 1 nm are achieved by using a 200 μm fiber and a 20 mm focal length collimator with two filters.

47 OTHER INSTRUMENTATION↗

A chemical perspective on the chiral induced spin selectivity effect

ABSTRACT This review discusses opportunities in chemistry that are enabled by the chiral induced spin selectivity (CISS) effect. First, the review begins with a brief overview of the seminal studies on CISS. Next, we discuss different chiral material systems whose properties can be tailored through chemical means, with a special emphasis on hybrid organic-inorganic layered materials that exhibit some of the largest spin filtering properties to date. Then, we discuss the promise of CISS for chemical reactions and enantioseparation before concluding.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

PhyLM: A Mission Design Concept for an Optical/Lidar Instrument to Measure Ocean Productivity and Aerosols from Space

The Physiology Lidar-Multispectral Mission (PhyLM) is intended to explore the complex ecosystems of our global oceans. New "inversion" methods and improved understanding of marine optics have opened the door to quantifying a range of critical ocean properties. This new information could revolutionize our understanding of global ocean processes, such as phytoplankton growth, harmful algal blooms, carbon fluxes between major pools and the productivity equation. The new science requires new measurements not addressed by currently planned space missions. PhyLM will combine active and advanced passive remote sensing technologies to quantify standing stocks and fluxes of climate-critical components of the Ocean carbon cycle to meet these science providing multispectral bands from the far UV through the near infrared (340 - 1250 nm) at a ground resolution of 250 m. Improved detectors, filters, mirrors, digitization and focal plane design will offer an overall higher-quality data product. The unprecedented accuracy and precision of the absolute water-leaving radiances will support inversion- based quantification of an expanded set of ocean carbon cycle components. The dual- wavelength (532 & 1064 nm) Nd:Yag Lidar will enhance the accuracy and precision of the passive data by providing aerosol profiles for atmospheric correction and coincident active measurements of backscattering. The Lidar will also examine dark-side fluorescence as an additional approach to quantifying phytoplankton biomass in highly productive regions.

Gervin, Janette C.↗

A Two-Dimensional Variational Analysis Method for NSCAT Ambiguity Removal: Methodology, Sensitivity, and Tuning

In this study, we apply a two-dimensional variational analysis method (2d-VAR) to select a wind solution from NASA Scatterometer (NSCAT) ambiguous winds. 2d-VAR determines a "best" gridded surface wind analysis by minimizing a cost function. The cost function measures the misfit to the observations, the background, and the filtering and dynamical constraints. The ambiguity closest in direction to the minimizing analysis is selected. 2d-VAR method, sensitivity and numerical behavior are described. 2d-VAR is compared to statistical interpolation (OI) by examining the response of both systems to a single ship observation and to a swath of unique scatterometer winds. 2d-VAR is used with both NSCAT ambiguities and NSCAT backscatter values. Results are roughly comparable. When the background field is poor, 2d-VAR ambiguity removal often selects low probability ambiguities. To avoid this behavior, an initial 2d-VAR analysis, using only the two most likely ambiguities, provides the first guess for an analysis using all the ambiguities or the backscatter data. 2d-VAR and median filter selected ambiguities usually agree. Both methods require horizontal consistency, so disagreements occur in clumps, or as linear features. In these cases, 2d-VAR ambiguities are often more meteorologically reasonable and more consistent with satellite imagery.

Hoffman, R. N.↗

Normal Mode Initialization

The objective of this research is the development and implementation of normal mode procedures for use with the GLAS analysis/forecast system. Specific tasks of this work include: (1) high latitude filtering of model fields to preserve the GLAS GCM's linear stability during integration; (2) development of nonlinear normal mode initialization (NLNMI) processes, both adiabatic and diabatic. Using NLNMI to initialize GLAS analyses; investigation of the impact of normal mode initialization on the GLAS analysis/forecast system, especially in regard to data assimilation; and (3) diagnosis of the 1-3 day systemic forecast errors of the GLAS GCM.

Bloom, S. C.↗

The Swift-BAT Hard X-Ray Transient Monitor

The Swift/Burst Alert Telescope (BAT) hard X-ray transient monitor provides near real-time coverage of the X-ray sky in the energy range 15-50 keV. The BAT observes 88% of the sky each day with a detection sensitivity of 5.3 mCrab for a full-day observation and a time resolution as fine as 64 s. The three main purposes of the monitor are (1) the discovery of new transient X-ray sources, (2) the detection of outbursts or other changes in the flux of known X-ray sources, and (3) the generation of light curves of more than 900 sources spanning over eight years. The primary interface for the BAT transient monitor is a public Web site. Between 2005 February 12 and 2013 April 30, 245 sources have been detected in the monitor, 146 of them persistent and 99 detected only in outburst. Among these sources, 17 were previously unknown and were discovered in the transient monitor. In this paper, we discuss the methodology and the data processing and filtering for the BAT transient monitor and review its sensitivity and exposure.We provide a summary of the source detections and classify them according to the variability of their light curves. Finally, we review all new BAT monitor discoveries. For the new sources that are previously unpublished, we present basic data analysis and interpretations.

black hole physics↗

File Specification for GEOS-5 FP (Forward Processing)

The GEOS-5 FP Atmospheric Data Assimilation System (GEOS-5 ADAS) uses an analysis developed jointly with NOAA's National Centers for Environmental Prediction (NCEP), which allows the Global Modeling and Assimilation Office (GMAO) to take advantage of the developments at NCEP and the Joint Center for Satellite Data Assimilation (JCSDA). The GEOS-5 AGCM uses the finite-volume dynamics (Lin, 2004) integrated with various physics packages (e.g, Bacmeister et al., 2006), under the Earth System Modeling Framework (ESMF) including the Catchment Land Surface Model (CLSM) (e.g., Koster et al., 2000). The GSI analysis is a three-dimensional variational (3DVar) analysis applied in grid-point space to facilitate the implementation of anisotropic, inhomogeneous covariances (e.g., Wu et al., 2002; Derber et al., 2003). The GSI implementation for GEOS-5 FP incorporates a set of recursive filters that produce approximately Gaussian smoothing kernels and isotropic correlation functions. The GEOS-5 ADAS is documented in Rienecker et al. (2008). More recent updates to the model are presented in Molod et al. (2011). The GEOS-5 system actively assimilates roughly 2 × 10(exp 6) observations for each analysis, including about 7.5 × 10(exp 5) AIRS radiance data. The input stream is roughly twice this volume, but because of the large volume, the data are thinned commensurate with the analysis grid to reduce the computational burden. Data are also rejected from the analysis through quality control procedures designed to detect, for example, the presence of cloud. To minimize the spurious periodic perturbations of the analysis, GEOS-5 FP uses the Incremental Analysis Update (IAU) technique developed by Bloom et al. (1996). More details of this procedure are given in Appendix A. The assimilation is performed at a horizontal resolution of 0.3125-degree longitude by 0.25- degree latitude and at 72 levels, extending to 0.01 hPa. All products are generated at the native resolution of the horizontal grid. The majority of data products are time-averaged, but four instantaneous products are also available. Hourly data intervals are used for two-dimensional products, while 3-hourly intervals are used for three-dimensional products. These may be on the model's native 72-layer vertical grid or at 42 pressure surfaces extending to 0.1 hPa. This document describes the gridded output files produced by the GMAO near real-time operational FP, using the most recent version of the GEOS-5 assimilation system. Additional details about variables listed in this file specification can be found in a separate document, the GEOS-5 File Specification Variable Definition Glossary. Documentation about the current access methods for products described in this document can be found on the GMAO products page: http://gmao.gsfc.nasa.gov/products/.

GSI↗

File Specification for GEOS-5 FP-IT (Forward Processing for Instrument Teams)

The GEOS-5 FP-IT Atmospheric Data Assimilation System (GEOS-5 ADAS) uses an analysis developed jointly with NOAA's National Centers for Environmental Prediction (NCEP), which allows the Global Modeling and Assimilation Office (GMAO) to take advantage of the developments at NCEP and the Joint Center for Satellite Data Assimilation (JCSDA). The GEOS-5 AGCM uses the finite-volume dynamics (Lin, 2004) integrated with various physics packages (e.g, Bacmeister et al., 2006), under the Earth System Modeling Framework (ESMF) including the Catchment Land Surface Model (CLSM) (e.g., Koster et al., 2000). The GSI analysis is a three-dimensional variational (3DVar) analysis applied in grid-point space to facilitate the implementation of anisotropic, inhomogeneous covariances (e.g., Wu et al., 2002; Derber et al., 2003). The GSI implementation for GEOS-5 FP-IT incorporates a set of recursive filters that produce approximately Gaussian smoothing kernels and isotropic correlation functions. The GEOS-5 ADAS is documented in Rienecker et al. (2008). More recent updates to the model are presented in Molod et al. (2011). The GEOS-5 system actively assimilates roughly 2 × 10(exp 6) observations for each analysis, including about 7.5 × 10(exp 5) AIRS radiance data. The input stream is roughly twice this volume, but because of the large volume, the data are thinned commensurate with the analysis grid to reduce the computational burden. Data are also rejected from the analysis through quality control procedures designed to detect, for example, the presence of cloud. To minimize the spurious periodic perturbations of the analysis, GEOS-5 FP-IT uses the Incremental Analysis Update (IAU) technique developed by Bloom et al. (1996). More details of this procedure are given in Appendix A. The analysis is performed at a horizontal resolution of 0.625-degree longitude by 0.5-degree latitude and at 72 levels, extending to 0.01 hPa. All products are generated at the native resolution of the horizontal grid. The majority of data products are time-averaged, but four instantaneous products are also available. Hourly data intervals are used for two-dimensional products, while 3-hourly intervals are used for three-dimensional products. These may be on the model's native 72-layer vertical grid or at 42 pressure surfaces extending to 0.1 hPa. This document describes the gridded output files produced by the GMAO near real-time operational GEOS-5 FP-IT processing in support of the EOS instrument teams. Additional details about variables listed in this file specification can be found in a separate document, the GEOS-5 File Specification Variable Definition Glossary.

3D Var↗