Search NASA⌕ Search

SEARCH · Search NASA

Results for “false data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 253 records · Page 14

pyFLANK, a graph neural network based null distribution inference model for F ST outlier detection

Detecting genomic regions under selection is essential for understanding how populations adapt to different environments, yet it remains challenging due to the confounding effects of demographic history and linkage disequilibrium (LD). Fixation index (F ST ) is a widely used statistic to identify genomic regions under adaptation. However, identifying genes under selection by defining F ST outliers often remains challenging, owing to confounding effects of underlying demographic history. Traditional methods assume independence among loci and rely on simple demographic models, while newer models perform much better but are computationally expensive and not easily scalable. Here, we present pyFLANK, an open-source and automated Python implementation which detects F ST outliers using a null distribution inferred from quasi-independent loci. Our tool integrates three approaches to identify loci obeying a null distribution: graph neural network (GNN) inference, linkage disequilibrium (LD)-based inference, and user-defined input. Because pyFLANK uses GNN-based inference of quasi-independent loci, it yields a more accurate null model with less need for user parameter input. In simulation experiments, pyFLANK achieved lower false positive rates than current methods while maintaining comparable detection power, indicating that its refined null model better distinguishes true adaptive loci from background variation. The GNN-based model, in particular, detected additional loci associated with phenotypic variance that were not identified by existing methods. Assessments of simulation and real data from different species demonstrate that pyFLANK achieves lower false positive rates compared with other commonly used F ST outlier detectors, while maintaining comparable detection power and excellent computational performance, providing a robust and user-friendly tool for identifying loci under divergent selection. It extends existing F ST outlier frameworks by incorporating explicit LD-aware strategies for null model calibration. The method is intended as a practical and scalable complement to existing genome scan approaches.

FST↗

Condition monitoring and anomaly detection in cyber-physical systems

The modern industrial environment is equipping myriads of smart manufacturing machines where the state of each device can be monitored continuously. Such monitoring can help identify possible future failures and develop a cost-effective maintenance plan. However, it is a daunting task to perform early detection with low false positives and negatives from the huge volume of collected data. This requires developing a holistic machine learning framework to address the issues in condition monitoring of high priority components and develop efficient techniques to detect anomalies that can detect and possibly localize the faulty components. This paper presents a comparative analysis of recent machine learning approaches for robust, cost-effective anomaly detection in cyber-physical systems. While detection has been extensively studied, very few researchers have analyzed the localization of the anomalies. We show that supervised learning outperforms unsupervised algorithms. For supervised cases, we achieve near-perfect accuracy of 98% (specifically for tree-based algorithms). In contrast, the best-case accuracy in the unsupervised cases was 63%—the area under the receiver operating characteristic curve (AUC) exhibits similar outcomes as an additional metric.

Marfo, William↗

PECAN2

PECAN2, Pose Classification with 3D Atomic Network 2, presents an innovative approach to pose classification in computational modeling, particularly in the context of molecular docking. Traditional methods like docking, which rely on physics-based calculations, are prone to inaccuracies in predicting binding poses during experimental testing. While machine learning (ML) approaches, such as ML-driven pose classification, have been introduced to address these issues, they often depend on the availability of crystal structure data. In contrast, PECAN2 introduces a novel pose classification method that operates independently of crystal structures. Instead, it utilizes a 3D atomic neural network with Point Cloud Network (PCN) to establish correlations between docking scores and experimental data. This approach marks a departure from previous studies that heavily relied on crystal structures for labeling. The key innovation lies in the ability of PECAN2 to enhance the performance of molecular docking by filtering out false positives and false negatives through the correlation between docking scores and experimental data.

Shim, Heesung↗

A Data-driven Approach for Predicting Nepheline Crystallization in High-Level Waste Glasses

High-level waste (HLW) glasses with high aluminum content are prone to nepheline crystallization during the slow canister cooling that is experienced during large-scale production. Due to its detrimental effects on glass durability, nepheline precipitation must be avoided; however, developing robust, predictive models for nepheline crystallization behavior in HLW glasses is difficult due to their compositional complexity. Using overly conservative constraints to predict nepheline formation can limit the waste loading to lower than the achievable capacity. In this study, a robust data-driven model using five compositional features has been developed to predict nepheline formation. The analysis of the model and the data show that there is an overlap, instead of a distinct compositional boundary, between glasses that form and do not form nepheline. As a result, the model’s predictive accuracy is not the same throughout the feature space and instead is dependent on the location of the glass composition in the dimensionally reduced feature space.

Sargin, Irmak↗

DETECTABILITY OF TRANSITING JUPITERS AND LOW-MASS ECLIPSING BINARIES IN SPARSELY SAMPLED PAN-STARRS-1 SURVEY DATA

We present detailed simulations of the Pan-STARRS-1 (PS1) multi-epoch, multiband 3pi Survey in order to assess its potential yield of transiting planets and eclipsing binaries. This survey differs from dedicated transit surveys in that it will cover the entire northern sky but provide only sparsely sampled light curves. Since most eclipses would be detected at only a single epoch, the 3pi Survey will be most sensitive to deep eclipses (approx>0.10 mag) caused by Jupiters transiting M dwarfs and eclipsing stellar/substellar binaries. The survey will measure parallaxes for the approx4 x 10{sup 5} stars within 100 pc, which will enable a volume-limited eclipse search, reducing the number of astrophysical false positives compared with previous magnitude-limited searches. Using the best available empirical data, we constructed a model of the extended solar neighborhood that includes stars, brown dwarfs, and a realistic binary population. We computed the yield of deeply eclipsing systems using both a semianalytic and a full Monte Carlo approach. We examined statistical tests for detecting single-epoch eclipses in sparsely sampled data and assessed their vulnerability to false positives due to stellar variability. Assuming a short-period planet frequency of 0.5% for M dwarfs, our simulations predict that about a dozen transiting Jupiters around low-mass stars (M {sub *} < 0.3 M {sub sun}) within 100 pc are potentially detectable in the PS1 3pi Survey, along with approx300 low-mass eclipsing binaries (both component masses <0.5 M {sub sun}), including approx10 eclipsing field brown dwarfs. Extensive follow-up observations would be required to characterize these candidate eclipsing systems, thereby enabling comprehensive tests of structural models and novel insights into the planetary architecture of low-mass stars.

79 ASTRONOMY AND ASTROPHYSICS↗

Enhancing searches for resonances with machine learning and moment decomposition

A key challenge in searches for resonant new physics is that classifiers trained to enhance potential signals must not induce localized structures. Such structures could result in a false signal when the background is estimated from data using sideband methods. A variety of techniques have been developed to construct classifiers which are independent from the resonant feature (often a mass). Such strategies are sufficient to avoid localized structures, but are not necessary. We develop a new set of tools using a novel moment loss function (Moment Decomposition or MoDe) which relax the assumption of independence without creating structures in the background. Here, by allowing classifiers to be more flexible, we enhance the sensitivity to new physics without compromising the fidelity of the background estimation.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Machine learning for photovoltaic single axis tracker fault detection and classification

More than 81% of the annual capacity of utility-scale photovoltaic (PV) power plants in the U.S. use single-axis trackers (SATs) due to SATs delivering 4% in capacity factor on average over fixed-array systems. However, SATs are subject to faults, such as software misconfigurations and mechanical failures, resulting in suboptimal tracking. If left undetected, the overall power yield of the PV power plant is reduced significantly. Minimizing downtime and ensuring efficient operation of SATs requires robust detection and diagnosis mechanisms for SAT faults. We present a machine learning framework for implementing real-time SAT fault detection and classification. Our implementation of the proposed framework reliably identifies measurements taken from a test PV system undergoing emulated SAT faults relative to state-of-the-art algorithms and produces nearly zero false positives on our testing days. Code and data are available at https://pvpmc.sandia.gov/tools.

Fault classification↗

Satellite Embedding-Based Population Imputation for Areas with Missing Building Footprint Data: A Computer Vision-Based Approach

High-resolution population modeling is important for supporting effective decision-making across diverse sectors. LandScan Mosaic generates population estimates at the level of individual buildings and aggregates them to 3 arc-second grids, and this approach performs well in regions where building footprint data are comprehensive and reliable. However, large portions of the globe still suffer from incomplete, sparse, or entirely missing building stock datasets, creating a structural limitation for strictly building-based population models. To address this research gap, this study proposes a computer vision-based framework that employs Google Earth Engine satellite embeddings and UNet, which allows us to directly impute grid-level population estimates in building-data-deficient areas. Applied to Taiwan as a case study, the framework achieved strong predictive performance with R$^{2}$ of 0.89, RMSE of 18.70, and MAE of 8.41, outperforming traditional machine learning approaches. Notably, the proposed framework effectively addressed building false-positive errors inherent in Global Human Settlement Layer (GHSL) data, correctly identifying uninhabited areas that were erroneously classified as populated. The framework also offers significant advantages for global population mapping, particularly in terms of scalability and temporal consistency, thereby extending the coverage and accuracy of high-resolution population products in data-scarce regions worldwide. Urban planners, decision makers, and related stakeholders can obtain granular population distributions to support more accurate and targeted infrastructure investment, service delivery, resource allocation, and risk assessment decisions.

97 MATHEMATICS AND COMPUTING↗

Reconstruction of false spring occurrences over the southeastern United States, 1901 2007: an increasing risk of spring freeze damage?

Near-record warmth over much of the United States during March 2007 promoted early growth of crops and vegetation. A widespread arctic air outbreak followed in early April, resulting in extensive agricultural losses over much of the south-central and southeastern US. This false spring event also resulted in widespread damage to newly grown tissues of native deciduous forest species, shown by previous researchers to have had measurable effects on the terrestrial carbon cycle. The current study reconstructed the historical occurrence of false springs over most of the southeastern quarter of the conterminous US (32 39 N; 75 98 W) from 1901 to 2007 using daily maximum and minimum temperature records from 176 stations in the Global Historical Climatology Network database, and enhanced vegetation index (EVI) data derived from Moderate Resolution Imaging Spectroradiometer (MODIS) satellite observations. A false spring index was derived that examined the timing of the start of the growing season (SGS), or leaf emergence, relative to the timing of a potentially damaging last hard freeze (minimum temperature 2.2 C). SGS was modeled for the domain by combining EVI data with ground-based temperature degree day calculations reflecting the rate of springtime warming. No significant area-wide, long-term SGS trend was found; however, over much of a contiguous region stretching from Mississippi eastward to the Carolinas, the timing of the last hard freeze was found to occur significantly later, this change occurring along with increased frequency of false springs. Earlier last hard freeze dates and decreased frequency of false springs were found over much of the northwestern part of the study region, including Arkansas and southern Missouri.

54 ENVIRONMENTAL SCIENCES↗

Adaptive error detection for HDR/PDR brachytherapy: Guidance for decision making during real-time in vivo point dosimetry

Purpose: This study presents an adaptive error detection algorithm (AEDA) for real-timein vivo point dosimetry during high dose rate (HDR) or pulsed dose rate (PDR) brachytherapy (BT) where the error identification, in contrast to existing approaches, does not depend on an a priori reconstruction of the dosimeter position. Instead, the treatment is judged based on dose rate comparisons between measurements and calculations of the most viable dosimeter position provided by the AEDA in a data driven approach. As a result, the AEDA compensates for false error cases related to systematic effects of the dosimeter position reconstruction. Given its nearly exclusive dependence on stable dosimeter positioning, the AEDA allows for a substantially simplified and time efficient real-time in vivo BT dosimetry implementation. Methods: In the event of a measured potential treatment error, the AEDA proposes the most viable dosimeter position out of alternatives to the original reconstruction by means of a data driven matching procedure between dose rate distributions. If measured dose rates do not differ significantly from the most viable alternative, the initial error indication may be attributed to a mispositioned or misreconstructed dosimeter (false error). However, if the error declaration persists, no viable dosimeter position can be found to explain the error, hence the discrepancy is more likely to originate from a misplaced or misreconstructed source applicator or from erroneously connected source guide tubes (true error). Results: The AEDA applied on twoin vivo dosimetry implementations for pulsed dose rate BT demonstrated that the AEDA correctly described effects responsible for initial error indications. The AEDA was able to correctly identify the major part of all permutations of simulated guide tube swap errors and simulated shifts of individual needles from the original reconstruction. Unidentified errors corresponded to scenarios where the dosimeter position was sufficiently symmetric with respect to error and no-error source position constellations. The AEDA was able to correctly identify all false errors represented by mispositioned dosimeters contrary to an error detection algorithm relying on the original reconstruction. Conclusions: The study demonstrates that the AEDA error identification during HDR/PDR BT relies on a stable dosimeter position rather than on an accurate dosimeter reconstruction, and the AEDA’s capacity to distinguish between true and false error scenarios. The study further shows that the AEDA can offer guidance in decision making in the event of potential errors detected with real-timein vivo point dosimetry.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Inter-Domain Fusion for Enhanced Intrusion Detection in Power Systems: An Evidence Theoretic and Meta-Heuristic Approach

False alerts due to misconfigured or compromised intrusion detection systems (IDS) in industrial control system (ICS) networks can lead to severe economic and operational damage. However, research using deep learning to reduce false alerts often requires the physical and cyber sensor data to be trustworthy. Implicit trust is a major problem for artificial intelligence or machine learning (AI/ML) in cyber-physical system (CPS) security, because when these solutions are most urgently needed is also when they are most at risk (e.g., during an attack). To address this, the Inter-Domain Evidence theoretic Approach for Inference (IDEA-I) is proposed that reframes the detection problem as how to make good decisions given uncertainty. Specifically, an evidence theoretic approach leveraging Dempster–Shafer (DS) combination rules and their variants is proposed for reducing false alerts. A multi-hypothesis mass function model is designed that leverages probability scores obtained from supervised-learning classifiers. Using this model, a location-cum-domain-based fusion framework is proposed to evaluate the detector’s performance using disjunctive, conjunctive, and cautious conjunctive rules. The approach is demonstrated in a cyber-physical power system testbed, and the classifiers are trained with datasets from Man-In-The-Middle attack emulation in a large-scale synthetic electric grid. For evaluating the performance, we consider plausibility, belief, pignistic, and general Bayesian theorem-based metrics as decision functions. To improve the performance, a multi-objective-based genetic algorithm is proposed for feature selection considering the decision metrics as the fitness function. Finally, we present a software application to evaluate the DS fusion approaches with different parameters and architectures.

42 ENGINEERING↗

MINIMUM REQUIREMENTS FOR DETECTING A STOCHASTIC GRAVITATIONAL WAVE BACKGROUND USING PULSARS

We assess the detectability of a nanohertz gravitational wave (GW) background in a pulsar timing array (PTA) program by considering the shape and amplitude of the cross-correlation function summed over pulsar pairs. The distribution of correlation amplitudes is found to be non-Gaussian and highly skewed, which significantly influences detection and false-alarm probabilities. When only white noise combines with GWs in timing data, our detection results are consistent with those found by others. Contamination by red noise from spin variations and from any uncorrected interstellar plasma effects significantly increases the false-alarm probability. The number of arrival times (and thus the observing cadence) is important only as long as the residuals are dominated by white noise. When red noise and GWs dominate, the statistical significance of the correlation estimate can be improved only by increasing the number of pulsars. We characterize plausible detection regimes by evaluating the number of millisecond pulsars (MSPs) that must be monitored in a high-cadence, five-year timing program to detect a GW background spectrum h{sub c} (f) = A(f/f{sub 0}){sup -2/3} with f{sub 0} = 1 yr{sup -1} and A = 10{sup -15}. Our results indicate that a sample of 20 super-stable MSPs-those with rms timing residuals {sigma}{sub r} {approx}< 20 ns(A/10{sup -15}) from red-noise contributions over a five-year span-will allow detection of the GW background and study of its spectrum. However, a timing program on {approx}> 50-100 MSPs is likely needed for a complete PTA program, particularly if red noise is generally present in MSPs.

79 ASTRONOMY AND ASTROPHYSICS↗

Adaptive Impact-Driven Detection of Silent Data Corruption for HPC Applications

For exascale HPC applications, silent data corruption (SDC) is one of the most dangerous problems because there is no indication that there are errors during the execution. We propose an adaptive impact-driven method that can detect SDCs dynamically. The key contributions are threefold. (1) We carefully characterize 18 real-world HPC applications and discuss the runtime data features, as well as the impact of the SDCs on their execution results. (2) We propose an impact-driven detection model that does not blindly improve the prediction accuracy, but instead detects only influential SDCs to guarantee user-acceptable execution results. (3) Our solution can adapt to dynamic prediction errors based on local runtime data and can automatically tune detection ranges for guaranteeing low false alarms. Experiments show that our detector can detect 80-99.99% of SDCs with a false alarm rate less that 1% of iterations for most cases. The memory cost and detection overhead are reduced to 15% and 6.3%, respectively, for a large majority of applications.

Exascale HPC↗

Verification of the Effectiveness of X-ray Machine Collimation

The Oak Ridge National Laboratory (ORNL) was tasked by the United States National Nuclear Security Administration’s Second Line of Defense (SLD) program with reducing the false alarm rate experienced by radiation portal monitors (RPM) installed near x-ray machines. Although the mechanism for interference between RPMs and x-ray machines is well understood, x-ray machines are often installed too near RPMs due to a variety of facility constraints. The result is underutilized RPMs that are plagued by a high false alarm rate from x-rays that escape the x-ray machine and are detected by the RPM. In this work, the effectiveness of an x-ray machine collimation system designed at ORNL is validated through the analysis of field data collected before and after the installation. The significant reduction in false alarm rate due to the collimation system is explicitly shown, and examples of field installations are provided. Utilization of the results presented here allows RPM systems to operate effectively even when installed near x-ray machines.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Using support vector machines for anomalous change detonation

We cast anomalous change detection as a binary classification problem, and use a support vector machine (SVM) to build a detector that does not depend on assumptions about the underlying data distribution. To speed up the computation, our SVM is implemented, in part, on a graphical processing unit. Results on real and simulated anomalous changes are used to compare performance to algorithms which effectively assume a Gaussian distribution. In this paper, we investigate the use of support vector machines (SVMs) with radial basis kernels for finding anomalous changes. Compared to typical applications of SVMs, we are operating in a regime of very low false alarm rate. This means that even for relatively large training sets, the data are quite meager in the regime of operational interest. This drives us to use larger training sets, which in turn places more of a computational burden on the SVM. We initially considered three different approaches to to address the need to work in the very low false alarm rate regime. The first is a standard SVM which is trained at one threshold (where more reliable estimates of false alarm rates are possible) and then re-thresholded for the low false alarm rate regime. The second uses the same thresholding approach, but employs a so-called least squares SVM; here a quadratic (instead of a hinge-based) loss function is employed, and for this model, there are good theoretical arguments in favor of adjusting the threshold in a straightforward manner. The third approach employs a weighted support vector machine, where the weights for the two types of errors (false alarm and missed detection) are automatically adjusted to achieve the desired false alarm rate. We have found in previous experiments (not shown here) that the first two types can in some cases work well, while in other cases they do not. This renders both approaches unreliable for automated change detection. By contrast, the third approach reliably produces good results, but at the cost of larger computational requirements caused by the need to estimate very small false alarm rates. To address these computational requirements, we employ a recently developed in-house solver for SVMs that is significantly faster than freely available standard solvers. But these computational issues are secondary to the larger question: do kernelized solutions provide better performance, in terms of detection rates and false alarm rates, than more traditional methods for change detection that effectively assume Gaussian data distributions? To this end, we will compare ROC curves obtained from the SVM with those from chronochrome, covariance equalization, and hyperbolic anomalous change detection.

97 MATHEMATICS AND COMPUTING↗

A bi-level data-driven framework for fault-detection and diagnosis of HVAC systems

Long-term operation of heating, ventilation, and air conditioning (HVAC) systems will eventually lead to a range of HVAC system failures, resulting in excessive energy consumption and maintenance costs. Here, to avoid HVAC malfunctioning, fault detection diagnostic (FDD) is utilized as a common practice. Machine learning methods have lately received considerable interest for FDD analysis of HVAC systems due to their high detection accuracy. Meanwhile, HVAC malfunctions are regarded as rare occurrences, hence normal operating data samples are much more accessible than data samples in faulty and malfunctioning conditions. The dominating frequency of normal operation in HVAC datasets has also led to heavily biased classification algorithms within the literature. Moreover, the focus of previous literature has been on increasing the accuracy of the models which leads to a high number of false positives (misleading alarms) in the system. In order to enhance the performance of diagnostic procedures and fill the mentioned gaps, this study proposes a novel data-driven framework. A bi-level machine learning framework is developed for diagnosing faults in air handling units (AHUs) and rooftop units (RTUs) based on principal component analysis (PCA), time series anomaly detection, and random forest (RF). It is shown that PCA can reduce the dataset dimension with one principal component accounting for 95% of data variance. Also, the random forest could classify the faults with 89% precision for single-zone AHU, 85% precision for RTU, and 79% for multi-zone AHU. By proposing this framework, three persistent challenges are addressed: (I) minimizing false positives; (II) accounting for data imbalance; and (III) normal condition monitoring of equipment.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

GOTTCHA Database, Version 1

One major challenge in the field of shotgun metagenomics is the accurate identification of the organisms present within the community, based on classification of short sequence reads. Though microbial community profiling methods have emerged to attempt to rapidly classify the millions of reads output from contemporary sequencers, the combination of incomplete databases, similarity among otherwise divergent genomes, and the large volumes of sequencing data required for metagenome sequencing has led to unacceptably high false discovery rates (FDR). Here we present the application of a novel, gene-independent and signature-based metagenomic taxonomic profiling tool with significantly smaller FDR, which is also capable of classifying never-before seen genomes into the appropriate parent taxa.The algorithm is based upon three primary computational phases: (I) genomic decomposition into bit vectors, (II) bit vector intersections to identify shared regions, and (III) bit vector subtractions to remove shared regions and reveal unique, signature regions.In the Decomposition phase, genomic data is first masked to highlight only the valid (non-ambiguous) regions and then decomposed into overlapping 24-mers. The k-mers are sorted along with their start positions, de-replicated, and then prefixed, to minimize data duplication. The prefixes are indexed and an identical data structure is created for the start positions to mimic that of the k-mer data structure.During the Intersection phase -- which is the most computationally intensive phase -- as an all-vs-all comparison is made, the number of comparisons is first reduced by four methods: (a) Prefix restriction, (b) Overlap detection, (c) Overlap restriction, and (d) Result recording. In Prefix restriction, only k-mers of the same prefix are compared. Within that group, potential overlap of k-mer suffixes that would result in a non-empty set intersection are screened for. If such an overlap exists, the region which intersects is first reduced by performing a binary search of the boundary suffixes of the smaller set into the larger set, which defines the limits of the zipper-based intersection process. Rather than recording the actual k-mers of the intersection, another data structure of identical "shape" is created which consists of only bit vectors so that only a 1 or 0 will be stored in the location of the k-mer suffix that was found in the intersection. This reduces the amount of data generated and stored considerably.During the Subtraction phase, relevant intersection bitmasks are first unionized together to form a single bitmask which is then applied over the original genome to reveal only those regions of the genome that are unique. These regions are then exported to disk in FASTA format and used in the application of determining the constituents of an unknown metagenomic community.The DATABASE provided is the result of the algorithm described.

Freitas, Tracey↗

GOTTCHA Database, Version 1

One major challenge in the field of shotgun metagenomics is the accurate identification of the organisms present within the community, based on classification of short sequence reads. Though microbial community profiling methods have emerged to attempt to rapidly classify the millions of reads output from contemporary sequencers, the combination of incomplete databases, similarity among otherwise divergent genomes, and the large volumes of sequencing data required for metagenome sequencing has led to unacceptably high false discovery rates (FDR). Here we present the application of a novel, gene-independent and signature-based metagenomic taxonomic profiling tool with significantly smaller FDR, which is also capable of classifying never-before seen genomes into the appropriate parent taxa.The algorithm is based upon three primary computational phases: (I) genomic decomposition into bit vectors, (II) bit vector intersections to identify shared regions, and (III) bit vector subtractions to remove shared regions and reveal unique, signature regions.In the Decomposition phase, genomic data is first masked to highlight only the valid (non-ambiguous) regions and then decomposed into overlapping 24-mers. The k-mers are sorted along with their start positions, de-replicated, and then prefixed, to minimize data duplication. The prefixes are indexed and an identical data structure is created for the start positions to mimic that of the k-mer data structure.During the Intersection phase -- which is the most computationally intensive phase -- as an all-vs-all comparison is made, the number of comparisons is first reduced by four methods: (a) Prefix restriction, (b) Overlap detection, (c) Overlap restriction, and (d) Result recording. In Prefix restriction, only k-mers of the same prefix are compared. Within that group, potential overlap of k-mer suffixes that would result in a non-empty set intersection are screened for. If such an overlap exists, the region which intersects is first reduced by performing a binary search of the boundary suffixes of the smaller set into the larger set, which defines the limits of the zipper-based intersection process. Rather than recording the actual k-mers of the intersection, another data structure of identical "shape" is created which consists of only bit vectors so that only a 1 or 0 will be stored in the location of the k-mer suffix that was found in the intersection. This reduces the amount of data generated and stored considerably.During the Subtraction phase, relevant intersection bitmasks are first unionized together to form a single bitmask which is then applied over the original genome to reveal only those regions of the genome that are unique. These regions are then exported to disk in FASTA format and used in the application of determining the constituents of an unknown metagenomic community.The DATABASE provided is the result of the algorithm described.

Freitas, Tracey↗