Search NASA⌕ Search

SEARCH · Search NASA

Results for “Clustering algorithms”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 721 records · Page 40
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

The Atacama Cosmology Telescope: arcminute-resolution maps of 18 000 square degrees of the microwave sky from ACT 2008–2018 data combined with Planck

This paper presents a maximum-likelihood algorithm for combining sky maps with disparate sky coverage, angular resolution and spatially varying anisotropic noise into a single map of the sky. We use this to merge hundreds of individual maps covering the 2008–2018 ACT observing seasons, resulting in by far the deepest ACT maps released so far. We also combine the maps with the full Planck maps, resulting in maps that have the best features of both Planck and ACT: Planck’s nearly white noise on intermediate and large angular scales and ACT’s high-resolution and sensitivity on small angular scales. The maps cover over 18 000 square degrees, nearly half the full sky, at 100, 150 and 220 GHz. Furthermore, they reveal 4 000 optically-confirmed clusters through the Sunyaev Zel’dovich effect (SZ) and 18 500 point source candidates at > 5σ, the largest single collection of SZ clusters and millimeter wave sources to date. The multi-frequency maps provide millimeter images of nearby galaxies and individual Milky Way nebulae, and even clear detections of several nearby stars. Other anticipated uses of these maps include, for example, thermal SZ and kinematic SZ cluster stacking, CMB cluster lensing and galactic dust science. The method itself has negligible bias. However, due to the preliminary nature of some of the component data sets, we caution that these maps should not be used for precision cosmological analysis. The maps are part of ACT DR5, and will be made available on LAMBDA no later than three months after the journal publication of this article, along with an interactive sky atlas.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Smart Pixels: In-pixel AI for on-sensor data filtering

We present a smart pixel prototype readout integrated circuit (ROIC) designed in CMOS 28 nm bulk process, with in-pixel implementation of an artificial intelligence (AI) / machine learning (ML) based data filtering algorithm designed as proof-of-principle for a Phase III upgrade at the Large Hadron Collider (LHC) pixel detector. The first version of the ROIC consists of two matrices of 256 smart pixels, each 25$\times$25 µm\textsuperscript{2} in size. Each pixel consists of a charge-sensitive preamplifier with leakage current compensation and three auto-zero comparators for a 2-bit flash-type ADC. The frontend is capable of synchronously digitizing the sensor charge within 25 ns. Measurement results show an equivalent noise charge (ENC) of $\sim$30e\textsuperscript{-} and a total dispersion of $\sim$100e\textsuperscript{-} The second version of the ROIC uses a fully connected two-layer neural network (NN) to process information from a cluster of 256 pixels to determine if the pattern corresponds to highly desirable high-momentum particle tracks for selection and readout. The digital NN is embedded in-between analog signal processing regions of the 256 pixels without increasing the pixel size and is implemented as fully combinatorial digital logic to minimize power consumption and eliminate clock distribution, and is active only in the presence of an input signal. The total power consumption of the neural network is $\sim$ 300 $\mu$W. The NN performs momentum classification based on the generated cluster patterns and even with a modest momentum threshold, it is capable of 54.4\% – 75.4\% total data rejection, opening the possibility of using the pixel information at 40MHz for the trigger. The total power consumption of analog and digital functions per pixel is $\sim$ 6 $\mu$W per pixel, which corresponds to $\sim$ 1 W/cm\textsuperscript{2} staying within the experimental constraints.

Parpillon, Benjamin↗
DOE OSTI2022

Machine learning on FPGA for event selection

Real-time data processing is a frontier field in experimental particle physics. The application of FPGAs at the trigger level is used by many current and planned experiments (CMS, LHCb, Belle2, PANDA). Usually they use conventional processing algorithms. LHCb has implemented Machine Learning (ML) elements for real-time data processing with a triggered readout system that runs most of the ML algorithms on a computer farm. The work described in this article aims to test the ML-FPGA algorithms for streaming data acquisition. Herein, there are many experiments working in this area and they have a lot in common, but there are many specific solutions for detector and accelerator parameters that are worth exploring further. This report describes the purpose of the work and progress in evaluating the ML-FPGA application.

47 OTHER INSTRUMENTATION↗
DOE OSTI2023

Distributed out-of-memory NMF on CPU/GPU architectures

We propose an efficient distributed out-of-memory implementation of the non-negative matrix factorization (NMF) algorithm for heterogeneous high-performance-computing systems. The proposed implementation is based on prior work on NMFk, which can perform automatic model selection and extract latent variables and patterns from data. In this work, we extend NMFk by adding support for dense and sparse matrix operation on multi-node, multi-GPU systems. The resulting algorithm is optimized for out-of-memory problems where the memory required to factorize a given matrix is greater than the available GPU memory. Memory complexity is reduced by batching/tiling strategies, and sparse and dense matrix operations are significantly accelerated with GPU cores (or tensor cores when available). Input/output latency associated with batch copies between host and device is hidden using CUDA streams to overlap data transfers and compute asynchronously, and latency associated with collective communications (both intra-node and inter-node) is reduced using optimized NVIDIA Collective Communication Library (NCCL) based communicators. Benchmark results show significant improvement, from 32X to 76x speedup, with the new implementation using GPUs over the CPU-based NMFk. Good weak scaling was demonstrated on up to 4096 multi-GPU cluster nodes with approximately 25,000 GPUs when decomposing a dense 340 Terabyte-size matrix and an 11 Exabyte-size sparse matrix of density 10 -6 .

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Combining shallow-water and analytical wake models for tidal array micro-siting

For tidal-stream energy to become a competitive renewable energy source, clustering multiple turbines into arrays is paramount. Array optimisation is thus critical for achieving maximum power performance and reducing cost of energy. However, ascertaining an optimal array layout is a complex problem, subject to specific site hydrodynamics and multiple inter-disciplinary constraints. In this work, we present a novel optimisation approach that combines an analytical-based wake model, FLORIS, with an ocean model, Thetis. The approach is demonstrated through applications of increasing complexity. By utilising the method of analytical wake superposition, the addition or alteration of turbine position does not require re-calculation of the entire flow field, thus allowing the use of simple heuristic techniques to perform optimisation at a fraction of the computational cost of more sophisticated methods. Using a custom condition-based placement algorithm, this methodology is applied to the Pentland Firth for arrays with turbines of 3.05 m/s rated speed, demonstrating practical implications whilst considering the temporal variability of the tide. For a 24-turbine array case, micro-siting using this technique delivered an array 15.8% more productive on average than a staggered layout, despite flow speeds regularly exceeding the rated value. Performance was evaluated through assessment of the optimised layout within the ocean model that treats turbines through a discrete turbine representation. Used iteratively, this methodology could deliver improved array configurations in a manner that accounts for local hydrodynamic effects.

16 TIDAL AND WAVE POWER↗
DOE OSTI2020

Scalable Heterogeneous Execution of a Coupled-Cluster Model with Perturbative Triples

The CCSD(T) coupled-cluster model with perturbative triples is considered a gold standard for computational modeling of the correlated behavior of electrons in molecular systems. A fundamental constraint is the relatively small global-memory capacity in GPUs compared to the main-memory capacity on host nodes, necessitating relatively smaller tile sizes for high-dimensional tensor contractions in NWChem's GPU-accelerated implementation of the CCSD(T) method. A coordinated redesign is described to address this limitation and associated data movement overheads, including a novel fused GPU kernel for a set of tensor contractions, along with inter-node communication optimization and data caching. The new implementation of GPU-accelerated CCSD(T) improves overall performance by 3.4x. Finally, we discuss the trade-offs in using this fused algorithm on current and future supercomputing platforms.

Kim, Jinsung↗
DOE OSTI2025

Geometric Interpretation of the Cluster Location Problem Part II: Application to the Pahala, Hawaii, Earthquake Sequence

In the companion “Theory” article, we presented a new framing of the seismic location problem in terms of differential geometry (Harris et al., 2025). From that viewpoint, we developed a “project and correct” approach for estimating the relative locations of earthquakes. Here, in this study, we use project and correct to estimate high-precision relative locations of events from an earthquake sequence beneath the town of Pahala, Hawaii, using high-precision correlation-derived picks. The sequence was active from 2020 through 2022 and produced many highly correlated signals at Hawaii Volcano Observatory (HVO) stations on the island of Hawaii. The data we inverted consisted of 2882 events with observations at 5 HVO stations. For comparison with the travel-time image, we also produced conventional hypocenter solutions using both the Bayesloc program (Myers et al., 2007, 2009) and a purpose-built double-difference code. There were obvious structural elements in the resulting image, the resolution of which we used to test the performance of the project and the correct algorithm. For the projection step, we first produced a 3D local basis using an singular value decomposition (SVD) of the 2882 groups of times. Projection of the travel-time vectors into this basis resulted in an image with structures similar to those produced by our conventional locators, but with distortion as predicted by theory. Removing the distortion requires an inverse operator generated from the metric tensor at the geometric centroid of the events. We compared two approaches to obtaining such an inverse operator. The first uses an estimate of the geographic centroid of the event cloud from the centroid of the travel-time data. The second approach uses the centroid of the conventionally produced locations. The first approach produces a corrected image very similar to the conventional results, but with a rotation. The corrected image produced using the conventionally derived centroid is a near-exact match to the conventional locations.

Dodge, Douglas A. [Lawrence Livermore National Lab↗
DOE OSTI2021

Noise-induced barren plateaus in variational quantum algorithms

Abstract Variational Quantum Algorithms (VQAs) may be a path to quantum advantage on Noisy Intermediate-Scale Quantum (NISQ) computers. A natural question is whether noise on NISQ devices places fundamental limitations on VQA performance. We rigorously prove a serious limitation for noisy VQAs, in that the noise causes the training landscape to have a barren plateau (i.e., vanishing gradient). Specifically, for the local Pauli noise considered, we prove that the gradient vanishes exponentially in the number of qubits n if the depth of the ansatz grows linearly with n . These noise-induced barren plateaus (NIBPs) are conceptually different from noise-free barren plateaus, which are linked to random parameter initialization. Our result is formulated for a generic ansatz that includes as special cases the Quantum Alternating Operator Ansatz and the Unitary Coupled Cluster Ansatz, among others. For the former, our numerical heuristics demonstrate the NIBP phenomenon for a realistic hardware noise model.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2022

Semi-Supervised Domain Adaptation for Cross-Survey Galaxy Morphology Classification and Anomaly Detection

In the era of big astronomical surveys, our ability to leverage artificial intelligence algorithms simultaneously for multiple datasets will open new avenues for scientific discovery. Unfortunately, simply training a deep neural network on images from one data domain often leads to very poor performance on any other dataset. Here we develop a Universal Domain Adaptation method DeepAstroUDA, capable of performing semi-supervised domain alignment that can be applied to datasets with different types of class overlap. Extra classes can be present in any of the two datasets, and the method can even be used in the presence of unknown classes. For the first time, we demonstrate the successful use of domain adaptation on two very different observational datasets (from SDSS and DECaLS). We show that our method is capable of bridging the gap between two astronomical surveys, and also performs well for anomaly detection and clustering of unknown data in the unlabeled dataset. We apply our model to two examples of galaxy morphology classification tasks with anomaly detection: 1) classifying spiral and elliptical galaxies with detection of merging galaxies (three classes including one unknown anomaly class); 2) a more granular problem where the classes describe more detailed morphological properties of galaxies, with the detection of gravitational lenses (ten classes including one unknown anomaly class).

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2020

Robust Group Subspace Recovery: A New Approach for Multi-Modality Data Fusion

Robust Subspace Recovery (RoSuRe) algorithm was recently introduced as a principled and numerically efficient algorithm that unfolds underlying Unions of Subspaces (UoS) structure, present in the data. The union of Subspaces (UoS) is capable of identifying more complex trends in data sets than simple linear models. In this work, we build on and extend RoSuRe to prospect the structure of different data modalities individually. We propose a novel multi-modal data fusion approach based on group sparsity which we refer to as Robust Group Subspace Recovery (RoGSuRe). Relying on a bi-sparsity pursuit paradigm and non-smooth optimization techniques, the introduced framework learns a new joint representation of the time series from different data modalities, respecting an underlying UoS model. We subsequently integrate the obtained structures to form a unified subspace structure. The proposed approach exploits the structural dependencies between the different modalities data to cluster the associated target objects. The resulting fusion of the unlabeled sensors’ data from experiments on audio and magnetic data has shown that our method is competitive with other state of the art subspace clustering methods. The resulting UoS structure is employed to classify newly observed data points, highlighting the abstraction capacity of the proposed method.

47 OTHER INSTRUMENTATION↗
DOE OSTI2024

Wholesale Electricity Analysis via Simulation & Learning Experiments (WEASLE): Platform Development and Pilot Competition

This document reports the development of the Wholesale Electricity Analysis via Simulation and Learning Experiments (WEASLE) platform and the pilot competition that was conducted to test the platform. Due to the increasing reliance on variable renewable energy resources for bulk power, the pilot competition, called the Energy Storage Participation Algorithm Competition (ESPA-Comp), was used to test the effect of various market designs on storage utilization and market efficiency. Basic details of the platform are provided, including an overview of the market clearing engine, the battery dispatch and degradation models, electric grid topology and resource mix, and software architecture. Two market designs were tested: a two-settlement market analogous to typical ISO design today, and a multi-settlement market that allows additional forward-trading periods during the real-time market. Results from the pilot competition show that the storage bidding problem is nontrivial and is well suited for future challenges. We find that: 1) all four teams utilized different approaches to the bidding problem, 2) different methodological approaches led to substantially different offer behaviors, 3) resource profits are clustered by team and methodological approach, 4) simulated offers reduced market surplus by about 0.5%, 5) substantially different prices between two-settlement and multi-settlement markets albeit minimal difference in overall market surplus.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2023

CEGANN: CRYSTAL EDGE GRAPH ATTENTION NEURAL NETWORK

SF-22-156 Machine learning (ML) models and applications in materials design and discovery typically involve the use of feature representations or descriptors followed by a learning algorithm that maps them to user desired properties of interest. Most popular mathematical formulation-based descriptors are not unique across atomic environments and suffer from transferability issues across different application domains and/or material classes. The CEGANN code provides a unified interface to facilitate material characterization across materials across multiple scales (from atomic to mesoscale) and diverse classes of materials ranging from metals oxides, non-metals, and even hierarchical materials such as zeolites and semi ordered materials such as mesophases. CEGANN implements a Graph Attention Network (GAT) type convolution architecture. The details of network architecture can be found in the paper https://doi.org/10.48550/arXiv.2207.10168. The software comes with pretrained examples and dataset for the classification of the following representative systems: (1) Structure-level representation such as space group (2) Structural dimensionality (e.g., bulk, 2D, clusters etc.) (3) Grain boundary identification (4) Nucleation and growth of a zeolite polymorph (5) Characterization of binary mesophases and their phase transitions (6) Growth of ice. The code is written in python programming language.

CHAN, HENRYT↗
DOE OSTI2024

Plasma Surface Interactions: Predicting the Performance and Impact of Dynamic PFC Surfaces

The objective of this project is to develop, and integrate, high-performance simulation tools capable of predicting plasma-facing component (PFC) operating lifetime and the impact of the evolving surface morphology of tungsten-based PFCs on plasma contamination, including the dynamic recycling of fuel species and tritium retention, in future magnetic fusion devices. Establishing a fundamental physical understanding and developing predictive capabilities of plasma-surface interactions (PSI) requires simultaneously addressing complex and diverse physics occurring over a wide range of length (Angstroms to meters) and time (femtoseconds to years) scales, as well as integrating extensive physical processes across the plasma–surface interface. This requires development of not only detailed physics models and computational strategies at each scale, but also algorithms and methods to couple them effectively in a way that can be robustly validated. Deploying these tools requires the continued development and coupling of leadership-scale computational codes to describe the boundary plasma and the evolving PFC surface, as well as a host of simulations that bridge disparate scales to address complex physical and computational issues at the plasma–surface interface in multi-component materials systems for magnetic fusion energy development beyond ITER.

36 MATERIALS SCIENCE↗
DOE OSTI2024

FORESTR: Finding, Organizing, Representing, Explaining, Summarizing, and Thinning Random forests

Random forests have become popular models used for data driven predictions. As a result, random forests are currently used or being considered for high-consequence mission applications in national security, such as the prediction of yield from optical signals and malware detection. While random forests may provide accurate predictions, the complexity of the algorithm causes a lack of interpretability. Random forests are an ensemble of regression or decision trees. Individual regression and decision trees are interpretable, but ensembles are inherently difficult to interpret due to the compilation of many models. We aim to increase the interpretability of random forests by finding patterns in the ensemble of trees that can be used to “thin” (or remove) trees. As a starting point, in this report, we develop a new distance metric for quantifying the similarity between trees based on their topologies (i.e., shapes). We base the metric on a novel distance metric for graphs that is a proper mathematical distance, is invariant to transformations, has registration between graphs, and computes topological evolutions between graphs. We use the tree distance metric to compute tree statistics such as a “mean tree” and to identify clusters of trees. We apply the developed methodology to a toy dataset and a mission relevant product inspection dataset to demonstrate how the metric can provide insight into random forests. Furthermore, we discuss the limitations of the approach and ideas for future research into how the metric could be used as a thinning tool to develop less complex models.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Unified architecture for data-driven metadata tagging of building automation systems

This article presents a Unified Architecture (UA) for automated point tagging of Building Automation System (BAS) data, based on a combination of data-driven approaches. Advanced energy analytics applications—including fault detection and diagnostics and supervisory control—have emerged as a significant opportunity for improving the performance of our built environment. Effective application of these analytics depends on harnessing structured data from the various building control and monitoring systems, but typical BAS implementations do not employ any standardized metadata schema. While standards such as Project Haystack and Brick Schema have been developed to address this issue, the process of structuring the data, i.e., tagging the points to apply a standard metadata schema, has, to date, been a manual process. This process is typically costly, labor-intensive, and error-prone. In this work we address this gap by proposing a UA that automates the process of point tagging by leveraging the data accessible through connection to the BAS, including time-series data and the raw point names. The UA intertwines supervised classification and unsupervised clustering techniques from machine learning and leverages both their deterministic and probabilistic outputs to inform the point tagging process. Furthermore, we extend the UA to embed additional input and output data-processing modules that are designed to address the challenges associated with the real-time deployment of this automation solution. We test the UA on two datasets for real-life buildings: (i) commercial retail buildings and (ii) office buildings from the National Renewable Energy Laboratory (NREL) campus. We report the proposed methodology correctly applied 85–90% and 70–75% of the tags in each of these test scenarios, respectively for two significantly different building types used for testing UA's fully-functional prototype. The proposed UA, therefore, offers promising approach for automatically tagging BAS data as it reaches close to 90% accuracy. Further building upon this framework to algorithmically identify the equipment type and their relationships is an apt future research direction to pursue.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2020

Propagating sample variance uncertainties in redshift calibration: simulations, theory, and application to the COSMOS2015 data

ABSTRACT Cosmological analyses of galaxy surveys rely on knowledge of the redshift distribution of their galaxy sample. This is usually derived from a spectroscopic and/or many-band photometric calibrator survey of a small patch of sky. The uncertainties in the redshift distribution of the calibrator sample include a contribution from shot noise, or Poisson sampling errors, but, given the small volume they probe, they are dominated by sample variance introduced by large-scale structures. Redshift uncertainties have been shown to constitute one of the leading contributions to systematic uncertainties in cosmological inferences from weak lensing and galaxy clustering, and hence they must be propagated through the analyses. In this work, we study the effects of sample variance on small-area redshift surveys, from theory to simulations to the COSMOS2015 data set. We present a three-step Dirichlet method of resampling a given survey-based redshift calibration distribution to enable the propagation of both shot noise and sample variance uncertainties. The method can accommodate different levels of prior confidence on different redshift sources. This method can be applied to any calibration sample with known redshifts and phenotypes (i.e. cells in a self-organizing map, or some other way of discretizing photometric space), and provides a simple way of propagating prior redshift uncertainties into cosmological analyses. As a worked example, we apply the full scheme to the COSMOS2015 data set, for which we also present a new, principled SOM algorithm designed to handle noisy photometric data. We make available a catalogue of the resulting resamplings of the COSMOS2015 galaxies.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Capturing many-body correlation effects with quantum and classical computing

Theoretical descriptions of excited states of molecular systems in high-energy regimes are crucial for supporting and driving many experimental efforts at light source facilities. However, capturing their complicated correlation effects requires formalisms that provide a hierarchical infrastructure of approximations. These approximations lead to an increased overhead in classical computing methods and, therefore, decisions regarding the ranking of approximations and the quality of results must be made on purely numerical grounds. The emergence of quantum computing methods has the potential to change this situation. Here, in this study, we demonstrate the efficiency of the quantum phase estimator (QPE) in identifying core-level states relevant to x-ray photoelectron spectroscopy. We compare and validate the QPE predictions with exact diagonalization and real-time equation-of-motion coupled-cluster formulations, which are some of the most accurate methods for states dominated by collective correlation effects.

74 ATOMIC AND MOLECULAR PHYSICS↗
DOE OSTI2023

Conserved unique peptide patterns (CUPP) online platform 2.0: implementation of +1000 JGI fungal genomes

Carbohydrate-processing enzymes, CAZymes, are classified into families based on sequence and three-dimensional fold. Because many CAZyme families contain members of diverse molecular function (different EC-numbers), sophisticated tools are required to further delineate these enzymes. Such delineation is provided by the peptide-based clustering method CUPP, Conserved Unique Peptide Patterns. CUPP operates synergistically with the CAZy family/subfamily categorizations to allow systematic exploration of CAZymes by defining small protein groups with shared sequence motifs. The updated CUPP library contains 21,930 of such motif groups including 3,842,628 proteins. The new implementation of the CUPP-webserver, https://cupp.info/, now includes all published fungal and algal genomes from the Joint Genome Institute (JGI), genome resources MycoCosm and PhycoCosm, dynamically subdivided into motif groups of CAZymes. This allows users to browse the JGI portals for specific predicted functions or specific protein families from genome sequences. Thus, a genome can be searched for proteins having specific characteristics. All JGI proteins have a hyperlink to a summary page which links to the predicted gene splicing including which regions have RNA support. The new CUPP implementation also includes an update of the annotation algorithm that uses only a fourth of the RAM while enabling multi-threading, providing an annotation speed below 1 ms/protein.

59 BASIC BIOLOGICAL SCIENCES↗