Search NASA⌕ Search

SEARCH · Search NASA

Results for “HDBSCAN”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

17 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

PANDORA: A Parallel Dendrogram Construction Algorithm for Single Linkage Clustering on GPU

This paper introduces Pandora, a parallel algorithm for computing dendrograms, the hierarchical cluster trees for single linkage clustering (SLC). Current parallel approaches construct dendrograms by partitioning a minimum spanning tree and removing edges. However, they struggle with skewed, hard-to-parallelize real-world dendrograms. Consequently, computing dendrograms is the sequential bottleneck in HDBSCAN*[21], a popular SLC variant. Pandora uses recursive tree contraction to address this limitation. Pandora contracts nodes to construct progressively smaller trees. It computes the smallest contracted dendrogram and expands it by inserting contracted edges. This recursive strategy is highly parallel, skew-independent, work-optimal, and well-suited for GPUs and multicores. We develop a performance portable implementation of Pandora in Kokkos[31] and evaluate its performance on multicore CPUs and multi-vendor GPUs (e.g., Nvidia, AMD) for dendrogram construction in HDBSCAN*. Multithreaded Pandora is 2.2x faster than the current best-multithreaded implementation. Our GPU version achieves 6-20x speedup on AMD GPUs and 10-37x on NVIDIA GPUs over multithreaded Pandora. Pandora removes HDBSCAN*’s sequential bottleneck, greatly boosting efficiency, particularly with GPUs.

Sao, Piyush↗
NASA NTRSDate not supplied

Visualizing Corridors in Terminal Airspace using Trajectory Clustering

Context: Advances in battery and automation technology have made routine air taxi and cargo transport in urban areas a business model that can be attained by emerging aviation innovators. The community vision and work to enable these novel operations is discussed using the term ‘Urban Air Mobility’ or UAM. Small, piloted, airspace vehicles that fly with a few passengers do operate in urban areas today, and these vehicles can be studied as an early proxy for this future UAM traffic. Aim: We seek to identify corridors already in daily operation and their properties. Method: We applied DBSCAN and HDBSCAN to Dallas Forth-Worth TRACON flight data to identify corridors in use, their density, and devised a method to annotate landing sites used in these corridors with site metadata. Results: While DBSCAN was unable to group similar trajectories, we we were able to successfully identify corridors using HDBSCAN, measure their density and annotate them. Conclusion: The applied method can successfully identify corridors in daily operation with additional metadata to help domain expert understand the intent of UAM corridors.

UAM, Trajectory, TRACON, Clustering, DBSCAN, HDBSC↗
NASA NTRSDate not supplied

Visualizing Corridors in Terminal Airspace Using Trajectory Clustering

Context: Advances in battery and automation technology have made routine air taxi and cargo transport in urban areas a business model that can be attained by emerging aviation innovators. The community vision and work to enable these novel operations is discussed using the term ‘Urban Air Mobility’ or UAM. Small, piloted, airspace vehicles that fly with a few passengers do operate in urban areas today, and these vehicles can be studied as an early proxy for this future UAM traffic. Aim: We seek to identify corridors already in daily operation and their properties. Method: We applied DBSCAN and HDBSCAN to Dallas Forth-Worth TRACON flight data to identify corridors in use, their density, and devised a method to annotate landing sites used in these corridors with site metadata. Results: While DBSCAN was unable to group similar trajectories, we we were able to successfully identify corridors using HDBSCAN, measure their density and annotate them. Conclusion: The applied method can successfully identify corridors in daily operation with additional metadata to help domain expert understand the intent of UAM corridors.

UAM Trajectory, TRACON, Clustering, DBSCAN, HDBSCA↗
DOE OSTI2025

Enhancing Cluster Identification in Atom Probe Tomography Data Using Transfer Learning

Atom Probe Tomography (APT) is a powerful technique for visualizing the atomic-scale distribution of solutes in materials, but quantitative cluster analysis of APT datasets remains a challenge due to the need for subjective parameter selection in clustering algorithms. While distance-based and density-based methods such as HDBSCAN are widely used, their performance is highly sensitive to user-defined parameters, which undermines reproducibility and accuracy. This study proposes an image-based, deep learning-aided workflow for automating parameter selection and cluster detection in APT data analysis. By projecting 3D APT point clouds onto 2D planes, we leverage pretrained convolutional neural networks (ConvNeXt-Tiny and ResNet-50) through transfer learning to predict the number of clusters present in synthetic datasets. The output is used to guide K-means clustering and estimate HDBSCAN parameters, specifically minimum cluster size and minimum sample points. This approach reduces reliance on manual parameter tuning, improving consistency and scalability. The methodology demonstrates the feasibility of using image-based deep learning for interpreting complex spatial patterns in APT data, enabling faster and more objective analysis. The complete workflow and code are made publicly available to support reproducibility and future research.

Density-based clustering↗
DOE OSTI2023

Unsupervised learning of representative local atomic arrangements in molecular dynamics data

Molecular dynamics (MD) simulations present a data-mining challenge, given that they can generate a considerable amount of data but often rely on limited or biased human interpretation to examine their information content. By not asking the right questions of MD data we may miss critical information hidden within it. Here we combine dimensionality reduction (UMAP) and unsupervised hierarchical clustering (HDBSCAN) to quantitatively characterize prevalent coordination environments of chemical species within MD data. By focusing on local coordination, we significantly reduce the amount of data to be analyzed by extracting all distinct molecular formulas within a given coordination sphere. We then efficiently combine UMAP and HDBSCAN with alignment or shape-matching algorithms to partition these formulas into structural isomer families indicating their relative populations. The method was employed to reveal details of cation coordination in electrolytes based on molecular liquids.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

Robust clustering of the local Milky Way stellar kinematic substructures with Gaia eDR3

Understanding local stellar kinematic substructures in the solar neighbourhood helps build a complete picture of the formation of the Milky Way, as well as an empirical phase space distribution of dark matter that would inform detection experiments. We apply the clustering algorithm HDBSCAN on the Gaia early third data release to identify a list of stable clusters in velocity space and action-angle space by taking into account the measurement uncertainties and studying the stability of the clustering results. We find 1405 (497) stars in 23 (6) robust clusters in velocity space (action-angle space) that are consistently not associated with noise. We discuss the kinematic properties of these structures and study whether many of the small clusters belong to a similar larger cluster based on their chemical abundances. They are attributed to the known structures: the Gaia Sausage-Enceladus, the Helmi Stream, and globular cluster NGC 3201 are found in both spaces, while NGC 104 and the thick disc (Sequoia) are identified in velocity space (action-angle space). Although we do not identify any new structures, we find that the HDBSCAN member selection of already known structures is unstable to input kinematics of the stars when resampled within their uncertainties. We therefore present the stable subset of local kinematic structures, which are consistently identified by the clustering algorithm, and emphasize the need to take into account error propagation during both the manual and automated identification of stellar structures, both for existing ones as well as future discoveries.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

Nearby stellar substructures in the Galactic halo from DESI Milky Way Survey Year 1 Data Release

We report five nearby ($d_{\mathrm{helio}} < 5$ kpc) stellar substructures in the Galactic halo from a subset of 138 661 stars in the Dark Energy Spectroscopic Instrument (DESI) Milky Way Survey Year 1 Data Release. With an unsupervised clustering algorithm, HDBSCAN*, these substructures are independently identified in Integrals of Motion ($E_{\rm tot}$, $L_{\rm z}$, $\log {J_r}$, $\log {J_z}$) space and Galactocentric cylindrical velocity space ($V_{R}$, $V_{\phi }$, $V_{z}$). We associate all identified clusters with known nearby substructures (Helmi streams, M18-Cand10/MMH-1, Sequoia, Antaeus, and ED-2) previously reported in various studies. With metallicities precisely measured by DESI, we confirm that the Helmi streams, M18-Cand10, and ED-2 are chemically distinct from local halo stars. We have characterized the chemodynamic properties of each dynamic group, including their metallicity dispersions, to associate them with their progenitor types (globular cluster or dwarf galaxy). Our approach for searching substructures with HDBSCAN* reliably detects real substructures in the Galactic halo, suggesting that applying the same method can lead to the discovery of new substructures in future DESI data. With more stars from future DESI data releases and improved astrometry from the upcoming Gaia Data Release 4, we will have a more detailed blueprint of the Galactic halo, offering a significant improvement in our understanding of the formation and evolutionary history of the Milky Way Galaxy.

dynamics↗
DOE OSTI2026

Disentangling the Distant Stellar Halo Using K Giants in DESI Year 3 Data

We present a sample of 88,959 K giants from Dark Energy Spectroscopic Instrument Milky Way Survey Year 3 data, which we use to characterize the chemo-dynamical properties of the stellar halo at Galactocentric distances of 12 to ∼100 kpc. Using HDBSCAN, we identify five prominent stellar halo substructures: Aleph, the Sagittarius stream, Gaia-Sausage-Enceladus (GSE), Cetus-Palca, and the Orphan–Chenab stream. We present the properties of each of these structures as they appear in our catalog, and examine how uncertainties on distance affect the characterization of substructure with this approach. We also examine regions associated with previously reported overdensities (such as the Virgo Overdensities and the Sagittarius spur) that we do not recover with HDBSCAN. The size and distance range of our catalog allows us to explore in detail the residual stellar halo, comprising stars that we do not associate with any substructure. We find that samples of ∼2000 outer halo stars with both highly prograde and highly retrograde angular momenta have similar metallicity distribution functions (MDFs), which do not resemble the MDFs of either GSE or Sagittarius. Both the prograde and retrograde residual halo MDFs are bimodal, with a metal-poor peak at [Fe/H] ∼ −2 and a metal-rich peak at [Fe/H] ∼ −1.3 (prograde) or −1.5 (retrograde). The MDF for lower-angular-momentum residual halo K giants does not show clear evidence for a metal-poor peak, and broadly resembles the MDF of GSE, even at much lower binding energies than GSE itself. We discuss possible interpretations of these findings for GSE accretion scenarios.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Substructure in the stellar halo near the Sun: I. Data-driven clustering in integrals-of-motion space

Context. Merger debris is expected to populate the stellar haloes of galaxies. In the case of the Milky Way, this debris should be apparent as clumps in a space defined by the orbital integrals of motion of the stars. Aims. Our aim is to develop a data-driven and statistics-based method for finding these clumps in integrals-of-motion space for nearby halo stars and to evaluate their significance robustly. Methods. We used data from Gaia EDR3, extended with radial velocities from ground-based spectroscopic surveys, to construct a sample of halo stars within 2.5 kpc from the Sun. We applied a hierarchical clustering method that makes exhaustive use of the single linkage algorithm in three-dimensional space defined by the commonly used integrals of motion energy E, together with two components of the angular momentum, L z and L ⊥ . To evaluate the statistical significance of the clusters, we compared the density within an ellipsoidal region centred on the cluster to that of random sets with similar global dynamical properties. By selecting the signal at the location of their maximum statistical significance in the hierarchical tree, we extracted a set of significant unique clusters. By describing these clusters with ellipsoids, we estimated the proximity of a star to the cluster centre using the Mahalanobis distance. Additionally, we applied the HDBSCAN clustering algorithm in velocity space to each cluster to extract subgroups representing debris with different orbital phases. Results. Our procedure identifies 67 highly significant clusters (> 3σ), containing 12% of the sources in our halo set, and 232 subgroups or individual streams in velocity space. In total, 13.8% of the stars in our data set can be confidently associated with a significant cluster based on their Mahalanobis distance. Inspection of the hierarchical tree describing our data set reveals a complex web of relations between the significant clusters, suggesting that they can be tentatively grouped into at least six main large structures, many of which can be associated with previously identified halo substructures, and a number of independent substructures. This preliminary conclusion is further explored in a companion paper, in which we also characterise the substructures in terms of their stellar populations. Conclusions. Our method allows us to systematically detect kinematic substructures in the Galactic stellar halo with a data-driven and interpretable algorithm. The list of the clusters and the associated star catalogue are provided in two tables available at the CDS.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

StarHorse results for spectroscopic surveys and Gaia DR3: Chrono-chemical populations in the solar vicinity, the genuine thick disk, and young alpha-rich stars

The Gaia mission has provided an invaluable wealth of astrometric data for more than a billion stars in our Galaxy. The synergy between Gaia astrometry, photometry, and spectroscopic surveys gives us comprehensive information about the Milky Way. Using the Bayesian isochrone-fitting code StarHorse, we derive distances and extinctions for more than 10 million unique stars listed in both Gaia Data Release 3 and public spectroscopic surveys: 557 559 in GALAH+ DR3, 4 531 028 in LAMOST DR7 LRS, 347 535 in LAMOST DR7 MRS, 562 424 in APOGEE DR17, 471 490 in RAVE DR6, 249 991 in SDSS DR12 (optical spectra from BOSS and SEGUE), 67 562 in the Gaia-ESO DR5 survey, and 4 211 087 in the Gaia RVS part of the Gaia DR3 release. StarHorse can increase the precision of distance and extinction measurements where Gaia parallaxes alone would be uncertain. We used StarHorse for the first time to derive stellar ages for main-sequence turnoff and subgiant branch stars, around 2.5 million stars, with age uncertainties typically around 30%; the uncertainties drop to 15% for subgiant-branch-only stars, depending on the resolution of the survey. With the derived ages in hand, we investigated the chemical-age relations. In particular, the α and neutron-capture element ratios versus age in the solar neighbourhood show trends similar to previous works, validating our ages. We used the chemical abundances from local subgiant samples of GALAH DR3, APOGEE DR17, and LAMOST MRS DR7 to map groups with similar chemical compositions and StarHorse ages, using the dimensionality reduction technique t-SNE and the clustering algorithm HDBSCAN. We identify three distinct groups in all three samples, confirmed by their kinematic properties: the genuine chemical thick disk, the thin disk, and a considerable number of young alpha-rich stars (427) that are also a part of the delivered catalogues. We confirm that the genuine thick disk’s kinematics and age properties are radically different from those of the thin disk and compatible with high-redshift (z ≈ 2) star-forming disks with high dispersion velocities. We also find a few extra chemical populations in GALAH DR3 thanks to the availability of neutron-capture element information.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

Scalable edge clustering of dynamic graphs via weighted line graphs

Timestamped relational datasets consisting of records (or connections) between pairs of entities are ubiquitous in network science. For applications like peer-to-peer communication, email, various social network interactions, and computer network security, it is useful to organize these records into groups based on how and when they are occurring. Weighted line graphs offer a natural way to model how records are related in such datasets but for large real-world graph topologies, building and utilizing the line graph is prohibitively expensive. Here, we present the framework to cluster the edges of a dynamic graph via the associated line graph that contains two major contributions. The first is a method to work with the line graph implicitly and the second is a distributed scale implementation of an agglomerative hierarchical graph clustering algorithm. We outline a novel hierarchical dynamic graph edge clustering approach that efficiently breaks massive relational datasets into small sets of edges containing events at various timescales. This is in stark contrast to traditional graph clustering algorithms that prioritize highly connected (clique-like) community structures. Our approach relies on constructing a sufficient subgraph of a weighted line graph and applying a hierarchical agglomerative clustering. This approach is related to scalable techniques from spatial clustering, nonlinear-dimension reduction, topological data analysis, and draws particular inspiration from HDBSCAN. As an edge clustering, this method yields an overlapping node clustering. Our algorithm is parallelizable and we demonstrate efficient clustering of a billion-scale, real-world dynamic graph into small edge sets that correlate in topology and time. The entire clustering process for a graph with tens of billions of edges takes just a few minutes of run time on 256 nodes of a distributed compute environment. We argue how the output of the edge clustering is useful for a multitude of data visualization and powerful machine learning tasks, both involving the original massive dynamic graph data and metadata associated with the nodes and edges. Finally, we describe how this approach can be extended to dynamic hypergraphs and dynamic graphs/hypergraphs with unstructured data living on vertices and edges.

Data Analysis↗
DOE OSTI2025

ArborX 2.0

ArborX library tackles a problem of efficiently finding geometric objects that are close in space. Variations of this problem, such as finding the nearest neighbors of a point, or finding all objects within a certain distance, are inherent components of applications in many fields. The data may be large so that solving the problem efficiently may require significant computational resources, such as multiple processors or accelerators such as general purpose GPUs. ArborX' main advantage in its ability to solve large problems efficiently utilizing a combination of distributed and on-node parallelism. ArborX can be run efficiently on a wide variety of hardware, including GPUs from different vendors, which distinguishes it from other available libraries which typically choose only few of these. The other advantage is that it supports both types of user problems: spatial problems (useful for intersections and finding objects within certain distance), and nearest neighbor problems. ArborX also supports flexible interface in its interaction with a user. Particularly, it allows a user to call user's own function on a positive match, a functionality not rarely available in other libraries. ArborX implements construction and traversal algorithms using efficient tree structures, such as bounding volume hierarchy (BVH). At its core, ArborX uses linear BVH for its low construction cost and sufficient quality. ArborX implements both spatial and nearest-neighbor traversal algorithms. ArborX also provides several clustering algorithms (minimum spanning tree, DBSCAN, HDBSCAN*), interpolation using minimum least squares and ray tracing. ArborX is written using C++, and is parallelized using the message passing interface (MPI) for the distributed communication, and the Kokkos library for on-node parallelism. This approach allows ArborX to be run on a wide variety of hardware, from common laptops and desktops to supercomputers while using the same codebase.

Prokopenko, Andrey [Oak Ridge National Laboratory ↗
DOE OSTI2025

Hiperclust

This software leverages transfer learning to analyze atom probe tomography (APT) data. It is trained on synthetic data and then applies this knowledge to predict the optimal number of clusters for a given APT dataset. Initially, the software used preliminary clustering to estimate the general structure of the data. Based on this, it provides suggestions for key parameters like minimum cluster size and minimum number of points. These parameters are critical for algorithms like HDBSCAN, ensuring accurate cluster formation without the need for trial-and-error testing. The software runs on High-Performance computing (HPC) systems, enabling fast, scalable analysis of large APT datasets, ultimately saving time and improving the reliability of clustering outcomes.

Tang, Yalei [Idaho National Laboratory (INL), Idah↗
DOE OSTI2025

Clustering at Massive Scale

ClaMS provides hierarchical clustering technology for use on massive, high-dimensional datasets that require distributed memory for processing. The algorithm employed is inspired by the popular HDBSCAN algorithm but makes use of computational kernels better suited for distributed computing. ClaMS is built on scalable nearest neighbor graph construction, metric forest completion, and approximate minimum spanning tree techniques.

Stanley, ThomasA [Lawrence Livermore National Labo↗
DOE OSTI2021

Diagnosing Open Cluster Stock 2: Member Candidates and Mass Distribution with Gaia DR2 and LAMOST

We identify 1325 member candidates of the open cluster (OC) Stock 2 using data from Gaia DR2. We use the algorithms Clusterix 2.0 and HDBSCAN to select cluster candidates and further refine the final cluster membership by defining neighbors in 5D phase space (X {sub cp}, Y {sub cp}, Z{sub cp},κ⋅μ{sub α}{sup ∗}/ϖ, κ · μ {sub δ}/ϖ). Among these candidates, less than half have G, G {sub BP}, and G {sub RP} extinctions from Gaia. When Gaia extinctions are unavailable, we compute extiction using empirical formulas and E(B − V) = 0.350. We analyze the spatial distribution and mass profile of Stock 2. Our results reveal Stock 2 is still a bound OC and we find evidence of mass segregation. By comparing initial mass functions, the present-day mass function indicates that Stock 2 is a massive stellar cluster with a mass of 4000 M {sub ⊙}. The core radius and tidal radius, calculated via the radial density profile and total mass, are 3.97 pc and 22.65 pc, respectively. Common stars between our selected member candidates and the Large Sky Area Multi-Object Fiber Spectroscopic Telescope DR7 medium-resolution catalog give a metalliclity of [Fe/H] = −0.040 ± 0.147.

47 OTHER INSTRUMENTATION↗
DOE OSTI2022

Possibilities and Limitations of Kinematically Identifying Stars from Accreted Ultra-faint Dwarf Galaxies

Abstract The Milky Way has accreted many ultra-faint dwarf galaxies (UFDs), and stars from these galaxies can be found throughout our Galaxy today. Studying these stars provides insight into galaxy formation and early chemical enrichment, but identifying them is difficult. Clustering stellar dynamics in 4D phase space ( E , L z , J r , J z ) is one method of identifying accreted structure that is currently being utilized in the search for accreted UFDs. We produce 32 simulated stellar halos using particle tagging with the Caterpillar simulation suite and thoroughly test the abilities of different clustering algorithms to recover tidally disrupted UFD remnants. We perform over 10,000 clustering runs, testing seven clustering algorithms, roughly twenty hyperparameter choices per algorithm, and six different types of data sets each with up to 32 simulated samples. Of the seven algorithms, HDBSCAN most consistently balances UFD recovery rates and cluster realness rates. We find that, even in highly idealized cases, the vast majority of clusters found by clustering algorithms do not correspond to real accreted UFD remnants and we can generally only recover 6% of UFDs remnants at best. These results focus exclusively on groups of stars from UFDs, which have weak dynamic signatures compared to the background of other stars. The recoverable UFD remnants are those that accreted recently, z accretion ≲ 0.5. Based on these results, we make recommendations to help guide the search for dynamically linked clusters of UFD stars in observational data. We find that real clusters generally have higher median energy and J r , providing a way to help identify real versus fake clusters. We also recommend incorporating chemical tagging as a way to improve clustering results.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

Enhancing Cluster Identification in Atom Probe Tomography Data Using Transfer Learning

Atom probe tomography (APT) has enabled the direct visualization of solute clusters, providing valuable insights into material structures. This clustering is crucial for understanding the nanoscale composition and behavior of materials, which can significantly influence their mechanical and physical properties. However, the widely used clustering methods in the APT community face challenges such as subjective parametric selection and limited applicability, particularly in dealing with overlapping clusters, nested clusters, and artifacts across different scales, such as precipitates and dislocations. To address these challenges, we present a framework based on density-based cluster analysis that aims to be less dependent on user input, reproducible, and robust.

Density-based clustering↗