SEARCH · Search NASA

Results for “Self-Organizing Maps”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

All sources NASA NTRS DOE OSTI

DOE OSTI2026

Enhancing weak lensing redshift distribution characterization by optimizing the Dark Energy Survey Self-Organizing Map Photo-z method

Characterization of the redshift distribution of ensembles of galaxies is pivotal for large scale structure cosmological studies. In this work, we focus on improving the Self-Organizing Map (SOM) methodology for photometric redshift estimation (SOMPZ), specifically in anticipation of the Dark Energy Survey Year 6 (DES Y6) data. This data set, featuring deeper and fainter galaxies than DES Year 3 (DES Y3), demands adapted techniques to ensure accurate recovery of the underlying redshift distribution. We investigate three strategies for enhancing the existing SOM-based approach used in DES Y3: 1) Replacing the Y3 SOM algorithm with one tailored for redshift estimation challenges; 2) Incorporating $\textit{g}$-band flux information to refine redshift estimates (i.e. using $\textit{griz}$ fluxes as opposed to only $\textit{riz}$); 3) Augmenting redshift data for galaxies where available. These methods are applied to DES Y3 data, and results are compared to the Y3 fiducial ones. Our analysis indicates significant improvements with the first two strategies, notably reducing the overlap between redshift bins. By combining strategies 1 and 2, we have successfully managed to reduce redshift bin overlap in DES Y3 by up to 66$\%$. Conversely, the third strategy, involving the addition of redshift data for selected galaxies as an additional feature in the method, yields inferior results and is abandoned. Our findings contribute to the advancement of weak lensing redshift characterization and lay the groundwork for better redshift characterization in DES Year 6 and future stage IV surveys, like the Rubin Observatory.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2026

Linking mixtures of air pollution exposures and preterm birth with a self-organizing map

Explore the source record for details and available documents.

Kelly, Brenna [University of Utah, Salt Lake City,↗

DOE OSTI2026

A procedure for rule extraction from a Self-Organising plasma disruption predictor for JET

In a previous paper, a Self-Organizing Map had proven to be able to identify the regions of the plasma operative space characterizing the pre-disruptive phase at JET without relying on any a priori information. One of the strengths of this disruption predictor lies in its inherent self-organization capability. The Self-Organizing Map discovers non-trivial relationships and captures the complicated interplay of device diagnostics on the internal plasma states directly from the experimental data. Moreover, the provided model allows the visualization of high-dimensional plasma parameters and facilitates easy interrogation of the model to understand the reasons behind its correlations. In this paper, an additional step is taken towards the interpretability of models for predicting disruptions by training a Decision Tree to classify the plasma states according to the interpretation provided by the Self-Organizing Map (stable or at high risk of disruptions). The Decision tree provides a set of rules which describe the transition of the plasma towards the pre-disruptive phase as visualized in the Self-Organizing Map. The obtained rules for the database explored in the study identify four regions in the map, two of which are at risk of disruption. These regions correspond to partitions of a 3D space based on the peaking factors of the core and divertor radiation, as well as the Locked Mode. The agreement between the Self-Organizing Map answers and the rules supplied by the Decision Tree is confirmed by the comparison of the performance exhibited by the two models in the prediction of disruptions.

Setzu, Samuele [Univ. of Cagliari, Monserrato, Cag↗

DOE OSTI2026

Dark Energy Survey Year 6 results: Clustering redshifts and importance sampling of self-organized-maps 𝑛⁡(𝑧) realizations for 3 × 2 ⁢pt samples

This work is part of a series establishing the redshift framework for the 3 × 2 ⁢pt analysis of the Dark Energy Survey Year 6 (DES Y6). For DES Y6, photometric redshift distributions are estimated using self-organizing maps (SOMs), calibrated with spectroscopic and many-band photometric data. To overcome limitations from color-redshift degeneracies and incomplete spectroscopic coverage, we enhance this approach by incorporating clustering-based redshift constraints (clustering-z, or WZ) from angular cross-correlations with BOSS and eBOSS galaxies and eBOSS quasar samples. We define a WZ likelihood and apply importance sampling to a large ensemble of SOM-derived 𝑛⁡(𝑧) realizations, selecting those consistent with the clustering measurements to produce a posterior sample for each lens and source bin. The analysis uses angular scales corresponding to 1.5–5 Mpc to optimize signal-to-noise ratio while mitigating modeling uncertainties and marginalizes over redshift-dependent galaxy bias and other systematics informed by the N-body simulation CARDINAL . While a sparser spectroscopic reference sample limits WZ constraining power at 𝑧 >1.1, particularly for source bins, we demonstrate that combining SOM with WZ improves redshift accuracy and enhances the overall cosmological constraining power of DES Y6. As a result, we estimate an improvement in 𝑆 8 of approximately 10% for cosmic shear and 3 ×2⁢pt analysis, primarily due to the WZ calibration of the source samples.

Cosmological parameters↗

DOE OSTI2026

Spring Dust in Colorado Plateau: Transport Pathways and Interannual Variability Derived From Two Decades of MERRA‐2 Reanalysis

Spring dust storms in the U.S. Southwest significantly impact environmental and human systems, yet their climatological patterns and driving mechanisms remain poorly understood. Using two decades of MERRA-2 reanalysis data and self-organizing map clustering, we identified four distinct dust transport pathways from surrounding and remote deserts for the top 10% of spring dust events impacting the Colorado Plateau: (a) intense southwesterly near-surface transport where eastward upward transport is blocked by topography, (b) substantial remote transport plus weaker surface emissions, (c) moderate southwesterly transport with stronger upper-level transport, and (d) rare northwesterly pattern linked to the Great Basin sources, which accounts for 38%, 14%, 37%, and 11% of total events, respectively. The frequency and intensity of these events strongly correlate with large-scale climate variability, with increased dust events during La Niña. Our results provide a robust framework for improving seasonal dust forecasts and understanding future dust dynamics under climate change.

aerosol transport↗

DOE OSTI2025

Using active learning to improve quasar identification for the DESI spectra processing pipeline

The Dark Energy Spectroscopic Instrument (DESI) survey uses an automatic spectral classification pipeline to classify spectra. QuasarNET is a convolutional neural network used as part of this pipeline originally trained using data from the Baryon Oscillation Spectroscopic Survey (BOSS). In this paper we implement an active learning algorithm to optimally select spectra to use for training a new version of the QuasarNET weights file using only DESI data, with the goal of improving classification accuracy. This active learning algorithm includes a novel outlier rejection step using a Self-Organizing Map to ensure we label spectra representative of the larger quasar sample observed in DESI. We perform two iterations of the active learning pipeline, assembling a final dataset of 5600 labeled spectra, a small subset of the approximately 1.3 million quasar targets in DESI's Data Release 1. When splitting the spectra into training and validation subsets we achieve similar performance to the previously trained weights file in completeness and purity calculated on the validation dataset but do so with less than one tenth of the amount of training data. The new weights also more consistently classify objects in the same way when used on unlabeled data compared to the old weights file. In the process of improving QuasarNET's classification accuracy we discovered a systemic error in QuasarNET's redshift estimation and used our findings to improve our understanding of QuasarNET's redshifts.

Machine learning↗

DOE OSTI2026

Dark Energy Survey Year 6 results: Redshift calibration of the MagLim++ lens sample

In this work, we derive and calibrate the redshift distribution of the MagLim++ lens galaxy sample used in the Dark Energy Survey Year 6 (DES Y6) 3 x 2pt cosmology analysis. The 3 x 2pt analysis combines galaxy clustering from the lens galaxy sample and weak gravitational lensing. The redshift distributions are inferred using the SOMPZ method - a Self-Organizing Map framework that combines deep-field multi-band photometry, wide-field data, and a synthetic source injection ( B alrog) catalog. Key improvements over the DES Year 3 (Y3) calibration include a noise-weighted SOM metric, an expanded Balrog catalogue, and an improved scheme for propagating systematic uncertainties, which allows us to generate O(10 8 ) redshift realizations that collectively span the dominant sources of uncertainty. These realizations are then combined with independent clustering-redshift measurements via importance sampling. The resulting calibration achieves typical uncertainties on the mean redshift of 1-2%, corresponding to a 20-30% average reduction relative to DES Y3. We compress the n(z) uncertainties into a small number of orthogonal modes for use in cosmological inference. Marginalizing over these modes leads to only a minor degradation in cosmological constraints. Here, this analysis establishes the MagLim++ sample as a robust lens sample for precision cosmology with DES Y6 and provides a scalable framework for future surveys.

dark energy↗

DOE OSTI2025

Regime Characterization of Offshore Wind Resource Using Unsupervised Learning

Predictability of wind resource conditions is critical for offshore wind design and operations. While many studies of extreme wind conditions focus on specific events such as low-level jets or ramps, these rely on threshold definitions that limit generality. Here we present a data-driven framework that combines principal component analysis (PCA), self-organizing maps (SOM), and k-means clustering to classify wind resource conditions as typical and anomalous from climatological data. Anomalies are defined not by fixed thresholds but by flagging samples located far from SOM node centers inside the baseline SOM structure. This reframes extremes as rare ebents and hence, likely difficult to anticipate by numerical weather prediction models. We applied this approach to 23 years (2000–2022) of hourly profiles from the NOW-23 hindcast model at the Humboldt Wind Energy Area. Classification is conducted on a feature space consisting of 10 m wind speed and direction, bulk shear and veer across 30–270 m, and a low-level jet index. Dimensionality reduction is achieved through PC. A 2 × 3 OM lattice trained on the PCA vectors identified six baseline regimes spanning weak to strong flow states. High quantization-error profiles are identified and re-clustered into four anomalous regimes. The baseline regimes exhibited clear seasonal and diurnal cycles. Meanwhile, the anomalous regimes represented <10 % of all hours but showed distinct combinations of speed, shear, and veer, when compared to the baseline regimes. Anomalous regimes are typically short-lived (~few hours), yet their transitions can lead to hub-height wind changes of −18 to +9 m s -1 . For a representative 15 MW turbine, these shifts imply rapid swings in capacity factor from near-full output to negligible generation. Validation with lidar buoy data showed 51% agreement in SOM labels across ~6,000 overlapping hours, with most mismatches confined to adjacent speed classes. HRRR comparisons further revealed that anomalous regimes were disproportionately associated with forecast biases exceeding 5 m s -1 . Together, these results reframe extremes in offshore wind from absolute maxima or minima to weather states that are difficult to anticipate from models.

17 WIND ENERGY↗

DOE OSTI2025

Toward Drilling the Perfect Geothermal Well: An International Research Coordination Network for Geothermal Drilling Optimization Supported by Deep Machine Learning and Cloud Based Data Aggregation

The EDGE project, supported by the U.S. Department of Energy Geothermal Technologies Office under award DE-EE0008793, established a data-driven framework for improving the efficiency, cost-effectiveness, and reliability of geothermal well drilling. The project focused on developing scalable data infrastructure, advanced machine learning and probabilistic models, and integrated analytics tools to support continuous drilling optimization. A central objective was to reduce geothermal drilling costs by up to seventy percent while minimizing the risk of well failure through predictive diagnostics and adaptive planning. Over the project period, a comprehensive data repository was designed and deployed, incorporating records from over one hundred geothermal wells across varied geological settings. This repository supported both structured and unstructured data and adhered to FAIR data principles, enabling provenance tracking, quality control, and standardized metadata. The project introduced automated ingestion pipelines and a cloud-hosted platform that facilitated access to raw, processed, and derived datasets. This infrastructure served as the foundation for model development and analysis. Machine learning workflows were developed to predict key drilling metrics including rate of penetration, non-productive time, and total drilling costs. Self-organizing maps and dimensionality reduction methods were used to uncover operational patterns and outliers, while supervised learning algorithms such as random forests and deep neural networks were applied to forecast performance outcomes. The models were validated on heterogeneous datasets from both U.S. and Icelandic fields, demonstrating variable but significant predictive accuracy. The results indicated that finer temporal resolution, inclusion of lithological data, and consistency in operational annotations could substantially improve model performance. The project also implemented process mining techniques to reconstruct state-transition models from drilling event logs. These models enabled the identification of deviations from optimal workflows and provided insights into recurring failure modes. Analysis of non-productive time highlighted the impact of equipment failures, geological challenges, and human factors, offering opportunities for targeted mitigation strategies. The EDGE Dashboard was developed as a web-based expert system integrating data visualization, model outputs, and user-driven queries. It provided an accessible interface for operators to explore historical data, evaluate predicted outcomes, and compare drilling scenarios. Initial feedback from project partners suggested that the dashboard could serve as a foundation for more advanced advisory and optimization tools. Overall, the EDGE project demonstrated the feasibility and value of applying modern data science techniques to geothermal drilling. It delivered a set of interoperable tools and models that can support more efficient, lower-risk well development. The findings point toward a viable path for transitioning from advisory analytics to semi-autonomous drilling systems, contingent on continued collaboration, expanded datasets, and field validation. The project results have immediate relevance for drilling operations, data management practices, and future geothermal R&D efforts aimed at achieving reliable, cost-competitive geothermal energy at scale.

15 GEOTHERMAL ENERGY↗

DOE OSTI2026

Stakeholder-guided holistic, Adaptive Framework for enhancing community Energy Resilience (SAFER) (Final Technical Report)

The Stakeholder-guided holistic, Adaptive Framework for enhancing community Energy Resilience (SAFER) project advances resilience science and engineering by addressing challenges in rural Kansas communities where aging infrastructure, extreme weather, and socioeconomic disparities heighten vulnerability to energy disruptions. Traditional approaches often focus on technical performance while overlooking community concerns and priorities. SAFER responds by integrating community perspectives with advanced analytical frameworks to create a holistic model for measuring and improving resilience. Project objectives included developing novel resilience metrics, advancing modeling frameworks that capture interdependencies across infrastructures, and embedding community-centric indicators directly into planning processes for distributed energy resources. The key technical innovations included the creation of self-organizing map (SOM)-based indices for objective resilience quantification, hetero-functional graph theory (HFGT) models linking power, water, transportation, and community assets, and graph neural network (GNN) tools for identifying critical nodes in complex systems. Community-centric energy planning was demonstrated through optimal siting and sizing of (photovoltaic) PV and battery storage, ensuring resilience enhancements also addressed energy burden and energy insecurity. SAFER engaged community partners in Dodge City and Ford County through surveys, focus groups, and workshops, generating more than 600 responses that established baseline measures of energy burden, financial insecurity, and willingness-to-pay to avoid outages. This data, organized in terms of a community capitals framework, informed the development of weighted reliability indices that better reflect community costs than traditional utility metrics. SAFER’s GNN-based critical node identification framework identified expert-labelled critical nodes with over 99% accuracy, while also uncovering additional functionalities essential for proactive resilience planning. The project’s models demonstrated that optimal PV and storage deployment could improve resilience indices by over 11 percent, with dispatch strategies further enhancing outcomes, confirming both the technical effectiveness and economic feasibility of these approaches. Through its combined emphasis on rigorous modeling, community-focused planning, and community engagement, SAFER advances the state of resilience research while delivering direct benefits to rural communities. The project provides tools, guidelines, and resilience heatmaps that help utilities, local governments, and residents better anticipate disruptions, prioritize investments, and strengthen the capacity to withstand and recover from energy-related hazards. Furthermore, the developed HFG and GNN frameworks are designed for transferability, allowing them to be adapted for resilience planning in other communities with minimal retraining. This inductive learning capability provides a scalable pathway to extend the SAFER project’s impact. Thus, creating a foundation for a nationally applicable model of infrastructure resilience. Additionally, the HFG can also be extended to include other FEMA community lifelines.

14 SOLAR ENERGY↗

DOE OSTI2025

The DECADE cosmic shear project II: photometric redshift calibration of the source galaxy sample

We present the photometric redshift characterization and calibration for the Dark Energy Camera All Data Everywhere (DECADE) weak lensing dataset: a catalog of 107 million galaxies observed by the Dark Energy Camera (DECam) in the northern Galactic cap. The redshifts are estimated from a combination of wide-field photometry, deep-field photometry with associated redshift estimates, and a transfer function between the wide field and deep field that is estimated using a source injection catalog. We construct four tomographic bins for the galaxy catalog, and estimate the redshift distribution, $n(z)$, within each one using the Self-organizing Map Photo-Z (SOMPZ) methodology. Our estimates include the contributions from sample variance, zeropoint calibration uncertainties, and redshift biases, as quantified for the deep-field dataset. The total uncertainties on the mean redshifts are $σ_{\langle}$$_z$$_{\rangle} ≈ 0.01$. The SOMPZ estimates are then compared to those from the clustering redshift method, obtained by cross-correlating our source galaxies with galaxies in spectroscopic surveys, and are shown to be consistent with each other.

Anbajagane, D. [Univ. of Chicago, IL (United State↗

DOE OSTI2025

Synoptic Weather Regime Classifications for June, July and August, from 2000 to 2025

The synoptic weather regime classification has become a highly demanded product for the ARM site in recent years. This type of regime classification has shown applications in various studies and topics, including aerosol-cloud interactions, land-atmosphere interactions, and cloud radiative effects. The VAP employs an unsupervised machine learning method, Self-organizing map (SOM), to classify weather regimes for each day of the AMF campaigns and fixed sites, using ERA5 data. This idea is mainly based on our published study for TRACER in Wang et al. (2022, JGR-A).

node_som_ml↗

DOE OSTI2025

Synoptic Weather Regime Classifications for December, January and February, from 2000 to 2025

{node_som_ml,synop_wea_reg}↗

DOE OSTI2025

Synoptic Weather Regime Classifications for September, October and November, from 2000 to 2025

node_som_ml↗

DOE OSTI2025

Dark Energy Survey Year 6 Results: Redshift Calibration of the MagLim++ Lens Sample

In this work, we derive and calibrate the redshift distribution of the MagLim++ lens galaxy sample used in the Dark Energy Survey Year 6 (DES Y6) 3x2pt cosmology analysis. The 3x2pt analysis combines galaxy clustering from the lens galaxy sample and weak gravitational lensing. The redshift distributions are inferred using the SOMPZ method - a Self-Organizing Map framework that combines deep-field multi-band photometry, wide-field data, and a synthetic source injection (Balrog) catalog. Key improvements over the DES Year 3 (Y3) calibration include a noise-weighted SOM metric, an expanded Balrog catalogue, and an improved scheme for propagating systematic uncertainties, which allows us to generate O($10^8$) redshift realizations that collectively span the dominant sources of uncertainty. These realizations are then combined with independent clustering-redshift measurements via importance sampling. The resulting calibration achieves typical uncertainties on the mean redshift of 1-2%, corresponding to a 20-30% average reduction relative to DES Y3. We compress the $n(z)$ uncertainties into a small number of orthogonal modes for use in cosmological inference. Marginalizing over these modes leads to only a minor degradation in cosmological constraints. This analysis establishes the MagLim++ sample as a robust lens sample for precision cosmology with DES Y6 and provides a scalable framework for future surveys.

Giannini, G. [Chicago U., Astron. Astrophys. Ctr.;↗

DOE OSTI2025

Dark Energy Survey Year 6 Results: Redshift Calibration of the Weak Lensing Source Galaxies

Determining the distribution of redshifts for galaxies in wide-field photometric surveys is essential for robust cosmological studies of weak gravitational lensing. We present the methodology, calibrated redshift distributions, and uncertainties of the final Dark Energy Survey Year 6 (Y6) weak lensing galaxy data, divided into four redshift bins centered at $\langle z \rangle = [0.414, 0.538, 0.846, 1.157]$. We combine independent information from two methods on the full shape of redshift distributions: optical and near-infrared photometry within an improved Self-Organizing Map $p(z)$ (SOMPZ) framework, and cross-correlations with spectroscopic galaxy clustering measurements (WZ), which we demonstrate to be consistent both in terms of the redshift calibration itself and in terms of resulting cosmological constraints within 0.1$σ$. We describe the process used to produce an ensemble of redshift distributions that account for several known sources of uncertainty. Among these, imperfection in the calibration sample due to the lack of faint, representative spectra is the dominant factor. The final uncertainty on mean redshift in each bin is $σ_{\langle z\rangle} = [0.012, 0.008,0.009, 0.024]$. We ensure the robustness of the redshift distributions by leveraging new image simulations and a cross-check with galaxy shape information via the shear ratio (SR) method.

Yin, B. [Duke U.] (ORCID:0009000656049980)↗

DOE OSTI2025

The Power of DESI for Photometric Redshift Calibration: A Case Study with KiDS-1000

Accurate redshift estimates are a critical requirement for weak lensing surveys and one of the main uncertainties in constraints on dark energy and large-scale cosmic structure. In this paper, we study the potential to calibrate photometric redshift (photo-z) distributions for gravitational lensing using the Dark Energy Spectroscopic Instrument (DESI). Since beginning its science operations in 2021, DESI has collected more than 50 million redshifts, adding about one million monthly. In addition to its large-scale structure samples, DESI has also acquired over 256k high-quality spectroscopic redshifts (spec-zs) in the COSMOS and XMM and VVDS fields. This is already a factor of 3 larger than previous spec-z calibration compilations in these two regions. Here, we explore calibrating photo-zs for the subset of KiDS-1000 galaxies that fall into joint self-organizing map (SOM) cells overlapping the DESI COSMOS footprint using the DESI COSMOS observations. Estimating the redshift distribution in KiDS-1000 with the new DESI data, we find broad consistency with previously published results while also detecting differences in the mean redshift in some tomographic bins with an average shifts of Delta Mean(z) = -0.028 in the mean and Delta Median(z) = +0.011 in the median across tomographic bins. However, we also find that incompleteness per SOM cell, i.e., groups of galaxies with similar colors and magnitudes, can modify n(z) distributions. Finally, we comment on the fact that larger photometric catalogs, aligned with the DESI COSMOS and DESI XMM and VVDS footprints, would be needed to fully exploit the DESI dataset and would extend the coverage to nearly eight times the area of existing 9-band photometry.

Blanco, Diana [UC, Santa Cruz; UC, Santa Cruz, Ins↗

DOE OSTI2026

A Unified Photometric Redshift Calibration for Weak Lensing Surveys Using the Dark Energy Spectroscopic Instrument

The effective redshift distribution n(z) of galaxies is a critical component in the study of weak gravitational lensing. Here, we introduce a new method for determining n(z) for weak lensing surveys based on high-quality redshifts and neural-network-based importance weights. Additionally, we present the first unified photometric redshift calibration of the three leading stage-III weak lensing surveys, the Dark Energy Survey (DES), the Hyper Suprime-Cam (HSC) survey, and the Kilo-Degree Survey (KiDS), with state-of-the-art spectroscopic data from the Dark Energy Spectroscopic Instrument (DESI). We verify our method using a new, data-driven approach and obtain n(z) constraints with statistical uncertainties of the order of $σ_z$ ~ 0.01 and smaller. Our analysis is largely independent of previous photometric redshift calibrations and, thus, provides an important cross-check in light of recent cosmological tensions. Overall, we find excellent agreement with previously published results on the DES Y3 and HSC Y1 data sets, while there are some differences on the mean redshift with respect to the previously published KiDS-1000 results. We attribute the latter to mismatches in photometric noise properties in the COSMOS field compared to the wider KiDS self-organizing map-gold catalog. At the same time, the new n(z) estimates for KiDS do not significantly change estimates of cosmic structure growth from cosmic shear. Finally, we discuss how our method can be applied to future weak lensing calibrations with DESI data.

Lange, J. U. [American Univ., Washington, DC (Unit↗