Search NASA⌕ Search

SEARCH · Search NASA

Results for “ATTENTION”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 271 records · Page 15
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

SHF: Symmetrical Hierarchical Forest with Pretrained Vision Transformer Encoder for High-Resolution Medical Segmentation

This paper presents a novel approach to addressing the long-sequence problem in high-resolution medical images for Vision Transformers (ViTs). Using smaller patches as tokens can enhance ViT performance, but quadratically increases computation and memory requirements. Therefore, the common practice for applying ViTs to high-resolution images is either to: (a) employ complex sub-quadratic attention schemes or (b) use large to medium-sized patches and rely on additional mechanisms within the model to capture the spatial hierarchy of details. We propose Symmetrical Hierarchical Forest (SHF), a lightweight approach that adaptively patches the input image to increase token information density and encode hierarchical spatial structures into the input embedding. We then apply a reverse depatching scheme to the output embeddings of the transformer encoder, eliminating the need for convolution-based decoders. Unlike previous methods that modify attention mechanisms or use a complex hierarchy of interacting models, SHF can be retrofitted to any ViT model to allow it to learn the hierarchical structure of details in high-resolution images without requiring architectural changes. Experimental results demonstrate significant gains in computational efficiency and performance: on the PAIP WSI dataset, we achieved a 3∼32×speedup or a 2.95%∼7.03% increase in accuracy (measured by Dice score) at a 64K2 resolution with the same computational budget, compared to state-of-the-art production models. On the 3D medical datasets BTCV and KiTS, training was 6×faster, with accuracy gains of 6.93% and 5.9%, respectively, compared to models without SHF.

Zhang, Enzhi [Hokkaido University, Japan]↗
DOE OSTI2025

Sub-microsecond Transformers for Jet Tagging on FPGAs

We present the first sub-microsecond transformer implementation on an FPGA achieving competitive performance for state-of-the-art high-energy physics benchmarks. Transformers have shown exceptional performance on multiple tasks in modern machine learning applications, including jet tagging at the CERN Large Hadron Collider (LHC). However, their computational complexity prohibits use in real-time applications, such as the hardware trigger system of the collider experiments up until now. In this work, we demonstrate the first application of transformers for jet tagging on FPGAs, achieving $\mathcal{O}(100)$ nanosecond latency with superior performance compared to alternative baseline models. We leverage high-granularity quantization and distributed arithmetic optimization to fit the entire transformer model on a single FPGA, achieving the required throughput and latency. Furthermore, we add multi-head attention and linear attention support to hls4ml, making our work accessible to the broader fast machine learning community. This work advances the next-generation trigger systems for the High Luminosity LHC, enabling the use of transformers for real-time applications in high-energy physics and beyond.

Laatu, Lauri [Imperial Coll., London]↗
DOE OSTI2024

Harnessing graph convolutional neural networks for identification of glassy states in metallic glasses

Graph Convolutional Neural Networks (GCNNs) have emerged as powerful tools for analyzing materials. In this study, we employ GCNNs to examine structural characteristics of CuZr metallic glasses (MGs) and identify their states. We use molecular dynamics to simulate the quenching process of CuZr, using cooling rates ranging from 10 9 to 10 15 K/s, to produce six unique glassy states. For each state, we create a dataset comprising 1,800 distinct samples. We evaluate the effectiveness of various GCNNs, including Graph Attention Neural Network (GANN), Graph Sample and AggreGatE (GraphSAGE), Graph Isomorphism Network (GIN), and Relational Graph Convolutional Neural Network (RGCN). GANN and GraphSAGE demonstrate comparable performance, achieving an overall accuracy of 81% in classifying the MG states. Furthermore, these results underscore the potential of GCNNs to detect subtle structural variances in disordered materials and point to broader application of deep learning in the analysis of MGs and other amorphous substances.

36 MATERIALS SCIENCE↗
DOE OSTI2026

AGFormer: Adaptive Spatiotemporal graph informed transformer for multi-reservoir inflow forecasting

Accurate reservoir inflow forecasting is crucial for effective water resource management, yet most machine learning models focus on single-reservoir prediction and overlook spatial dependencies among hydrologically connected reservoirs. Here, we propose AGFormer (Adaptive Graph-Informed Transformer), an end-to-end framework that integrates adaptive graph learning with temporal sequence modeling for multi-reservoir inflow forecasting. A shared encoder and graph attention mechanism generate reservoir-specific embeddings, which are then processed by the Transformer-based encoder–decoder for multi-step inflow forecasting. We also introduce a pretraining paradigm to learn robust temporal embeddings from misaligned historical records. Evaluated on 30 reservoirs in the Upper Colorado River Basin, AGFormer achieves superior seven-day-ahead forecasts, with NSE > 0.75 for 20 reservoirs—outperforming Encoder–Decoder LSTM, GCN+LSTM, and Transformer baselines. Adaptive graph learning captures dynamic inter-reservoir dependencies, and feature attribution aligns with snowmelt-driven hydrology. Incorporating forecasted meteorological inputs further enhances accuracy, demonstrating AGFormer’s potential to support reservoir management under dynamic hydrological conditions.

Adaptive graph learning↗
DOE OSTI2025

River Dissolved Oxygen Prediction Using Machine Learning Models and Wireless Sensor Measurements

Simultaneous flooding&heat and droughts&heat events can potentially destabilize hydro-meteorological conditions to deteriorate the water quality of Neches River. Machine learning (ML) models utilizing wireless sensor measurements have been applied to predict water quality and optimize various water management strategies. This study aims to develop ML models to predict dissolved oxygen (DO) prediction under various hydro-meteorological conditions and enhance water management decision-making. Wireless sensor measurements of DO, water temperature, sample depth, conductivity, turbidity, and pH, along with discharge from the United States Geological Survey stations, are collected for model inputs at the Pine Island Bayou C749 station (PIB-C749) and Neches River Saltwater Barrier (SWB). Multilayer perceptron neural networks, recurrent neural networks, long short-term memory (LSTM), and bidirectional LSTM (BiLSTM) with and without attention mechanism (AT) are tested to determine the best model, which is applied the rolling forecast method to predict 14-day DO. Traditional and recurrent transfer learning (TL and RTL) methods are adopted to overcome insufficient data at the SWB. The input feature importance analysis using the integrated gradients (IG) algorithm is applied to determine dominant inputs. The results show LSTM-based models are capable handling long sequential data. AT-BiLSTM and RTL-LSTM demonstrate the best performance at the PIB-C749 (RMSE=0.054) and the SWB (RMSE=0.028), respectively. TL and RTL methods significantly improve model performance at the SWB. DO, temperature, and pH show higher importance, consistent with hydrodynamics and water chemistry. Both best models are applied to predict 14-day DO and demonstrate reasonable performance for decision-making. Hydro-meteorological conditions of 2017 flood and 2012 drought events are simulated and reveal that possible hypoxia occurs after flooding due to increasing temperature and turbidity, and DO concentration decreases significantly under heat and drought conditions. In conclusion, LSTM-based models utilizing wireless sensor data can be a timely and effective approach to make appropriate decisions on water resource management.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Load Profile Inpainting for Missing Load Data Restoration and Baseline Estimation

This paper introduces a Generative Adversarial Nets (GAN) based, Load Profile Inpainting Network (Load-PIN) for restoring missing load data segments and estimating the baseline for a demand response event. The inputs are time series load data before and after the inpainting period together with explanatory variables (e.g., weather data). Here, we propose a Generator structure consisting of a coarse network and a fine-tuning network. The coarse network provides an initial estimation of the data segment in the inpainting period. The fine-tuning network consists of self-attention blocks and gated convolution layers for adjusting the initial estimations. Loss functions are specially designed for the fine-tuning and the discriminator networks to enhance both the point-to-point accuracy and realisticness of the results. We test the Load-PIN on three real-world data sets for two applications: patching missing data and deriving baselines of conservation voltage reduction (CVR) events. We benchmark the performance of Load-PIN with five existing deep-learning methods. Our simulation results show that, compared with the state-of-the-art methods, Load-PIN can handle varying-length missing data events and achieve 15-30% accuracy improvement.

14 SOLAR ENERGY↗
DOE OSTI2022

Monitoring Carbon Storage Sites with Time-Lapse Gravity Surveys

Monitoring technologies are deployed on carbon storage sites to demonstrate that the injected CO2 is contained within the storage complex during and long after the active life of the project and that the CO2 behavior is conforming to expectations. Detection of gravity variations accompanying mass redistributions in the subsurface caused by fluid movements in reservoirs provides a unique means for monitoring the dynamics of a carbon sequestration site. Time-lapse gravity surveys have a long history of effectively monitoring temporal density changes in the subsurface. However, monitoring of carbon storage sites with time-lapse gravity surveys using surface or borehole measurements remains limited, although it gained more attention during the last decade. With the recent progresses in instrument capabilities, and the increased number of fluid storage sites (e.g., wastewater, CO2, natural gas, etc.), the gravity method (with its unique ability to remotely estimate both the mass of an injected fluid and its location) has a promising future in monitoring. This chapter provides background information regarding the gravity method and discusses its modeling and application to various carbon storage sites.

Carbon Sequestration, time-lapse gravity, monitori↗
DOE OSTI2023

Ion Mobility Spectrometry-Mass Spectrometry for High-Throughput Analysis

Ion mobility spectrometry is a widely used analytical technique providing gas phase separation of molecules. It has received increasing attention in the recent years with the advancement in technology development and the availability of commercial instruments. In this chapter, we introduced the ion mobility fundamental theory and provided examples of IMS applications, especially for isomer separation, collision cross section database generation, high throughput analysis workflows, software tools for IMS data analysis, and ongoing high resolution SLIM IMS development. While IMS is not yet routinely utilized in drug discovery and pharmaceutical industry, there has been increased interest in high throughput library screening and antibody characterization. With all the ongoing development in IMS technology and informatics, we foresee more and more exciting applications of high throughput IMS analysis in different fields including omics studies, drug discovery and clinical applications in the near future.

Ross, Dylan H.↗
DOE OSTI2024

Non-Invasive Biophysical Techniques to Monitor the Structural Plasticity of the Photosynthetic Machinery of Live Diatom Cells

The photosynthetic performance of diatoms depends largely on the organization and structural flexibility of their thylakoid membranes, the densely packed, highly organized membrane vesicles in which light reactions of photosynthesis occur. Different regulatory mechanisms that fine tune the photosynthetic functions affect the organization of the photosynthetic machinery at different levels of structural complexity, from the level of individual protein complexes to the macroarray of membrane proteins and the remodeling of the entire thylakoid membrane system. To monitor these reorganizations, non-invasive techniques are of special value. In this chapter, we focus our attention on three of these techniques, which have been demonstrated to provide unique and useful information on the structure and structural and functional plasticity of live diatom cells: (i) circular dichroism (CD) spectroscopy, which has provided unique information on the chiral (macro-)organization of protein complexes and on their rapid, reversible reorganizations, fine-tuning the light-harvesting processes, as well as on variations in the short-range excitonic interactions in the antenna complexes; (ii) small-angle neutron scattering (SANS), which has been used to determine the periodic organization of the thylakoid membranes and to monitor reversible ultrastructural changes on the time-scale of minutes, induced by variations in the environmental conditions such as changes in temperature or light intensity; and (iii) electrochromic shift absorbance transients (ΔA ECS ), a spectroscopic tool which has been shown to be capable of identifying distinct functional groups of the light-harvesting carotenoid fucoxanthin in different diatoms and in cells exposed to different light intensities. Future use of these techniques will most certainly contribute to the deeper understanding of key regulatory mechanisms of photosynthesis in diatoms.

Szabo, Milan↗
DOE OSTI2024

Circular Economy Contributions to Decarbonizing the US Steel Sector

The potential benefits of the circular economy (CE) for decarbonization have recently attracted much attention in the academic and grey literature. The department of energy (DOE) industrial decarbonization roadmap highlight that in addition to the four pillars (energy efficiency, electrification, low-carbon fuels, feedstocks, and energy sources, and carbon capture, utilization, and storage (CCUS)) identified in the roadmap, "scenarios [...] will need to be developed that incorporate materials efficiency and circular economy strategies". Moreover, the roadmap identifies hard-to-abate CO2 emissions across the five industrial subsectors included in the analysis (iron & steel, chemical, food & beverage, refining, and cement manufacturing). Thus, additional options to decarbonize the industrial sector, like the CE, could prove beneficial to address those hard-to-abate emissions. Moreover, they may be less costly, require less R&D or incentives, and be more readily adopted than other strategies such as energy efficiency and CCUS. While the Industrial Decarbonization Roadmap includes some CE strategies (e.g., the increasing market share of steel from electric arc furnaces - which incorporate steel scrap), a deeper dive into what role the CE could play in the United States (US) industrial decarbonization is needed. In this work, we present several industrial decarbonization scenarios that incorporate materials efficiency and circular economy strategies for the iron & steel subsector. The scenarios identify barriers (including technical limits and constraints), opportunities, and R&D needs.Moreover, we estimate the contribution of those scenarios to reducing the industrial sector's CO2 emissions and their potential synergies with the Industrial Decarbonization Roadmap's four pillars. Iron & steel manufacturing was responsible for 90 million tons of industrial CO2 emissions in 2020 - 7% of the industrial sector's total process- related emissions. Besides recycling - which presents technical limitations due to trace contaminants - CE strategies on the demand side could lower iron & steel manufacturing emissions (e.g., the development of lighter products or business models that encourage a more intensive use). This study reviews barriers, opportunities, and trade-offs for 5 material efficiency strategies. Many topics explored in the review call for further research.

barriers↗
DOE OSTI2023

Chapter 7: Recent Advances in Electrocatalysts for Hydrogen Oxidation Reaction in Alkaline Electrolytes

The recent development in electrocatalysts for the hydrogen oxidation reaction (HOR) attracted significant attention. In this sense, this chapter provides a fundamental understanding of the state-of-the-art PGM and non-PGM-based electrocatalysts for the HOR in alkaline conditions. Moreover, a strong focus has been made on the strategies to overcome the slow kinetics of the HOR to commercialize anion-exchange membrane fuel cells.

anion-exchange membrane↗
DOE OSTI2023

Additive Manufacturing of Self‐Sensing Carbon Fiber Composites

Carbon fiber-reinforced polymer (CFRP) composites have gained substantial attention across various industries owing to their exceptional mechanical properties and lightweight nature. The emergence of additive manufacturing technologies brings new opportunities to the industry, offering advantages such as design freedom, rapid prototyping, and customization. However, the fabrication of CFRP composites through 3D printing techniques poses challenges pertaining to low resolution and limitations in complex geometry realization. This work introduces digital light processing printing as a versatile, high-resolution method ideal for CFRP composite fabrication. Furthermore, the development and characterization of CFRP are focused on and the manipulation of mechanical properties through variations in matrix resins and fiber loadings is investigated, showcasing the versatility of CFRP composites for tailored applications. Additionally, the integration of self-sensing capabilities in CFRP structures is explored, which opens up opportunities for applications in smart components for automotive and structural health monitoring.

carbon fiber composites↗
DOE OSTI2020

Printing 2D Conjugated Polymer Monolayers and Their Distinct Electronic Properties

Abstract Recently, 2D monolayer films of conjugated polymers have gained increasing attention owing to the preeminence of 2D inorganic films that exhibit unique optoelectronic and mechanical properties compared to their bulk analogs. Despite numerous efforts, crystallization of semiconducting polymers into highly ordered 2D monolayer films still remains challenging. Herein, a dynamic‐template‐assisted meniscus‐guided coating is utilized to fabricate continuous, highly ordered 2D monolayer films of conjugated polymers over a centimeter scale with enhanced backbone π–π stacking. In contrast, monolayer films printed on solid substrates confer upon the 1D fiber networks strong alkyl side‐chain stacking at the expense of backbone packing. From single‐layers to multilayers, the polymer π‐stacks change from edge‐on to bimodal orientation as the film thickness reaches ≈20 nm. Spectroscopic and cyclic voltammetry analysis reveals an abrupt increase in J‐aggregation and absorption coefficient and a decrease in bandgap and highest occupied molecular orbital level until critical thickness, possibly arising from the straightened polymer backbone. This is corroborated by an abrupt increase in hole mobility with film thickness, reaching a maximum of 0.7 cm 2 V −1 s −1 near the critical thickness. Finally, fabrication of chemical sensors incorporating polymer films of various thicknesses is demonstrated, and an ultrahigh sensitivity of the ≈7 nm thick ultrathin film (bilayers) to 1 ppb ammonia is shown.

Kafle, Prapti↗
DOE OSTI2020

Ultralow-Strain Zn-Substituted Layered Oxide Cathode with Suppressed P2–O2 Transition for Stable Sodium Ion Storage

Layered transition metal oxides have drawn much attention as a promising candidate cathode material for sodium-ion batteries. However, their performance degradation originating from strains and lattice phase transitions remains a critical challenge. Herein, a high-concentration Zn-substituted Na x MnO 2 cathode with strongly suppressed P2–O2 transition is investigated, which exhibits a volume change as low as 1.0% in the charge/discharge process. Additionally, such ultralow strain characteristics ensure a stable host for sodium ion storage, which significantly improves the cycling stability and rate capability of the cathode material. Also, the strong coupling between the highly reversible capacity and the doping content of Zn in Na x MnO 2 is investigated. It is suggested that a reversible anionic redox reaction can be effectively triggered by Zn ions and is also highly dependent on the Zn content. Such an ion doping strategy could shed light on the design and construction of stable and high-capacity sodium ion host.

25 ENERGY STORAGE↗
DOE OSTI2020

Over 15% Efficiency Polymer Solar Cells Enabled by Conformation Tuning of Newly Designed Asymmetric Small‐Molecule Acceptors

Abstract The prosperous period of polymer solar cells (PSCs) has witnessed great progress in molecule design methods to promote power conversion efficiency (PCE). Designing asymmetric structures has been proved effective in tuning energy level and morphology, which has drawn strong attention from the PSC community. Two hepta‐ring and octa‐ring asymmetric small molecular acceptors (SMAs) (IDTP‐4F and IDTTP‐4F) with S‐shape and C‐shape confirmations are developed to study the relationship between conformation shapes and PSC efficiencies. The similarity of absorption and energy levels between two SMAs makes the conformation a single variable. Additionally, three wide‐bandgap polymer donors (PM6, S1, and PM7) are chosen to prove the universality of the relationship between conformation and photovoltaic performance. Consequently, the champion PCE afforded by PM7: IDTP‐4F is as high as 15.2% while that of PM7: IDTTP‐4F is 13.8%. Moreover, the S‐shape IDTP‐4F performs obviously better than their IDTTP‐4F counterparts in PSCs regardless of the polymer donors, which confirms that S‐shape conformation performs better than the C‐shape one. This work provides an insight into how conformations of asymmetric SMAs affect PCEs, specific functions of utilizing different polymer donors to finely tune the active‐layer morphology and another possibility to reach an excellent PCE over 15%.

Guo, Qing↗
DOE OSTI2020

Secondary Ion Mass Spectrometry (SIMS) for Chemical Characterization of Metal Halide Perovskites

Metal halide perovskite (MHP) solar cells have attracted much attention due to the rapidly growing power conversion efficiency that has reached 25.2% in a decade, comparable to established commercial photovoltaic modules. Compositional engineering is one of the most effective methods to boost the performance of MHP solar cells. Further improving the efficiency and the stability of MHP solar cells necessitates good understanding of the chemical–efficiency correlation and the chemical evolution during the degradation of MHP solar cells. In this regard, time-of-flight secondary ion mass spectrometry (ToF-SIMS) is a powerful tool to investigate the chemical aspect of MHPs and has played an important role in advancing the development of MHP optoelectronics. However, up to date, a review that can guide future utilization of ToF-SIMS in the MHP development is missing. In this paper, the capabilities of ToF-SIMS in MHP investigations are summarized and analyzed from simple material synthesis and chemical distribution to more complicated device operation mechanism and stability. The strength of ToF-SIMS in resolving important issues in this field, such as interface composition, ion migration, and degradation in MHP is highlighted. Finally, an outlook with an emphasis on making the utmost of ToF-SIMS in developing MHP devices is provided.

36 MATERIALS SCIENCE↗
DOE OSTI2020

Toward the Prediction and Control of Glass Transition Temperature for Donor–Acceptor Polymers

Semiconducting donor–acceptor (D–A) polymers have attracted considerable attention toward the application of organic electronic and optoelectronic devices. However, a rational design rule for making semiconducting polymers with desired thermal and mechanical properties is currently lacking, which greatly limits the development of new polymers for advanced applications. Here, polydiketopyrrolopyrrole (PDPP)-based D–A polymers with varied alkyl side-chain lengths and backbone moieties are systematically designed, followed by investigating their thermal and thin film mechanical responses. The experimental results show a reduction in both elastic modulus and glass transition temperature (Tg) with increasing side-chain length, which is further verified through coarse-grained molecular dynamics simulations. Informed from experimental results, a mass-per-flexible bond model is developed to capture such observation through a linear correlation between Tg and polymer chain flexibility. Using this model, a wide range of backbone Tg over 80 °C and elastic modulus over 400 MPa can be predicted for PDPP-based polymers. This study highlights the important role of side-chain structure in influencing the thermomechanical performance of conjugated polymers, and provides an effective strategy to design and predict Tg and elastic modulus of future new D–A polymers.

36 MATERIALS SCIENCE↗