Search NASA⌕ Search

SEARCH · Search NASA

Results for “attention mechanisms”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Short-Term Rainfall Prediction Based on Radar Echo Using an Improved Self-Attention PredRNN Deep Learning Model

Accurate short-term precipitation forecast is extremely important for urban flood warning and natural disaster prevention. In this paper, we present an innovative deep learning model named ISA-PredRNN (improved self-attention PredRNN) for precipitation nowcasting based on radar echoes on the basis of the advanced PredRNN-V2. We introduce the self-attention mechanism and the long-term memory state into the model and design a new set of gating mechanisms. To better capture different intensities of precipitation, the loss function with weights was designed. We further train the model using a combination of reverse scheduled sampling and scheduled sampling to learn the long-term dynamics from the radar echo sequences. Experimental results show that the new model (ISA-PredRNN) can effectively extract the spatiotemporal features of radar echo maps and obtain radar echo prediction results with a small gap from the ground truths. From the comparison with the other six models, the new ISA-PredRNN model has the most accurate prediction results with a critical success index (CSI) of 0.7001, 0.5812 and 0.3052 under the radar echo thresholds of 10 dBZ, 20 dBZ and 30 dBZ, respectively.

Wu, Dali (ORCID:0000000231177074)↗
DOE OSTI2024

Quantum Vision Transformers for Quark–Gluon Classification

We introduce a hybrid quantum-classical vision transformer architecture, notable for its integration of variational quantum circuits within both the attention mechanism and the multi-layer perceptrons. The research addresses the critical challenge of computational efficiency and resource constraints in analyzing data from the upcoming High Luminosity Large Hadron Collider, presenting the architecture as a potential solution. In particular, we evaluate our method by applying the model to multi-detector jet images from CMS Open Data. The goal is to distinguish quark-initiated from gluon-initiated jets. We successfully train the quantum model and evaluate it via numerical simulations. Using this approach, we achieve classification performance almost on par with the one obtained with the completely classical architecture, considering a similar number of parameters.

Comajoan Cara, Marçal (ORCID:0009000126263752)↗
DOE OSTI2021

Hardware-Based Emulator with Deep Learning Model for Building Energy Control and Prediction Based on Occupancy Sensors’ Data

Heating, ventilation, and air conditioning (HVAC) is the largest source of residential energy consumption. Occupancy sensors’ data can be used for HVAC control since it indicates the number of people in the building. HVAC and sensors form a typical cyber-physical system (CPS). In this paper, we aim to build a hardware-based emulation platform to study the occupancy data’s features, which can be further extracted by using machine learning models. In particular, we propose two hardware-based emulators to investigate the use of wired/wireless communication interfaces for occupancy sensor-based building CPS control, and the use of deep learning to predict the building energy consumption with the sensor data. We hypothesize is that the building energy consumption may be predicted by using the occupancy data collected by the sensors, and question what type of prediction model should be used to accurately predict the energy load. Another hypothesis is that an in-lab hardware/software platform could be built to emulate the occupancy sensing process. The machine learning algorithms can then be used to analyze the energy load based on the sensing data. To test the emulator, the occupancy data from the sensors is used to predict energy consumption. The synchronization scheme between sensors and the HVAC server will be discussed. We have built two hardware/software emulation platforms to investigate the sensor/HVAC integration strategies, and used an enhanced deep learning model—which has sequence-to-sequence long short-term memory (Seq2Seq LSTM)—with an attention model to predict the building energy consumption with the preservation of the intrinsic patterns. Because the long-range temporal dependencies are captured, the Seq2Seq models may provide a higher accuracy by using LSTM architectures with encoder and decoder. Meanwhile, LSTMs can capture the temporal and spatial patterns of time series data. The attention model can highlight the most relevant input information in the energy prediction by allocating the attention weights. The communication overhead between the sensors and the HVAC control server can also be alleviated via the attention mechanism, which can automatically ignore the irrelevant information and amplify the relevant information during CNN training. Our experiments and performance analysis show that, compared with the traditional LSTM neural network, the performance of the proposed method has a 30% higher prediction accuracy.

Ye, Zhijing↗
NASA NTRS1982

Damping of thermoacoustic oscillations

A commonly encountered and troublesome problem in cryogenic systems is related to the occurrence of thermoacoustic oscillations (TAO). The oscillations are accompanied by large heat fluxes which can cause large increases in the boiloff from dewars. Such a boiloff can lead to a serious degradation in performance. It appears, therefore, highly advisable to incorporate mechanisms for damping TAO in those parts of the system where there oscillations might occur. The present investigation is concerned with the criteria for the design of such damping mechanisms. Attention is given to the theory regrading the suppression of TAO, a damper consisting of a capillary with a surge tank, and porous solid dampers.

Tward, E.↗
NASA NTRS1991

Polar stratospheric clouds and the ozone hole

An account is given of physical processes governing the formation of stratospheric particles, in order to dramatize the interactions between polar stratospheric clouds and the Antarctic ozone-destruction mechanism. Attention is given to the successive stages of particle nucleation, condensation/evaporation and sedimentation/coagulation phenomena, and the ways in which polar stratospheric clouds are observed. Considerable evidence exists that polar stratospheric cloud particles are composed of nitric acid. The relatively small Arctic ozone hole depletion is due to the much smaller duration of Arctic stratospheric clouds.

Hamill, Patrick↗
NASA NTRS1993

Fiber pushout and interfacial shear in metal-matrix composites

Recent thin-slice pushout tests have suggested that MMC matrix-fiber interface failure processes depend not only on such intrinsic factors as bond strength and toughness, and matrix plasticity, but such extrinsic factors as specimen configuration, thermally-induced residual stresses, and the mechanics associated with a given test. After detailing the contrasts in fiber-pullout and fiber-pushout mechanics, attention is given to selected aspects of thin-slice fiber pushout behavior illustrative of the physical nature of interfacial shear response and its dependence on both intrinsic and extrinsic factors.

Koss, Donald A.↗
NASA NTRSDate not supplied

Flight Trajectory Prediction Based on Hybrid-Recurrent Networks

The development of future technologies for the National Airspace System (NAS) will be reliant on a new communications infrastructure capable of managing the limited available spectrum for communications among aircraft and ground systems. Emerging approaches to autonomous allocation of aviation spectrum mostlyrely on machine learning techniques, where 4D (longitude, latitude, altitude, time) trajectory prediction is an important data input to enable real-time resource allocation. This study explores and evaluates effective data sources and deep recurrent neural network techniques when determining flight trajectories. Specifically, data are collected and evaluated in a 100-day and 14-day period. Sources of data include NASA Sherlock Data Warehouse, MIT Lincoln Labs Corridor Integrated Weather Service (CIWS), and assorted NOAA weather datasets. Deep learning models for 4D predictions all utilize a hybrid-recurrent technique. A baseline model is considered via the convolutional-LSTM design from the existing literature. The modified design considers Gated Recurrent Units (GRU), Independently Recurrent Neural Networks (IndRNN), and stand-alone self-attention layers. Results indicatethe effectiveness of LSTM and GRUcells for state-of-the-art data processing (interpolation). Additionally, GRUs may be quickly trained with limited data, allowing for exacting improvements with optimizer selection. Attention mechanisms provide notable performance improvements to convolutional layers and may extend dimensional capabilities of a learning model. Finally, NOAA measurements provide only a supplemental value, requiring support from tailored measurements for Air Traffic Management.

Nathan Schimpf↗
DOE OSTI2023

Identification of Blue Horizontal Branch Stars with Multimodal Fusion

Blue Horizontal Branch stars (BHBs) are ideal tracers to probe the global structure of the milky Way (MW), and the increased size of the BHB star sample could be helpful to accurately calculate the MW's enclosed mass and kinematics. Large survey telescopes have produced an increasing number of astronomical images and spectra. However, traditional methods of identifying BHBs are limited in dealing with the large scale of astronomical data. A fast and efficient way of identifying BHBs can provide a more significant sample for further analysis and research. Therefore, in order to fully use the various data observed and further improve the identification accuracy of BHBs, we have innovatively proposed and implemented a Bi-level attention mechanism-based Transformer multimodal fusion model, called Bi-level Attention in the Transformer with Multimodality (BATMM). The model consists of a spectrum encoder, an image encoder, and a Transformer multimodal fusion module. The Transformer enables the effective fusion of data from two modalities, namely image and spectrum, by using the proposed Bi-level attention mechanism, including cross-attention and self-attention. As a result, the information from the different modalities complements each other, thus improving the accuracy of the identification of BHBs. The experimental results show that the F1 score of the proposed BATMM is 94.78%, which is 21.77% and 2.76% higher than the image and spectral unimodality, respectively. It is therefore demonstrated that higher identification accuracy of BHBs can be achieved by means of using data from multiple modalities and employing an efficient data fusion strategy.

79 ASTRONOMY AND ASTROPHYSICS↗
NASA NTRS1987

Mechanisms of selective attention and space motion sickness

The neural mismatch theory of space motion sickness asserts that the central and peripheral autonomic sequelae of discordant sensory input arise from central integrative processes falling to reconcile patterns of incoming sensory information with existing memory. Stated differently, perceived novelty reaches a stress level as integrative mechanisms fail to return a sense of control to the individual in the new environment. Based on evidence summarized here, the severity of the neural mismatch may be dependent upon the relative amount of attention selectively afforded to each sensory input competing for control of behavior. Components of the limbic system may play important roles in match-mismatch operations, be therapeutically modulated by antimotion sickness drugs, and be optimally positioned to control autonomic output.

NASA Discipline Neuroscience↗
DOE OSTI2022

Spatial Graph Attention and Curiosity-driven Policy for Antiviral Drug Discovery

We developed Distilled Graph Attention Policy Network (DGAPN), a reinforcement learning model to generate novel graph-structured chemical representations that optimize user-defined objectives by efficiently navigating a physically constrained domain. The framework is examined on the task of generating molecules that are designed to bind, noncovalently, to functional sites of SARS-CoV-2 proteins. We present a spatial Graph Attention (sGAT) mechanism that leverages self-attention over both node and edge attributes as well as encoding the spatial structure --- this capability is of considerable interest in synthetic biology and drug discovery. An attentional policy network is introduced to learn the decision rules for a dynamic, fragment-based chemical environment, and state-of-the-art policy gradient techniques are employed to train the network with stability. Exploration is driven by the stochasticity of the action space design and the innovation reward bonuses learned and proposed by random network distillation. In experiments, our framework achieved outstanding results compared to state-of-the-art algorithms, while reducing the complexity of paths to chemical synthesis.

Wu, Yulun↗
DOE OSTI2022

SPATIAL GRAPH ATTENTION AND CURIOSITY-DRIVEN POLICY FOR ANTIVIRAL DRUG DISCOVERY

We developed Distilled Graph Attention Policy Network (DGAPN), a reinforcement learning model to generate novel graph-structured chemical representations that optimize user-defined objectives by efficiently navigating a physically constrained domain. The framework is examined on the task of generating molecules that are designed to bind, noncovalently, to functional sites of SARS-CoV-2 proteins. We present a spatial Graph Attention (sGAT) mechanism that leverages self-attention over both node and edge attributes as well as encoding the spatial structure - this capability is of considerable interest in synthetic biology and drug discovery. An attentional policy network is introduced to learn the decision rules for a dynamic, fragment-based chemical environment, and state-of-the-art policy gradient techniques are employed to train the network with stability. Exploration is driven by the stochasticity of the action space design and the innovation reward bonuses learned and proposed by random network distillation. In experiments, our framework achieved outstanding results compared to state-of-the-art algorithms, while reducing the complexity of paths to chemical synthesis.

Wu, Y↗
NASA NTRS1992

Fracture mechanics expert system

Attention is given to fracture mechanics, an analytical method used extensively in the National Space Transportation System to conservatively predict the remaining service life of an article when a flaw or a material defect is detected. These analyses are performed on hardware containing material defects that have been detected by various nondestructive inspection techniques. An expert system being developed to streamline the process so that hardware dispositions may be obtained in a timely and consistent manner is discussed. The expert system reduces the potential for errors due to the manual transcription between the various software programs involved in completing a fracture mechanics analysis. NEXPERT Object, the expert system development shell selected for this purpose, allows the various software programs used in fracture mechanics analyses to be accessed and manipulated from the same platform.

Powers, E.↗
DOE OSTI2026

Attention-based 3D – convolutional neural network model for mechanical property predictions using visible light images in metal additive manufacturing

Additive manufacturing (AM), while commonly used for rapid prototyping and creating components with complex geometries, has not been widely adopted for critical applications across the aerospace, automotive, defense, energy, and medical industries. This is, in part, due to the challenges of controlling flaws and uncertainty in the mechanical behavior of additively manufactured components. In recent years, there has been an increase in research aimed at predicting the final mechanical properties of additively manufactured components during the printing process. To address these issues, a 3D-CNN model was trained using low-cost in situ visible-light camera data, anomaly classifications, and the chosen process parameters to predict the ultimate tensile strength (UTS), yield strength (YS), total elongation (TE), and uniform elongation (UE). The 3D-CNN layers of the model employed attention mechanisms to prioritize features in the data, thereby improving prediction accuracy. Furthermore, the effect of each process parameter and anomaly class is investigated using attention-based dynamic sigmoid weighted gates to interpret the influence each class has on the final prediction. Different combinations of the in situ data were fed into the 3D-CNN, with varying amounts of image layers, to determine the ideal combination for predicting mechanical properties in situ. Here, the 3D-CNN model achieved mean absolute percentage errors (MAPE) below 5% for both UTS and YS while using only a single camera input and under half of the available image layers.

36 MATERIALS SCIENCE↗
DOE OSTI2023

A comparison of histopathology imaging comprehension algorithms based on multiple instance learning

Whole slide imaging (WSI), also called digital virtual microscopy, is a new imaging modality. It allows for the application of AI and machine learning methods to cancer pathology to help establish a means for the automatic diagnosis of cancer cases. However, designing machine-learning models for WSI is computationally challenging due to its required ultra-high resolution. The current state-of-the-art models use multiple instance learning (MIL). MIL is a weakly-supervised learning method in which the model uses an array of inferences from many smaller instances to make a final classification about the entire set. In the context of WSI, researchers divide the ultra-high-resolution image into many patches. The model then classifies the slide based on an array of inferences from the patches. Among several ways of making the final classification, attention-based mechanisms have resulted in superb accuracy scores. The Transformer, one attention-based algorithm, has reported substantial improvements for WSI comprehension tasks. In this project, we studied and compared several WSI comprehension algorithms. We used the following three datasets: CAMELYON16+17, TCGALung, and TCGA-Kidney. We found that attention-based MIL algorithms performed better than standard MIL algorithms for classifying WSI images, achieving a higher mean accuracy and AUC. However, none of the attention-based algorithms performed significantly better than the others, reporting accuracy scores that varied widely. Presumably, it is due to the limited availability of training samples in the data corpus. Since it is not easy to increase the samples from human subjects, some machine learning techniques like transfer learning could help mitigate this issue.

Saunders, Adam↗
DOE OSTI2021

Attention-based quantum tomography

Abstract With rapid progress across platforms for quantum systems, the problem of many-body quantum state reconstruction for noisy quantum states becomes an important challenge. There has been a growing interest in approaching the problem of quantum state reconstruction using generative neural network models. Here we propose the ‘attention-based quantum tomography’ (AQT), a quantum state reconstruction using an attention mechanism-based generative network that learns the mixed state density matrix of a noisy quantum state. AQT is based on the model proposed in ‘Attention is all you need’ by Vaswani et al (2017 NIPS ) that is designed to learn long-range correlations in natural language sentences and thereby outperform previous natural language processing (NLP) models. We demonstrate not only that AQT outperforms earlier neural-network-based quantum state reconstruction on identical tasks but that AQT can accurately reconstruct the density matrix associated with a noisy quantum state experimentally realized in an IBMQ quantum computer. We speculate the success of the AQT stems from its ability to model quantum entanglement across the entire quantum system much as the attention model for NLP captures the correlations among words in a sentence.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1993

The Stimulus Movement Effect: Allocation of Attention or Artifact?

In previous reports, including one by the author, learning has been shown to benefit by having discriminanda move rather than remain stationary. This stimulus movement effect might be attributed to several theoretical mechanisms, including attention, topological memory, and exposure duration. The series of experiments reported in this article was designed to Contrast these potential explanatory factors. Ten rhesus monkeys (Macaca mulatta) were tested on a variety of computerized tasks in which the stimuli remained stationary, flashed, or moved at systematically varied speeds. Performance was significantly best when the sample stimulus moved quickly and was poorest when the stimulus remained stationary. Further analysis of these data and other previously published data revealed that the distribution of the stimulus movement effect across trials supported an attention allocation interpretation.

Washburn, David A.↗
NASA NTRS1982

The faint young sun-climate paradox - Continental influences

We examine the various mechanisms which have been proposed to compensate for the climatic effects of a 30% increase in the solar luminosity over the past 4 1/2 billion years. Although atmospheric greenhouse effects have received most attention, other mechanisms may have played a role of comparable importance. In particular, we note that the development of continents during the past 2 1/2 billion years could have had a significant secular effect on the atmosphere-ocean heat transport system. As a result, past climates may have been less susceptible to complete freeze-over. A simple energy balance model is used to demonstrate the magnitude of this effect. Because the CO2 greenhouse effect is not the only means of compensating for solar evolution, the faint-young-sun problem should not be used to infer past levels of atmospheric CO2.

Endal, A. S.↗
NASA NTRS1988

Sensor deployment mechanism for Surfer satellite

A design is presented for a sensor-deployment mechanism to be used aboard the Surfer satellite, from which scientific instruments will be extended to study the earth ionosphere during the Space Shuttle Tether Experiment. The design discussed uses four folding arms to extend the radial sensors, as well as two storable tubular extendible members or spirally wound self-extending tube booms to project the axial sensors outward. The design solution chosen, a folding arm, is discussed in detail with attention to mechanical operation and component functions. Test program results are presented.

Dill, Robert↗