Results for “RoBERTa”

NASA NTRSDate not supplied

Towards an Aviation Large Language Model by Fine-tuning and Evaluating Transformers

In the aviation domain, there are many applications for machine learning and artificial intelligence tools that utilize natural language. For example, there is a desire to know the commonalities in written safety reports such as voluntary post incidents reports or aerial wildfire operations reports to better understand the risks present. Another use-case is the possibility of extracting airspace procedures and constraints currently written in documents such as Letters of Agreement. These applications can benefit from the use of state-of-the-art natural language processing techniques when adapted to the language/phraseology specific to the aviation domain. This paper evaluates the viability of adaptation of NLP tools to the aviation domain by fine-tuning transformer based models using aviation data sets. In 2018, a novel language model based on neural units (also called transformers) was created and became known as “Bidirectional Encoder Representations from Transformers” or BERT. This architecture combined with large amounts of English training data and innovative semi-supervised training tasks set the standard for what would later emerge as Large Language Models. The performance of these models was further improved by hyperparameter tuning and refinement of the semi-supervised training task and resulted in “Robustly Optimized BERT Pre-training Approach through hyperparameter tuning” or RoBERTa models. These pre-trained Large Language Models proved to be useful for a wide variety of natural language processing tasks such as text classification and question answering through a process called fine-tuning. The transformer architecture with pre-trained weights served as the basis with the last few layers replaced with layers fine-tuned to perform a new task e.g., a layer that provides a label for the entire input text. This process of fine-tuning can also be used to adapt the models to new domains; e.g., BioBERT started with the pre-trained BERT model and was completed by additional fine-tuning and training on biomedical documents. Transformer-based architectures can also be used to create rich representations of text called embeddings which can serve as the input to other machine learning models. This allows simpler algorithms such as logistic regression to use context-rich representations of the text while still remaining quick to train and evaluate. In the world of aviation, there is a growing demand for natural language processing and understanding but the domain presents unique challenges. Due to the technical content (and specialized language) of most aviation documents, fine-tuning pre-trained Large Language Models to specific tasks has not met the benchmark on natural language processing tasks set by simpler models trained from scratch on the data. To address this deficiency, this paper evaluates the improvements from fine-tuning a Large Language Model on a large set of aviation documents using the original semi-supervised training tasks before performing specific natural language tasks. In fine-tuning, a domain-specific dataset is used on the original training task but with the pre-trained Large Language Model instead of starting from a random initialization. This approach allows the model to be adapted to the specific domain language without discarding the information gained from training on general English data. This paper utilized two major dataset types to train and assess the RoBERTa fine-tuning performance. The first are 7,057 Letters of Agreement which are Federal Aviation Administration (FAA) documents that formalize airspace operations across the national airspace system. They contain many examples of ‘aviation English’ using domain specific terminology and phrasing which serves as a representative basis to perform the semi-supervised fine-tuning. The second type is the 494 document classification labels to be used for evaluation. This down-stream evaluation aims to show the performance of the fine-tuned model, better understand how much data is needed for an effective fine-tuning, and how fine-tuning can be adapted for different applications in-the domain. After semi-supervised training, evaluation begins by encoding the documents for classification using the fine-tuned RoBERTa model. Then a logistic regression classifier is trained to label the document type and compared against our ground truth labels. This currently leads to a 82.8% accuracy on 10-fold cross validation showing improvement over baseline RoBERTa which achieved 81.0%. We plan to measure the improvements on additional tasks and it is expected that these improvements will lead to more robust models that can tackle the natural language processing challenges present in aviation datasets.

ATM↗

NASA NTRS2025

A Preliminary Study on the Feasibility of Large Language Models for Detecting Micro-Behaviors Among Team Members in Space Missions

Large-language models (LLMs) have been recently used for spoken language understanding (SLU) to infer meaning and semantics from speech in tasks such as speaker intent and sentiment classification. Due to being trained on large amounts of data, and their ability to understand context and relationships between words, LLMs are competent, enabling them to generalize across tasks without requiring many task-specific training samples. This research examines the feasibility of few-shot learning in LLMs for detecting subtle, brief, and possibly unconscious interactions between team members, called ``micro-behaviors," and provides insights into the appropriate design of LLMs for this task. Our data came from 5 teams participating in a 45-day mission at the US National Aeronautics and Space Administration’s (NASA) Human Exploration Research Analog (HERA). More specifically we used data collected from team interaction battery (TIB) tasks teams performed five times in-mission which comprise an average 1.5 hours of conversation data per day. Micro-behaviors were coded according to an adapted version of Smith & Griffins (2022) theoretical framework in terms of Violation (i.e., presence of valenced behavior, uplifting/positive or discouraging/negative), Intensity (i.e., force of behavior in terms of how uplifting or discouraging is the behavior), and Intent (i.e., motive of the behavior in terms of whether it was deliberate or unintentional). We explore the ability of LLMs to detect the presence and intensity of micro-behaviors. We examine employing and fine-tuning readily available LLMs (i.e., RoBERTa, DistilBERT), as well as prompting state-of-the-art sequence classification models (i.e., Llama-2, Llama-3). In a total of 13,058 conversational turns (17.8% uplifting, 3.3% discouraging, 75.76% neutral, 3.14% nulls), we compute the macro F1-score of the 3-way micro-behavior classification task (i.e., classifying among uplifting, discouraging, and neutral; 33% chance). Results indicate that the RoBERTa model achieves a F1-score of 36.2% (uplift: 43.3% precision (P), 15.1% recall (R); discourage: 20% P, 0.5% R). These results significantly improve when we augment the data via paraphrasing in the RoBERTa model, reaching a 41.2% macro F1-score (uplift: 37.7% P, 86.3% R; discourage: 3.5% P, 1.8% R). Finally, the Llama-2 model with 3-shot prompting yields 38% macro F1-score (uplift: 28.7% P, 20% R; discourage: 7.2% P, 18% R), which is slightly better compared to the RoBERTa model without data augmentation, highlighting the effectiveness of sequence classification models in detecting minority classes with a small sample size. Findings indicate that LLMs hold potential to detect subtle behaviors in conversations, which could be valuable in assessing team behavior in space exploration missions. Future studies will evaluate the performance of different LLM prompting strategies or fine-tuning methods.

Ankush Raut↗

NASA NTRSDate not supplied

Towards an Aviation Large Language Model by Fine-tuning and Evaluating Transformers

In the aviation domain, there are many applications for machine learning and artificial intelligence tools that utilize natural language. For example, there is a desire to know the commonalities in written safety reports such as voluntary post incidents reports or create more accurate transcripts of air traffic management conversations. Another use-case is the possibility of extracting airspace procedures and constraints currently written in documents such as Letters of Agreement (LOA) which is used as the evaluation case in this paper. These applications can benefit from the use of state-of-the-art Natural Language Processing (NLP) techniques when adapted to the language/phraseology specific to the aviation domain. This paper evaluates the viability of transferring pre-trained large language models to the aviation domain by adapting transformer based models using aviation datasets. This paper utilized two datasets to adapt a ‘Robustly Optimized Bidirectional Encoder Representations from Transformers Approach’ (RoBERTa) model and two down-stream classification tasks to assess its performance. These datasets are all built upon Letters of Agreement which are Federal Aviation Administration (FAA) documents that formalize airspace operations across the national airspace system. The first two datasets are used for the adaptation of RoBERTa to the aviation domain and were of different sizes to assess the number of documents needed to adapt to the aviation domain. They contain many examples of ‘aviation English’ using domain specific terminology and phrasing which serves as a representative basis to perform the unsupervised adaptation. The second dataset is a separate set of LOA documents with two sets of classification labels to be used for evaluation; one at the document level and one at the line level. These down-stream evaluations allowed the measurement of improvement by adapting RoBERTa. The accuracy increased by 4-6% on both tasks and the F1 score on the class of interest increased by 4-8% from the adaptation.

Air Traffic Management↗

NASA NTRSDate not supplied

Towards an Aviation Large Language Model by Fine-tuning and Evaluating Transformers

In the aviation domain, there are many applications for machine learning and artificial intelligence tools that utilize natural language. For example, there is a desire to know the commonalities in written safety reports such as voluntary post incidents reports or create more accurate transcripts of air traffic management conversations. Another use-case is the possibility of extracting airspace procedures and constraints currently written in documents such as Letters of Agreement (LOA) which is used as the evaluation case in this paper. These applications can benefit from the use of state-of-the-art Natural Language Processing (NLP) techniques when adapted to the language/phraseology specific to the aviation domain. This paper evaluates the viability of transferring pre-trained large language models to the aviation domain by adapting transformer based models using aviation datasets. This paper utilized two datasets to adapt a ‘Robustly Optimized Bidirectional Encoder Representations from Transformers Approach’ (RoBERTa) model and two down-stream classification tasks to assess its performance. These datasets are all built upon Letters of Agreement which are Federal Aviation Administration (FAA) documents that formalize airspace operations across the national airspace system. The first two datasets are used for the adaptation of RoBERTa to the aviation domain and were of different sizes to assess the number of documents needed to adapt to the aviation domain. They contain many examples of ‘aviation English’ using domain specific terminology and phrasing which serves as a representative basis to perform the unsupervised adaptation. The second dataset is a separate set of LOA documents with two sets of classification labels to be used for evaluation; one at the document level and one at the line level. These down-stream evaluations allowed the measurement of improvement by adapting RoBERTa. The accuracy increased by 4-6% on both tasks and the F1 score on the class of interest increased by 4-8% from the adaptation.

Air Traffic Management↗

NASA NTRSDate not supplied

2022 Spring Internship Exit Presentation

As efforts of the National Aeronautics and Space Administration (NASA) and the Federal Aviation Administration (FAA) continue to digitize the air traffic management (ATM) domain, there is countless times of need for downstream natural language processing (NLP) tasks such as named entity recognition, text summarization, classification, and more. Although there are a plethora of open-sourced pre-trained transformer models in the NLP field such as BERT, RoBERTa, XLNet, and GPT-3, these models are trained on general corpora and perform poorly on domain-specific terminology and phraseology seen in ATM documents such as Notice to Airmen (NOTAMs) and Letters of Agreement (LoA). Our proposed research objective will be to first gather a large corpus of air traffic management related documents, orders, notices, books, technical papers, conference papers, articles, and other miscellaneous sources of text data from the FAA, NASA, and accredited conference and publication societies. After gathering this data, many steps will have to be taken to collate and preprocess the data into a format understandable by our test transformer models. Thirdly, we will set up training pipelines to train the RoBERTa model on its unsupervised training task masked language modelling (MLM) using resources provided by the NASA Advanced Supercomputing (NAS) facilities. Finally, these fine-tuned transformer models will be evaluated on their performance on down-stream NLP tasks as mentioned above, to show whether they will be effective when working with ATM related data or not. Once complete, this model could be made open-sourced on the HuggingFace website, where the rest of the ATM community can access and utilize this tool.

NLP↗

NASA NTRS1986

Planet-crossing asteroids: Interrelationships within the solar system

Near-infrared reflectance spectra 0.6 to 2.5 micrometer were acquired of asteroids 1627 Ivar (Amor), 43 Ariadne, 335 Roberta, 386 Siegena and 695 Bella (3:1 Kirkwood Gap) with the IRTF, Mauna Kea. CCD spectra 0.5-1.0 micrometer were acquired of 1866 Sisyphus (Apollo), 17 Thetis, 695 Bella, 797 Montana, and 877 Walkure (3:1 Kirkwood Gap) using facilities at Cerro Tololo Inter-American Observatory. An upper limit on the production rate of CN in asteroid 3200 Phaeton of < 4 x 10 to the 23rd power sec was determined based on photometric measurements at 3871A using facilities at Lowell Observatory. This value is in the range of the lowest production rate measured for a comet, however, it does not constitute a positive detection of CN in this asteroid. A first attempt of look for companion objects or evidence of dust debris associated with this asteroid was made with a CCD camera. Whereas the search extended to 19th magnitude (corresponding to 150m and 330m for albedos of 0.15 and 0.03 respectively), a look close enough to the asteroid was not attained to definitively eliminate the presence of coorbiting dust debris.

Mcfadden, L. A.↗

NASA NTRSDate not supplied

Natural Language Processing Analysis of Notices to Airmen for Air Traffic Management Optimization

With new emerging technologies in the field of NLP, we explore their applications to digitize and analyze heritage Air Traffic Management (ATM) documents for planning and optimizing airspace operations. Specifically, this research focuses on harvesting semi-structured or un-structured information contained in Notices to Airmen (NOTAMs). Using NLP and other advanced data analytics, we will construct a data-driven framework which facilitates finding language patterns and the use of pretrained language models for classification and extraction of useful airspace constraints and restrictions. These may lead to tools that assist airspace users in understanding the constraints more efficiently, contributing to better route planning and safer execution. This paper explores three workflows entailing different NLP tasks. First, unsupervised techniques like word embedding and topic modeling are used for pattern finding and document classification. Second, a dataset is created by extracting information from the semi-structured NOTAM format as metadata for categorizing, visualizing, and extracting key entities driving NOTAM content. Third, modern pre-built deep learning based transformer models such as BERT, RoBERTa, and XLNet are evaluated on the question answering task, an even more robust approach to information extraction, as well as their respective fine-tuning tasks. In this work we include various performance metrics for the trained models to evaluate both accuracy and precision and we show that the models can be generalized for their respective tasks. The research work developed shows promise in uncovering trends in digital NOTAMs in the NAS and also offers a new framework for digitizing and inferring insights from free-form legacy NOTAMs, that are yet to be digitized.

Natural Language Processing↗

NASA NTRSDate not supplied

Natural Language Processing (NLP) Analysis of NOTAMs for Air Traffic Management Optimization

With new emerging technologies in the field of NLP, we explore their applications to digitize and analyze heritage Air Traffic Management (ATM) documents for planning and optimizing airspace operations. Specifically, this research focuses on harvesting semi-structured or un-structured information contained in Notices to Airmen (NOTAMs). Using NLP and other advanced data analytics, we will construct a data-driven framework which facilitates finding language patterns and the use of pretrained language models for classification and extraction of useful airspace constraints and restrictions. These may lead to tools that assist airspace users in understanding the constraints more efficiently, contributing to better route planning and safer execution. This paper explores three workflows entailing different NLP tasks. First, unsupervised techniques like word embedding and topic modeling are used for pattern finding and document classification. Second, a dataset is created by extracting information from the semi-structured NOTAM format as metadata for categorizing, visualizing, and extracting key entities driving NOTAM content. Third, modern pre-built deep learning based transformer models such as BERT, RoBERTa, and XLNet are evaluated on the question answering task, an even more robust approach to information extraction, as well as their respective fine-tuning tasks. In this work we include various performance metrics for the trained models to evaluate both accuracy and precision and we show that the models can be generalized for their respective tasks. The research work developed shows promise in uncovering trends in digital NOTAMs in the NAS and also offers a new framework for digitizing and inferring insights from free-form legacy NOTAMs, that are yet to be digitized. Video is an mp4 download, with a play time of 9 min 35 secs.

Natural Language Processing↗

NASA NTRSDate not supplied

Document Classification Techniques for Aviation Letters of Agreement

Often when working with technical documents, it is helpful to classify them into specific categories. In this paper, we conduct a thorough review of natural language processing techniques to perform this classification task on Letters of Agreement (LOAs), technical aviation documents outlining rules for utilizing US airspace. We evaluate multiple techniques, including Transfer Learning, for representing the text in the documents as embeddings: unigram and bigram Term Frequency Inverse Document Frequency (TFIDF), Word2Vec, Doc2Vec, GloVe and RoBERTa. We investigate a wide range of classification models: K-Nearest Neighbors, Random Forest, Support Vector Machines (SVM), Logistic Regression, Naive Bayes, Feed-Forward Neural Network, Convolutional Neural Networks (CNNs) and Long-Short Term Memory (LSTM). By comparing the different methods, we found the best overall approach for our task was to use unigram TFIDF representations with SVM while also gaining insight into how the other methodologies performed on a small technical datasets.

Aayushi Batra↗

NASA NTRSDate not supplied

Document Classification Techniques for Aviation Letters of Agreement

Often when working with historic air traffic management (ATM) documents, it is helpful to classify them into specific categories. In this paper, we conduct a thorough review of natural language processing techniques to perform this classification task on Letters of Agreement (LOAs), technical aviation documents outlining rules for utilizing US airspace. We evaluate multiple techniques for representing the text in the documents as embeddings: unigram and bigram Term Frequency Inverse Document Frequency (TFIDF), Word2Vec, Doc2Vec, GloVe and RoBERTa. We investigate a wide range of classification models: K-Nearest Neighbors, Random Forest, Support Vector Machines (SVM), Logistic Regression, Naive Bayes, Feed-Forward Neural Network, Convolutional Neural Networks (CNNs) and Long-Short Term Memory (LSTM). By comparing the different methods, we found the best overall approach for our task was to use unigram TFIDF representations with SVM while also gaining insight into how the other methodologies performed on a small technical datasets.

ATM↗

NASA NTRSDate not supplied

Document Classification Techniques for Aviation Letters of Agreement

Often when working with historic air traffic management (ATM) documents, it is helpful to classify them into specific categories. In this paper, we conduct a thorough review of natural language processing techniques to perform this classification task on Letters of Agreement (LOAs), technical aviation documents outlining rules for utilizing US airspace. We evaluate multiple techniques for representing the text in the documents as embeddings: unigram and bigram Term Frequency Inverse Document Frequency (TFIDF), Word2Vec, Doc2Vec, GloVe and RoBERTa. We investigate a wide range of classification models: K-Nearest Neighbors, Random Forest, Support Vector Machines (SVM), Logistic Regression, Naive Bayes, Feed-Forward Neural Network, Convolutional Neural Networks (CNNs) and Long-Short Term Memory (LSTM). By comparing the different methods, we found the best overall approach for our task was to use unigram TFIDF representations with SVM while also gaining insight into how the other methodologies performed on a small technical datasets.

ATM↗

NASA NTRS1976

Stability of elastic bending and torsion of uniform cantilever rotor blades in hover with variable structural coupling

The stability of elastic flap bending, lead-lag bending, and torsion of uniform, untwisted, cantilever rotor blades without chordwise offsets between the elastic, mass, tension, and areodynamic center axes is investigated for the hovering flight condition. The equations of motion are obtained by simplifying the general, nonlinear, partial differential equations of motion of an elastic rotating cantilever blade. The equations are adapted for a linearized stability analysis in the hovering flight condition by prescribing aerodynamic forces, applying Galerkin's method, and linearizing the resulting ordinary differential equations about the equilibrium operating condition. The aerodynamic forces are obtained from strip theory based on a quasi-steady approximation of two-dimensional unsteady airfoil theory. Six coupled mode shapes, calculated from free vibration about the equilibrium operating condition, are used in the linearized stability analysis. The study emphasizes the effects of two types of structural coupling that strongly influence the stability of hingeless rotor blades. The first structural coupling is the linear coupling between flap and lead-lag bending of the rotor blade. The second structural coupling is a nonlinear coupling between flap bending, lead-lag bending, and torsion deflections. Results are obtained for a wide variety of hingeless rotor configurations and operating conditions in order to provide a reasonably complete picture of hingeless rotor blade stability characteristics.

Hodges, D. H., Roberta.↗

NASA NTRS1987

Determination of hydrogen abundance in selected lunar soils

Hydrogen was implanted in lunar soil through solar wind activity. In order to determine the feasibility of utilizing this solar wind hydrogen, it is necessary to know not only hydrogen abundances in bulk soils from a variety of locations but also the distribution of hydrogen within a given soil. Hydrogen distribution in bulk soils, grain size separates, mineral types, and core samples was investigated. Hydrogen was found in all samples studied. The amount varied considerably, depending on soil maturity, mineral types present, grain size distribution, and depth. Hydrogen implantation is definitely a surface phenomenon. However, as constructional particles are formed, previously exposed surfaces become embedded within particles, causing an enrichment of hydrogen in these species. In view of possibly extracting the hydrogen for use on the lunar surface, it is encouraging to know that hydrogen is present to a considerable depth and not only in the upper few millimeters. Based on these preliminary studies, extraction of solar wind hydrogen from lunar soil appears feasible, particulary if some kind of grain size separation is possible.

Bustin, Roberta↗

NASA NTRS1986

Rare earth element patterns in Archean high-grade metasediments and their tectonic significance

REE data on metasedimentary rocks from two different types of high-grade Archean terrains are presented and analyzed. The value of REEs as indicators of crustal evolution is explained; the three geologic settings (in North America, Southern Africa, and Australia) from which the samples were obtained are described; and the data are presented in extensive tables and graphs and discussed in terms of metamorphic effects, the role of accessory phases, provenance, and tectonic implications (recycling, the previous extent of high-grade terrains, and a model of Archean crustal growth). The diversity of REE patterns in shallow-shelf metasediments is attributed to local provenance, while the Eu-depleted post-Archean patterns are associated with K-rich plutons from small, stable early Archean terrains.

Taylor, Stuart Ross↗

NASA NTRS1987

The enigmatic object variable A in M33

Variable A is a very luminous (about -9.5 mag), highly unstable star in M33. In 1950, it was one of the visibly brightest stars in M33, with the spectrum of a very luminous F supergiant. It then rapidly declined in brightness by 3.5 mag, becoming faint and red after slowly increasing in brightness during the previous 50 yr. This paper reports current spectroscopy and visual and infrared photometry. Variable A is still faint and red and now has the spectrum of an M supergiant. It also has a large infrared excess and is today as bright at 10 microns as it was at its visual maximum in 1950. Its energy distribution, luminosities, and mass-loss rate, which is quite large at 0.0002 solar mass/yr, are discussed. Its present late-type spectrum is probably produced in a cool pseudophotosphere, and it is shedding its mass in a high-density, low-velocity wind. A possible explanation is presented for Variable A's bizarre behavior as a massive star near the critical limit to its stabilty for its initial mass and temperature, where the balance between the radiation pressure, turbulence, and gravity of its atmosphere is very uncertain.

Humphreys, Roberta M.↗

NASA NTRS1988

Growth of the lower continental crust

One of the largest uncertainties in crustal composition and growth models is the nature of the lower continental crust. Specifically, by what processes is it formed and modified, and when is it formed, particularly in reference to the upper crust? The main reason for this lack of information is the scarcity of lower crustal rock samples. These are restricted to two types: rocks which outcrop in granulite facies terrains and granulite facies xenoliths which are transported to the earth's surface by young volcanics. The important conclusions arising from the xenolith studies are: the majority of mafic lower crustal xenoliths formed through cumulate process, resitic xenoliths are rare; and formation and metamorphism of the deep crust is intimately linked to igneous activity and/or orogeny which are manifest in one form or another at the earth's surface. Therefore, estimates of crustal growth based on surface exposures is representative, although the proportion of remobilized pre-existing crust may be significantly greater at the surface than in the deep crust.

Rudnick, Roberta L.↗

NASA NTRS1988

Methods of extracting hydrogen from lunar soil

Increasing interest in establishing a lunar base has generated considerable study on the utilization of lunar resources. Because of its importance in producing water, reducing oxides, and serving as a fuel for orbital tranfer vehicles, hydrogen is of prime importance as a resource. Lowman (1985) states that hydrogen would greatly facilitate the establishment of an autonomous permanent colony, and he calls hydrogen the most valuable lunar resource. Through the centuries, hydrogen has been embedded in lunar soil by the solar wind. The hydrogen can be extracted by heating the soil to 900 C (Carr et al, 1987). In order to obtain hydrogen on the lunar surface, an extraction method must be developed which will not only be reliable but also economically feasible. Three heating methods are examined for possible use in extracting hydrogen from lunar soil.

Bustin, Roberta↗

NASA NTRS1988

Lunar hydrogen: A resource for future use at lunar bases and space activities

Hydrogen abundances were determined for grain size separates of five lunar soils and one soil breccia. The hydrogen abundance studies have provided important baseline information for engineering models undergoing study at the present time. From the studies is appears that there is sufficient hydrogen present in selected lunar materials which could be recovered to support future space activities. It is well known that hydrogen can be extracted from lunar soils by heating between 400 and 800 C. Recovery of hydrogen for regolith materials would involve heating with solar mirrors and collecting the released hydrogen. Current baseline models for the lunar base are requiring the production of 1000 metric tons of oxygen per year. From this requirement it follows that around 117 metric tons per year of hydrogen would be required for the production of water. The ability to obtain hydrogen from the lunar regolith would assist in lowering the operating costs of any lunar base.

Gibson, Everett K., Jr.↗