Search NASA⌕ Search

SEARCH · Search NASA

Results for “Benchmark dataset”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 541 records · Page 30
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Scaling laws in jet classification

We demonstrate the emergence of scaling laws in the benchmark top versus QCD jet classification problem in collider physics. Six distinct physically-motivated classifiers exhibit power-law scaling of the binary cross-entropy test loss as a function of training set size, with distinct power law indices. This result highlights the importance of comparing classifiers as a function of dataset size rather than for a fixed training set, as the optimal classifier may change considerably as the dataset is scaled up. We speculate on the interpretation of our results in terms of previous models of scaling laws observed in natural language and image datasets.

Batson, Joshua↗
NASA NTRS2026

Hydrology Copilot: A Cloud-Native Ai System for Hydrological Data Analysis

The emergence of AI-driven Earth observation systems promises to broaden access to petabyte-scale geospatial data beyond domain specialists. However, translating this vision into operational scientific infrastructure requires addressing fundamental challenges in data virtualization, code transparency, and domain-specific reasoning. We present Hydrology Copilot, a cloud-native AI framework for natural-language-driven analysis of Earth observation data. To demonstrate operational capabilities at scale, we implement the system using NASA's North American Land Data Assimilation System version 3 (NLDAS-3), which provides surface meteorological forcing and land-surface model output across North and Central America at 1-km resolution, from which drought diagnostics are derived. The system integrates five core contributions: (1) scalable data virtualization using Kerchunk-based cloud optimized access, achieving a 1.5 to 4.6 times improvement in I/O latency across benchmark queries spanning regional single-day extractions (4.6 times speedup) to continental monthly aggregations (1.5 times speedup); (2) transparent code generation through Microsoft Azure AI Foundry agents that expose executable Python workflows for scientific verification; (3) persistent conversational memory enabling multi-turn analytical discourse across sessions; (4) intelligent query validation that enforces dataset boundaries and resolves ambiguous requests before execution; and (5) a multi-agent architecture coordinating query parsing, code generation, and visualization. We evaluate the system through drought-monitoring workflows, demonstrating reliable code generation, accurate results validated against reference computations and the operational U.S. Drought Monitor, and efficient operation across increasingly complex tasks. By bridging natural-language interfaces with rigorous hydrological analysis, Hydrology Copilot advances beyond proof-of-concept demonstrations to provide a deployable framework for operational Earth science applications.

Data virtualization↗
DOE OSTI2025

High-Fidelity, Large-Scale, Realistic Dataset Development

The final report summarizes the work performed for supporting the ARPA-E Grid Optimization Competition (Challenge 2 and Challenge 3) within the stated period. Challenge 2 For the challenge period, the main responsibility of the team is to investigate, gen- erate, and deliver parts of the data sets for the competition, based on the competition model for Challenge 2, existing data sets from Challenge 1, and data source supplied by other data set teams. Challenge 3 For the challenge period, the main responsibility of the team is to propose, create, deliver, and maintain the data format during the competition period. The data format will specify how the benchmark data will be represented and communicated to competitors. It will also specify how competitors should report back the solutions. The data format will be closely aligned with the problem formulation (maintained by the formulation team) and the solution validation process (maintained by the validation team). Our team is also responsible in investigating, generating, and delivering parts of the data sets for the competition. The data sets will be created based on the competition model for Challenge 3, existing data sets from Challenge 1 and Challenge 2, and data source supplied by other data set teams.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2024

Predicting metabolic modules in incomplete bacterial genomes with MetaPathPredict

The reconstruction of complete microbial metabolic pathways using ‘omics data from environmental samples remains challenging. Computational pipelines for pathway reconstruction that utilize machine learning methods to predict the presence or absence of KEGG modules in incomplete genomes are lacking. Here, we present MetaPathPredict, a software tool that incorporates machine learning models to predict the presence of complete KEGG modules within bacterial genomic datasets. Using gene annotation data and information from the KEGG module database, MetaPathPredict employs deep learning models to predict the presence of KEGG modules in a genome. MetaPathPredict can be used as a command line tool or as a Python module, and both options are designed to be run locally or on a compute cluster. Benchmarks show that MetaPathPredict makes robust predictions of KEGG module presence within highly incomplete genomes.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

BUTTER-E - Energy Consumption Data for the BUTTER Empirical Deep Learning Dataset

The BUTTER-E - Energy Consumption Data for the BUTTER Empirical Deep Learning Dataset adds node-level energy consumption data from watt-meters to the primary sweep of the BUTTER - Empirical Deep Learning Dataset. This dataset contains energy consumption and performance data from 63,527 individual experimental runs spanning 30,582 distinct configurations: 13 datasets, 20 sizes (number of trainable parameters), 8 network "shapes", and 14 depths on both CPU and GPU hardware collected using node-level watt-meters. This dataset reveals the complex relationship between dataset size, network structure, and energy use, and highlights the impact of cache effects. BUTTER-E is intended to be joined with the BUTTER dataset (see "BUTTER - Empirical Deep Learning Dataset on OEDI" resource below) which characterizes the performance of 483k distinct fully connected neural networks but does not include energy measurements.

Array↗
DOE OSTI2021

Nitrogen and phosphorus cycling in an ombrotrophic peatland: a benchmark for assessing change

Aims Slow decomposition and isolation from groundwater mean that ombrotrophic peatlands store a large amount of soil carbon (C) but have low availability of nitrogen (N) and phosphorus (P). To better understand the role these limiting nutrients play in determining the C balance of peatland ecosystems, we compile comprehensive N and P budgets for a forested bog in northern Minnesota, USA. Methods N and P within plants, soils, and water are quantified based on field measurements. The resulting empirical dataset are then compared to modern-day, site-level simulations from the peatland land surface version of the Energy Exascale Earth System Model (ELM-SPRUCE).

Salmon, Verity G.↗
DOE OSTI2021

Benchmarking graph neural networks for materials chemistry

Graph neural networks (GNNs) have received intense interest as a rapidly expanding class of machine learning models remarkably well-suited for materials applications. To date, a number of successful GNNs have been proposed and demonstrated for systems ranging from crystal stability to electronic property prediction and to surface chemistry and heterogeneous catalysis. However, a consistent benchmark of these models remains lacking, hindering the development and consistent evaluation of new models in the materials field. Here, we present a workflow and testing platform, MatDeepLearn, for quickly and reproducibly assessing and comparing GNNs and other machine learning models. We use this platform to optimize and evaluate a selection of top performing GNNs on several representative datasets in computational materials chemistry. From our investigations we note the importance of hyperparameter selection and find roughly similar performances for the top models once optimized. We identify several strengths in GNNs over conventional models in cases with compositionally diverse datasets and in its overall flexibility with respect to inputs, due to learned rather than defined representations. Meanwhile several weaknesses of GNNs are also observed including high data requirements, and suggestions for further improvement for applications in materials chemistry are discussed.

36 MATERIALS SCIENCE↗
NASA NTRS2012

The Violent Early Solar System, as Told by Lunar Sample Geochronology

One of the legacies of the samples collected by the Apollo and Luna missions is the link forged between radiometric ages of rocks and relative ages according to stratigraphic relationships and impact crater size-frequency distributions. Our current understanding of the history of the inner solar system is based on the relative chronology of individual planets, tied to the absolute geochronology of the Moon via these important samples. Samples from these nearside locations reveal a preponderance of impact-disturbed or recrystallized ages between 3.75 and 3.95 billion years. Argon and lead loss (and correlated disturbances in the Rb-Sr system) have been attributed to metamorphism of the lunar crust by an enormous number of impacts in a brief pulse of time, called the Lunar Cataclysm or Late Heavy Bombardment. Subsequent high-precision geochronometric analyses of Apollo samples and lunar highlands meteorites show a wider range of ages, but very few older than 4 Ga. The paucity of ancient impact melt rocks has been interpreted to mean that either that most impact basins formed at this time, or that ejecta from the large, near-side, young basins dominates the Apollo samples. Selenochronology is getting more complicated: new results question meaning of sample ages, crater counts, crater production functions, and the solar system itself. Improved geological mapping of lunar geologic units and boundaries using multiple remote sensing datasets. High-resolution image-based crater counting of discrete geologic units and relating them to location. Improved understanding of the regolith thickness and its global variation (GRAIL). Tying the sampling of impact-melt rocks to the lunar impact flux. Using improved techniques (magnetic fields, diffusion studies, isotopic analysis) on existing samples. New sample return from benchmark craters, particularly SPA, which appears in 2013 Decadal Survey.

Cohen, Barbara↗
DOE OSTI2022

Evaluation of Ocean Biogeochemistry and Carbon Cycling in CMIP Earth System Models With the International Ocean Model Benchmarking (IOMB) Software System

Abstract The International Ocean Model Benchmarking (IOMB) software package is a new community resource that we use here to evaluate surface and upper ocean biogeochemical variables and integrated anthropogenic carbon uptake from earth system models (ESMs) contributing to the 5th and 6th phases of the Coupled Model Intercomparison Project (CMIP5 and CMIP6). IOMB generates graphics and tables for systematically comparing model predictions against multiple datasets. Our analysis reveals some improvement in the multi‐model mean from CMIP5 to CMIP6 for most of the variables we examined. Compared to data‐constrained estimates of ocean anthropogenic carbon uptake for the 1994–2007 period, negative biases exist for many models between 30 and 50°S. Global model estimates of anthropogenic carbon uptake for the same period do not change significantly from CMIP5 to CMIP6, with the combined ensemble mean estimate of 27.8 ± 0.5 Pg C lower than a data‐constrained estimate of 33.0 ± 4.0 Pg C. At the same time, the change in the natural carbon inventory from CMIP is estimated to be a source of 0.7 ± 0.3 Pg C, which is considerably smaller in magnitude than a data‐constrained estimate of 5.0 ± 3.0 Pg C. With chlorofluorocarbon (CFC) predictions available for several models, we demonstrate that negative anthropogenic dissolved inorganic carbon biases coincide with negative biases in CFC concentration, highlighting the importance of weak exchange between the surface and interior ocean in regulating rates of anthropogenic carbon uptake. To examine the robustness of this attribution across the CMIP models, we calculate the global vertical temperature gradient between 200 and 1,000 m as a metric for global stratification and exchange between the surface and deeper waters. We find a linear relationship between the bias of the vertical temperature gradients and the bias in global anthropogenic carbon uptake, consistent with the hypothesis that model biases in anthropogenic carbon uptake are related to biases in surface‐to‐interior exchange by physical processes.

58 GEOSCIENCES↗
DOE OSTI2025

A comparison of probabilistic generative frameworks for molecular simulations

Generative artificial intelligence is now a widely used tool in molecular science. Despite the popularity of probabilistic generative models, numerical experiments benchmarking their performance on molecular data are lacking. Here, in this work, we introduce and explain several classes of generative models, broadly sorted into two categories: flow-based models and diffusion models. We select three representative models: neural spline flows, conditional flow matching, and denoising diffusion probabilistic models, and examine their accuracy, computational cost, and generation speed across datasets with tunable dimensionality, complexity, and modal asymmetry. Our findings are varied, with no one framework being the best for all purposes. In a nutshell, (i) neural spline flows do best at capturing mode asymmetry present in low-dimensional data, (ii) conditional flow matching outperforms other models for high-dimensional data with low complexity, and (iii) denoising diffusion probabilistic models appear the best for low-dimensional data with high complexity. Our datasets include a Gaussian mixture model and the dihedral torsion angle distribution of the Aib9 peptide, generated via a molecular dynamics simulation. We hope our taxonomy of probabilistic generative frameworks and numerical results may guide model selection for a wide range of molecular tasks.

Artificial intelligence↗
DOE OSTI2025

Airborne imaging spectroscopy surveys of Arctic and boreal Alaska and northwestern Canada 2017–2023

Since 2015, NASA’s Arctic Boreal Vulnerability Experiment (ABoVE) has investigated how climate change impacts the vulnerability and/or resilience of the permafrost-affected ecosystems of Alaska and northwestern Canada. ABoVE conducted extensive surveys with the Next Generation Airborne Visible/Infrared Imaging Spectrometer (AVIRIS-NG) during 2017, 2018, 2019, and 2022 and with AVIRIS-3 in 2023 to characterize tundra, taiga, peatlands, and wetlands in unprecedented detail. The ABoVE AVIRIS dataset comprises ~1700 individual flight lines covering ~120,000 km 2 with nominal 5 m × 5 m spatial resolution. Data include individual transects to capture important gradients like the tundra-taiga ecotone and maps of up to 10,000 km 2 for key study areas like the Mackenzie Delta. The ABoVE AVIRIS surveys enable diverse ecosystem science, provide crucial benchmark data for validating retrievals from the PACE, PRISMA, and EnMAP satellite sensors and help prepare for the SBG and CHIME missions. This paper guides interested researchers to fully explore the ABoVE AVIRIS spectral imagery and complements our guide to the ABoVE airborne synthetic aperture radar surveys.

Miller, Charles E. [California Institute of Techno↗
DOE OSTI2024

EV Forecasting-Based Model Predictive Control for Distribution System Congestion Mitigation

The uncoordinated charging of electric vehicles (EVs) in time and space brings congestion issues to the distribution network. This paper proposes an EV charging demand forecasting-based model predictive control (MPC) method for distribution system congestion management. To effectively forecast the time-series EV station charging demand, a hybrid forecasting model that integrates the long short-term memory network (LSTM) and Transformer is proposed. The Transformer-LSTM model is trained using a one-year real historical charging dataset of EV stations to forecast future charging demand in 15-minute intervals. This informs the MPC for distribution network congestion management and minimization of PV curtailment. Numerical results carried out on the modified IEEE 123-bus distribution system demonstrate that the proposed method can effectively resolve line congestion issues through EV smart charging and PV curtailment while outperforming other benchmarks.

ADVANCED PROPULSION SYSTEMS,SOLAR ENERGY↗
DOE OSTI2025

Advancing the Frontiers of Deep Learning for Low-Dose 3D Cone-Beam CT Reconstruction

X-ray computed tomography (CT) is an important noninvasive medical imaging modality for studying the structural details of internal organs. Image reconstruction in CT is an inverse problem of recovering an object's internal structure from the absorption profile of X-ray beams (sinogram) measured using a detector. The classical variational approach for CT reconstruction minimizes an energy functional using an appropriate iterative algorithm. Motivated by the success of deep learning (DL), researchers have begun to leverage training data and enhanced computing capabilities in recent years to produce high-fidelity reconstructed images. Nonetheless, much of the academic research in DL algorithms for CT has focused primarily on the two-dimensional setting (with simplified forward operators and noise model) for proofs-of-concept, and a comprehensive benchmarking of various classical and data-driven CT reconstruction approaches has not beenundertaken. The key objective of our CT reconstruction grand challenge was to promote methodological advancements for both classical and DL-based approaches for clinical CT with a reasonably accurately simulated 3D CT forward operator and noise model. We have utilized the publicly available LIDC-IDRI dataset and simulated sinograms and FDK images corresponding to two dose levels (clinical- and low-dose, constituting two tracks of the challenge) starting from the normal-dose images as the ground truth. In this paper, we summarize the motivation, context, and results of our challenge, and highlight the future research directions in DL for clinical CT.

X-ray tomography↗
DOE OSTI2020

Development of a Comprehensive Two-Phase Flow Database for the Validation of NEK-2P

Three-dimensional (3-D) two-phase Computational Fluid Dynamics (CFD) codes are emerging as a powerful and potentially practical tool for applications in which detailed local flow information is needed. However, two-phase flow models and associated closure relations are not well established for CFD applications, which is partly due to the lack of high-quality validation data. The main objective of this work is to develop a comprehensive database of two-phase flows that can be used to validate two-phase CFD codes such as NEK-2P. In this project, four advanced local measurement systems, including Particle Image Velocimetry and Planar Laser-Induced Fluorescence (PIV-PLIF), high-speed imaging, x-ray densitometry, and multi-sensor conductivity probe are employed to measure the local two-phase flow parameters of both gas and liquid phases. By combining these techniques, the local void fraction, bubble velocity, interfacial area concentration, bubble frequency, liquid velocity, turbulence intensity, etc., in various two-phase flow regimes can be obtained. These local measurement techniques are first used in a 25.4 mm circular pipe test section. Seven air-water two-phase flow conditions spanning the bubbly, slug, churn-turbulent, and annular flow regimes are measured in this facility. The obtained database contains the radial profiles of both gas- and liquid-phase parameters at three axial locations along the test section. The second facility used in this work contains a 30 mm × 10 mm rectangular test section, in which three two-phase flow and two single-phase flow conditions are measured. Two-dimensional distributions of local two-phase flow parameters in the cross-sectional plane are measured in this facility at three axial locations as well. A facility featuring a 3×3 electrically heated rod bundle is also designed and being constructed in this project. This facility is specially designed for optical measurements and is expected to provide high-quality boiling data in the future. Preliminary analyses have been performed for the data taken in the 25.4 mm circular pipe. Both center-peaked and wall-peaked void fraction profiles have been observed in the data depending on the two-phase flow conditions and/or developing lengths. The 1-D drift-flux model was evaluated with the newly obtained datasets, in which both gas- and liquid-phases data were directly measured. The distribution parameter model has been optimized based on a new void-profile classification method proposed in this study. The optimized drift-flux model shows a significant improvement in predicting both gas velocity and void fraction. The measured liquid-phase turbulence was used to benchmark Sato’s turbulence model considering the bubble-induced shear stress for the three tested bubbly flows. The benchmark results showed good agreement between the PIV measurements and model predictions. In the bubbly flows tested that have low void fractions less than 3%, the effect of the bubble-induced turbulence was found not significant. However, the bubble-induced shear stress becomes important with the increase of the void fraction. The Conjugate Heat Transfer (CHT) model was developed and implemented in NEK-2P. This model allows the coupled simulation of the solid domain and two-phase fluid domain, allowing the specification of realistic boundary conditions. The CHT implementation was verified first with non-boiling simulations. The predicted temperatures in the fluid domain were shown to be identical in simulations with or without the CHT model. The CHT model was then validated through simulations of three Becker benchmark CHF tests performed under both Dryout (DO) and Departure from Nucleate Boiling (DNB) conditions. Reasonably good agreement was obtained between calculated wall temperatures and corresponding experimental data.

42 ENGINEERING↗
DOE OSTI2025

Generalizable, fast, and accurate DeepQSPR with fastprop

Abstract Quantitative Structure–Property Relationship studies (QSPR), often referred to interchangeably as QSAR, seek to establish a mapping between molecular structure and an arbitrary target property. Historically this was done on a target-by-target basis with new descriptors being devised to specifically map to a given target. Today software packages exist that calculate thousands of these descriptors, enabling general modeling typically with classical and machine learning methods. Also present today are learned representation methods in which deep learning models generate a target-specific representation during training. The former requires less training data and offers improved speed and interpretability while the latter offers excellent generality, while the intersection of the two remains under-explored. This paper introduces , a software package and general Deep-QSPR framework that combines a cogent set of molecular descriptors with deep learning to achieve state-of-the-art performance on datasets ranging from tens to tens of thousands of molecules. provides both a user-friendly Command Line Interface and highly interoperable set of Python modules for the training and deployment of feedforward neural networks for property prediction. This approach yields improvements in speed and interpretability over existing methods while statistically equaling or exceeding their performance across most of the tested benchmarks. is designed with Research Software Engineering best practices and is free and open source, hosted at github.com/jacksonburns/fastprop.

Burns, Jackson W. (ORCID:0000000206579426)↗
DOE OSTI2026

Ultra-High Vacuum Outgassing Characterization of Thermally Processed Low-Carbon Steel for Advanced Particle Accelerator and Gravitational Wave Detector Applications

This dissertation investigated AISI 1020 low-carbon steel as an alternative vacuum chamber material to conventional stainless steel for ultra-high vacuum (UHV) and extreme-high vacuum (XHV) applications. After a 400 °C/48 h bake, AISI 1020 tube chambers achieved a hydrogen outgassing rate of 2.4 × 10¿¹6 Torr·L·s¿¹·cm¿², approximately 2,300 times lower than the prebaked 316L stainless-steel comparator, among the lowest hydrogen outgassing rates ever reported for an uncoated metallic vacuum chamber. Bare and magnetite-coated AISI 1020 chambers were then compared using throughput and rate-of-rise methods. The magnetite coating yielded 5× lower water outgassing at room temperature, but this advantage disappeared after 80 °C baking. After full thermal conditioning (400 °C/48 h prebake followed by 150 °C/96 h and 200 °C/110 h), bare steel achieved 25× lower hydrogen outgassing than the magnetite-coated chamber (9.6 × 10¿¹6 Torr·L·s¿¹·cm¿²) and >99% H2 purity with carbon species below RGA detection. Monte Carlo molecular flow simulations of a CEBAF photogun beamline (96 scenarios) showed that replacing 304L stainless steel with AISI 1020 reduces equilibrium H2 pressure by a factor of 833; a single 304L electrode contributes 98.8% of the gas load despite occupying only 9.1% of the internal surface area. A 500-m Einstein Telescope beampipe screening showed that corrugated bellows contribute 18% of the gas load from only 0.7% of the surface area. A five-model adsorption isotherm framework applied to 22 pumpdown datasets (164 fits with AR(1)-GLS correction) established that the experimental protocol, not the material, controls isotherm identifiability: Dubinin–Radushkevich wins isothermal pipe pumpdowns; Langmuir wins thermally dominated chamber bakes. Cross-dataset joint fitting of the AISI 1020 pipe pumpdowns yielded an H2 diffusion activation energy Ed = 7.24 ± 1.28 kcal·mol¿¹, consistent with trap dominated diffusion in commercial low-carbon steels. Two companion innovations were developed: a Variable Conductance Device (VCD, patent pending IDF-00723) for XHV outgassing measurement, and VacuumDesignerPro (VDP), a MATLAB-based design tool validated against LIGO benchmarks.

Al-Allaq, Aiman H [Old Dominion University]↗
DOE OSTI2020

A Causal Inference Model Based on Random Forests to Identify the Effect of Soil Moisture on Precipitation

Soil moisture influences precipitation mainly through its impact on land–atmosphere interactions. Understanding and correctly modeling soil moisture–precipitation (SM–P) coupling is crucial for improving weather forecasting and subseasonal to seasonal climate predictions, especially when predicting the persistence and magnitude of drought. However, the sign and spatial structure of SM–P feedback are still being debated in the climate research community, mainly due to the difficulty in establishing causal relationships and the high degree of nonlinearity in land–atmosphere processes. To this end, we developed a causal inference model based on the Granger causality analysis and a nonlinear machine learning model. This model includes three steps: nonlinear anomaly decomposition, nonlinear Granger causality analysis, and evaluation of the quality of SM–P feedback, which eliminates the nonlinear response of interannual and seasonal variability and the memory effects of climatic factors and isolates the causal relationship of local SM–P feedback. We applied this model by using National Climate Assessment–Land Data Assimilation System (NCA-LDAS) datasets over the United States. Here, the results highlight the importance of nonlinear atmosphere responses in land–atmosphere interactions. In addition, the strong feedback over the southwestern United States and the Great Plains both highlight the impacts of topographic factors rather than only the sensitivity of evapotranspiration to soil moisture. Furthermore, the SM–P index defined by our framework is used to benchmark Earth system models (ESMs), which provides a new metric for efficiently identifying potential model biases in modeling local land–atmosphere interactions and may help the development of ESMs in improving simulations of water cycle variability and extremes.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRSDate not supplied

Towards an Aviation Large Language Model by Fine-tuning and Evaluating Transformers

In the aviation domain, there are many applications for machine learning and artificial intelligence tools that utilize natural language. For example, there is a desire to know the commonalities in written safety reports such as voluntary post incidents reports or aerial wildfire operations reports to better understand the risks present. Another use-case is the possibility of extracting airspace procedures and constraints currently written in documents such as Letters of Agreement. These applications can benefit from the use of state-of-the-art natural language processing techniques when adapted to the language/phraseology specific to the aviation domain. This paper evaluates the viability of adaptation of NLP tools to the aviation domain by fine-tuning transformer based models using aviation data sets. In 2018, a novel language model based on neural units (also called transformers) was created and became known as “Bidirectional Encoder Representations from Transformers” or BERT. This architecture combined with large amounts of English training data and innovative semi-supervised training tasks set the standard for what would later emerge as Large Language Models. The performance of these models was further improved by hyperparameter tuning and refinement of the semi-supervised training task and resulted in “Robustly Optimized BERT Pre-training Approach through hyperparameter tuning” or RoBERTa models. These pre-trained Large Language Models proved to be useful for a wide variety of natural language processing tasks such as text classification and question answering through a process called fine-tuning. The transformer architecture with pre-trained weights served as the basis with the last few layers replaced with layers fine-tuned to perform a new task e.g., a layer that provides a label for the entire input text. This process of fine-tuning can also be used to adapt the models to new domains; e.g., BioBERT started with the pre-trained BERT model and was completed by additional fine-tuning and training on biomedical documents. Transformer-based architectures can also be used to create rich representations of text called embeddings which can serve as the input to other machine learning models. This allows simpler algorithms such as logistic regression to use context-rich representations of the text while still remaining quick to train and evaluate. In the world of aviation, there is a growing demand for natural language processing and understanding but the domain presents unique challenges. Due to the technical content (and specialized language) of most aviation documents, fine-tuning pre-trained Large Language Models to specific tasks has not met the benchmark on natural language processing tasks set by simpler models trained from scratch on the data. To address this deficiency, this paper evaluates the improvements from fine-tuning a Large Language Model on a large set of aviation documents using the original semi-supervised training tasks before performing specific natural language tasks. In fine-tuning, a domain-specific dataset is used on the original training task but with the pre-trained Large Language Model instead of starting from a random initialization. This approach allows the model to be adapted to the specific domain language without discarding the information gained from training on general English data. This paper utilized two major dataset types to train and assess the RoBERTa fine-tuning performance. The first are 7,057 Letters of Agreement which are Federal Aviation Administration (FAA) documents that formalize airspace operations across the national airspace system. They contain many examples of ‘aviation English’ using domain specific terminology and phrasing which serves as a representative basis to perform the semi-supervised fine-tuning. The second type is the 494 document classification labels to be used for evaluation. This down-stream evaluation aims to show the performance of the fine-tuned model, better understand how much data is needed for an effective fine-tuning, and how fine-tuning can be adapted for different applications in-the domain. After semi-supervised training, evaluation begins by encoding the documents for classification using the fine-tuned RoBERTa model. Then a logistic regression classifier is trained to label the document type and compared against our ground truth labels. This currently leads to a 82.8% accuracy on 10-fold cross validation showing improvement over baseline RoBERTa which achieved 81.0%. We plan to measure the improvements on additional tasks and it is expected that these improvements will lead to more robust models that can tackle the natural language processing challenges present in aviation datasets.

ATM↗