Search NASA⌕ Search

SEARCH · Search NASA

Results for “transformer encoder”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

Simulating fermions with a digital quantum computer

Quantum computers are expected to become a powerful tool for studying physical quantum systems. Consequently, a number of quantum algorithms to determine the physical properties of such systems have been developed. Although qubit-based quantum computers are naturally suited to the study of spin-1/2 systems, systems containing other degrees of freedom must first be encoded into qubits. Transformations to and from fermionic degrees of freedom have long been an important tool in physics and chemistry, which is now finding another application in the simulation of fermionic systems on quantum computers based on qubits. In this work, we discuss methods for encoding fermionic degrees of freedom into qubits.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Exploring Semantic Search Capability of Graph Convolutions Over a Knowledge Graph Built Using Earth Science Corpora

Traditional knowledge graphs tend to be too generic, and often perform poorly on complex scientific queries. Often times, precedence is given to pop culture over scientific knowledge for queries. This is predominantly due to the use of internet sources for building the knowledge graph. With this work, we aim to explore the effectiveness of combining a knowledge graph generated from earth science corpora with a language model and graph convolutions for the purpose of surfacing latent and related sentences given a natural language query. In this model, sentences are conceptualized in the graph as nodes which are connected through entities—words and phrases of interest found in the text—extracted using Google Cloud’s entity extraction model. The language model we used for this is Bidirectional Encoder Representations from Transformers (BERT).The sentences are given a numeric representation by the BERT model. Graph convolutions are then applied to sentence embeddings in order to obtain a vector representation of the sentence as well as the surrounding graph structure, thereby leveraging the power of adjacency inherently encoded in graph structures. With this presentation, we demonstrate the ability of graph convolutions and their improved ability to surface relevant, latent information based on the subject of the input query.

Muthukumaran Ramasubramanian↗

Device and method for constructing and displaying high quality images from imaging data by transforming a data structure utilizing machine learning techniques

Constructing a computer image from raw imaging data or encoded imaging data by transforming a first data structure in which the raw imaging data or the encoded imaging data is stored into a second data structure storing reorganized imaging data. The raw imaging data or the encoded imaging data is received, stored in the first data structure. The computer reorganizes the raw imaging data or the encoded imaging data into the reorganized data and stores the reorganized data in the second data structure, which is a multi-dimensional array having subarrays containing local information needed by a convolutional neural network for processing the reorganized data. Other portions of the multi-dimensional array store other portions of the raw imaging data or the encoded imaging data. The computer also processes the reorganized data using the convolutional neural network to construct the image, whereby a constructed image is formed.

Korbin, John P.↗

Context Modeler for Wavelet Compression of Spectral Hyperspectral Images

A context-modeling sub-algorithm has been developed as part of an algorithm that effects three-dimensional (3D) wavelet-based compression of hyperspectral image data. The context-modeling subalgorithm, hereafter denoted the context modeler, provides estimates of probability distributions of wavelet-transformed data being encoded. These estimates are utilized by an entropy coding subalgorithm that is another major component of the compression algorithm. The estimates make it possible to compress the image data more effectively than would otherwise be possible. The following background discussion is prerequisite to a meaningful summary of the context modeler. This discussion is presented relative to ICER-3D, which is the name attached to a particular compression algorithm and the software that implements it. The ICER-3D software is summarized briefly in the preceding article, ICER-3D Hyperspectral Image Compression Software (NPO-43238). Some aspects of this algorithm were previously described, in a slightly more general context than the ICER-3D software, in "Improving 3D Wavelet-Based Compression of Hyperspectral Images" (NPO-41381), NASA Tech Briefs, Vol. 33, No. 3 (March 2009), page 7a. In turn, ICER-3D is a product of generalization of ICER, another previously reported algorithm and computer program that can perform both lossless and lossy wavelet-based compression and decompression of gray-scale-image data. In ICER-3D, hyperspectral image data are decomposed using a 3D discrete wavelet transform (DWT). Following wavelet decomposition, mean values are subtracted from spatial planes of spatially low-pass subbands prior to encoding. The resulting data are converted to sign-magnitude form and compressed. In ICER-3D, compression is progressive, in that compressed information is ordered so that as more of the compressed data stream is received, successive reconstructions of the hyperspectral image data are of successively higher overall fidelity.

Kiely, Aaron↗

Advancing quantum simulations of the nuclear shell model with Gray-code–based resource-efficient protocols

Background: Some of the computational limitations in solving the nuclear many-body problem could be overcome by utilizing quantum computers. The nuclear shell-model calculations providing deeper insights into the properties of atomic nuclei are one such case with high demand for resources, as the size of the Hilbert space grows exponentially with the number of particles involved. Quantum algorithms are being developed to overcome these challenges and advance such calculations. Purpose: To develop quantum circuits for the nuclear shell-model, leveraging the capabilities of noisy intermediate-scale quantum (NISQ) devices. Here, we aim to minimize resource requirements (specifically in terms of qubits and gates) and strive to reduce the impact of noise by employing relevant mitigation techniques. Methods: We achieve noise resilience by designing an optimized Ansatz for the variational quantum eigensolver (VQE) based on Givens rotations and incorporating qubit-ADAPT-VQE in combination with variational quantum deflation (VQD) to compute ground and excited states, incorporating the zero-noise extrapolation mitigation technique. Furthermore, the qubit requirements are significantly reduced by mapping the basis states to qubits using Gray-code encoding and generalizing transformations of fermionic operators to efficiently represent many-body states. Results: By employing the resource-efficient protocols, we achieve the ground and excited state energy levels of 38 Ar and 6 Li with better accuracy. These energy levels are presented for noiseless simulations, noisy conditions, and after applying noise mitigation techniques. Results are compared for Jordan-Wigner and Gray-code encoding using VQE, qubit-ADAPT-VQE, and VQD. Conclusions: Our work highlights the potential of resource-efficient protocols to leverage the full potential of NISQ devices in scaling the nuclear shell model calculations, offering a pathway toward more complex quantum simulations in nuclear physics. This approach establishes a framework for studying other nuclear systems with improved quantum resource efficiency, marking a significant advancement in applying quantum computing to realistic nuclear physics applications.

Physics - Nuclear physics and radiation physics↗

Masked Symbol Modeling for Demodulation of Oversampled Baseband Communication Signals in Impulsive Noise-Dominated Channels

Recent breakthroughs in natural language processing show that attention mech- anism in Transformer networks, trained via masked-token prediction, enables models to capture the semantic context of the tokens and internalize the grammar of language. While the application of Transformers to communication systems is a burgeoning field, the notion of context within physical waveforms remains under-explored. This paper addresses that gap by re-examining inter-symbol con- tribution (ISC) caused by pulse-shaping overlap. Rather than treating ISC as a nuisance, we view it as a deterministic source of contextual information embedded in oversampled complex baseband signals. We propose Masked Symbol Model- ing (MSM), a framework for the physical (PHY) layer inspired by Bidirectional Encoder Representations from Transformers methodology. In MSM, a subset of symbol-aligned samples is randomly masked, and a Transformer predicts the missing symbol identifiers using the surrounding “in-between” samples. Through this objective, the model learns the latent syntax of complex baseband waveforms. We illustrate MSM’s potential by applying it to the task of demodulating sig- nals corrupted by impulsive noise, where the model infers corrupted segments by leveraging the learned context. Our results suggest a path toward receivers that interpret, rather than merely detect communication signals, opening new avenues for context-aware PHY layer design.

Bedir, Oguz↗

MechBERT: Language Models for Extracting Chemical and Property Relationships about Mechanical Stress and Strain

Language models are transforming materials-aware naturallanguage processing by enabling the extraction of dynamic, context-rich information from unstructured text, thus, moving beyond the limitations of traditional information-extraction methods. Moreover, small language models are on the rise because some of them can perform better than large language models (LLMs) when given domain-specific questionanswer tasks, especially about an application area that relies on a highly specialized vernacular, such as materials science. We therefore present a new class of MechBERT language models for understanding mechanical stress and strain in materials. These employ Bidirectional Encoder Representations for transformer (BERT) architectures. We showcase four MechBERT models, all of which were pretrained on a corpus of documents that are textually rich in chemicals and their stress–strain properties and were fine-tuned on question-answering tasks. We evaluated the level of performance of our models on domain-specific as well as general English-language question-answer tasks and also explored the influence of the size and type of BERT architectures on model performance. We find that our MechBERT models outperform BERT-based models of the same size and maintain relevancy better than much larger BERT-based models when tasked with domain-specific question-answering tasks within the stress–strain engineering sector. These small language models also enable much faster processing and require a much smaller fraction of data to pretrain them, affording them greater operational efficiency and energy sustainability than LLMs.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Quantum-Assisted Variational Segmentation for Image-to-Image Wildfire Detection Using Satellite Data

The quantum computing community has been searching for suitable applications to demonstrate the potential of near-term quantum devices. Quantum machine learning is a potential candidate, particularly using models that cannot be efficiently simulated with classical computers [1, 2]. This work focuses on a transition phase of quantum computers where the quantum machine learning model is still simulable classically but projected not to be simulable as the size of the model grows. Ultimately quantum computers may have advantages for high-dimensional real-world problems. Due to the limited number of qubits in current noisy intermediate-scale quantum (NISQ) devices, the direct application of quantum computers in high dimensional data is not feasible. To remedy this problem, an encoder-decoder architecture can be utilized. The encoder model would transform the high-dimensional data into a compact representation, to a level that small quantum computers can be used today (or in the near future), and the decoder would take the quantum processed outputs back to the high-dimensional space. Addressing the two challenges of quantum machine learning, this work investigates a hybrid supervised generative model with a quantum Ising Born machine embedded as the latent distribution. The model contains four main parts (Figure 1.a.): (1) a U-NET architecture responsible for learning segmentation flow, (2) a Prior network responsible for learning an encoded latent distribution of the input data, (3) a Born machine which represents the latent distribution, and (4) a Posterior network in charge of learning the joint encoded latent distribution of inputs and target data. The initial model, proposed by [3], is optimized by (1) maximizing the overlap of the prior and posterior latent distributions, and (2) minimizing the segmentation loss. The proposed model is designed to be investigated in a simulation environment applied to the real-world application of wildfire segmentation. Specifically, the model is designed to solve the patchy wildfire segmentations of Moderate Resolution Imaging Spectroradiometer (MODIS) by taking the MODIS observations and using Visible Infrared Imaging Radiometer Suite’s (VIIRS) consistent wildfire product as the target. The model solves patchy wildfire segmentations and provides insight into the epistemic errors sourced from model variation. The model utilizes the Born machine as a QUBO solver to represent the latent space as a Bernoulli distribution. The proposed configuration allows the variational segmentation model to leverage the true quantum probabilistic nature and derive a more expressive latent configuration, increasing the model performance in describing wildfire segmentations. The quantum probabilistic information of the Born machine is directly incorporated in the Kullback-Leibler divergence loss in the prior and posterior distributions, forcing the Bernoulli latent distribution to maximize the overlap of input and joint input-target distributions. The proposed model is then trained and compared with a baseline only consisting of direct Bernoulli latent distribution with no Born machine representing the latent space. The models are evaluated based on the segmentation metrics, such as precision, recall, intersect of union, with uncertainty boundaries accounting for the stochastic nature of the model. Our findings show that even in low latent-dimensional space (due to the limit in computational power of the classical quantum simulator), we are able to effectively capture the latent representation and hence the model performs better than the baseline. The findings are a projection for scaling the model into higher dimensional latent space with the Born machine surpassing the baseline performance. Figure 1. Sub-figure (a) demonstrates the architecture for the training phase. The model consists of a Prior and Posterior network that encode inputs and joint input-target data into compact representations, respectively. The Born machine represents the latent distribution, and the U-NET branch learns the segmentation patterns of the data. The stochasticity is introduced to the U-NET through its last layer to create meaningful but stochastic segmentations. Sub-figure (b) represents the inference phase where the model takes the stochastic behavior from the prior network and injects that into the U-NET. Each attempt of inference will generate different but similar segmentations from the same distribution of the wildfire event. REFERENCES [1] Coyle, B., Mills, D., Danos, V., & Kashefi, E. (2020). The Born supremacy: quantum advantage and training of an Ising Born machine. npj Quantum Information, 6(1), 1-11. [2] Liu, J. G., & Wang, L. (2018). Differentiable learning of quantum circuit born machines. Physical Review A, 98(6), 062324. [3] Kohl, S., Romera-Paredes, B., Meyer, C., De Fauw, J., Ledsam, J. R., Maier-Hein, K., ... & Ronneberger, O. (2018). A probabilistic u-net for segmentation of ambiguous images. Advances in neural information processing systems, 31.

quantum machine learning↗

ATAT: Astronomical Transformer for time series and Tabular data

Context. The advent of next-generation survey instruments, such as theVera C. RubinObservatory and its Legacy Survey of Space and Time (LSST), is opening a window for new research in time-domain astronomy. The Extended LSST Astronomical Time-Series Classification Challenge (ELAsTiCC) was created to test the capacity of brokers to deal with a simulated LSST stream. Aims. Our aim is to develop a next-generation model for the classification of variable astronomical objects. We describe ATAT, the Astronomical Transformer for time series And Tabular data, a classification model conceived by the ALeRCE alert broker to classify light curves from next-generation alert streams. ATAT was tested in production during the first round of the ELAsTiCC campaigns. Methods. ATAT consists of two transformer models that encode light curves and features using novel time modulation and quantile feature tokenizer mechanisms, respectively. ATAT was trained on different combinations of light curves, metadata, and features calculated over the light curves. We compare ATAT against the current ALeRCE classifier, a balanced hierarchical random forest (BHRF) trained on human-engineered features derived from light curves and metadata. Results. When trained on light curves and metadata, ATAT achieves a macro F1 score of 82.9 ± 0.4 in 20 classes, outperforming the BHRF model trained on 429 features, which achieves a macro F1 score of 79.4 ± 0.1. Conclusions. The use of transformer multimodal architectures, combining light curves and tabular data, opens new possibilities for classifying alerts from a new generation of large etendue telescopes, such as theVera C. RubinObservatory, in real-world brokering scenarios.

Astronomy & Astrophysics↗

PRIMED for the Future: Purposing Raw Intake for Machine Learning-Enabled Detection (Final Report)

The COVID-19 pandemic demonstrated how a novel, elusive, and diffuse biological threat can engender uncertainty and misinformation, and it underscored the need for flexible analytical modalities agnostic to the identity of biological material. Yet even before the pandemic, recognition of the limitations of the current, list-based approach, which focuses on known pathogens and biotoxins, and of the importance of agent-agnostic biodetection was growing within the biosecurity community. In a 2018 report on “Biodefense in the Age of Synthetic Biology,” for example, the National Academy of Sciences stated that “an overreliance on the Select Agent List is a systematic weakness affecting many aspects of the United States’ current biodefense mitigation capability”. More recently, a group of biodefense researchers proposed the identification and adoption of “bioagent-agnostic signatures (BASs)” as a way of detecting and characterizing not only existing agents but also novel ones, an approach they believe will “enable a more flexible and resilient biodefense posture”. Indeed, the future of biodetection requires us to begin developing novel analytics that can identify anomalies and/or characteristics that indicate a potential threat, whether known or unknown, without looking for a specific signature that has been identified previously. To assess potential threats more rapidly, it is critical to develop agnostic artificial intelligence (AI)/machine learning (ML) systems that can be employed for real-time assessment of the nature and source of a perturbation. Such systems should be multi scale and multi-dimensional, integrating sensor data from a range of biological, chemical, and physical application spaces. Emerging deep learning (DL) models demonstrate exceptional promise for identification of discriminatory features within multi-dimensional datasets. DL models have the capacity to recognize and encode highly complex patterns in a wide range of input data modalities, including images, text, and biological/chemical/physical spectra. As such, they can execute a wide range of assessments and determinations that have traditionally required a human operator. The promise of advances in DL is apparent in the realm of human health and medicine. DL models have been validated for evaluating a variety of clinical threats to human health in a range of contexts, including infection and cancer, and they demonstrated improved performance in predicting stroke relative to human neurologists in some categories of data. Continuously evolving advances in AI/ML are expected to support more efficient evaluation of raw sequence, spectroscopy, and spectrometry data. For instance, recent advances and deployment of large language models (LLM) such as Generative Pre training Transformer (GPT) and Bidirectional Encoder Representations from Transformers (BERT) have already motivated application of these models for biological function prediction. As frameworks such as LLMs become larger and more complex in their representations, their capacity to serve as pre-trained models that can be fine-tuned for biological/biodetection purposes will similarly be amplified. While existing and emerging AI/ML have found broad applicability and use cases in the clinical sciences, development for environmental evaluation and biodetection has been limited. Functionalizing such capabilities for this purpose requires an understanding of the existing technical landscape and how the respective tools and algorithms are currently being employed. This landscape awareness then allows an assessment of the current practical capabilities of existing models and the anticipated requirements and development efforts that will be needed to adapt available algorithms for biodetection applications relevant to DHS. Leveraging expertise in biodetection, ML, and operational biodetection, the effort described in this report is comprised of a systematic landscape assessment (Subtask 2.1), comparative evaluation (Subtask 2.2), and formulation of a value proposition (Subtask 2.3) for the prospect of ML-enabled, agnostic biodetection from raw, or minimally-processed, datasets.

59 BASIC BIOLOGICAL SCIENCES↗

Large CMOS imager using hadamard transform based multiplexing

We have developed a concept design for a large (~10k x 10k) CMOS imaging array whose elements are grouped in small subarrays with N pixels in each. The subarrays are code-division multiplexed using the Hadamard Transform (HT) based encoding. The Hadamard code improves the signal-to-noise (SNR) ratio to the reference of the read-out amplifier by a factor of N^1/2. This way of grouping pixels reduces the number of hybridization bumps by N. A single chip layout has been designed and the architecture of the imager has been developed to accommodate the HT base multiplexing into the existing CMOS technology. The imager architecture allows for a trade-off between the speed and the sensitivity. The envisioned imager would operate at a speed >100 fps with the pixel noise < 20 e-. The power dissipation would be ~100 pW/pixe1. The combination of the large format, high speed, high sensitivity and low power dissipation can be very attractive for space reconnaissance applications.

CMOS imager↗

Navigating collinear superspace

We introduce a new set of effective field theory rules for constructing Lagrangians with \( \mathcal{N} \) = 1 supersymmetry in collinear superspace. In the standard superspace treatment, superfields are functions of the coordinates \( \left({x}^{\mu },{\theta}^{\alpha },{\theta}^{\dagger \overset{\cdot }{\alpha }}\right) \) , and supersymmetry preservation is manifest at the Lagrangian level in part due to the inclusion of auxiliary F - and D -term components. By contrast, collinear superspace depends on a smaller set of coordinates ( x μ , η , η † ), where η is a complex Grassmann number without a spinor index. This provides a formulation of supersymmetric theories that depends exclusively on propagating degrees of freedom, at the expense of obscuring Lorentz invariance and introducing inverse momentum scales. After establishing the general framework, we construct collinear superspace Lagrangians for free chiral matter and non-Abelian gauge fields. For the latter construction, an important ingredient is a superfield representation that is simultaneously chiral, anti-chiral, and real; this novel object encodes residual gauge transformations on the light cone. Additionally, we discuss a fundamental obstruction to constructing inter- acting theories with chiral matter; overcoming these issues is the subject of our companion paper, where we introduce a larger set of superfields to realize the full range of interactions compatible with \( \mathcal{N} \) = 1. Along the way, we provide a novel framing of reparametrization invariance using a spinor decomposition, which provides insight into this important light-cone symmetry.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

EHR-BERT: A BERT-based model for effective anomaly detection in electronic health records

Objective: Physicians and clinicians rely on data contained in electronic health records (EHRs), as recorded by health information technology (HIT), to make informed decisions about their patients. The reliability of HIT systems in this regard is critical to patient safety. Consequently, better tools are needed to monitor the performance of HIT systems for potential hazards that could compromise the collected EHRs, which in turn could affect patient safety. In this paper, we propose a new framework for detecting anomalies in EHRs using sequence of clinical events. This new framework, EHR-Bidirectional Encoder Representations from Transformers (BERT), is motivated by the gaps in the existing deep-learning related methods, including high false negatives, sub-optimal accuracy, higher computational cost, and the risk of information loss. EHR-BERT is an innovative framework rooted in the BERT architecture, meticulously tailored to navigate the hurdles in the contemporary BERT method; thus, enhancing anomaly detection in EHRs for healthcare applications.Methods: The EHR-BERT framework was designed using the Sequential Masked Token Prediction (SMTP) method. This approach treats EHRs as natural language sentences and iteratively masks input tokens during both training and prediction stages. This method facilitates the learning of EHR sequence patterns in both directions for each event and identifies anomalies based on deviations from the normal execution models trained on EHR sequences.Results: Extensive experiments on large EHR datasets across various medical domains demonstrate that EHR-BERT markedly improves upon existing models. It significantly reduces the number of false positives and enhances the detection rate, thus bolstering the reliability of anomaly detection in electronic health records. This improvement is attributed to the model’s ability to minimize information loss and maximize data utilization effectively.Conclusion: EHR-BERT showcases immense potential in decreasing medical errors related to anomalous clinical events, positioning itself as an indispensable asset for enhancing patient safety and the overall standard of healthcare services. The framework effectively overcomes the drawbacks of earlier models, making it a promising solution for healthcare professionals to ensure the reliability and quality of health data.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Identifying COVID-19 cases and extracting patient reported symptoms from Reddit using natural language processing

We used social media data from “covid19positive” subreddit, from 03/2020 to 03/2022 to identify COVID-19 cases and extract their reported symptoms automatically using natural language processing (NLP). We trained a Bidirectional Encoder Representations from Transformers classification model with chunking to identify COVID-19 cases; also, we developed a novel QuadArm model, which incorporates Question-answering, dual-corpus expansion, Adaptive rotation clustering, and mapping, to extract symptoms. Our classification model achieved a 91.2% accuracy for the early period (03/2020-05/2020) and was applied to the Delta (07/2021–09/2021) and Omicron (12/2021–03/2022) periods for case identification. We identified 310, 8794, and 12,094 COVID-positive authors in the three periods, respectively. The top five common symptoms extracted in the early period were coughing (57%), fever (55%), loss of sense of smell (41%), headache (40%), and sore throat (40%). During the Delta period, these symptoms remained as the top five symptoms with percent authors reporting symptoms reduced to half or fewer than the early period. During the Omicron period, loss of sense of smell was reported less while sore throat was reported more. Our study demonstrated that NLP can be used to identify COVID-19 cases accurately and extracted symptoms efficiently.

60 APPLIED LIFE SCIENCES↗

A knowledge-informed large language model framework for U.S. nuclear power plant shutdown initiating event classification for probabilistic risk assessment

Identifying and classifying shutdown initiating events (SDIEs) is critical for developing shutdown probabilistic risk assessment for nuclear power plants. Existing computational approaches cannot achieve satisfactory performance due to the challenges of unavailable large, labeled datasets, imbalanced event types, and label noise. To address these challenges, we propose a hybrid pipeline that integrates a knowledge-informed machine learning model to prescreen non-SDIEs and a large language model (LLM) to classify SDIEs into four types. In the prescreening stage, we proposed a set of 44 SDIE text patterns that consist of the most salient keywords and phrases from six SDIE types. Text vectorization based on the SDIE patterns generates feature vectors that are highly separable by using a simple binary classifier. The second stage builds Bidirectional Encoder Representations from Transformers (BERT)-based LLM, which learns generic English language representations from self-supervised pretraining on a large dataset and adapts to SDIE classification by fine-tuning it on an SDIE dataset. The proposed approaches are evaluated on a dataset with 10,928 events using precision, recall ratio, F 1 score, and average accuracy. In conclusion, the results demonstrate that the prescreening stage can exclude more than 97% non-SDIEs, and the LLM achieves an average accuracy of 95.1% for SDIE classification.

99 - GENERAL AND MISCELLANEOUS↗

Deployment of Dynamic Neural Network Optimization to Minimize Heat Rate During Ramping for Coal Power Plants (Final Technical Report)

Much success was achieved throughout the course of this project. A successful implementation of Dynamic Neural Network Optimization (D-NNO) was coupled with Adaptive Predictive Controls (APC) and a novel hardware installation comprised of an advanced sensor network (ASN) measuring mass-weighted averages of flue gas constituents above the horizontal superheater of a coal-fired utility boiler. From 2019 through 2023 (including an extension due to COVID delays), the team was able to prototype, evaluate, deploy, iterate, and ultimately finalize an advanced closed-loop control D-NNO system which demonstrated the ability to: •improve unit efficiency ~2.0% relative to unoptimized operation (represented as total fuel fired per MWh generated) •improve unit NOx emission rates 10%+ beyond static optimization baselines •improve unit temperature stability as much as 58% and on average 12% •improve operating load stability as much as 35% The culmination of this project has generated an advanced methodology of deploying specially designed recurrent neural networks (long short-term memory, gated recurrent unit, encoder-decoder networks, transformers, etc.), customized trajectory planning and closed-loop optimization modules capable of adapting to live electric grid responses and demands, self-tuning and adaptive expert controls constantly adjusting prediction parameters to real-time unit behavior, and a hardware/software package able to reliably calculate net unit heat rate (NUHR) in real-time using flue gas constituents, machine learning, and known combustion relationships. Through this real-time NUHR value, immediate feedback on system adjustments relative to operating efficiency was available, allowing for rapid improvements to system performance. In addition to development and deployment of the advanced D-NNO system, the approach methodology has been readily commercialized through the project platform Griffin Open Systems, LLC, the D-NNO software platform host. Similar methodologies to those developed by this project have already been deployed at 5 other units across the United States, with another 6 implementations scheduled, and more expected. Over the course of the project, multiple academic papers were submitted and accepted for publication within esteemed academic journals, and PhD students were trained and graduated, as well as undergraduate students becoming involved and participating to project objectives.

01 COAL, LIGNITE, AND PEAT↗

Automating the Analysis of Large Language Models Responses through Zero-Shot Question Answering

Recent advancements in Large Language Models (LLMs) have shown significant potential in various applications, yet their evaluation, particularly in zero-shot question answering scenarios, remains a challenging task. In this study, our objective was to explore precision metrics for Large Language Models (LLM) and design and implement a software pipeline to automatically evaluate LLMs' outputs under zero-shot question answering. Zero-shot question answering involves a model providing answers to questions about topics it hasn't seen during training. It leverages the principles of zero-shot learning by relying on semantic understanding and generalization from related knowledge. The data used was metadata from medical databases on congenital heart disease. We explored eleven LLM metrics and selected three for our evaluation: BLEU, BERTScore, and MoverScore. BLEU calculates a score based on the overlap of n-grams (contiguous sequences of n items, typically words) between the machine-generated translation and the reference translations. Higher BLEU scores indicate better correspondence between the machine-generated and human-generated translations. BERTScore is a metric used to evaluate the quality of machine-generated text by measuring the similarity of token embeddings produced by BERT (Bidirectional Encoder Representations from Transformers) between the generated text and reference text. MoverScore is a metric that quantifies the dissimilarity between the distributions of word embeddings from machine-generated text and reference text, emphasizing semantic similarity over exact token overlap. We also introduced HBKI, a composite metric summarizing these approaches. We tested five models —GPT-3, Llama-2, Gemini 1.5 Pro, Solar 10.7B, and Mixtral-8x7b. Our software pipeline, designed and implemented using Object-Oriented Programming principles, allows users to customize the selection and extraction of features for topics of interest in their own research. Our results show that MoverScore delivered the most precise evaluation of the LLM's outputs, while Mixtral-8x7b achieved the best overall performance in extracting metadata from the databases.

97 MATHEMATICS AND COMPUTING↗

CyBERT: Cybersecurity Claim Classification by Fine-Tuning the BERT Language Model

We introduce CyBERT, a cybersecurity feature claims classifier based on bidirectional encoder representations from transformers and a key component in our semi-automated cybersecurity vetting for industrial control systems (ICS). To train CyBERT, we created a corpus of labeled sequences from ICS device documentation collected across a wide range of vendors and devices. This corpus provides the foundation for fine-tuning BERT’s language model, including a prediction-guided relabeling process. We propose an approach to obtain optimal hyperparameters, including the learning rate, the number of dense layers, and their configuration, to increase the accuracy of our classifier. Fine-tuning all hyperparameters of the resulting model led to an increase in classification accuracy from 76% obtained with BertForSequenceClassification’s original architecture to 94.4% obtained with CyBERT. Furthermore, we evaluated CyBERT for the impact of randomness in the initialization, training, and data-sampling phases. CyBERT demonstrated a standard deviation of ±0.6% during validation across 100 random seed values. Finally, we also compared the performance of CyBERT to other well-established language models including GPT2, ULMFiT, and ELMo, as well as neural network models such as CNN, LSTM, and BiLSTM. The results showed that CyBERT outperforms these models on the validation accuracy and the F1 score, validating CyBERT’s robustness and accuracy as a cybersecurity feature claims classifier.

97 MATHEMATICS AND COMPUTING↗