Search NASA⌕ Search

SEARCH · Search NASA

Results for “data quality metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 307 records · Page 17

Imputation of urban environmental sensor data using gated attention bidirectional long short-term memory (GA-BiLSTM): methods, performance, and implications

Urban environmental monitoring networks frequently encounter significant data gaps due to sensor malfunctions, environmental disturbances, and communication failures. Reliable approaches to address these gaps are essential for ensuring the continuity and quality of environmental data streams. In this study, we developed a gated attention bidirectional long short-term memory (GA-BiLSTM) model to impute missing data in a dense urban monitoring network. Using observations from the CROCUS network in Chicago, we evaluated GA-BiLSTM against widely used approaches (XGBoost and K-nearest neighbors) under scenarios of both short-term intermittent gaps and prolonged outages. GA-BiLSTM consistently outperformed comparative methods, particularly during extended outages of up to ten days, demonstrating its ability to capture spatiotemporal dependencies across sensor nodes. Beyond performance metrics, feature importance and spatial network analyses highlighted the unexpected but critical predictive role of peripheral rural nodes, underlining their strategic value for maintaining robust urban monitoring systems. These results emphasize that advanced imputation methods can substantially improve the reliability of environmental monitoring networks and support more resilient data infrastructures for urban sustainability.

Data imputation↗

QWIP: A Quantitative Metric for Quality Control of Aquatic Reflectance Spectral Shape using the Apparent Visible Wavelength

The colors of the ocean and inland waters span clear blue to turbid brown, and the corresponding spectral shapes of the waterleaving signal are diverse depending on the various types and concentrations of phytoplankton, sediment, detritus and colored dissolved organic matter. Here we present a simple metric developed from a global dataset spanning blue, green and brown water types to assess the quality of a measured or derived aquatic spectrum. The Quality Water Index Polynomial (QWIP) is founded on the Apparent Visible Wavelength (AVW), a one-dimensional geophysical metric of color that is inherently correlated to spectral shape calculated as a weighted harmonic mean across visible wavelengths. The QWIP represents a polynomial relationship between the hyperspectral AVW and a Normalized Difference Index (NDI) using red and green wavelengths. The QWIP score represents the difference between a spectrum’s AVW and NDI and the QWIP polynomial. The approach is tested extensively with both raw and quality controlled field data to identify spectra that fall outside the general trends observed in aquatic optics. For example, QWIP scores less than or greater than 0.2 would fail an initial screening and be subject to additional quality control. Common outliers tend to have spectral features related to: 1) incorrect removal of surface reflected skylight or 2) optically shallow water. The approach was applied to hyperspectral imagery from the Hyperspectral Imager for the Coastal Ocean (HICO), as well as to multispectral imagery from the Visual Infrared Imaging Radiometer Suite (VIIRS) using sensor-specific extrapolations to approximate AVW. This simple approach can be rapidly implemented in ocean color processing chains to provide a level of uncertainty about a measured or retrieved spectrum and flag questionable or unusual spectra for further analysis.

remote sensing reflectance↗

Energy and power quality measurement for electrical distribution in AC and DC microgrid buildings

Today's selection of DC microgrid buildings features a diverse set of electrical topologies and turnkey solutions, each with specific design trade-offs and optimizations. Designers desperately need standardized metrics and procedures for measurement and verification (M&V) to analyze and compare the advantages of each DC solution to traditional AC building networks. This work develops M&V procedures for quantifying and comparing the energy efficiency and power quality in buildings. To calculate full-building efficiency, this work introduces the measurement-informed modeling method, a procedure that develops and refines a building's energy model with metered data. To quantify power quality, this work defines a new voltage quality index that applies to both AC and DC buildings. This article describes the equipment, instrumentation, and operation necessary to calculate the efficiency and power quality. It then demonstrates these methods with a set of field tests. We report these M&V procedures can ultimately be used to compare and improve the efficiency and power quality of various DC topologies.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Atmospheric condition identification in multivariate data through a metric for total variation

Identification of atmospheric conditions within a multivariable atmospheric data set is a necessary step in the validation of emerging and existing high-fidelity models used to simulate wind plant flows and operation.Atmospheric conditions relevant for wind energy research include stationary conditions, given the need for well-converged statistics for model validation, as well as conditions observed less frequently, such as extreme atmospheric events, which are used in wind turbine and wind plant design.Aggregation of observations without regard to covariance between time series discounts the dynamical nature of the atmosphere and is not sufficiently representative of atmospheric conditions.Identification and characterization of continuous time periods with atmospheric conditions that have a high value for analysis or simulation set the stage for more advanced model validation and the development of real-time control and operational strategies.The current work explores a single metric for variation in a multivariate data sample that quantifies variability within each channel as well as covariance between channels.The total variation is used to identify conditions of interest that conform to desired objective functions, such as stationary conditions, ramps or waves of wind speed, and changes in wind direction.Total variation is somewhat sensitive to the presence of outliers in the input data, and the method is best complemented by quality-control procedures to ensure reliable results.The direct detection and classification of events or conditions of interest within atmospheric data sets is vital to developing our understanding of wind plant response and to the formulation of forecasting and control models.

17 WIND ENERGY↗

Retrieval of Aerosol Optical Properties From GOCI-II Observations: Continuation of Long-Term Geostationary Aerosol Monitoring Over East Asia

Since the Geostationary Ocean Color Imager (GOCI) was successfully launched in 2010, the GOCI Yonsei aerosol retrieval (YAER) algorithm has been continuously updated to retrieve hourly aerosol optical properties. GOCI-II has 4 more channels including UV, finer spatial resolution (250 m), and daily full disk coverage as compared to GOCI, and was launched in February 2020, onboard the GEO-KOMPSAT-2B (GK-2B) satellite. In this study, we extended the YAER algorithm to GOCI-II data based on its improved performance in many aspects and present the first results of aerosol optical properties retrieved from GOCI-II data. Utilizing the overlapping period between the GOCI-II and GOCI in geostationary Earth orbit, we present GOCI-II aerosol retrievals for high aerosol-loading cases over East Asia and show that these have a consistent spatial distribution with those from GOCI. Furthermore, GOCI-II provides AOD at an even higher spatial resolution, revealing finer changes in aerosol concentrations. Validation results for one year data show that the GOCI-II AOD has a correlation coefficient of 0.83 and a ratio within the expected error (EE) of 59.4 % when compared with the aerosol robotic network (AERONET) data. We compared statistical metrics for the GOCI and GOCI-II AODs to assess the consistency between the two datasets. In addition, it was found that there is a strong correlation between the two datasets from the comparison of gridded GOCI and GOCI-II AOD products. It is expected that data from GOCI-II will continue long-term aerosol records with high accuracy that can be used to address air-quality issues over East Asia.

AOD↗

A Study on the Development of Service Quality Index for Incheon International Airport

The main purpose of this study is located at developing Ominibus Monitors System(OMS) for internal management, which will enable to establish standards, finding out matters to be improved, and appreciation for its treatment in a systematic way. It is through developing subjective or objective estimation tool with use importance, perceived level, and complex index at international airport by each principal service items. The direction of this study came towards for the purpose of developing a metric analysis tool, utilizing the Quantitative Second Data, Analysing Perceived Data through airport user surveys, systemizing the data collection-input-analysis process, making data image according to graph of results, planning Service Encounter and endowing control attribution, and ensuring competitiveness at the minimal international standards. It is much important to set up a pre-investigation plan on the base of existent foreign literature and actual inspection to international airport. Two tasks have been executed together on the base of this pre-investigation; one is developing subjective estimation standards for departing party, entering party, and airport residence and the other is developing objective standards as complementary methods. The study has processed for the purpose of monitoring services at airports regularly and irregularly through developing software system for operating standards after ensuring credibility and feasibility of estimation standards with substantial and statistical way.

Lee, Kang Seok↗

Summary of NDC Capacity Building Workshop and Regional Seismic Travel Time (RSTT) in combination with Data Sharing and Integration Training

The NNSA Seismic Cooperation Program (SCP) sponsored Stephen Myers (LLNL), Michael Begnaud (LANL), Brian Young (SNL) and Istvan Bondar (Research Center for Astronomy and Earth Sciences, Hungary) to serve as a presenters/trainers at the “NDC Capacity Building Workshop and Regional Seismic Travel Time (RSTT) in combination with Data Sharing and Integration Training” September 4-8 2022 in Muscat, Oman (See Appendix A for the agenda). The workshop and training (workshop from here forward) was organized by the Comprehensive Nuclear-Test-Ban Treaty Organization (CTBTO) Provisional Technical Secretariat (PTS). The first half of the week was devoted to NDC workshop activities, and the second half was devoted to RSTT training. Fifty-five participants from 27 countries and the CTBTO-PTS attended the 5-day workshop (See Appendix B for list of participants and countries of origin). Presentations from the PTS described the International Monitoring System (IMS), International Data Centre (IDC) products, and metrics of regional data utilization. Contributed presentations from each country’s scientists included descriptions of regional and national networks, methods of data analysis, and needs for material and technical assistance. Training included an overview of the RSTT method and instruction on how to locate seismic events with the iLoc program, which utilizes RSTT travel times to reduce bias in event location estimates. Methods of seismic tomography and the need for a high-quality tomographic set, including seismological “ground truth”, were emphasized. Seismological “ground truth” or “GT” is a term that has come to mean both events with known location and events with well-characterized locations that are estimated using seismological data, typically with epicenter accuracy of 5 km or better. Notably, the instructional platform has migrated from UNIX shell scripts to Jupyter Notebooks. Jupyter Notebooks have the advantage being more visually intuitive, including display of graphics within the notebook. Each notebook includes every processing step that participants need to reproduce the entire exercise.

58 GEOSCIENCES↗

Leveraging Large Language Models for Real-World Data Evidence: A Framework for Automated Treatment Extraction and Data Harmonization

Background: The ability to comprehensively collect treatment information from cancer patient medical records would enable studies to evaluate real-world benefits and risks tied to specific treatments. Currently, it is difficult to system- atically collect high-quality treatment information because it is often stored in unstructured text. Manually extracting and standardizing drug and regimen data is time-intensive. Recent advances in large language models (LLMs) offer a potential solution for automated extraction of structured treatment information from clinical text. Objective: This study systematically evaluates the utility of four LLMs from the Llama family for automated extraction of oncology treatment information from clinical text. This information can guide researchers using cancer registry data to provide insights into cancer care and outcomes beyond clinical trials. Methods: Four instruction-tuned Llama models with varying parameter counts (1B, 3B, 8B, and 70B) were evaluated for their ability to extract treatment information from clinical documents. A unified oncology knowledge base integrating seven major public data sources was developed to standardize and normalize extracted entities—a critical step for harmonizing data from diverse sources. Extracted treatment data were compared against expert-annotated ground truth. Model performance was assessed using accuracy metrics (Precision, Recall, F1-Score) and opera- tional feasibility metrics, including processing speed and structural compliance of the output. Results: A strong positive correlation was observed between model size and extraction accuracy. F1-score improved from 0.609 for the 1B model to 0.710 (3B), 0.807 (8B), and 0.828 (70B). While larger models demonstrated superior accuracy and compliance, they incurred higher computational costs. The modest performance difference between 8B and 70B suggests diminishing returns with increasing model size. Conclusions: LLMs represent a viable technology for automating oncology treatment extraction. The 8B-parameter model emerged as a highly effective option, balancing high accuracy and computational efficiency. Selecting an appropriate LLM for deployment in cancer registries involves a trade-off between desired accuracy and available operational resources. Harmonizing extracted entities with the oncology knowledge base facilitates standardized integration into common data models, enhancing data quality for real-world evidence analyses.

artificial intelligence↗

Trimming and Decontamination of Metagenomic Data can Significantly Impact Assembly and Binning Metrics, Phylogenomic and Functional Analysis

Background: Investigators using metagenomic sequencing to study microbiomes often trim and decontaminate reads without knowing their effect on downstream analyses. Objective: This study was designed to evaluate the impacts JGI trimming and decontamination procedures have on assembly and binning metrics, placement of MAGs into species trees, and functional profiles of MAGs extracted from complex rhizosphere metagenomes, as well as how more aggressive trimming impacts these binning metrics. Methods: Twenty-three Miscanthus x giganteus rhizosphere metagenomes were subjected to different combinations and thresholds of force, kmer, and quality trimming and decontamination using BBDuk. Reads were assembled and binned in KBase. Phylogenomic and statistical analyses were applied to evaluate the effects of trimming and decontamination on downstream analyses. Results: We found that JGI trimmed and decontaminated reads had significant impacts on assembly and binning metrics compared to raw reads, including significantly higher total contig counts, more contigs greater than 10k bp in length, and larger total lengths of raw assemblies compared to QC assemblies, and 2.0% lower average contamination of QC MAGs compared to raw MAGs. We also found that differences in the placement of MAGs in species trees increased with decreasing completeness and contamination thresholds. Furthermore, aggressive trimming (Q20) was found to significantly reduce MAG counts. Conclusion: Trimming and decontamination of metagenomics reads prior to assembly can change an investigator’s answer to the questions, “Who is there and what are they doing?” However, mild trimming and decontamination of metagenomic reads with high-quality scores are recommended for removing sample processing and sequencing artifacts.

Whitham, Jason M.↗

Sensitivity of Optical Satellites to Estimate Windthrow Tree-Mortality in a Central Amazon Forest

Windthrow (i.e., trees broken and uprooted by wind) is a major natural disturbance in Amazon forests. Images from medium-resolution optical satellites combined with extensive field data have allowed researchers to assess patterns of windthrow tree-mortality and to monitor forest recovery over decades of succession in different regions. Although satellites with high spatial-resolution have become available in the last decade, they have not yet been employed for the quantification of windthrow tree-mortality. Here, we address how increasing the spatial resolution of satellites affects plot-to-landscape estimates of windthrow tree-mortality. We combined forest inventory data with Landsat 8 (30 m pixel), Sentinel 2 (10 m), and WorldView 2 (2 m) imagery over an old-growth forest in the Central Amazon that was disturbed by a single windthrow event in November 2015. Remote sensing estimates of windthrow tree-mortality were produced from Spectral Mixture Analysis and evaluated with forest inventory data (i.e., ground true) by using Generalized Linear Models. Field measured windthrow tree-mortality (3 transects and 30 subplots) crossing the entire disturbance gradient was 26.9 ± 11.1% (mean ± 95% CI). Although the three satellites produced reliable and statistically similar estimates (from 26.5% to 30.3%, p < 0.001), Landsat 8 had the most accurate results and efficiently captured field-observed variations in windthrow tree-mortality across the entire gradient of disturbance (Sentinel 2 and WorldView 2 produced the second and third best results, respectively). As expected, mean-associated uncertainties decreased systematically with increasing spatial resolution (i.e., from Landsat 8 to Sentinel 2 and WorldView 2). However, the overall quality of model fits showed the opposite pattern. We suggest that this reflects the influence of a relatively minor disturbance, such as defoliation and crown damage, and the fast growth of natural regeneration, which were not measured in the field nor can be captured by coarser resolution imagery. Our results validate the reliability of Landsat imagery for assessing plot-to-landscape patterns of windthrow tree-mortality in dense and heterogeneous tropical forests. Satellites with high spatial resolution can improve estimates of windthrow severity by allowing the quantification of crown damage and mortality of lower canopy and understory trees. However, this requires the validation of remote sensing metrics using field data at compatible scales.

54 ENVIRONMENTAL SCIENCES↗

Exploring Sustainability in Scientific Software through Code Quality & Test Coverage Metrics

Context: Scientific open-source software (SciOSS) plays a foundational role in research and engineering, yet its long-term sustainability has often been overlooked and remains a significant concern. Objective: This study investigates the long-term sustainability of SciOSS through code and test quality metrics. Method: We analyze CASS Software Portfolio projects, classifying them by sustainability and comparing their code structure, test coverage, and links between code quality and testing across the dataset. Results: Sustainable projects show higher, more consistent test coverage and clearer code-test correlations, while unsustainable ones show weaker patterns. Overall, test coverage is low in scientific software, and high complexity and coupling reduce testability. Conclusion: In this study, we present a practical, data-driven approach for assessing sustainability in scientific software, offering a foundation for evaluating long-term software health and supporting future efforts in quality assurance and sustainability monitoring.

Md mushfiqur rahman, Sheikh [University of Tenness↗

IRDFF-II: A New Neutron Metrology Library

High quality nuclear data is the most fundamental underpinning for all neutron metrology applications. This paper describes the release of version II of the International Reactor Dosimetry and Fusion File (IRDFF-II) that contains a consistent set of nuclear data for fission and fusion neutron metrology applications up to 60 MeV neutron energy. The library is intended to support: a) applications in research reactors; b) safety and regulatory applications in the nuclear power generation in commercial fission reactors; and c) material damage studies in support of the research and development of advanced fusion concepts. The paper describes the contents of the library, documents the thorough verification process used in its preparation, and provides an extensive set of validation data gathered from a wide range of neutron benchmark fields. The new IRDFF-II library includes 119 metrology reactions, four cover material reactions to support self-shielding corrections, five metrology metrics used by the dosimetry community, and cumulative fission products yields for seven fission products in three different neutron energy regions. In support of characterizing the measurement of the residual nuclei from the dosimetry reactions and the fission product decay modes, the present document lists the recommended decay data, particle emission energies and probabilities for 68 activation products. It also includes neutron spectral characterization data for 29 neutron benchmark fields for the validation of the library contents. Additional six reference fields were assessed (four from plutonium critical assemblies, two measured fields for thermal-neutron induced fission on 233U and 239Pu targets) but not used for validation due to systematic discrepancies in C/E reaction rate values or lack of reaction-rate experimental data. Another ten analytical functions are included that can be useful for calculating average cross sections, average energy, thermal spectrum average cross sections and resonance integrals. The IRDFF-II library and comprehensive documentation is available online at www-nds.iaea.org/IRDFF/. Evaluated cross sections can be compared with experimental data and other evaluations at www-nds.iaea.org/exfor/endf.htm. The new library is expected to become the international reference in neutron metrology for multiple applications.

IAEA IRDFF↗

Satellites for long-term monitoring of inland U.S. lakes: The MERIS time series and application for chlorophyll-a

Lakes and other surface fresh waterbodies provide drinking water, recreational and economic opportunities, food, and other critical support for humans, aquatic life, and ecosystem health. Lakes are also productive ecosystems that provide habitats and influence global cycles. Chlorophyll concentration provides a common metric of water quality, and is frequently used as a proxy for lake trophic state. Here, we document the generation and distribution of the complete MEdium Resolution Imaging Spectrometer (MERIS; Appendix A provides a complete list of abbreviations) radiometric time series for over 2300 satellite resolvable inland bodies of water across the contiguous United States (CONUS) and more than 5,000 in Alaska. This contribution greatly increases the ease of use of satellite remote sensing data for inland water quality monitoring, as well as highlights new horizons in inland water remote sensing algorithm development. We evaluate the performance of satellite remote sensing Cyanobacteria Index (CI)-based chlorophyll algorithms, the retrievals for which provide surrogate estimates of phytoplankton concentrations in cyanobacteria dominated lakes. Our analysis quantifies the algorithms' abilities to assess lake trophic state across the CONUS. As a case study, we apply a bootstrapping approach to derive a new CI-to-chlorophyll relationship, ChlBS, which performs relatively well with a multiplicative bias of 1.11 (11%) and mean absolute error of 1.60 (60%). While the primary contribution of this work is the distribution of the MERIS radiometric timeseries, we provide this case study as a roadmap for future stakeholders' algorithm development activities, as well as a tool to assess the strengths and weaknesses of applying a single algorithm across CONUS.

MERIS timeseries↗

Counter Data Paucity through Adversarial Invariance Encoding: A Case Study on Modeling Battery Thermal Runaway

Lithium-ion batteries, widely used for their durability and high energy storage, face the risk of internal short circuits leading to catastrophic thermal runaway events. These events, triggered by external stimuli like mechanical loads, pose safety concerns in applications such as electric vehicles. Detecting and understanding thermal runaway events is crucial, but physics-driven models struggle to explain the non-linear evolution of battery temperature during these events, considering factors like material composition and state-of-charge. Due to the rarity of these events and the cost of data collection, we propose a deep learning (DL) model to predict battery temperature responses during thermal runaway. The challenge lies in the scarcity of data, making traditional DL models prone to overfitting and learning low-quality representations of the complex process.Our approach introduces a novel few-shot architecture that incorporates an adversarially governed invariant encoding process. This architecture aims to distill "invariant" relationships by addressing distributional shifts in data across various battery properties, facilitating the detection of thermal runaway events. Specifically, our results demonstrate that deep learning models conditioned on these "invariant" representations outperform state-of-the-art baselines, achieving a remarkable 96.8% performance improvement in terms of the popular metric MAPE. This framework presents a promising direction for enhancing battery safety modeling, particularly in the context of rare and complex events like thermal runaway. Our code and code and dataset used for the paper are public1.

Tabassum, Anika [ORNL] (ORCID:0000000254600955)↗

Developing Lightning Prediction Tools for the CCAFS Dual-Polarimetric Radar

NASA Marshall Space Flight Center and the University of Alabama Huntsville are collaborating with the 45th Weather Squadron (45WS) to develop improved lightning prediction capabilities for the new C-band dual-polarimetric weather radar being acquired for use by 45WS and launch weather forecasters at Cape Canaveral Air Force Station (CCAFS). In particular, these algorithms will focus on lightning onset, cessation and combined lightning-radar applications for convective winds assessment. Research using radar reflectivity (Z) data for prediction of lightning onset has been extensively discussed in the literature and subsequently applied by launch weather forecasters as it pertains to lightning nowcasting. Currently the forecasters apply a relatively straight forward but effective temperature-Z threshold algorithm for assessing the likelihood of lightning onset in a given storm. In addition, a layered VIL above the freezing level product is used as automated guidance for the onset of lightning. Only limited research and field work has been conducted on lightning cessation using Z and vertically-integrated Z for determining cessation. Though not used operationally vertically-integrated Z (basis for VIL) has recently shown promise as a tool for use in nowcasting lightning cessation. The work discussed herein leverages and expands upon these and similar reflectivity-threshold approaches via the application/addition of over two decades of polarimetric radar research focused on distinct multi-parameter radar signatures of ice/mixed-phase initiation and ice-crystal orientation in highly electrified convective clouds. Specifically, our approach is based on numerous previous studies that have observed repeatable patterns in the behavior of the vertical hydrometeor column as it relates to the temporal evolution of differential reflectivity and depolarization (manifested in either LDR or p(sub hv)), development of in-situ mixed and ice phase microphysics, electric fields, and ensuing lightning in the sub-tropical/tropical convection typical of the southeastern U.S., Maritime Continent, and southwestern Amazon. The polarimetric signatures detected in this setting provide a basis for automated 3-D detection of hydrometeor types in fuzzy logic hydrometeor identification algorithms (HID). Our working hypothesis is that improvement in lightning onset warning lead time and specificity for a given storm, relative to application of a Z-threshold algorithm, should arise as a consequence of the ability of dual-polarimetric radar to unambiguously detect and identify (through HID algorithms) the updraft elevation of rain-water cores above the freezing level and subsequent onset of drop freezing, riming, and robust mixed phase processes leading to significant charge separation and lightning. This type of algorithm, though dependent on the quality of the polarimetric data should be less susceptible to variable Z-calibration that can impact a given Z-threshold approach. To facilitate development of the algorithm while the 45WS dual-pol radar is in its current test stages and to evaluate the impact of polarimetric data quality (e.g., modified scan parameters and sampling) on the ensuing algorithms, we are using the ARMOR C-band dual-pol radar in Huntsville combined with N. Alabama LMA data and ARMOR HID algorithms [NCAR algorithm modified for application at C-band] in a testbed fashion. For lightning cessation we are revisiting the application of differential propagation phase variables for the monitoring of ice crystal alignment driven by in-cloud electric fields combined with metrics of ice water path (i.e., vertically integrated reflectivity). Importantly it should be noted that this approach is still very much a research topic and as such, we will explore operational applications that involve radar frequencies other than C-Band by using the UAH MAX X-band dual-pol radar in slow staring modes.

Petersen, W. A.↗

An experimental evaluation of the Sternberg task as a workload metric for helicopter Flight Handling Qualities (FHQ) research

The objective was to determine whether the Sternberg item-recognition task, employed as a secondary task measure of spare mental capacity for flight handling qualities (FHQ) simulation research, could help to differentiate between different flight-control conditions. FHQ evaluations were conducted on the Vertical Motion Simulator at Ames Research Center to investigate different primary flight-control configurations, and selected stability and control augmentation levels for helicopters engaged in low-level flight regimes. The Sternberg task was superimposed upon the primary flight-control task in a balanced experimental design. The results of parametric statistical analysis of Sternberg secondary task data failed to support the continued use of this task as a measure of pilot workload. In addition to the secondary task, subjects provided Cooper-Harper pilot ratings (CHPR) and responded to workload questionnaire. The CHPR data also failed to provide reliable statistical discrimination between FHQ treatment conditions; some insight into the behavior of the secondary task was gained from the workload questionnaire data.

Hemingway, J. C.↗

Automating the Analysis of Large Language Models Responses through Zero-Shot Question Answering

Recent advancements in Large Language Models (LLMs) have shown significant potential in various applications, yet their evaluation, particularly in zero-shot question answering scenarios, remains a challenging task. In this study, our objective was to explore precision metrics for Large Language Models (LLM) and design and implement a software pipeline to automatically evaluate LLMs' outputs under zero-shot question answering. Zero-shot question answering involves a model providing answers to questions about topics it hasn't seen during training. It leverages the principles of zero-shot learning by relying on semantic understanding and generalization from related knowledge. The data used was metadata from medical databases on congenital heart disease. We explored eleven LLM metrics and selected three for our evaluation: BLEU, BERTScore, and MoverScore. BLEU calculates a score based on the overlap of n-grams (contiguous sequences of n items, typically words) between the machine-generated translation and the reference translations. Higher BLEU scores indicate better correspondence between the machine-generated and human-generated translations. BERTScore is a metric used to evaluate the quality of machine-generated text by measuring the similarity of token embeddings produced by BERT (Bidirectional Encoder Representations from Transformers) between the generated text and reference text. MoverScore is a metric that quantifies the dissimilarity between the distributions of word embeddings from machine-generated text and reference text, emphasizing semantic similarity over exact token overlap. We also introduced HBKI, a composite metric summarizing these approaches. We tested five models —GPT-3, Llama-2, Gemini 1.5 Pro, Solar 10.7B, and Mixtral-8x7b. Our software pipeline, designed and implemented using Object-Oriented Programming principles, allows users to customize the selection and extraction of features for topics of interest in their own research. Our results show that MoverScore delivered the most precise evaluation of the LLM's outputs, while Mixtral-8x7b achieved the best overall performance in extracting metadata from the databases.

97 MATHEMATICS AND COMPUTING↗

Hydrological connectivity: a review and emerging strategies for integrating measurement, modeling, and management

This review synthesizes methods for measuring, modeling, and managing hydrologic connectivity, offering pathways to improve practices and address environmental challenges (e.g., climate change) and sustainability. As a key driver of water movement and nutrient cycling, hydrologic connectivity influences flood mitigation, water quality regulation, and biodiversity conservation. However, traditional field-based methods (e.g., dye tracing), indirect measurements (e.g., runoff analysis), and remote sensing techniques (e.g., InSAR) often struggle to capture the complexity of catchment-scale interactions. Similarly, modeling approaches—including process-based and percolation theory-based models, graph theory, and entropy-based metrics—face limitations in fully representing these interconnected processes. Both modeling and measurement techniques are constrained by inadequate spatial and temporal coverage, high data demands, computational complexity, and difficulties in representing subsurface connectivity. Subsequently, we critique current management practices that prioritize isolated variables (e.g., streamflow, sediment transport) over system-wide strategies and emphasize the need for adaptive, connectivity-based approaches in water resource planning and restoration. Moving forward, we highlight the importance of interdisciplinary collaboration, technological innovations (e.g., AI-driven modeling, real-time monitoring), and integrated frameworks to improve connectivity measurement, modeling, and adaptive management to restore fragmented hydrologic networks. This integrated approach sets the stage for transformative water resource management, fostering proactive policy development and stakeholder engagement.

Dwivedi, Dipankar↗