Search NASA⌕ Search

SEARCH · Search NASA

Results for “Benchmarking”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 541 records · Page 30

Roadmap and Benchmarking: Privacy in Federated Load Forecasting

Data-driven techniques for energy demand forecasting continue to emerge with promising impacts on distribution grid planning. However, the development of robust and generalizable machine learning models requires that representative high quality training data are available. Distributed energy resources have begun to embed intelligence, gathering large amounts of data on customer demand, behavior, and household devices that are connected to the grid. Though utilities aggregate meter-level demand data for load shaping, demand response, outage management, reliability planning, and billing applications, there lies an inherent privacy concern in sharing consumption data that may identify individual consumer behavioral patterns. Hence, while sharing the data is crucial, the private sensitive customer data must be safeguarded from being exposed or manipulated. In this study, we propose a roadmap for implementing a based privacy preserving framework to support the advancement of data-driven analytics in data-sensitive distributed energy resources environments. The roadmap incorporates federated learning–a distributed training framework, differential privacy–a statistical framework that provides guarantees to safeguard the leakage of sensitive data, secure multiparty computation and homomorphic encryption– techniques for encrypting model gradients and applying secure aggregation on the server. Moreover, we perform baseline experiments on the federated short-term load forecasting (STLF) task using open-source residential load profile datasets, offering insights into the challenges of integrating differential privacy into federated learning.

Abebe, Waqwoya [Oak Ridge National Laboratory (ORN↗

Forest aboveground biomass estimation through integration of sentinel-2 and PALSAR-2 time series: assessing models trained on GEDI and field inventory benchmarks

Accurate and spatially explicit forest Aboveground Biomass (AGB) mapping through remote sensing is critical for quantifying terrestrial carbon stocks and informing effective forest management strategies. However, AGB estimation in dense forests with complex terrain remains challenging due to satellite sensor signal saturation problem (saturation issue occurs in high biomass forests), structural complexity, and limited ground truth for calibration. This study presents a novel framework that integrates multi-temporal Sentinel-2 optical imagery, ALOS PALSAR-2 Synthetic Aperture Radar (SAR) data, and topographic variables with explainable Machine Learning to map AGB across mountainous forests within subtropical and temperate oceanic climate zones of Mexico. We evaluate the effects of temporal granularity and sensor synergy by comparing multiple temporal inputs and sensor configurations (Sentinel-2, PALSAR-2, and their fusion), and assess model performance using two reference datasets: NASA GEDI LiDAR-derived biomass and Mexico’s National Forest and Soil Inventory (INFyS). Our results showed that models trained on INFyS consistently outperformed those trained on GEDI, highlighting limitations in GEDI’s reliability in biomass estimates within this study region. Furthermore, the integration of Sentinel-2 and PALSAR-2 provided improved predictions compared to single-sensor models, particularly when combined with temporally explicit yearly statistics. The best-performing model, which was trained on INFyS data, and considered both Sentinel-2 and PALSAR-2 yearly statistics, as well as topographic variables, achieved an R2 of 0.64, RMSE of 51.10 Mg/ha, and relative RMSE (rRMSE) of 58.69%. Explainable ML analysis identified Sentinel-2 spectral indices and topographic features as key predictors, while PALSAR-2 metrics provided complementary information, partially mitigating saturation effects in high-biomass areas. Specifically, integrating both sensors substantially improved AGB estimation in high biomass forest (≥200 Mg/ha), yielding 98% gains over optical-only model, with resulting estimates exceeding GEDI L4B by 29% and ESA-CCI-BIOMASS by 174%. Terrain-stratified analysis indicated close agreement with GEDI in low-slope areas, with increasing divergence as slope steepness increased, while estimates remained consistently higher than ESA-CCI-BIOMASS across all slope classes. The proposed approach advances multi-sensor fusion and temporal feature engineering for AGB mapping using open-access satellite datasets, providing a scalable and reproducible framework for annual biomass monitoring in topographically complex mountainous forests. The resulting 25 m resolution biomass product has the potential to provide spatially detailed information for forest monitoring and may support applications in carbon accounting and forest management.

54 ENVIRONMENTAL SCIENCES↗

Fundamental benchmarking of the discharge properties of negative electrodes in lead acid batteries

The unprecedented scale of energy storage deployment needed to allow high penetration of intermittent renewable energy sources into the electrical grid places significant economic cost and performance targets on battery technologies. Lead batteries, owing to their highly abundant and inexpensive raw materials, can be considered an important solution to grid storage as long as they achieve high material utilization, fast recharge rates, and long cycle life. To address the limitations in material utilization, we need to revisit the electrochemical processes during the discharge step and answer the following question: what are the fundamental limits of the discharge reaction? In this work we discuss how well-defined lead metal surfaces with nanometer scale roughness allow us to resolve the accessible discharge capacity for both faradaic and non-faradaic processes as a function of electrolyte concentration and in the presence of traditional additives. The direct connection between PbSO 4 layer morphology and discharge rates gives us important insights on the mechanism of discharge as related to the dissolution and passivation events. In conclusion, this work sets the stage for reimagining the design of lead batteries with implications to grid storage applications.

25 ENERGY STORAGE↗

Benchmarking optimization methods for materials research: Gradient descent and Bayesian optimization for lithium-ion battery aging diagnostics

Accurate and efficient parameter estimation is essential for battery diagnostics and aging analysis. Here, in this study, we compare two optimization-based approaches—gradient descent and Bayesian optimization—for extracting parameters from differential voltage analysis in lithium-ion batteries. While these techniques are widely used, their relative strengths and limitations for this application are not well understood. The study evaluates the trade-offs between these methods in terms of result quality, computational cost, and reliability within this specific application. The diagnostic results from our battery data suggest adopting gradient descent as an initial method for rapid and efficient analysis, while employing more stable optimization techniques, such as Bayesian optimization, as a verification step to mitigate potential instability. Comparing the two methods provides information on algorithmic choice, while inspiring further discussions on selecting appropriate techniques for specific research tasks.

Zhao, Ziqing [Boston Univ., MA (United States)] (O↗

Benchmarking state-of-the-art theory and empirical models of pionless neutrino-argon scattering in GENIE

Upcoming experiments need improved simulations of neutrino scattering. This work uses the popular GENIE event generator to test a variety of neutrino interaction models against recent MicroBooNE measurements of pionless charged-current interactions. The GENIE code can easily interchange model components, including nucleon form factor parameterizations, quasielastic cross-section calculations, treatments of the nuclear ground state and hadronic final-state interactions. Leveraging this software capability in comparisons with MicroBooNE data, the performance of some of GENIE’s most theoretically sophisticated model components is evaluated and contrasted with more empirically-driven alternatives.

Neutrino cross section↗

Worldwide benchmarking of cost-effective radiometers for direct and diffuse irradiance

Solar energy projects can benefit from direct normal irradiance (DNI) and diffuse horizontal irradiance (DHI) measurements during all project phases. Several commercial measurement systems for DNI and DHI are available. Sun trackers with pyranometers and pyrheliometers can provide highly accurate measurements but are often impractical in solar energy applications. For less expensive and more robust sensors, it is often unclear which accuracy can be expected under a project site's specific atmospheric conditions. We address this challenge through our dedicated experimental comparison of relevant sensor systems (rotating shadowband irradiometer [short RSI], Delta-T SPN1, EKO MS-90, PyranoCam, Sunto CaptPro, Kipp & Zonen CSD3) at up to six sites worldwide. The RSI systems (rRMSD 3 to 8.6%, DNI; 4.8 to 7.6%, DHI) and PyranoCam (rRMSD 2.6 to 5.2%, DNI; 4.4 to 5.8%, DHI) exhibit similar error metrics and are the most accurate systems in the test. Delta-T SPN1 and EKO MS-90 (rRMSD 6.8 to 15%, DNI; 10.6 to 20.1%, DHI) but especially Kipp & Zonen CSD3 and Sunto CaptPro show significant deviations (rRMSD 17.7 to 20%, DNI; 33 to 58%, DHI). We evaluate the influence of relevant atmospheric parameters on the sensors' accuracies by a rather unique measurement setup. MS-90's DNI errors depend on DNI itself, with overestimations for low reference DNI. The deviations of SPN1's DHI and DNI measurements increase sharply in situations with high circumsolar irradiance. Also CaptPro and CSD3's increased measurement errors are related to circumsolar irradiance. For RSI and PyranoCam, only moderate influences on the measurements are identified, indicating a general applicability of these instruments.

14 SOLAR ENERGY↗

Benchmarking Concentration and Extraction Methods for Wastewater-Based Surveillance of Eight Human Respiratory Viruses: Implications for Rapid Application to Novel Pathogens

To provide early warning and support a rapid response to a novel virus through wastewater surveillance, it would be ideal to understand in advance which concentration and extraction methods are likely to be effective for dPCR-based methods, depending on virus characteristics. In this study, we spiked raw wastewater samples with eight human respiratory viruses and processed them with four methods that concentrate and/or extract nucleic acids from both liquid and solid fractions (Promega, Nanotrap, and InnovaPrep) or only the solid fraction of wastewater (Solids). Our findings provide encouraging evidence that all four methods combined with dPCR could detect an emerging virus in wastewater, although they differed in sensitivity. The pattern of recovery efficiency for adenoviruses, coronaviruses, and influenza A viruses was consistent across methods, with Promega producing higher median recovery efficiencies, while distinct patterns were observed for coxsackieviruses. We also normalized the concentration data with two endogenous fecal indicators, PMMoV and Carjivirus (formerly crAssphage). We found that normalization could reduce method-associated differences if the indicator exhibited a recovery pattern similar to that of the target virus. These findings can guide the selection of concentration and extraction methods for wastewater monitoring based on the properties of target viruses, thus enhancing pandemic preparedness.

Biological and medical sciences↗

Benchmarking Correlation-Consistent Basis Sets for Frequency-Dependent Polarizabilities with Multiresolution Analysis

This paper presents the first converged frequency-dependent HF polarizability results for general molecules, on a set of 89 closed-shell atoms and molecules. The solver employs multiresolution analysis (MRA) in a multiwavelet basis to compute both ground and response states to a guaranteed precision, which are validated against independent numerical grid calculations on atoms and linear molecules. The MRA ground-state energies and response properties are used to evaluate results in correlation-consistent basis sets up to 5Z augmented with either single or double diffuse functions and core-polarization functions. Systematic trends are revealed through consideration of chemical composition as well as the use of machine learning to cluster convergence trends, the latter suggesting the possibility of learning and correcting basis-set error.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Benchmarking Guanidinium Organosulfonate Hydrogen-Bonded Frameworks for Structure Determination of Encapsulated Guests

Single crystal X-ray diffraction (SCXRD) is arguably the most definitive method for molecular structure determination, but it is often challenged by compounds that are liquids or oils at room temperature or do not form crystals adequate for analysis. Our laboratory previously reported a simple, cost-effective, single-step crystallization method based on guanidinium organosulfonate (GS) hydrogen bonded frameworks for structure determination of a wide range of encapsulated guest molecules, including assignment of the absolute configuration of chiral centers. Herein, we expand on those results and report a head-to-head comparison of the GS method with adamantoid “molecular chaperones”, which have been reported to be useful hosts for structure determination. Inclusion compounds limited to only two GS hosts are characterized by low R1 values and Flack parameters, infrequent disorder of the host and guest, and manageable disorder when it does exist. The structures of some target molecules that were not included or resolved using the adamantoid chaperones were successfully included and resolved by the GS hosts, and vice versa. Of the 32 guests attempted by the GS method, 31 inclusion compounds afforded successful guest structure solutions, a 97% success rate. The GS hosts and adamantoid chaperones are complementary with respect to guest inclusion, arguing that both should be employed in the arsenal of methods for structure determination. Furthermore, the low cost of organosulfonate host components promises an accessible route to molecular structure determination for a wide range of users.

36 MATERIALS SCIENCE↗

Complexity-calibrated benchmarks for machine learning reveal when prediction algorithms succeed and mislead

Abstract Recurrent neural networks are used to forecast time series in finance, climate, language, and from many other domains. Reservoir computers are a particularly easily trainable form of recurrent neural network. Recently, a “next-generation” reservoir computer was introduced in which the memory trace involves only a finite number of previous symbols. We explore the inherent limitations of finite-past memory traces in this intriguing proposal. A lower bound from Fano’s inequality shows that, on highly non-Markovian processes generated by large probabilistic state machines, next-generation reservoir computers with reasonably long memory traces have an error probability that is at least $$\sim 60\%$$ ∼ 60 % higher than the minimal attainable error probability in predicting the next observation. More generally, it appears that popular recurrent neural networks fall far short of optimally predicting such complex processes. These results highlight the need for a new generation of optimized recurrent neural network architectures. Alongside this finding, we present concentration-of-measure results for randomly-generated but complex processes. One conclusion is that large probabilistic state machines—specifically, large $$\epsilon$$ ϵ -machines—are key to generating challenging and structurally-unbiased stimuli for ground-truthing recurrent neural network architectures.

97 MATHEMATICS AND COMPUTING↗