Search NASA⌕ Search

SEARCH · Search NASA

Results for “reproducibility”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

PQML: Enabling the Predictive Reproducibility on NISQ Machines for Quantum ML Applications

Quantum computing represents a groundbreaking approach to high-performance computing. In recent years, quantum computers have progressed from single-qubit processors to systems boasting over 400 qubits. The presence of such a large number of qubits offers significant advantages, including enhanced computational speed—a capability beyond classical computing methods. However, the current stage of quantum computing is referred to as the noisy intermediate-scale quantum (NISQ) era. The existence of noise in this era presents challenges in testing quantum computing applications, leading to considerable variance in application results. Furthermore, the diverse noise characteristics observed across different machines exacerbate this issue, complicating the selection of the appropriate machine for application execution. In response to these challenges, we introduce our Predictive Quantum Machine Learning (PQML) tool. This tool is designed to predict outcomes when executing identical quantum machine learning applications—specifically, a critical suite of variational quantum algorithms—across various quantum computers during the NISQ era. This effort relies on data collected over a 12-month period. To the best of our knowledge, this study represents the first attempt to ensure reproducibility across quantum computers for complex circuits. Additionally, we have developed a model capable of forecasting the accuracy of quantum computers for variational quantum algorithms, with a particular emphasis on quantum machine learning as a case study.

Senapati, Priyabrata [Kent State University]↗
DOE OSTI2023

Important Factors for the Reliable and Reproducible Preparation of Non-aqueous Electrolyte Solutions for Lithium Batteries

Electrolyte solutions play a crucial role in rechargeable lithium batteries. Driven by an ever-increasing demand for better batteries, more and more stringent performance requirements are being asked of electrolytes. This demand to achieve electrolytes beyond the state-of-the-art has driven an increasing number of publications over the past decade. Nevertheless, a detailed description of electrolyte preparation is frequently missing when reporting new electrolytes for batteries, creating a barrier for others to reproduce important findings. In addition, false or misleading information in a paper may result from unintentional inaccuracies during the electrolyte preparation process. This article highlights important factors for the reliable and reproducible preparation of non-aqueous electrolyte solutions for lithium batteries, with the aim of encouraging researchers to follow more rigorous reporting standards.

Cao, Xia↗
DOE OSTI2024

Recommendations for improving rigor and reproducibility in site specific characterization

Heterogeneous catalysis is driven by the interaction of reactant molecules and the catalyst surface. The locus of this interaction as well as the surrounding ensemble of atoms is referred to as the catalyst active site. Active site characterization attempts to distinguish active catalytic sites from inactive surface sites, to elucidate the structural and chemical nature of active sites, and to quantify active site concentration. Numerous techniques have been demonstrated to provide compositional and structural information about the active sites within a catalyst. However, each technique has its own limitations and experimental pitfalls that can lead to data misinterpretation or irreproducible results. Further, this work aims to provide an overview of the types of data that can be collected, to outline common experimental challenges and how to avoid them, and to assemble relevant references for the most used active site characterization techniques. More broadly, we aim to outline best practices for researchers to collect, interpret, and report active site characterization data in a way that provides the most benefit to the broader catalysis community. Increasing the rigor and reproducibility of active site characterization offers a strategy to better link properties with catalytic performance and to enable the community to develop consensus concerning these relationships.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2024

MISIP: a data standard for the reuse and reproducibility of any stable isotope probing-derived nucleic acid sequence and experiment

DNA/RNA-stable isotope probing (SIP) is a powerful tool to link in situ microbial activity to sequencing data. Every SIP dataset captures distinct information about microbial community metabolism, process rates, and population dynamics, offering valuable insights for a wide range of research questions. Data reuse maximizes the information derived from the labor and resource-intensive SIP approaches. Yet, a review of publicly available SIP sequencing metadata showed that critical information necessary for reproducibility and reuse was often missing. Here, we outline the Minimum Information for any Stable Isotope Probing Sequence (MISIP) according to the Minimum Information for any (x) Sequence (MIxS) framework and include examples of MISIP reporting for common SIP experiments. Our objectives are to expand the capacity of MIxS to accommodate SIP-specific metadata and guide SIP users in metadata collection when planning and reporting an experiment. The MISIP standard requires 5 metadata fields—isotope, isotopolog, isotopolog label, labeling approach, and gradient position—and recommends several fields that represent best practices in acquiring and reporting SIP sequencing data (e.g., gradient density and nucleic acid amount). The standard is intended to be used in concert with other MIxS checklists to comprehensively describe the origin of sequence data, such as for marker genes (MISIP-MIMARKS) or metagenomes (MISIP-MIMS), in combination with metadata required by an environmental extension (e.g., soil). The adoption of the proposed data standard will improve the reuse of any sequence derived from a SIP experiment and, by extension, deepen understanding of in situ biogeochemical processes and microbial ecology.

Simpson, Abigayle↗
DOE OSTI2021

Achieving Highly Reproducible Results in Graphite-based Li-ion Full Coin Cells

Electrode materials, cathodes or anodes, are mostly evaluated in half coin cells where lithium (Li) metal is used. However, due to the instability nature of Li metal, the cycling performances of the half cells is more likely reflecting Li anode stability instead of the coupled electrodes, questioning the conclusions. Full coin cells utilizing graphite as the anode effectively assesses materials. But the key factors of preparing full coin cells with reproducible performances have rarely been discussed and overlooked leading to many inconsistent results. This work discusses the scientific gap between cell preparation and observed electrochemical performances to unveil the key parameters for achieving consistent and reliable results in full coin cells, critical for effective evaluation of any new materials tested under relevant conditions. The knowledge from this work is broadly applicable in other full cells systems such as Si-based full cells or Na-ion batteries.

Hu, Jiangtao↗
DOE OSTI2020

Quieting a noisy antenna reproduces photosynthetic light-harvesting spectra

Photosynthesis achieves near unity light-harvesting quantum efficiency yet it remains unknown whether there exists a fundamental organizing principle giving rise to robust light harvesting in the presence of dynamic light conditions and noisy physiological environments. Here, we present a noise-canceling network model that relates noisy physiological conditions, power conversion efficiency, and the resulting absorption spectra of photosynthetic organisms. Using light conditions in full solar exposure, light filtered by oxygenic phototrophs, and light filtered under seawater, we derived optimal absorption characteristics for efficient solar power conversion. We show how light-harvesting antennae can be tuned to maximize power conversion efficiency by minimizing excitation noise, thus providing a unified theoretical basis for the observed wavelength dependence of absorption in green plants, purple bacteria, and green sulfur bacteria.

Science & Technology - Other Topics↗
DOE OSTI2022

Interlaboratory Reproducibility of Contour Method Data in a High Strength Aluminum Alloy

The contour method for residual stress measurement has seen significant development, but an experimental reproducibility study utilizing physical samples has not been published. A double-blind reproducibly study is reported, having scope beginning with EDM cutting and ending with residual stress calculation. A reinforced I-beam sample geometry is identified for its unique residual stress profile when extracted from residual stress bearing quenched aluminum bar (7050-T74). Contour measurements are prescribed on a midplane of symmetry with dimensions 24.0 mm by 50.0 mm. Fourteen identically prepared samples are fabricated from a single long bar with well characterized and uniform residual stress. Five samples throughout the bar are identified for planning measurements to validate sample uniformity and overall suitability of the residual stress field. The planning measurements employ a range of techniques: contour method, neutron diffraction, and hole-drilling. Eight samples are distributed to an international group of participants to execute their standard measurement practice. A double-blind process is followed to provide anonymity. Results are provided by eight participants: six being self-similar and two being quite different, the latter set aside as outliers. An average residual stress field is established from non-outlying results and the spatial distribution of reproducibility standard deviation is determined. The average stress field ranges from -60 to 70 MPa and the reproducibility standard deviation averages 8.1 MPa on the measurement plane. The average reproducibility standard deviation is about 3 × larger for points within 1.0 mm of plane boundaries (17.6 MPa) than for the remaining points (6.1 MPa). Reproducibility standard deviation (among different labs) for contour method residual stress measurement is found to be very similar to repeatability standard deviation (in a single lab) reported in prior work. The reproducibility observed here, for the entire measurement process, is also similar to that found in a prior reproducibility study limited to contour method data analysis.

36 MATERIALS SCIENCE↗
DOE OSTI2020

Algorithms for Efficient Reproducible Floating Point Summation

We define “reproducibility” as getting bitwise identical results from multiple runs of the same program, perhaps with different hardware resources or other changes that should not affect the answer. Many users depend on reproducibility for debugging or correctness. However, dynamic scheduling of parallel computing resources, combined with nonassociative floating point addition, makes reproducibility challenging even for summation, or operations like the BLAS. We describe a “reproducible accumulator” data structure (the “binned number”) and associated algorithms to reproducibly sum binary floating point numbers, independent of summation order. We use a subset of the IEEE Floating Point Standard 754-2008 and bitwise operations on the standard representations in memory. Our approach requires only one read-only pass over the data, and one reduction in parallel, using a 6-word reproducible accumulator (more words can be used for higher accuracy), enabling standard tiling optimization techniques. Summing n words with a 6-word reproducible accumulator requires approximately 9 n floating point operations (arithmetic, comparison, and absolute value) and approximately 3 n bitwise operations. The final error bound with a 6-word reproducible accumulator and our default settings can be up to 2 29 times smaller than the error bound for conventional (recursive) summation on ill-conditioned double-precision inputs.

Computer Science↗
DOE OSTI2020

Interlaboratory Reproducibility of Contour Method Data Analysis and Residual Stress Calculation

While the contour method for residual stress assessment has developed rapidly, no published study documents its interlaboratory reproducibility. Objective: Here we report an initial reproducibility experiment focused on contour method data analysis and residual stress calculation. The experiment uses surface topography data from a physical process simulation of elastic-plastic beam bending. The simulation provides surface topography, for input to the contour method data analysis, as well as a known residual stress field with 130 MPa peak magnitude. Additionally, to increase realism, noise and specific artifacts are added to the topography data. A group of participants received the topography data (without the known residual stress), independently analyzed the data, and submitted results as a two-dimensional residual stress field. Analysis of submissions provides a group average residual stress field and the spatial distribution of reproducibility standard deviation. The group average residual stress agrees with the known stress in magnitude and spatial trend. The reproducibility standard deviation ranges from 2 to 54 MPa over the measurement plane, with an average of 5.4 MPa. Reproducibility standard deviation is smaller in the cross-section interior (≤ 5 MPa), modest near local extrema in the stress field (5 to 10 MPa), and larger near the cross-section boundaries (10 to 30 MPa). Overall, the largest values of reproducibility standard deviation (up to 54 MPa) occur in limited areas where artifacts had been added to the topography data; while some participants identified and removed these artifacts, some did not, leading to systematic differences that elevated the standard deviation.

42 ENGINEERING↗
DOE OSTI2023

Workshop on Addressing Rigor and Reproducibility in Thermal, Heterogeneous Catalysis

Heterogeneous catalysis has long served as the bedrock of the manufacturing of energy carriers, fuels and chemicals, and various technologies for pollution abatement. The significant complexity and variability spanning the entire breadth of catalyst material properties, synthesis methods, characterization techniques, and evaluation procedures, has focused attention on the need to establish community-accepted best practices for ensuring high-quality, benchmarked, and reproducible data. In addition, increased societal urgency to transition to clean energy and reduce greenhouse gas concentrations has incentivized interdisciplinary, convergent, and translational approaches to catalysis research in recent years. Research engineers and scientists with expertise cutting broadly across materials science, chemical synthesis, interfacial science, spectroscopy, and methods of data science and computational simulation, all bring diverse and important perspectives to catalysis research, but often with little awareness of the complexity of catalytic systems, especially in their working environment. As has already occurred in other scientific fields, there has been growing recognition and consensus in the heterogeneous catalysis research community that mechanisms are needed to improve the rigor and reproducibility (R&R) of experimental measurements, to ensure alignment of the broader research community with a common core of best practices specific to the realization of high-quality catalysis research. Similarly, the field is moving rapidly toward computationally informed and data science-driven catalyst design, but the success of implementing such predictive tools hinges on model training and validation rooted in rigorously obtained and reproducible experimental data that are benchmarked to common specifications. As such, this workshop was convened to prepare a report summarizing best practices for reporting data and performing experiments that researchers can use to benchmark, validate, and reproduce data in specific sub-fields of thermal, heterogeneous catalysis. Additionally, we discussed recommendations for future actions that may improve R&R in this field. The workshop organizers and participants include a diverse range of catalysis researchers from various employment sectors (e.g., academia, industry, national laboratory), institutional mission and resources (e.g., PhD-granting research universities, non-PhD-granting teaching universities), career stage (e.g., early, mid and late-career), technical expertise, and demographic background. This diverse group was involved in the discussion of workshop agenda items, writing this report, and discussing possible future action items for the community to consider, which helped ensure that a broad range of perspectives were captured in the description of the problems at hand and the creation of actionable solutions that may be effectively adopted by the diverse practitioners in catalysis research. Importantly, this group of workshop participants also included very early career researchers (e.g., senior PhD students, postdoctoral scholars) who will become the next generation of scientific leaders in various sectors, thus capturing emerging perspectives of newcomers to the field to shape its future while positively impacting the development of its future workforce. We envision that this effort will help advance the field of catalysis science by improving the rigor and reproducibility of experimental data collected by current researchers and future newcomers to the field, which is of broad importance to health and vitality of any scientific discipline. Therefore, best practices identified in this endeavor for thermal heterogeneous catalysis can be translated to such efforts in other areas of catalysis and other scientific fields involving the study of materials, and vice versa. We also envision this to be an ongoing effort, with future workshops that are convened to discuss issues of rigor and reproducibility on technical topics that were unable to be covered in this workshop due to its scope limitations, and as emerging methods and materials become more prevalent in the research community.

36 MATERIALS SCIENCE↗
DOE OSTI2026

Reproducibility, Replicability, and Research Quality in Homogeneous Catalysis

Researchers from all sectors of homogeneous catalysis convened in response to concerns regarding reproducibility in science to analyze the issue and provide recommendations. In addition to an in- person workshop, the group engaged the broader homogeneous catalysis community through a webinar series and virtually during the workshop. Results of the project affirm that homogeneous catalysis is not in a reproducibility crisis, as evidenced by the field’s current and past contributions to society that have led to economic growth and advances in a range of industries from agriculture to consumer goods to human health. However, it is not uncommon for researchers to encounter obstacles related to reproducibility. Ensuring reproducibility remains the responsibility of the community, both in current work and in training future researchers. This report is intended to engage key stakeholders, including disciplinary societies, publishers, employers, research leaders, and researchers, in practices that maximize reproducible homogeneous catalysis and ensure continued innovation and translatable discoveries. Recommendations made herein are also framed to be applicable beyond homogeneous catalysis, empowering the broader chemical if not scientific community.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2022

Characterizing the Reproducibility of Noisy Quantum Circuits

The ability of a quantum computer to reproduce or replicate the results of a quantum circuit is a key concern for verifying and validating applications of quantum computing. Statistical variations in circuit outcomes that arise from ill-characterized fluctuations in device noise may lead to computational errors and irreproducible results. While device characterization offers a direct assessment of noise, an outstanding concern is how such metrics bound the reproducibility of a given quantum circuit. Here, we first directly assess the reproducibility of a noisy quantum circuit, in terms of the Hellinger distance between the computational results, and then we show that device characterization offers an analytic bound on the observed variability. We validate the method using an ensemble of single qubit test circuits, executed on a superconducting transmon processor with well-characterized readout and gate error rates. The resulting description for circuit reproducibility, in terms of a composite device parameter, is confirmed to define an upper bound on the observed Hellinger distance, across the variable test circuits. This predictive correlation between circuit outcomes and device characterization offers an efficient method for assessing the reproducibility of noisy quantum circuits.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

An extensive assessment of the performance of pairwise and many-body interaction potentials in reproducing ab initio benchmark binding energies for water clusters n = 2–25

We assess the performance of 7 pairwise additive (TIP3P, TIP4P, TIP4P-ice, TIP5P, OPC, SPC, SPC/E) and 8 families of many-body potentials (q-AQUA, HIPPO, AMOEBA, EFP, TTM, WHBB, MB-pol, MB-UCB) in reproducing high-level ab initio benchmark values, CCSD(T) or MP2 at the complete basis set (CBS) limit for the binding energy and the many-body expansion (MBE) of water clusters n = 2–11, 16–17, 20, 25. By including a large range of cluster sizes having dissimilar hydrogen bonding networks, we obtain an understanding of how these potentials perform for different hydrogen bonding arrangements that are mostly outside of their parameterization range. While it is appropriate to compare the results of ab initio based many-body potentials directly to the electronic binding energies (D e 's), the pairwise additive ones are compared to the enthalpies at T = 298 K, ΔH(298 K), as the latter class of force fields are parametrized to reproduce enthalpies (implicitly accounting for zero-point energy corrections) rather than binding energies. We find that all pairwise additive potentials considered overestimate the reference ΔH values for the n = 2–25 clusters by >13%. For the water dimer (n = 2) in particular, the errors are in the range 83–119% for the pairwise additive potentials studied since these are based on an effective rather than the true 2-body interaction specifically designed as a means of partially accounting for the missing many-body terms. This stronger 2-body interaction is achieved by an enhanced monomer dipole moment that mimics its increase from the gas phase monomer to the condensed phase value. Indeed, for cluster sizes n ≥ 4 the percent deviations become slightly smaller (albeit all exceeding 13%). In contrast, we find that the many-body potentials perform more accurately in reproducing the electronic binding energies (D e 's) throughout the entire cluster range (n = 2–25), all reproducing the ab initio benchmark binding energies within ±7% of the respective CBS values. We further assess the ability of a subset of the many-body potentials (MB-UCB, q-AQUA, MB-pol, and TTM2.1-F) to also reproduce the magnitude of the ab initio many-body energy terms for water cluster sizes n = 7, 10, 16 and 17. The potentials show an overall good agreement with the available benchmark values. However, we identify characteristic differences upon comparing the many-body terms at both the ab initio-optimized geometries and the respective potential-optimized geometries to the reference ab initio values. Additionally, by applying this analysis to a wide range of cluster sizes, trends in the MBE of the potentials with increasing cluster size can be identified. Finally, in an attempt to draw a parallel between the pairwise additive and many-body potentials, we report the analysis of the individual molecular dipole moments for water clusters with 1 to ~4 solvation shells with the TTM2.1-F potential. We find that the internally solvated water molecules have in general a larger molecular dipole moment ranging from 2.6–3.0 D. This justifies the use of an enhanced, with respect to the gas-phase value, molecular dipole moment for the pairwise additive potentials, which is intended to fold in the many body terms into an effective (enhanced) pairwise interaction through the choice of the charges. These results have important implications for the development of future generations of efficient, transferable, and highly accurate classical interaction potentials in both the pairwise additive and many-body categories.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2025

Reproducibility of fixed-node diffusion Monte Carlo across diverse community codes: The case of water–methane dimer

Fixed-node diffusion quantum Monte Carlo (FN-DMC) is a widely trusted many-body method for solving the Schrödinger equation, known for its reliable predictions of material and molecular properties. Furthermore, its excellent scalability with system complexity and near-perfect utilization of computational power make FN-DMC ideally positioned to leverage new advances in computing to address increasingly complex scientific problems. Even though the method is widely used as a computational gold standard, reproducibility across the numerous FN-DMC code implementations has yet to be demonstrated. This difficulty stems from the diverse array of DMC algorithms and trial wave functions, compounded by the method’s inherent stochastic nature. Here, this study represents a community-wide effort to assess the reproducibility of the method, affirming that yes, FN-DMC is reproducible (when handled with care). Using the water–methane dimer as the canonical test case, we compare results from eleven different FN-DMC codes and show that the approximations to treat the non-locality of pseudopotentials are the primary source of the discrepancies between them. In particular, we demonstrate that, for the same choice of determinantal component in the trial wave function, reliable and reproducible predictions can be achieved by employing the T-move, the determinant locality approximation, or the determinant T-move schemes, while the older locality approximation leads to considerable variability in results. These findings demonstrate that, with appropriate choices of algorithmic details, fixed-node DMC is reproducible across diverse community codes—highlighting the maturity and robustness of the method as a tool for open and reliable computational science.

Della Pia, Flaviano [Univ. of Cambridge (United Ki↗
DOE OSTI2026

Triboelectric Nanogenerator Repeatability and Reproducibility Study

The development of triboelectric nanogenerators (TENGs) has largely focused on optimizing output performance, often at the expense of other critical research considerations such as the development of reliable technical procedures. In particular, the reliability of reported results—specifically repeatability and reproducibility—remains underexplored and is frequently limited to brief discussion within available literature. Without rigorous validation through repeatability and reproducibility studies, the credibility and broader applicability of reported findings remain uncertain. This study addresses this gap by systematically evaluating the repeatability and reproducibility of TENG performance data. Five polymer materials—Kapton, polyethylene (PE), polyethylene terephthalate (PET), polytetrafluoroethylene (PTFE), and polyvinylidene fluoride (PVDF)—were investigated across all pairwise combinations of 25 total combinations for the reproducibility study and three selected pairs of the 25 samples were selected for the repeatability study. For each TENG pairing, we analyzed the methodology, experimental procedures, and resulting performance data to quantify consistency and reliability. The objective of this work is to assess the validity of the collected dataset and determine whether the observed performance trends are consistent for use in future TENG design and optimization studies. Establishing reliable and reproducible data is essential for advancing the development of high-output TENG systems and ensuring their dependable implementation in practical applications.

36 MATERIALS SCIENCE↗