Search NASA⌕ Search

SEARCH · Search NASA

Results for “Reproducibility”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

HPC-driven computational reproducibility in numerical relativity codes: a use case study with IllinoisGRMHD

Abstract Reproducibility of results is a cornerstone of the scientific method. Scientific computing encounters two challenges when aiming for this goal. Firstly, reproducibility should not depend on details of the runtime environment, such as the compiler version or computing environment, so results are verifiable by third-parties. Secondly, different versions of software code executed in the same runtime environment should produceconsistent numerical results for physical quantities. In this manuscript, we test the feasibility of reproducing scientific results obtained using theIllinoisGRMHDcode that is part of an open-source community software for simulation in relativistic astrophysics, theEinstein Toolkit. We verify that numerical results of simulating a single isolated neutron star withIllinoisGRMHDcan be reproduced, and compare them to results reported by the code authors in 2015. We use two different supercomputers: Expanse at SDSC, and Stampede2 at TACC. By compiling the source code archived along with the paper on both Expanse and Stampede2, we find thatIllinoisGRMHDreproduces results published in its announcement paper up to errors comparable to round-off level changes in initial data parameters. We also verify that a current version ofIllinoisGRMHDreproduces these results once we account for bug fixes which have occurred since the original publication.

Astronomy & Astrophysics↗

Improving reproducibility in synchrotron tomography using implementation-adapted filters

For reconstructing large tomographic datasets fast, filtered backprojection-type or Fourier-based algorithms are still the method of choice, as they have been for decades. These robust and computationally efficient algorithms have been integrated in a broad range of software packages. The continuous mathematical formulas used for image reconstruction in such algorithms are unambiguous. However, variations in discretization and interpolation result in quantitative differences between reconstructed images, and corresponding segmentations, obtained from different software. This hinders reproducibility of experimental results, making it difficult to ensure that results and conclusions from experiments can be reproduced at different facilities or using different software. In this paper, a way to reduce such differences by optimizing the filter used in analytical algorithms is proposed. These filters can be computed using a wrapper routine around a black-box implementation of a reconstruction algorithm, and lead to quantitatively similar reconstructions. Use cases for this approach are demonstrated by computing implementation-adapted filters for several open-source implementations and applying them to simulated phantoms and real-world data acquired at the synchrotron. Our contribution to a reproducible reconstruction step forms a building block towards a fully reproducible synchrotron tomography data processing pipeline.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Reproducibility Practice in High-Performance Computing: Community Survey Results

The integrity of science and engineering research is grounded in assumptions of rigor and transparency on the part of those engaging in such research. HPC community effort to strengthen rigor and transparency take the form of reproducibility efforts. In a recent survey of the SC conference community, we collected information about the SC Reproducibility Initiative activities. Here, we present the survey results in this paper. Results show that the reproducibility initiative activities have contributed to higher levels of awareness on the part of SC conference technical program participants, and hint at contributing to greater scientific impact for the published papers of the SC conference series. Stringent point-of-manuscript-submission verification is problematic for reasons we point out, as are inherent difficulties of computational reproducibility in HPC. Future efforts should better decouple the community educational goals from goals that specifically strengthen a research work's potential for long-term impact through reuse 5-10 years down the road.

97 MATHEMATICS AND COMPUTING↗

MemXCT: Design, Optimization, Scaling, and Reproducibility of X-Ray Tomography Imaging

Here, this work extends our previous research entitled "MemXCT: Memory-centric X-ray CT Reconstruction with Massive Parallelization" that was originally published at SC19 conference (Hidayetoglu et al., 2019) with reproducibility of the computational imaging performance. X-ray computed tomography (XCT) is regularly used at synchrotron light sources to study the internal morphology of materials at high resolution. However, experimental constraints, such as radiation sensitivity, can result in noisy or undersampled measurements. Further, depending on the resolution, sample size and data acquisition rates, the resulting noisy dataset can be in the order of terabytes. Advanced iterative reconstruction techniques can produce high-quality images from noisy measurements, but their computational requirements have made their use an exception rather than the rule. We propose a novel memory-centric approach that avoids redundant computations at the expense of additional memory complexity. We develop a memory-centric iterative reconstruction system, MemXCT, that uses an optimized SpMV implementation with two-level pseudo-Hilbert ordering and multi-stage input buffering. We evaluate MemXCT on various supercomputer architectures involving KNL and GPU. MemXCT can reconstruct a large (11Kx11K) mouse brain tomogram in 10 seconds using 4096 KNL nodes (256K cores). The results presented in our original article at the SC19 were based on large-scale supercomputing resources. The MemXCT application was selected for the Student Cluster Competition (SCC) Reproducibility Challenge and evaluated on a variety of cloud computing resources by universities around the world in the SC20 conference. We summarize the results of the top-ranked SCC Reproducibility Challenge teams and identify the most pertinent measures for ensuring the reproducibility of our experiments in this article.

47 OTHER INSTRUMENTATION↗

MO-FG-CAMPUS-JeP3-02: A Novel Setup Approach to Improve C-Spine Curvature Reproducibility for Head and Neck Radiotherapy Using Optical Surface Imaging with Two Regions of Interest

Purpose: To develop a novel approach to improve cervical spine (c-spine) curvature reproducibility for head and neck (HN) patients using optical surface imaging (OSI) with two regions of interests (ROIs). Methods: The OSI-guided, two-step setup procedure requires two ROIs: ROI-1 of the shoulders and ROI-2 of the face. The neck can be stretched or squeezed in superior-inferior (SI) direction using a specially-designed sliding head support. We hypothesize that when these two ROIs are aligned, the c-spine should fall into a naturally reproducible position under same setup conditions. An anthropomorphous phantom test was performed to examine neck pitch angles comparing with the calculated angles. Three volunteers participated in the experiments, which start with conventional HN setup using skin markers and room lasers. An OSI image and lateral photo-picture were acquired as the references. In each of the three replicate tests, conventional setup was first applied after volunteers got on the couch. ROI-1 was aligned by moving the body, followed by ROI-2 alignment via adjusting head position and orientation under real-time OSI guidance. A final static OSI image and lateral picture were taken to evaluate both anterior and posterior surface alignments. Three degrees of freedom can be adjusted if an open-face mask was applied, including head SI shift using the sliding head support and pitch-and-roll rotations using a commercial couch extension. Surface alignment was analyzed comparing with conventional setup. Results: The neck pitch angle measured by OSI is consistent with the calculated (0.2±0.6°). Volunteer study illustrated improved c-spine setup reproducibility using OSI comparing with conventional setup. ROI alignments with 2mm/1° tolerance are achieved within 3 minutes. Identical knee support is important to achieve ROI-1 pitch alignment. Conclusion: The feasibility of this novel approach has been demonstrated for c-spine curvature setup reproducibility. Further evaluation is necessary with bony alignment variation in patient studies. This study is in part supported by the NIH (U54CA137788).

60 APPLIED LIFE SCIENCES↗

Does Chemical Engineering Research Have a Reproducibility Problem?

Concerns have been raised in multiple scientific fields in recent years about the reproducibility of published results. Systematic efforts to examine this issue have been undertaken in biomedicine and psychology, but less is known about this important issue in the materials-oriented research that underpins much of modern chemical engineering. Here, we relate a dramatic historical episode from our own institution to illustrate the implications of performing reproducible research and describe two case studies based on literature analysis to provide concrete information on the reproducibility of modern materials-oriented research. The two case studies deal with the properties of metal-organic frameworks (MOFs), a class of materials that have generated tens of thousands of papers. We do not claim that research on MOFs is less (or more) reproducible than other subfields; rather, we argue that the characteristics of this subfield are common to many areas of materials-oriented research. We conclude with specific recommendations for action by individual researchers, journal editors, publishers, and research communities.

catalysis (heterogeneous), defects, membrane, carb↗

A multimodal and integrated approach to interrogate human kidney biopsies with rigor and reproducibility: guidelines from the Kidney Precision Medicine Project

Comprehensive and spatially mapped molecular atlases of organs at a cellular level are a critical resource to gain insights into pathogenic mechanisms and personalized therapies for diseases. The Kidney Precision Medicine Project (KPMP) is an endeavor to generate three-dimensional (3-D) molecular atlases of healthy and diseased kidney biopsies by using multiple state-of-the-art omics and imaging technologies across several institutions. Obtaining rigorous and reproducible results from disparate methods and at different sites to interrogate biomolecules at a single-cell level or in 3-D space is a significant challenge that can be a futile exercise if not well controlled. Here we describe a “follow the tissue” pipeline for generating a reliable and authentic single-cell/region 3-D molecular atlas of human adult kidney. Our approach emphasizes quality assurance, quality control, validation, and harmonization across different omics and imaging technologies from sample procurement, processing, storage, shipping to data generation, analysis, and sharing. We established benchmarks for quality control, rigor, reproducibility, and feasibility across multiple technologies through a pilot experiment using common source tissue that was processed and analyzed at different institutions and different technologies. A peer review system was established to critically review quality control measures and the reproducibility of data generated by each technology before their being approved to interrogate clinical biopsy specimens. The process established economizes the use of valuable biopsy tissue for multiomics and imaging analysis with stringent quality control to ensure rigor and reproducibility of results and serves as a model for precision medicine projects across laboratories, institutions and consortia.

59 BASIC BIOLOGICAL SCIENCES↗

Open Reproducible Electron Microscopy Data Analysis

Electron microscopy (EM) is a cornerstone technique in the materials and biological sciences capable of imaging structures at nano- to atomic-scale resolution. Advances in technologies mean that one acquires datasets at increasing data rates and sizes. These advancements present enormous opportunities for researchers to understand complex systems. However, processing the resulting large-scale, complex data in a reproducible and shareable way is a real challenge for researchers. The building, managing, and maintaining complex workflows in a reproducible manner requires extensive knowledge in several areas outside the researchers’ core skill sets, such as software engineering, data science, and high-performance computing (HPC). Our work demonstrates an innovative approach to solving these problems, enabling reproducible EM data analysis through container encapsulated pipelines. Using modern container technologies, we encapsulate processing elements and connect them using shared memory. We expose user-friendly, advanced algorithms and tools to allow end users to utilize without expert programming skills. The platform enables reproducible, scalable, shareable pipelines for the analysis and visualization of EM data. Focusing on interoperability, we leverage the DOE and other agencies’ existing investments to provide a powerful software platform for EM data analysis.

Harris, Christopher↗

Using AI to Reproduce Neutrino Cross Section Analysis - Prototyping the Neutrino Discovery Platform

The Neutrino Discovery Platform (NDP) aims to accelerate DUNE-era science by making the neutrino program's existing datasets analyzable through fast, reproducible, and auditable workflows. We report a working version of two of its layers, data curation and agentic orchestration, built and tested end to end on MINERvA open data. The guiding lesson throughout is that a cross section is a measurement, and not just a plotted shape, only if it carries a defensible systematic-uncertainty budget, a trustworthy unfolding, and a reproducible record. Using a single medium-energy playlist pair from the MINERvA open-data release (about $2.05\times10^{17}$ protons on target of data), we first reproduced the shapes of two published charged-current inclusive $\nu_\mu$ measurements through a complete extraction ladder: selection, background subtraction, D'Agostini unfolding, efficiency correction, and flux normalization. These shape-level reproductions ran and tracked the published results, but they lacked the systematic-uncertainty machinery that defines a MINERvA cross section. To supply it, we vendored and built the MINERvA Analysis Toolkit and developed a many-universe systematic-uncertainty tool that produces a portable covariance artifact, a parallel event-loop runner, and a per-run auditability harness. Validated against a published covariance release, the toolchain reproduces the released statistical, flux, and muon-energy-scale terms and shows that they account for roughly 63\% of the total variance, with the remainder unreleased. Using this same infrastructure, we then performed a measurement of our own design, the hadronic recoil-energy distribution of low-energy ($E_\nu<2.5$~GeV) charged-current inclusive events, and found data/simulation shape agreement of $\chi^2/\mathrm{ndf}=1.26$. Together these results show that the platform supports original physics and not only reproductions.

Breaux, Auto [Tulane U. (main)]↗

Teaching Computational Reproducibility for Neuroimaging

We describe a project-based introduction to reproducible and collaborative neuroimaging analysis. Traditional teaching on neuroimaging usually consists of a series of lectures that emphasize the big picture rather than the foundations on which the techniques are based. The lectures are often paired with practical workshops in which students run imaging analyses using the graphical interface of specific neuroimaging software packages. Our experience suggests that this combination leaves the student with a superficial understanding of the underlying ideas, and an informal, inefficient, and inaccurate approach to analysis. To address these problems, we based our course around a substantial open-ended group project. This allowed us to teach: (a) computational tools to ensure computationally reproducible work, such as the Unix command line, structured code, version control, automated testing, and code review and (b) a clear understanding of the statistical techniques used for a basic analysis of a single run in an MR scanner. The emphasis we put on the group project showed the importance of standard computational tools for accuracy, efficiency, and collaboration. The projects were broadly successful in engaging students in working reproducibly on real scientific questions. We propose that a course on this model should be the foundation for future programs in neuroimaging. We believe it will also serve as a model for teaching efficient and reproducible research in other fields of computational science.

60 APPLIED LIFE SCIENCES↗

Best practices in software development for robust and reproducible geoscientific models based on insights from the Global Carbon Budget's dynamic vegetation models

Computational models play an increasingly vital role in scientific research by enabling the numerical simulation of complex processes. Such models are also fundamental in geosciences. For instance, they offer critical insights into the impacts of global change on the Earth system today and in the future. Beyond their value as research tools, models are also software products and should therefore adhere to certain established software engineering standards. However, scientists are rarely trained as software developers, which can lead to potential deficiencies in software quality like unreadable, inefficient, or erroneous code. The complexity of models, coupled with their integration into broader workflows, also often makes it challenging to reproduce results, evaluate processes, and build upon them. In this paper, we review the state and current practices of the development processes of the state-of-the-art land surface models used by the Global Carbon Budget. We combine the experience of modelers from the respective research groups with the expertise of software engineers from tech companies to outline key principles and tools for improving software quality in research. We explore four main areas: (1) model testing and validation, (2) scientific, technical, and user documentation, (3) version control, continuous integration, and code review, and (4) the portability and reproducibility of workflows. Our review reveals that while modeling communities are incorporating many best practices, significant room for improvement remains in areas such as automated testing, automated documentation, and reproducibility. Therefore, we here identify and promote essential software engineering practices, including numerous examples of practices from within the community that can serve as guidelines for other models and could help streamline processes across the entire community. We conclude with an open-source example implementation of these principles, demonstrating portable and reproducible data flows, a continuous integration setup, and web-based visualizations. This example may serve as a practical resource for model developers, users, and all scientists engaged in scientific programming.

Gregor, Konstantin [Technical Univ. of Munich (Ger↗

Explaining and predicting human behavior and social dynamics in simulated virtual worlds: reproducibility, generalizability, and robustness of causal discovery methods

Ground Truth program was designed to evaluate social science modeling approaches using simulation test beds with ground truth intentionally and systematically embedded to understand and model complex Human Domain systems and their dynamics Lazer et al. (Science 369:1060–1062, 2020). Our multidisciplinary team of data scientists, statisticians, experts in Artificial Intelligence (AI) and visual analytics had a unique role on the program to investigate accuracy, reproducibility, generalizability, and robustness of the state-of-the-art (SOTA) causal structure learning approaches applied to fully observed and sampled simulated data across virtual worlds. In addition, we analyzed the feasibility of using machine learning models to predict future social behavior with and without causal knowledge explicitly embedded. In this paper, we first present our causal modeling approach to discover the causal structure of four virtual worlds produced by the simulation teams—Urban Life, Financial Governance, Disaster and Geopolitical Conflict. Our approach adapts the state-of-the-art causal discovery (including ensemble models), machine learning, data analytics, and visualization techniques to allow a human-machine team to reverse-engineer the true causal relations from sampled and fully observed data. We next present our reproducibility analysis of two research methods team’s performance using a range of causal discovery models applied to both sampled and fully observed data, and analyze their effectiveness and limitations. We further investigate the generalizability and robustness to sampling of the SOTA causal discovery approaches on additional simulated datasets with known ground truth. Our results reveal the limitations of existing causal modeling approaches when applied to large-scale, noisy, high-dimensional data with unobserved variables and unknown relationships between them. We show that the SOTA causal models explored in our experiments are not designed to take advantage from vasts amounts of data and have difficulty recovering ground truth when latent confounders are present; they do not generalize well across simulation scenarios and are not robust to sampling; they are vulnerable to data and modeling assumptions, and therefore, the results are hard to reproduce. Finally, when we outline lessons learned and provide recommendations to improve models for causal discovery and prediction of human social behavior from observational data, we highlight the importance of learning data to knowledge representations or transformations to improve causal discovery and describe the benefit of causal feature selection for predictive and prescriptive modeling.

97 MATHEMATICS AND COMPUTING↗

Rigor and Reproducibility in Electrocatalysis: Best Practices for Operando Studies

Operando measurements have rapidly expanded the scope of electrocatalysis by enabling direct observation of catalytic interfaces under working conditions and by linking structural, compositional, and spectroscopic observables to activity and selectivity. However, the growth of operando methods has outpaced the adoption of broadly shared experimental standards, creating persistent challenges in reproducibility, interpretation, and comparison across laboratories and platforms. This perspective synthesizes discussions from the 2025 National Science Foundation Workshop on Rigor and Reproducibility in Electrocatalysis and outlines a practical framework for the rigorous use of operando measurements in electrocatalysis. We highlight three recurring needs: careful implementation of complex methods to avoid overinterpretation; recognition that (subtle) differences in reactor architecture, hydrodynamics, and electrical boundary conditions can alter apparent kinetics and selectivity; and transparent reporting standards that enable meaningful cross-comparison without constraining measurement-specific cell innovation. Focusing on widely used techniques (including X-ray and vibrational spectroscopies, mass spectrometry, and electron microscopy), we discuss technique-specific pitfalls, cross-validation strategies, and recurring platform-agnostic considerations such as mass transport, current distribution, temporal-resolution mismatches, and catalyst evolution. This Perspective aims to strengthen the mechanistic inference and improve the reproducibility, comparability, and predictive value of operando electrocatalysis research.

X-ray absorption spectroscopy↗

Towards Interactive, Reproducible Analytics at Scale on HPC Systems

The growth in scientific data volumes has resulted in a need to scale up processing and analysis pipelines using High Performance Computing (HPC) systems. These workflows need interactive, reproducible analytics at scale. The Jupyter platform provides core capabilities for interactivity but was not designed for HPC systems. In this paper, we outline our efforts that bring together core technologies based on the Jupyter Platform to create interactive, reproducible analytics at scale on HPC systems. Our work is grounded in a real world science use case-applying geophysical simulations and inversions for imaging the subsurface. Our core platform addresses three key areas of the scientific analysis workflow-reproducibility, scalability, and interactivity. We describe our implemention of a system, using Binder, Science Capsule, and Dask software. We demonstrate the use of this software to run our use case and interactively visualize real-Time streams of HDF5 data.

containers↗

On Reproducibility of Collective Acceleration of Protons in a Plasma-Anode Diode System

The article deals with the investigation of the conditions of reproducibility of the results of collective acceleration of protons in a high-current relativistic electron beam of nanosecond duration passing through a plasma bunch. A quantitative criterion of reproducibility in the system of an anode with a dielectric insert and a cylindrical cathode was the neutron yield in a nuclear reaction with the threshold about 2 MeV, which was measured by a silver-activated detector after each operation of the accelerator. Generation of an azimuthally-homogeneous electron beam and plasma in the area of the dielectric insert on the anode and the use of lenses in the region of electron beam drift resulted in the stabilization of the accelerating process. After optimization, the neutron yield scatter in a series of 30 pulses was at the level ±5% of the average value, which corresponded to the accelerated proton energy scatter ±3%. The conditions of best reproducibility coincided with those where the main group of protons reached their maximum values.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Reliability of the {sup 123}I-mIBG heart/mediastinum ratio: Results of a multicenter test–retest reproducibility study

A quantitative measurement, the Heart-to-Mediastinum (H/M) ratio of counts derived from a planar acquisition approximately 4 hours after injection of {sup 123}I-mIBG, is a strong predictor of outcomes in patients with stable class II-III heart failure and LVEF ≤ 35%. This study assessed the test–retest reproducibility of the H/M ratio in such patients. 47 subjects with class II-III systolic heart failure and LVEF ≤ 35% were tested at two time intervals separated by 5 to 14 days. Subjects were imaged twice on the same camera using the same radionuclide dose. Images were sent to a core analysis lab, where three nuclear technologists independently determined the H/M ratios. The primary endpoint was test–retest H/M ratio reproducibility calculated as the absolute difference in mean value determined by the three readers. Mean subject age was 65 ± 12 years, 85% were male, and mean BMI was 29 ± 6 kg/m{sup 2}. Mean injected activity was 10.18 ± 0.43 mCi for first dose and 10.09 ± 0.52 mCi for the second dose. The mean and SD values for first and repeat studies were almost identical: the 95% confidence interval of the mean test–retest difference was 0.055 to 0.076. Bland–Altman plots showed no systematic effect of the H/M ratio on the magnitude of the difference between replicate measurements. Inter-reader measurements were nearly identical. There were no serious adverse events despite exposure to {sup 123}I-mIBG on 2 occasions in a short time period. The Heart-to-Mediastinum ratio of {sup 123}I-mIBG is a consistent and highly reproducible measurement in stable Class II to III heart failure patients.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Evaluating Separation Selectivity and Collision Cross Section Measurement Reproducibility in Helium, Nitrogen, Argon, and Carbon Dioxide Drift Gases for Drift Tube Ion Mobility–Mass Spectrometry

Previous ion mobility (IM) studies have demonstrated that varying the drift gas composition can be used to enhance chemical selectivity and resolution, yet there are few drift gas studies aimed at achieving quantitatively reproducible mobility measurements. Here, we critically evaluate the conditions necessary to achieve reproducible collision cross section (CCS) measurements in pure drift gases (helium, nitrogen, argon, and carbon dioxide) using a commercial uniform field drift tube instrument. Optimal experimental parameters are assessed based on the convergence of CCS measurements to reproducible values which are compared with literature values. A suite of calibration standards with diverse masses, biological classes, and charge states are examined to assess chemical selectivity and resolution achievable in each drift gas. Results indicate nitrogen and argon perform similarly and are sufficient for most applications where high resolving power and high peak capacity are desired. Carbon dioxide exhibits more selectivity for resolving structurally heterogeneous compounds, which may be preferable in specific analyte pair separations. Helium demonstrated modest separation capabilities but has utility for comparison to theoretical values and previously published work. In drift gases other than nitrogen, pressure differentials up to 230 mTorr between the drift tube and upstream chamber were optimal for improving correlation to literature values, while in nitrogen, the recommended pressure differential of 150 mTorr was found appropriate. We present recommended experimental parameters as well as gas-specific CCS measurements for structurally homogeneous sets of analytes which are suitable for use by other laboratories as standards for purposes of instrument calibration and overall assessment of IM separation performance. .

74 ATOMIC AND MOLECULAR PHYSICS↗

Is the Supporting Information the Venue for Reproducibility and Transparency?

Making research data, software, and data processing tools readily available to the public could significantly enhance the impact of scientific publications. Openness and transparency could address reproducibility concerns and accelerate scientific progress.(1, 2) While archives and repositories would be preferable for securing data in the long term, the Supporting Information (SI) already allocates significant space to provide auxiliary files, links, and essential information needed to make scientific findings immediately reproducible as well as data processing protocols and numerical procedures executable. Furthermore, the SI document could ensure that data that might not fit in the tight confines of a journal article lives online even when laboratories move on to other projects, close, and lose track of their data. The change for openness and transparency might be difficult to embrace, depending on the specific nature of the project. Nevertheless, the trend to make raw research data and open-source software packages available to the public has been building steady momentum. Openness in research data and software sharing is already making transformative contributions to the communication of research findings, critical for ensuring reproducibility as well as training of the next generation of scientists.

Rudshteyn, Benjamin↗