Search NASA⌕ Search

SEARCH · Search NASA

Results for “Reproducibility”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2017

Is the Supporting Information the Venue for Reproducibility and Transparency?

Making research data, software, and data processing tools readily available to the public could significantly enhance the impact of scientific publications. Openness and transparency could address reproducibility concerns and accelerate scientific progress.(1, 2) While archives and repositories would be preferable for securing data in the long term, the Supporting Information (SI) already allocates significant space to provide auxiliary files, links, and essential information needed to make scientific findings immediately reproducible as well as data processing protocols and numerical procedures executable. Furthermore, the SI document could ensure that data that might not fit in the tight confines of a journal article lives online even when laboratories move on to other projects, close, and lose track of their data. The change for openness and transparency might be difficult to embrace, depending on the specific nature of the project. Nevertheless, the trend to make raw research data and open-source software packages available to the public has been building steady momentum. Openness in research data and software sharing is already making transformative contributions to the communication of research findings, critical for ensuring reproducibility as well as training of the next generation of scientists.

Rudshteyn, Benjamin↗
DOE OSTI2018

Reproducible workflow for multiplexed deep-scale proteome and phosphoproteome analysis of tumor tissues by liquid chromatography–mass spectrometry

Here we present an optimized workflow for global proteome and phosphoproteome analysis of tissues or cell lines that uses isobaric tags (TMT (tandem mass tags)-10) for multiplexed analysis and relative quantification, and provides 3×higher throughput than iTRAQ (isobaric tags for absolute and relative quantification)-4-based methods with high intraand inter-laboratory reproducibility. The workflow was systematically characterized and benchmarked across three independent laboratories using two distinct breast cancer subtypes from patient-derived xenograft models to enable assessment of proteome and phosphoproteome depth and quantitative reproducibility. Each plex consisted of ten samples, each being 300 µg of peptide derived from <50 mg of wet-weight tissue. Of the 10,000 proteins quantified per sample, we could distinguish 7,700 human proteins derived from tumor cells and 3100 mouse proteins derived from the surrounding stroma and blood. The maximum deviation across replicates and laboratories was <7%, and the interlaboratory correlation for TMT ratio–based comparison of the two breast cancer subtypes was r > 0.88. The maximum deviation for the phosphoproteome coverage was <24% across laboratories, with an average of >37,000 quantified phosphosites per sample and differential quantification correlations of r > 0.72. The full procedure, including sample processing and data generation, can be completed within 10 d for ten tissue samples, and 100 samples can be analyzed in ~4 months using a single LC-MS/MS instrument. The high quality, depth, and reproducibility of the data obtained both within and across laboratories should enable new biological insights to be obtained from mass spectrometry-based proteomics analyses of cells and tissues together with proteogenomic data integration.

Mertins, Philipp↗
DOE OSTI2016

Reproducible and controllable induction voltage adder for scaled beam experiments

A reproducible and controllable induction adder was developed using solid-state switching devices and Finemet cores for scaled beam compression experiments. A gate controlled MOSFET circuit was developed for the controllable voltage driver. The MOSFET circuit drove the induction adder at low magnetization levels of the cores which enabled us to form reproducible modulation voltages with jitter less than 0.3 ns. Preliminary beam compression experiments indicated that the induction adder can improve the reproducibility of modulation voltages and advance the beam physics experiments.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2026

From Reproducible Edge–Cloud Experimentation to Real-World Practice: The E2Clab Experience

Reproducibility is already difficult in distributed systems; on the computing continuum, it becomes substantially harder. Applications that span sensing devices, edge and fog resources, and cloud platforms must be evaluated across heterogeneous hardware, variable network conditions, cross-layer orchestration decisions, and long-running workflow lifecycles. We use E2Clab as a case study to examine these challenges and their implications for experimental methodology. We explain why reproducible experimentation is harder on the continuum, then revisit E2Clab as an initial response based on explicit modeling of infrastructure, workflow lifecycle, and artifacts. Lastly, we discuss how its evolution toward more realistic application settings can be understood through the lens of Translational Computer Science. We argue that reproducible continuum experimentation requires methods that are rigorous enough for research while remaining adaptable to real-world practice.

42 ENGINEERING↗
DOE OSTI2023

Towards Redefining the Reproducibility in Quantum Computing: A Data Analysis Approach on NISQ Devices

Although the building of quantum computers has kept making rapid progress in recent years, noise is still the main challenge for any application to leverage the power of quantum computing. Existing works addressing noise in quantum devices proposed noise reduction when deploying a quantum algorithm to a specified quantum computer. The reproducibility issue of quantum algorithms has been raised since the noise levels vary on different quantum computers. Importantly, existing works largely ignore the fact that the noise of quantum devices varies as time goes by. Therefore, reproducing the results on the same hardware will even become a problem. We analyze the reproducibility of quantum machine learning (QML) algorithms based on daily model training and execution data collection. Our analysis shows a correlation between our QML models’ test accuracy and quantum computer hardware’s calibration features. We also demonstrate that noisy simulators for quantum computers are not a reliable tool for quantum machine learning applications.

Senapati, Priyabrata↗
DOE OSTI2024

Impacts of floating-point non-associativity on reproducibility for HPC and deep learning applications

Run to run variability in parallel programs caused by floating-point non-associativity has been known to significantly affect reproducibility in iterative algorithms, due to accumulating errors. Non-reproducibility can critically affect the efficiency and effectiveness of correctness testing for stochastic programs. Recently, the sensitivity of deep learning training and inference pipelines to floating-point non-associativity has been found to sometimes be extreme. It can prevent certification for commercial applications, accurate assessment of robustness and sensitivity, and bug detection. New approaches in scientific computing applications have coupled deep learning models with high-performance computing, leading to an aggravation of debugging and testing challenges. Here we perform an investigation of the statistical properties of floating-point non-associativity within modern parallel programming models, and analyze performance and productivity impacts of replacing atomic operations with deterministic alternatives on GPUs. We examine the recently-added deterministic options in PyTorch within the context of GPU deployment for deep learning, uncovering and quantifying the impacts of input parameters triggering run to run variability and reporting on the reliability and completeness of the documentation. Finally, we evaluate the strategy of exploiting automatic determinism that could be provided by deterministic hardware, using the Groq LPUTM accelerator for inference portions of the deep learning pipeline. We demonstrate the benefits that a hardware-based strategy can provide within reproducibility and correctness efforts.

Shanmugavelu, Sanjif↗
DOE OSTI2020

Consistency and reproducibility in atomic layer deposition

Atomic layer deposition (ALD) is a thin film synthesis technique that can provide exquisite accuracy and precision in film thickness and composition even on complex, large area substrates. Based on self-limiting surface chemistry, ALD can be insensitive to process conditions and reactor designs, allowing an ALD process developed in one lab to be easily reproduced in other labs. In practice, however, ALD is sometimes difficult to reproduce or replicate, and the results can vary substantially between ALD reactors and between labs. This is exemplified by large deviations in reports on the growth of, e.g., Al2O3, FeOx, and TiO2 given the same precursors under similar conditions. Furthermore, the problem of irreproducibility seems to be growing as ALD is adopted by more researchers and integrated into new applications. In this article, the authors highlight some of the major sources of variations and errors and common misconceptions related to ALD. In particular, the authors focus on issues related to precursors, substrates, and deposition tools. The authors illustrate these problems through examples from the literature, and they present results from numerical simulations that describe how nonidealities would manifest in thickness profiles in a typical cross-flow reactor. They also describe how reproducibility in ALD is linked to consistent experimental practice and reporting between labs. The authors’ hope is that by educating newcomers to ALD and advocating for consistent reporting of deposition conditions, they can minimize irreproducibility and enable ALD practitioners to realize the full potential afforded by self-limiting surface chemistry.

36 MATERIALS SCIENCE↗
DOE OSTI2016

SU-F-I-31: Reproducibility of An Automatic Exposure Control Technique in the Low-Dose CT Scan of Cardiac PET/CT Exams

Purpose: Two separate low-dose CT scans are usually performed for attenuation correction of rest and stress N-13 ammonia PET/CT myocardial perfusion imaging (PET/CT). We utilize an automatic exposure control (AEC) technique to reduce CT radiation dose while maintaining perfusion image quality. Our goal is to assess the reproducibility of displayed CT dose index (CTDI) on same-day repeat CT scans (CT1 and CT2). Methods: Retrospectively, we reviewed CT images of PET/CT studies performed on the same day. Low-dose CT utilized AEC technique based on tube current modulation called Smart-mA. The scan parameters were 64 × 0.625mm collimation, 5mm slice thickness, 0.984 pitch, 1-sec rotation time, 120 kVp, and noise index 50 with a range of 10–200 mA. The scan length matched with PET field of view (FOV) with the heart near the middle of axial FOV. We identified the reference slice number (RS) for an anatomical landmark (carina) and used it to estimate axial shift between two CTs. For patient size, we measured an effective diameter on the reference slice. The effect of patient positioning to CTDI was evaluated using the table height. We calculated the absolute percent difference of the CTDI (%diff) for estimation of the reproducibility. Results: The study included 168 adults with an average body-mass index of 31.72 ± 9.10 (kg/m{sup 2}) and effective diameter was 32.72 ± 4.60 cm. The average CTDI was 1.95 ± 1.40 mGy for CT1 and 1.97 ± 1.42mGy for CT2. The mean %diff was 7.8 ± 6.8%. Linear regression analysis showed a significant correlation between the table height and %diff CTDI. (r=0.82, p<0.001) Conclusion: We have shown for the first time in human subjects, using two same-day CT images, that the AEC technique in low-dose CT is reproducible within 10% and significantly depends on the patient centering.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2016

SU-F-J-131: Reproducibility of Positioning Error Due to Temporarily Indwelled Urethral Catheter for Urethra-Sparing Prostate IMRT

Purpose: The purpose of this study was to prospectively assess the reproducibility of positioning errors due to temporarily indwelled catheter in urethra-sparing image-guided (IG) IMRT. Methods: Ten patients received urethra-sparing prostate IG-IMRT with implanted fiducials. After the first CT scan was performed in supine position, 6-Fr catheter was indwelled into urethra, and the second CT images were taken for planning. While the PTV received 80 Gy, 5% dose reduction was applied for the urethral PRV along the catheter. Additional CT scans were also performed at 5th and 30th fraction. Positions of interests (POIs) were set on posterior edge of prostate at beam isocenter level (POI1) and cranial and caudal edge of prostatic urethra on the post-indwelled CT images. POIs were copied into the pre-indwelled, 5th and 30th fraction’s CT images after fiducial matching on these CT images. The deviation of each POI between pre- and post-indwelled CT and the reproducibility of prostate displacement due to catheter were evaluated. Results: The deviation of POI1 caused by the indwelled catheter to the directions of RL/AP/SI (mm) was 0.20±0.27/−0.64±2.43/1.02±2.31, respectively, and the absolute distances (mm) were 3.15±1.41. The deviation tends to be larger if closer to the caudal edge of prostate. Compared with the pre-indwelled CT scan, a median displacement of all POIs (mm) were 0.3±0.2/2.2±1.1/2.0±2.6 in the post-indwelled, 0.4±0.4/3.4±2.1/2.3±2.6 in 5th, and 0.5±0.5/1.7±2.2/1.9±3.1 in 30th fraction’s CT scan with a similar data distribution. There were 6 patients with 5-mm-over displacement in AP and/or CC directions. Conclusion: Reproducibility of positioning errors due to temporarily indwelling catheter was observed. Especially in case of patients with unusually large shifts by indwelling catheter at the planning process, treatment planning should be performed by using the pre-indwelled CT images with transferred contour of the urethra identified by post-indwelled CT images.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2016

SU-F-T-55: Reproducibility of Interstitial HDR Brachytherapy Plans

Purpose: Treating gynecological cancers with interstitial high-dose-rate (HDR) brachytherapy requires precise reconstruction of catheter positions to obtain accurate dosimetric plans. In this study, we investigated the degree of reproducibility of dosimetric plans for Syed HDR brachytherapy. Methods: We randomly selected five patients having cervix-vaginal cancer who were recently treated in our clinic with interstitial HDR brachytherapy with a prescription dose of 25–30 Gy in five fractions. Interstitial needles/catheters were placed under fluoroscopic guidance and intra-operative 3T MRI scan was performed to confirm the desired catheter placement for adequate target volume coverage. A CT scan was performed and fused with the MRI for delineating high-risk CTV (HR-CTV), intermediate-risk CTV (IR-CTV) and OARs. HDR treatment plans were generated using Oncentra planning software. A single plan was used for all five fractions of treatment for each patient. For this study, we took the original clinical plan and removed all the reconstructed catheters from the plan keeping the original contours unchanged. Then, we manually reconstructed all the catheters and entered the same dwell time from the first original clinical plan. The dosimetric parameters studied were: D90 for HR-CTV and IR-CV, and D2cc for bladder, rectum, sigmoid and bowel. Results: The mean of absolute differences in dosimetric coverage (D90) were (range): 1.3% (1.0–2.0%) and 2.0% (0.9–3.6%) for HR-CTV and IR-CTV, respectively. In case of OARs, the mean of absolute variations in D2cc were (range): 4.7% (0.7–8.9%) for bladder, 1.60% (0.3–3.2%) for rectum, 1.6% (0–3.9%) for sigmoid, and 1.8% (0–5.1%) for bowel. Conclusion: Overall, the reproducibility of interstitial HDR plans was within clinically acceptable limit. Observed maximum variation in D2cc for bladder. If number of catchers and dwell points were relatively low or any one catheter was heavily loaded, then reproducibility of the plan was more sensitive to the accuracy of catheter reconstruction.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2025

Breaking the reproducibility barrier with standardized protocols for plant–microbiome research

Inter-laboratory replicability is crucial yet challenging in microbiome research. Leveraging microbiomes to promote soil health and plant growth requires understanding underlying molecular mechanisms using reproducible experimental systems. In a global collaborative effort involving five laboratories, we aimed to help advance reproducibility in microbiome studies by testing our ability to replicate synthetic community assembly experiments. Our study compared fabricated ecosystems constructed using two different synthetic bacterial communities, the model grass Brachypodium distachyon, and sterile EcoFAB 2.0 devices. All participating laboratories observed consistent inoculum-dependent changes in plant phenotype, root exudate composition, and final bacterial community structure, where Paraburkholderia sp. OAS925 could dramatically shift microbiome composition. Comparative genomics and exudate utilization linked the pH-dependent colonization ability of Paraburkholderia, which was further confirmed with motility assays. The study provides detailed protocols, benchmarking datasets, and best practices to help advance replicable science and inform future multi-laboratory reproducibility studies.

Novak, Vlastimil↗
DOE OSTI2017

Enhancing reproducibility in scientific computing: Metrics and registry for Singularity containers

Here we present Singularity Hub, a framework to build and deploy Singularity containers for mobility of compute, and the singularity-python software with novel metrics for assessing reproducibility of such containers. Singularity containers make it possible for scientists and developers to package reproducible software, and Singularity Hub adds automation to this workflow by building, capturing metadata for, visualizing, and serving containers programmatically. Our novel metrics, based on custom filters of content hashes of container contents, allow for comparison of an entire container, including operating system, custom software, and metadata. First we will review Singularity Hub’s primary use cases and how the infrastructure has been designed to support modern, common workflows. Next, we conduct three analyses to demonstrate build consistency, reproducibility metric and performance and interpretability, and potential for discovery. This is the first effort to demonstrate a rigorous assessment of measurable similarity between containers and operating systems. We provide these capabilities within Singularity Hub, as well as the source software singularity-python that provides the underlying functionality. Singularity Hub is available at https://singularity-hub.org, and we are excited to provide it as an openly available platform for building, and deploying scientific containers.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

The Need for Reproducibility

The purpose of this presentation is to consider issues of reproducibility, specifically it determines whether bitwise reproducible computation is possible, if computational research in DOE improves its publication process, and if reproducible results can be achieved apart from the peer review process?

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2020

Reproducibility and Optimization Techniques for Evaluated Data in the Resolved Resonance Region [Slides]

This presentation discusses the experimental setup input parameters that are basic quantities for reproducibility. The goal is to increase quality of the evaluated data and decrease time needed for an evaluation. Full reproducibility should be achieved efficiently by including any type of constraints in the optimization procedure and to improve computational time to reach convergence. The goal of an automated reproducibility aligns to the goal of “certified” evaluated data, i.e., defined by a well-defined metric.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2021

A reproducibility study of "Augmenting Genetic Algorithms with Deep Neural Networks for Exploring the Chemical Space"

Nigam et al. reported a genetic algorithm (GA) utilizing the SELFIES representation and also propose an adaptive, neural network-based penalty that is supposed to improve the diversity of the generated molecules. The main claims of the paper are that this GA outperforms other generative techniques (as measured by the penalized logP) and that a neural network-based adaptive penalty increases the diversity of the generated molecules. In this work, we investigated the reproducibility of their claims. Overall, we were able to reproduce comparable results using the SELFIES-based GA, but mostly by exploiting deficiencies of the (easily optimizable) fitness function (i.e., generating long, sulfur containing chains). In addition, we reproduce results showing that the discriminator can be used to bias the generation of molecules to ones that are similar to the reference set. Lastly, we attempted to quantify the evolution of the diversity, understand the influence of some hyperparameters, and propose improvements to the adaptive penalty.

Jablonka, Kevin Maik↗
DOE OSTI2024

Image-based modeling of coupled electro-chemo-mechanical behavior of Li-ion battery cathode using an interface-modified reproducing kernel particle method

Abstract An interface-modified reproducing kernel particle method (IM-RKPM) is introduced in this work to allow for a direct model construction from image pixels of heterogeneous polycrystalline Li-ion battery microstructures. The interface-modified reproducing kernel (IM-RK) approximation is constructed through scaling of a kernel function by a regularized distance function in conjunction with strategic placement of interface node locations. This leads to RK shape functions with either weak or strong discontinuities across material interfaces, suitable for modeling various interface mechanics. With the placement of a triple junction node and distance-based scaling of kernel functions, the resulting IM-RK shape function also possesses proper discontinuities at the triple junctions. This IM-RK approximation effectively remedies the well-known Gibb’s oscillation in the smooth approximation of discontinuities. Different from the conventional meshfree approaches for interface discontinuities, this IM-RK approach is done without additional degrees of freedom associated with the enrichment functions, and it is formulated with the standard procedures in the RK shape function construction. This work focuses on identifying the accuracy and convergence properties of IM-RKPM for modeling the coupled electro-chemo-mechanical system. A linear patch test is formulated and numerically tested for the electro-chemo-mechanical coupled problem with a Butler–Volmer boundary condition representing the physical conditions in Li-ion battery microstructures. This is followed by verification of the optimal rates of convergence of IM-RKPM for solving the coupled problem with higher order solutions. The image-based modeling of Li-ion battery microstructures in the numerical examples demonstrates the applicability of the proposed method to realistic Li-ion battery materials modeling.

25 ENERGY STORAGE↗
DOE OSTI2017

Reproducible, Component-based Modeling with TopoFlow, A Spatial Hydrologic Modeling Toolkit

Modern geoscientists have online access to an abundance of different data sets and models, but these resources differ from each other in myriad ways and this heterogeneity works against interoperability as well as reproducibility. The purpose of this paper is to illustrate the main issues and some best practices for addressing the challenge of reproducible science in the context of a relatively simple hydrologic modeling study for a small Arctic watershed near Fairbanks, Alaska. This study requires several different types of input data in addition to several, coupled model components. All data sets, model components and processing scripts (e.g. for preparation of data and figures, and for analysis of model output) are fully documented and made available online at persistent URLs. Similarly, all source code for the models and scripts is open-source, version controlled and made available online via GitHub. Each model component has a Basic Model Interface (BMI) to simplify coupling and its own HTML help page that includes a list of all equations and variables used. The set of all model components (TopoFlow) has also been made available as a Python package for easy installation. Three different graphical user interfaces for setting up TopoFlow runs are described, including one that allows model components to run and be coupled as web services.

58 GEOSCIENCES↗
DOE OSTI2024

A quasi-meshfree method for constructing boundary-aware reproducing bases on geometrically complex domains using manifold geodesics

In many applications, physical domains are geometrically complex making it challenging to perform coarse-scale approximation. A defeaturing process is often used to simplify the domain in preparation for approximation and analysis at the coarse scale. In this paper, a methodology is presented for constructing a coarse-scale reproducing basis on geometrically complex domains given an initial fine-scale mesh of the fully featured domain. The initial fine-scale mesh can be of poor quality and extremely refined. The construction of the basis functions begins with a coarse-scale covering of the domain and generation of weighting functions with local support. Manifold geodesics are used to define distances within the local support for general applicability to non-convex domains. Conventional moving least squares is used to construct the coarse-scale reproducing basis. Applications in quasi-interpolation and linear elasticity are presented.

97 MATHEMATICS AND COMPUTING↗