Search NASA⌕ Search

SEARCH · Search NASA

Results for “Benchmarking Software”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Comparison of GSFC and JPL VLBI modeling software: Benchmark

The Very Long Base Interferometry. The VLBI modeling software packages CALC 6.0 (GSFC) and MASTERFIT (JPL) are compared in some detail. Theoretical model delays are calculated for a set of 120 fictitious observations which involve a variety of sources, baselines, and antennas. Discrepancies between the total delays given by the two programs are of the order of 2 cm (RMS). The modeling of antenna offsets appears to account for approximately half of this difference. Relativistic bending contributions to the delay differ by 3 cm (RMS), and the there appears to be some mutual cancellation of errors involving antenna offsets, bending, and the effects of the two different Solar System ephemerides employed by CALC and MASTERFIT. This cancellation has not been completely characterized here.

Sovers, O. J.↗

Evaluation of Ocean Biogeochemistry and Carbon Cycling in CMIP Earth System Models With the International Ocean Model Benchmarking (IOMB) Software System

Abstract The International Ocean Model Benchmarking (IOMB) software package is a new community resource that we use here to evaluate surface and upper ocean biogeochemical variables and integrated anthropogenic carbon uptake from earth system models (ESMs) contributing to the 5th and 6th phases of the Coupled Model Intercomparison Project (CMIP5 and CMIP6). IOMB generates graphics and tables for systematically comparing model predictions against multiple datasets. Our analysis reveals some improvement in the multi‐model mean from CMIP5 to CMIP6 for most of the variables we examined. Compared to data‐constrained estimates of ocean anthropogenic carbon uptake for the 1994–2007 period, negative biases exist for many models between 30 and 50°S. Global model estimates of anthropogenic carbon uptake for the same period do not change significantly from CMIP5 to CMIP6, with the combined ensemble mean estimate of 27.8 ± 0.5 Pg C lower than a data‐constrained estimate of 33.0 ± 4.0 Pg C. At the same time, the change in the natural carbon inventory from CMIP is estimated to be a source of 0.7 ± 0.3 Pg C, which is considerably smaller in magnitude than a data‐constrained estimate of 5.0 ± 3.0 Pg C. With chlorofluorocarbon (CFC) predictions available for several models, we demonstrate that negative anthropogenic dissolved inorganic carbon biases coincide with negative biases in CFC concentration, highlighting the importance of weak exchange between the surface and interior ocean in regulating rates of anthropogenic carbon uptake. To examine the robustness of this attribution across the CMIP models, we calculate the global vertical temperature gradient between 200 and 1,000 m as a metric for global stratification and exchange between the surface and deeper waters. We find a linear relationship between the bias of the vertical temperature gradients and the bias in global anthropogenic carbon uptake, consistent with the hypothesis that model biases in anthropogenic carbon uptake are related to biases in surface‐to‐interior exchange by physical processes.

58 GEOSCIENCES↗

NASA Software Engineering Benchmarking Study

To identify best practices for the improvement of software engineering on projects, NASA's Offices of Chief Engineer (OCE) and Safety and Mission Assurance (OSMA) formed a team led by Heather Rarick and Sally Godfrey to conduct this benchmarking study. The primary goals of the study are to identify best practices that: Improve the management and technical development of software intensive systems; Have a track record of successful deployment by aerospace industries, universities [including research and development (R&D) laboratories], and defense services, as well as NASA's own component Centers; and Identify candidate solutions for NASA's software issues. Beginning in the late fall of 2010, focus topics were chosen and interview questions were developed, based on the NASA top software challenges. Between February 2011 and November 2011, the Benchmark Team interviewed a total of 18 organizations, consisting of five NASA Centers, five industry organizations, four defense services organizations, and four university or university R and D laboratory organizations. A software assurance representative also participated in each of the interviews to focus on assurance and software safety best practices. Interviewees provided a wealth of information on each topic area that included: software policy, software acquisition, software assurance, testing, training, maintaining rigor in small projects, metrics, and use of the Capability Maturity Model Integration (CMMI) framework, as well as a number of special topics that came up in the discussions. NASA's software engineering practices compared favorably with the external organizations in most benchmark areas, but in every topic, there were ways in which NASA could improve its practices. Compared to defense services organizations and some of the industry organizations, one of NASA's notable weaknesses involved communication with contractors regarding its policies and requirements for acquired software. One of NASA's strengths was its software assurance practices, which seemed to rate well in comparison to the other organizational groups and also seemed to include a larger scope of activities. An unexpected benefit of the software benchmarking study was the identification of many opportunities for collaboration in areas including metrics, training, sharing of CMMI experiences and resources such as instructors and CMMI Lead Appraisers, and even sharing of assets such as documented processes. A further unexpected benefit of the study was the feedback on NASA practices that was received from some of the organizations interviewed. From that feedback, other potential areas where NASA could improve were highlighted, such as accuracy of software cost estimation and budgetary practices. The detailed report contains discussion of the practices noted in each of the topic areas, as well as a summary of observations and recommendations from each of the topic areas. The resulting 24 recommendations from the topic areas were then consolidated to eliminate duplication and culled into a set of 14 suggested actionable recommendations. This final set of actionable recommendations, listed below, are items that can be implemented to improve NASA's software engineering practices and to help address many of the items that were listed in the NASA top software engineering issues. 1. Develop and implement standard contract language for software procurements. 2. Advance accurate and trusted software cost estimates for both procured and in-house software and improve the capture of actual cost data to facilitate further improvements. 3. Establish a consistent set of objectives and expectations, specifically types of metrics at the Agency level, so key trends and models can be identified and used to continuously improve software processes and each software development effort. 4. Maintain the CMMI Maturity Level requirement for critical NASA projects and use CMMI to measure organizations developing software for NASA. 5.onsolidate, collect and, if needed, develop common processes principles and other assets across the Agency in order to provide more consistency in software development and acquisition practices and to reduce the overall cost of maintaining or increasing current NASA CMMI maturity levels. 6. Provide additional support for small projects that includes: (a) guidance for appropriate tailoring of requirements for small projects, (b) availability of suitable tools, including support tool set-up and training, and (c) training for small project personnel, assurance personnel and technical authorities on the acceptable options for tailoring requirements and performing assurance on small projects. 7. Develop software training classes for the more experienced software engineers using on-line training, videos, or small separate modules of training that can be accommodated as needed throughout a project. 8. Create guidelines to structure non-classroom training opportunities such as mentoring, peer reviews, lessons learned sessions, and on-the-job training. 9. Develop a set of predictive software defect data and a process for assessing software testing metric data against it. 10. Assess Agency-wide licenses for commonly used software tools. 11. Fill the knowledge gap in common software engineering practices for new hires and co-ops.12. Work through the Science, Technology, Engineering and Mathematics (STEM) program with universities in strengthening education in the use of common software engineering practices and standards. 13. Follow up this benchmark study with a deeper look into what both internal and external organizations perceive as the scope of software assurance, the value they expect to obtain from it, and the shortcomings they experience in the current practice. 14. Continue interactions with external software engineering environment through collaborations, knowledge sharing, and benchmarking.

Rarick, Heather L.↗

DeepSurveySim: Simulation Software and Benchmark Challenges for Astronomical Observation Scheduling

Modern astronomical surveys have multiple competing scientific goals. Optimizing the observation schedule for these goals presents significant computational and theoretical challenges, and state-of-the-art methods rely on expensive human inspection of simulated telescope schedules. Automated methods, such as reinforcement learning, have recently been explored to accelerate scheduling. However, there do not yet exist benchmark data sets or user-friendly software frameworks for testing and comparing these methods. We present DeepSurveySim -- a high-fidelity and flexible simulation tool for use in telescope scheduling. DeepSurveySim provides methods for tracking and approximating sky conditions for a set of observations from a user-supplied telescope configuration. We envision this tool being used to produce benchmark data sets and for evaluating the efficacy of ground-based telescope scheduling algorithms, particularly for machine learning algorithms that would suffer in efficacy if limited to real data for training.We introduce three example survey configurations and related code implementations as benchmark problems that can be simulated with DeepSurveySim.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Introduction to benchmark dose methods and U.S. EPA's benchmark dose software (BMDS) version 2.1.1

Traditionally, the No-Observed-Adverse-Effect-Level (NOAEL) approach has been used to determine the point of departure (POD) from animal toxicology data for use in human health risk assessments. However, this approach is subject to substantial limitations that have been well defined, such as strict dependence on the dose selection, dose spacing, and sample size of the study from which the critical effect has been identified. Also, the NOAEL approach fails to take into consideration the shape of the dose-response curve and other related information. The benchmark dose (BMD) method, originally proposed as an alternative to the NOAEL methodology in the 1980s, addresses many of the limitations of the NOAEL method. It is less dependent on dose selection and spacing, and it takes into account the shape of the dose-response curve. In addition, the estimation of a BMD 95% lower bound confidence limit (BMDL) results in a POD that appropriately accounts for study quality (i.e., sample size). With the recent advent of user-friendly BMD software programs, including the U.S. Environmental Protection Agency's (U.S. EPA) Benchmark Dose Software (BMDS), BMD has become the method of choice for many health organizations world-wide. This paper discusses the BMD methods and corresponding software (i.e., BMDS version 2.1.1) that have been developed by the U.S. EPA, and includes a comparison with recently released European Food Safety Authority (EFSA) BMD guidance.

60 APPLIED LIFE SCIENCES↗

Benchmarking of Software and Methods for Use in Transient Multidimensional Fuel Performance with Spatial Reactor Kinetics

The key physics involved in accurate prediction of reactor-fuel-element behavior includes neutron transport and thermal hydraulics. The thermal hydraulic feedback mechanism is primarily provided through cross sections to the neutron transport that are temperature and density dependent. Historically, this coupling was primarily seen only in reactor simulators, which are well suited to model the reactor core, giving only a coarse treatment to individual fuel pins as well as simple models for thermal distribution calculations. This poor resolution on the primary coupling mechanisms can lead to conservatisms that should be removed to improve fuel design and performance. This work seeks to address the resolution of space-time-dependent neutron kinetics with thermal feedback within the fuel pin scale in the multiphysics framework. The specific application of this new capability is transient performance analysis of space-time-dependent temperature distribution of fuel elements. The coupling between the neutron transport and the thermal feedback is extremely important in this highly coupled problem, primarily applicable to reactivity-initiated-accidents (RIAs) and loss-of-coolant-accidents (LOCAs). The capability developed will include the coupling of the time-dependent neutron transport with the time-dependent thermal diffusion capability. An improvement in resolution and coupling is proposed by developing neutron transport models that are internally coupled with high fidelity within fuel pin thermal calculations in a multiphysics framework. Good agreement is shown with benchmarks and problems from the literature of RIAs and LOCAs for the tools used.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Benchmarking of software and methods for use in transient multidimensional fuel performance with spatial reactor kinetics

The key physics involved in accurate prediction of reactor-fuel-element behavior includes neutron transport and thermal hydraulics. The thermal hydraulic feedback mechanism is primarily provided through cross sections to the neutron transport that are temperature and density dependent. Historically, this coupling was primarily seen only in reactor simulators, which are well suited to model the reactor core, giving only a coarse treatment to individual fuel pins as well as simple models for thermal distribution calculations. This poor resolution on the primary coupling mechanisms can lead to conservatisms that should be removed to improve fuel design and performance. This work seeks to address the resolution of space-time-dependent neutron kinetics with thermal feedback within the fuel pin scale in the multi-physics framework. The specific application of this new capability is transient performance analysis of space-time-dependent temperature distribution of fuel elements. The coupling between the neutron transport and the thermal feedback is extremely important in this highly coupled problem, primarily applicable to reactivity-initiated- accidents (RIAs) and loss-of-coolant-accidents (LOCAs). The capability developed will include the coupling of the time-dependent neutron transport with the time-dependent thermal diffusion capability. An improvement in resolution and coupling is proposed by developing neutron transport models that are internally coupled with high fidelity within fuel pin thermal calculations in a multi-physics framework. Good agreement is shown with benchmarks and problems from the literature of RIAs and LOCAs for the tools used. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

DeepSurveySim: Simulation Software and Benchmark Challenges for Astronomical Observation Scheduling

Modern astronomical surveys have multiple competing scientific goals. Optimizing the observation schedule for these goals presents significant computational and theoretical challenges, and state-of-the-art methods rely on expensive human inspection of simulated telescope schedules. Automated methods, such as Reinforcement Learning (RL), have recently been explored to accelerate scheduling. DeepSurveySim provides methods for tracking and approximating sky conditions for a set of observations from a user-supplied telescope configuration.

79 ASTRONOMY AND ASTROPHYSICS↗

NASA Software Engineering Benchmarking Effort

Benchmarking was very interesting and provided a wealth of information (1) We did see potential solutions to some of our "top 10" issues (2) We have an assessment of where NASA stands with relation to other aerospace/defense groups We formed new contacts and potential collaborations (1) Several organizations sent us examples of their templates, processes (2) Many of the organizations were interested in future collaboration: sharing of training, metrics, Capability Maturity Model Integration (CMMI) appraisers, instructors, etc. We received feedback from some of our contractors/ partners (1) Desires to participate in our training; provide feedback on procedures (2) Welcomed opportunity to provide feedback on working with NASA

Godfrey, Sally↗

Pianola Version 1.0

Pianola is a software benchmark for measuring the performance of input/output (I/O) hardware and software. It works by monitoring the I/O activity of an existing application and storing the sequence of events, along with their timing, in a script. After some additional processing, a separate piece of software in this package can replay these I/) events and replicate their timing without reference to the original application. Therefore, Pianola allows users to create standalone I/O benchmarks that mimic the behavior of applications that either cannot be released or that are difficult for nonspecialists to configure, build or run.

May, J.↗

Pianola Version 1.0

Pianola is a software benchmark for measuring the performance of input/output (I/O) hardware and software. It works by monitoring the I/O activity of an existing application and storing the sequence of events, along with their timing, in a script. After some additional processing, a separate piece of software in this package can replay these I/) events and replicate their timing without reference to the original application. Therefore, Pianola allows users to create standalone I/O benchmarks that mimic the behavior of applications that either cannot be released or that are difficult for nonspecialists to configure, build or run.

May, J.↗

The International Land Model Benchmarking (ILAMB) System: Design, Theory, and Implementation

Abstract The increasing complexity of Earth system models has inspired efforts to quantitatively assess model fidelity through rigorous comparison with best available measurements and observational data products. Earth system models exhibit a high degree of spread in predictions of land biogeochemistry, biogeophysics, and hydrology, which are sensitive to forcing from other model components. Based on insights from prior land model evaluation studies and community workshops, the authors developed an open source model benchmarking software package that generates graphical diagnostics and scores model performance in support of the International Land Model Benchmarking (ILAMB) project. Employing a suite of in situ, remote sensing, and reanalysis data sets, the ILAMB package performs comprehensive model assessment across a wide range of land variables and generates a hierarchical set of web pages containing statistical analyses and figures designed to provide the user insights into strengths and weaknesses of multiple models or model versions. Described here is the benchmarking philosophy and mathematical methodology embodied in the most recent implementation of the ILAMB package. Comparison methods unique to a few specific data sets are presented, and guidelines for configuring an ILAMB analysis and interpreting resulting model performance scores are discussed. ILAMB is being adopted by modeling teams and centers during model development and for model intercomparison projects, and community engagement is sought for extending evaluation metrics and adding new observational data sets to the benchmarking framework.

54 ENVIRONMENTAL SCIENCES↗

The Next Generation System Analysis Model: Capabilities for Simulating a Waste Management System - 19131

The U.S. Department of Energy (DOE) Office of Nuclear Energy Office of Spent Fuel and Waste Disposition's Integrated Waste Management System (IWMS) program is applying integrated waste management system architecture analysis, system engineering, and decision analysis principles to inform potential future decisions regarding potential nuclear waste management system architectures. Architecture analyses of the IWMS are being conducted to support the future deployment of a comprehensive system for managing nuclear waste that considers all major aspects of the back end of the nuclear fuel cycle (i.e., transportation, storage, and disposal). The system architecture analyses consider alternative disposition pathways for commercial spent nuclear fuel (SNF) being stored dry in at-reactor Independent Spent Fuel Storage Installations (ISFSIs) or wet in spent-fuel pools. One method for evaluating these alternatives is simulating the movement of SNF throughout the system components. Successfully simulating a system on the scale of a national waste management system requires - accurate baseline information (e.g., historic fuel discharges from reactors and dry storage loadings at reactor sites), - modeling all logistic interactions between components of the proposed system, and - tracking and reporting all relevant details regarding the simulation's execution. The Next Generation System Analysis Model (NGSAM) is an agent-based simulation software tool designed for the express purpose of modeling the IWMS. It is based on the Performance Analysis Tool, a simulation platform that has been used to model logistics for the Department of Defense and the Federal Emergency Management Agency. Using existing, government-owned software allows NGSAM developers to modify the software at the level of source code and to develop flexible usability and modeling solutions for developing and running simulations of the IWMS. NGSAM imports data from the Oak Ridge National Laboratory (ORNL) Unified Database (e.g., historic assembly information, thermal profiles for assembly heat, at-reactor dry storage loadings) to ensure that the simulation initializes with a realistic representation of the state of commercial SNF in the United States. NGSAM is designed with the flexibility to model and track the behavior of virtually any IWMS architecture assumptions or design considerations. For example, NGSAM can model multiple interim storage facilities or repositories or none at all. Similarly, all assumptions regarding shipment schedules, operating schedules, and reactor operation assumptions can be modeled and adjusted in NGSAM with relative ease. The development of NGSAM was targeted to waste management system analysts as the primary users. With that in mind, analysts from ORNL, Sandia National Laboratories (SNL), and Argonne National Laboratory (ANL) were intimately involved in the development process. That involvement extended beyond merely defining requirements. System analysts have been involved with design decisions and software benchmarking and tests, and they have provided subject-matter expertise where necessary. The result is a high degree of interaction between professional software developers and system analysts, with a multitude of benefits to both groups. This paper explores some of the underlying methodology of the agile software development process used in the creation of NGSAM. It illustrates the capabilities of NGSAM. Some example use cases are discussed as well, with relevant results providing context for how NGSAM is used by system analysts to gain a greater understanding of alternative waste management system architectures. This is a technical paper that does not take into account the contractual limitations under the Standard Contract for Disposal of Spent Nuclear Fuel and/or High-Level Radioactive Waste (Standard Contract) (10 CFR Part 961). Under the provisions of the Standard Contract, DOE does not consider spent nuclear fuel in multi-assembly canisters to be an acceptable waste form, absent a mutually agreed-to contract amendment. To the extent discussions or recommendations in this paper conflict with the provisions of the Standard Contract, the Standard Contract provisions prevail. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Qualitative and quantitative validation of the SINBAD code on complex HPGe gamma-ray spectra

Radionuclides identification and quantification is a serious concern for many applications as safety or security of nuclear power plant or fuel cycle facility, CBRN risk identification, environmental radioprotection and waste measurements. High resolution gamma-ray spectrometry based on HPGe detectors is a performing solution for all these topics. During last decades, a great number of software has been developed to improve gamma spectra analysis. However, some difficulties remain in the analysis when photoelectric peaks are folded together with a high ratio between theirs amplitudes, when the Compton background is much larger compared to the signal of a single peak and when spectra are composed of a great number of peaks. This study deals with the comparison between conventional methods in radionuclides identification and quantification and the code called SINBAD ('Spectrometrie par Inference Non parametrique Bayesienne Deconvolutive'). For many years, SINBAD has been developed by CEA LIST for unfolding complex spectra from HPGe detectors. Contrary to conventional methods using fitting procedures, SINBAD uses a probabilistic approach with Bayesian inference to describe spectrum data. This conventional fitting method founded for example in Genie 2000 is compared with the nonparametric SINBAD approach regarding some key figures of merit as the peak centroid evaluation (identification) and peak surface evaluation (quantification). Unfriendly cases are studied for nuclides detection with closed gamma-rays energies and high photoelectric peak intensity differences. Tests are performed with spectra from the International Atomic Energy Agency (IAEA) for gamma spectra analysis software benchmark and with spectra acquired at the laboratory. Results show that SINBAD and Genie 2000 performances are quite similar with sometimes best results for SINBAD with the important difference that to achieve same performances the nonparametric method is user-friendly compared to the conventional method which need an expert parameters adjustment. (authors)

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Low-Cost Parallel Algorithms for 2:1 Octree Balance

The logical structure of a forest of octrees can be used to create scalable algorithms for parallel adaptive mesh refinement (AMR), which has recently been demonstrated for several petascale applications. Among various frequently used octree-based mesh operations, including refinement, coarsening, partitioning, and enumerating nodes, ensuring a 2:1 size balance between neighboring elements has historically been the most expensive in terms of CPU time and communication volume. The 2:1 balance operation is thus a primary target to optimize. One important component of a parallel balance algorithm is the ability to determine whether any two given octants have a consistent distance/size relation. Based on new logical concepts we propose fast algorithms for making this decision for all types of 2:1 balance conditions in 2D and 3D. Since we are able to achieve this without constructing any parent nodes in the tree that would otherwise need to be sorted and communicated, we can significantly reduce the required memory and communication volume. In addition, we propose a lightweight collective algorithm for reversing the asymmetric communication pattern induced by non-local octant interactions. We have implemented our improvements as part of the opensource “p4est” software. Benchmarking this code with both synthetic and simulation-driven adapted meshes we are able to demonstrate much reduced runtime and excellent weak and strong scalability. On our largest benchmark problem with 5.13 × 10 11 octants the new 2:1 balance algorithm executes in less than 8 seconds on 112,128 CPU cores of the Jaguar Cray XT5 supercomputer.

Computer Science↗

Delivering Science on Day One

While benchmarking software is useful for testing the performance limits and stability of Argonne National Laboratory’s new Theta supercomputer, there is no substitute for running real applications to explore the system’s potential. The Argonne Leadership Computing Facility’s Theta Early Science Program, modeled after its highly successful code migration program for the Mira supercomputer, has one primary aim: to deliver science on day one. Here is a closer look at the type of science problems that will be getting early access to Theta, a next-generation machine being rolled out this year.

Williams, Timothy J.↗

Computational Fluid Dynamic Modeling of Dry Cask Simulator with Crosswind

The purpose of this study is to create a STAR-CCM+ model of a Belowground Vertical Dry Cask Simulator (BVDCS) at Sandia National Laboratories (SNL) and validate the model with SNL’s experimental results. The BVDCS consists of a single boiling water reactor assembly fitted with electric heaters encompassed by a containment vessel and shell to represent a belowground spent nuclear fuel (SNF) dry storage system. Blowers are located near the inlet and outlet of the BVDCS to simulate crosswind conditions. In addition to the experimental results, the STAR-CCM+ model developed for this study is compared with a previous computational fluid dynamics (CFD) model in a different software program, which is used as a software-to-software benchmark. The experimental results provide a dataset to compare the STAR-CCM+ model results for a variety of different conditions. The main objective is to validate and improve STAR-CCM+ CFD models for spent nuclear fuel storage systems with explicitly modeled external environments and “wind driven” crossflows. These CFD models aide in the study of external particle deposition in spent nuclear fuel storage systems, which is important to predicting the significance of chloride induced stress corrosion cracking (CISCC). In addition to experimental comparison, a sensitivity analysis study is performed using the STAR-CCM+ model. The sensitivity analysis provides a quantitative assessment of the sensitivity of various parameters. This helps provide information on various parameters that are of particular importance to constructing a model representative of real life systems. The STAR-CCM+ model compared well to the experimental results showing similar responses to changes in cross wind flow, and a number of parameters are identified for model improvement.

Jensen, Ben J.↗