Search NASA⌕ Search

SEARCH · Search NASA

Results for “software metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2011

SAPHIRE 8 Quality Assurance Software Metrics Report

The purpose of this review of software metrics is to examine the quality of the metrics gathered in the 2010 IV&V and to set an outline for results of updated metrics runs to be performed. We find from the review that the maintenance of accepted quality standards presented in the SAPHIRE 8 initial Independent Verification and Validation (IV&V) of April, 2010 is most easily achieved by continuing to utilize the tools used in that effort while adding a metric of bug tracking and resolution. Recommendations from the final IV&V were to continue periodic measurable metrics such as McCabe's complexity measure to ensure quality is maintained. The four software tools used to measure quality in the IV&V were CodeHealer, Coverage Validator, Memory Validator, Performance Validator, and Thread Validator. These are evaluated based on their capabilities. We attempted to run their latest revisions with the newer Delphi 2010 based SAPHIRE 8 code that has been developed and was successful with all of the Validator series of tools on small tests. Another recommendation from the IV&V was to incorporate a bug tracking and resolution metric. To improve our capability of producing this metric, we integrated our current web reporting system with the SpiraTest test management software purchased earlier this year to track requirements traceability.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2014

Software Quality Metrics for Geant4: An Initial Assessment - Paper 122

In the context of critical applications, such as shielding and radiation protection, ensuring the quality of simulation software they depend on is of utmost importance. The assessment of simulation software quality is crucial both to determine its adoption in experimental applications and to guarantee reproducibility of outcome over time. Geant4 is a simulation system used in a wide range of experimental applications, some of which, such as radiation protection and biomedical studies, may involve critical use cases and would especially profit from an objective assessment of the software quality. In this study, we present the first results from an ongoing analysis of Geant4 code based on established software metrics from the perspective of software engineering. The analysis evaluates the current status of the code to quantify its characteristics with respect to documented quality standards; further appraisals concern evolutions over a series of release distributions. The quantitative assessment of the software is initially focused on a subset of Geant4 packages, which play a key role in scientific applications and are representative of different software development processes. We describe the chosen software metrics that map the entities of a software system to numeric values through mathematical definitions, quantifying software attributes ranging from functionality to maintainability. To measure such metrics, we exploited a set of software metrics tools, i.e. programs that implement a set of software metrics definitions. They allow software engineers to assess a software system by extracting the required attributes from the software and providing the corresponding metric values. To correlate software metrics values in a well-defined way, we used software quality standard models, such as the ISO/IEC 9126 standard, which aggregate numerical values to aid quality analysis and assessment. With this work we aim to create a baseline for evaluating correlations between software quality embedded in the Geant4 development process and simulation observables produced by Geant4-based applications. Furthermore, we intend to highlight what metrics are most effective at evaluating radiation transport software quality. We are confident that the result will provide constructive guidance both to improve key software tools, such as Geant4, and to estimate their contribution in risk analysis concerning sensitive applications. The remainder of this paper is structured as follows. We first discuss the approach adopted to fulfil our assessment. Then, we present the setup of our experiments and the way we collected data. We provide an initial interpretation of a subset of data resulting from measurements made on the selected Geant4 packages. As the analysis is still ongoing at the time of submitting this abstract, some results will be documented in the final paper and in the conference presentation. With this work we aim to create a proper Geant4 data set for evaluating its software characteristics and identifying both strengths and weaknesses. In this initial study, we considered the geometry and processes packages, which play a major role in most simulation applications. The data analysis is in progress at the time of writing; its complexity is related to the differences among software metrics tools, to the large amount of data to be evaluated and to the pioneering character of this project. We are confident that the results will provide valuable indications for the Geant4 user community and constructive guidance to improve key Geant4 packages. Other Monte Carlo codes could profit from the methodology developed in this pilot project for similar assessments of their software quality. At the end of this study, we intend to create a baseline for evaluating correlations between software quality embedded in the Geant4 development process and simulation observables produced by Geant4-based applications. (authors)

61 RADIATION PROTECTION AND DOSIMETRY↗
DOE OSTI2016

Quantitative Analysis Guidance in Safety Software Validation

Quantitative analysis in software development has been a challenging topic over the years in safety software. While qualitative processes for vigorous safety software verification and validation (V and V) have been provided through standard such as IEEE 1012, quantitative analysis for V and V tasks such as correctness, completeness or accuracy evaluation have not been formally defined. In practice, there are quantitative analyses in terms of using software metrics. Many of these software metrics are useful in analyzing the test results. Some of them are used for providing guidance in code evaluation activities e.g. numbers of line of codes not exceeding certain limits, recursive calls are not allowed, etc. However, few can be used for providing quantitative analysis guidance for the V and V tasks such as completeness evaluation. AREVA Federal Services (AFS) has provided services for many nuclear projects in US. Through compliance with IEEE 1012, AFS V and V processes have provided vigorous qualitative processes ensuring quality level of the software products used in the project meeting the required nuclear software level. In addition, quantitative analysis is developed to provide structured validation framework ensuring 'zero defect' in the software products. This paper describes some aspects of the quantitative approaches using a sample safety application. (authors)

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2013

Commercial Building Energy Baseline Modeling Software: Performance Metrics and Method Testing with Open Source Models and Implications for Proprietary Software Testing

The overarching goal of this work is to advance the capabilities of technology evaluators in evaluating the building-level baseline modeling capabilities of Energy Management and Information System (EMIS) software. Through their customer engagement platforms and products, EMIS software products have the potential to produce whole-building energy savings through multiple strategies: building system operation improvements, equipment efficiency upgrades and replacements, and inducement of behavioral change among the occupants and operations personnel. Some offerings may also automate the quantification of whole-building energy savings, relative to a baseline period, using empirical models that relate energy consumption to key influencing parameters, such as ambient weather conditions and building operation schedule. These automated baseline models can be used to streamline the whole-building measurement and verification (M&V) process, and therefore are of critical importance in the context of multi-measure whole-building focused utility efficiency programs.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2015

SU-E-T-776: Use of Quality Metrics for a New Hypo-Fractionated Pre-Surgical Mesothelioma Protocol

Purpose: The “SMART” (Surgery for Mesothelioma After Radiation Therapy) approach involves hypo-fractionated radiotherapy of the lung pleura to 25Gy over 5 days followed by surgical resection within 7. Early clinical results suggest that this approach is very promising, but also logistically challenging due to the multidisciplinary involvement. Due to the compressed schedule, high dose, and shortened planning time, the delivery of the planned doses were monitored for safety with quality metric software. Methods: Hypo-fractionated IMRT treatment plans were developed for all patients and exported to Quality Reports™ software. Plan quality metrics or PQMs™ were created to calculate an objective scoring function for each plan. This allows for an objective assessment of the quality of the plan and a benchmark for plan improvement for subsequent patients. The priorities of various components were incorporated based on similar hypo-fractionated protocols such as lung SBRT treatments. Results: Five patients have been treated at our institution using this approach. The plans were developed, QA performed, and ready within 5 days of simulation. Plan Quality metrics utilized in scoring included doses to OAR and target coverage. All patients tolerated treatment well and proceeded to surgery as scheduled. Reported toxicity included grade 1 nausea (n=1), grade 1 esophagitis (n=1), grade 2 fatigue (n=3). One patient had recurrent fluid accumulation following surgery. No patients experienced any pulmonary toxicity prior to surgery. Conclusion: An accelerated course of pre-operative high dose radiation for mesothelioma is an innovative and promising new protocol. Without historical data, one must proceed cautiously and monitor the data carefully. The development of quality metrics and scoring functions for these treatments allows us to benchmark our plans and monitor improvement. If subsequent toxicities occur, these will be easy to investigate and incorporate into the metrics. This will improve the safe delivery of large doses for these patients.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2012

Analyses Of Two End-User Software Vulnerability Exposure Metrics

The risk due to software vulnerabilities will not be completely resolved in the near future. Instead, putting reliable vulnerability measures into the hands of end-users so that informed decisions can be made regarding the relative security exposure incurred by choosing one software package over another is of importance. To that end, we propose two new security metrics, average active vulnerabilities (AAV) and vulnerability free days (VFD). These metrics capture both the speed with which new vulnerabilities are reported to vendors and the rate at which software vendors fix them. We then examine how the metrics are computed using currently available datasets and demonstrate their estimation in a simulation experiment using four different browsers as a case study. Finally, we discuss how the metrics may be used by the various stakeholders of software and to software usage decisions.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

ProgLIMI: Programmable LInk Metric Identification in Software-Defined Networks

In this paper, we propose the Programmable LInk Metric Identification (ProgLIMI) infrastructure for software-defined networking (SDN) networks. ProgLIMI identifies round-trip link metrics (RTLMs) from accumulated end-to-end metrics of selected measurement paths by leveraging the flexible routing control capability of SDN networks. ProgLIMI mainly solves three sub-problems: (1) monitor placement; (2) linearly independent measurement path construction; and (3) flow rule design. To reduce measurement cost, ProgLIMI tries to minimize the number of required monitors and flow rules. In this paper, we address the three sub-problems for both full and hybrid SDN networks. For full SDN networks, ProgLIMI can achieve full RTLM identification using only one monitor and two flow rules in each SDN switch. In contrast, the RTLM identification in hybrid SDN networks is more complicated due to the routing constraint of hybrid SDN networks. We first prove that the monitor placement problem in hybrid SDN networks is NP-hard. We then formulate the monitor placement and measurement path selection problem in hybrid SDN networks and propose a greedy heuristic algorithm to solve the problem efficiently. Our evaluations on both physical testbed and simulation platform reveal that ProgLIMI can accurately identify the RTLMs (delay and loss rate). Besides, ProgLIMI is also resource efficient, i.e., it only requires two flow rules in each SDN switch and a small number of monitors, and the extra probing traffic load incurred by ProgLIMI is also low.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Exploring Sustainability in Scientific Software through Code Quality & Test Coverage Metrics

Context: Scientific open-source software (SciOSS) plays a foundational role in research and engineering, yet its long-term sustainability has often been overlooked and remains a significant concern. Objective: This study investigates the long-term sustainability of SciOSS through code and test quality metrics. Method: We analyze CASS Software Portfolio projects, classifying them by sustainability and comparing their code structure, test coverage, and links between code quality and testing across the dataset. Results: Sustainable projects show higher, more consistent test coverage and clearer code-test correlations, while unsustainable ones show weaker patterns. Overall, test coverage is low in scientific software, and high complexity and coupling reduce testability. Conclusion: In this study, we present a practical, data-driven approach for assessing sustainability in scientific software, offering a foundation for evaluating long-term software health and supporting future efforts in quality assurance and sustainability monitoring.

Md mushfiqur rahman, Sheikh [University of Tenness↗
DOE OSTI2020

Software classification using phylogenetic techniques

Software, such as malware, may be classified using phylogenetic techniques. An evolutionary history of a representative set of software programs may be reconstructed to generate a reference phylogeny. Dynamic traces of the representative software programs may be obtained. The dynamic traces may include time-ordered sequences of execution commands extracted from running software binaries. Metrics may be developed using the dynamic traces. One or more unknown software programs may then be classified against the reference phylogeny using the metrics developed using the dynamic traces of the representative set of software programs.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2013

Functional Testing Protocols for Commercial Building Efficiency Baseline Modeling Software

This document describes procedures for testing and validating proprietary baseline energy modeling software accuracy in predicting energy use over the period of interest, such as a month or a year. The procedures are designed according to the methodology used for public domain baselining software in another LBNL report that was (like the present report) prepared for Pacific Gas and Electric Company: ?Commercial Building Energy Baseline Modeling Software: Performance Metrics and Method Testing with Open Source Models and Implications for Proprietary Software Testing Protocols? (referred to here as the ?Model Analysis Report?). The test procedure focuses on the quality of the software?s predictions rather than on the specific algorithms used to predict energy use. In this way the software vendor is not required to divulge or share proprietary information about how their software works, while enabling stakeholders to assess its performance.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2016

SU-F-I-43: A Software-Based Statistical Method to Compute Low Contrast Detectability in Computed Tomography Images

Purpose: The low contrast detectability (LCD) of a CT scanner is its ability to detect and display faint lesions. The current approach to quantify LCD is achieved using vendor-specific methods and phantoms, typically by subjectively observing the smallest size object at a contrast level above phantom background. However, this approach does not yield clinically applicable values for LCD. The current study proposes a statistical LCD metric using software tools to not only to assess scanner performance, but also to quantify the key factors affecting LCD. This approach was developed using uniform QC phantoms, and its applicability was then extended under simulated clinical conditions. Methods: MATLAB software was developed to compute LCD using a uniform image of a QC phantom. For a given virtual object size, the software randomly samples the image within a selected area, and uses statistical analysis based on Student’s t-distribution to compute the LCD as the minimal Hounsfield Unit’s that can be distinguished from the background at the 95% confidence level. Its validity was assessed by comparison with the behavior of a known QC phantom under various scan protocols and a tissue-mimicking phantom. The contributions of beam quality and scattered radiation upon the computed LCD were quantified by using various external beam-hardening filters and phantom lengths. Results: As expected, the LCD was inversely related to object size under all scan conditions. The type of image reconstruction kernel filter and tissue/organ type strongly influenced the background noise characteristics and therefore, the computed LCD for the associated image. Conclusion: The proposed metric and its associated software tools are vendor-independent and can be used to analyze any LCD scanner performance. Furthermore, the method employed can be used in conjunction with the relationships established in this study between LCD and tissue type to extend these concepts to patients’ clinical CT images.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2018

KCNSC Automated RAIL

The KCNSC Automated RAIL (Rolling Action Item List) system provides an electronic platform to manage and escalate rolling action items within an business and manufacturing environment at Honeywell. The software enables a tiered approach to issue management where issues are escalated up a management chain based on team input and compared to business metrics. The software manages action items at different levels of the organization and allows all users to discuss action items concurrently. In addition, the software drives accountability through timely emails and proper visibility during team meetings.

Branson, Donald↗
DOE OSTI2022

Mesoflow: An Open-Source Reacting Flow Solver for Catalysis at Mesoscale

We present the capabilities and software performance metrics of our open-source continuum solver for catalysis, Mesoflow, developed specifically for modeling transport and chemistry at the mesoscale. Our solver utilizes Cartesian block-structured adaptive mesh refinement to resolve complex catalyst surface morphologies directly obtained from X-ray tomography data. An immersed boundary based formulation enables rapid representation of complex geometries prevalent in most mesoporous catalyst interfaces. The solver is developed on top of open-source performance portable library, AMReX, providing parallel execution capabilities on current and upcoming high-performance-computing (HPC) architectures. Our flexible software framework enables integration of complex chemical mechanisms at heterogenous interfaces and time-split algorithms for circumventing highly disparate reaction and flow time-scales. Our current studies indicate a ten-fold performance gain by using graphics-processing-units (GPUs) compared to a single processor for representative problem sizes (2 million cell mesh). We will also present a brief introduction on how to build and use this software for application problems pertaining to catalytic upgrading and gas transport within porous catalyst particles.

adaptive meshing↗
DOE OSTI2019

Tracking materials science data lineage to manage millions of materials experiments and analyses

Abstract In an era of rapid advancement of algorithms that extract knowledge from data, data and metadata management are increasingly critical to research success. In materials science, there are few examples of experimental databases that contain many different types of information, and compared with other disciplines, the database sizes are relatively small. Underlying these issues are the challenges in managing and linking data across disparate synthesis and characterization experiments, which we address with the development of a lightweight data management framework that is generally applicable for experimental science and beyond. Five years of managing experiments with this system has yielded the Materials Experiment and Analysis Database (MEAD) that contains raw data and metadata from millions of materials synthesis and characterization experiments, as well as the analysis and distillation of that data into property and performance metrics via software in an accompanying open source repository. The unprecedented quantity and diversity of experimental data are searchable by experiment and analysis attributes generated by both researchers and data processing software. The search web interface allows users to visualize their search results and download zipped packages of data with full annotations of their lineage. The enormity of the data provides substantial challenges and opportunities for incorporating data science in the physical sciences, and MEAD’s data and algorithm management framework will foster increased incorporation of automation and autonomous discovery in materials and chemistry research.

Soedarmadji, Edwin↗
DOE OSTI2023

QASMBench: A Low-Level Quantum Benchmark Suite for NISQ Evaluation and Simulation

The rapid development of quantum computing (QC) in the NISQ era urgently demands a low-level benchmark suite and insightful evaluation metrics for characterizing the properties of prototype NISQ devices, the efficiency of QC programming compilers, schedulers and assemblers, and the capability of quantum system simulators in a classical computer. In this work, we fill this gap by proposing a low-level, easy-to-use benchmark suite called QASMBench based on the OpenQASM assembly representation. It consolidates commonly used quantum routines and kernels from a variety of domains including chemistry, simulation, linear algebra, searching, optimization, arithmetic, machine learning, fault tolerance, cryptography, and so on, trading-off between generality and usability. To analyze these kernels in terms of NISQ device execution, in addition to circuit width and depth, we propose four circuit metrics including gate density, retention lifespan, measurement density, and entanglement variance, to extract more insights about the execution efficiency, the susceptibility to NISQ error, and the potential gain from machine-specific optimizations. Applications in QASMBench can be launched and verified on several NISQ platforms, including IBM-Q, Rigetti, IonQ and Quantinuum. For evaluation, we measure the execution fidelity of a subset of QASMBench applications on 12 IBM-Q machines through density matrix state tomography, comprising 25K circuit evaluations. In addition we also compare the fidelity of executions among the IBM-Q machines, the IonQ QPU and the Rigetti Aspen M-1 system.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Process Mining in Healthcare - A Case Study for the Corporate Data Warehouse of the Veterans Affairs Office

Researchers at Oak Ridge National Laboratory are working with the Veteran’s Affairs Administration Office (VA) on a series of studies having the objective of detecting harmful anomalies (i.e., hazards) in VA health information technology systems using its Corporate Data Warehouse (CDW). This progress report describes an ORNL study focused on applying a process mining methodology to that CDW database. In the approach presented herein, process mining methodology is combined with additional software tools, metrics, and filters to permit a quick examination of large volumes of data to address specific research questions. In this work, a case study is presented in which the performance of the combined ORNL approach is evaluated by applying it to the CDW database. We performed an evidence-based study to effectively identify process models and to define metrics of frequency and performance for four health care domains: Consults, Radiology, Laboratory, and Outpatient Medication (RxOut) orders. Additionally, we classified the termination classes of the different cases by mapping to the OASIS Human Task Specification standard. We demonstrated, via process mining, that extracted raw data can aid the understanding of the flow of information in different clinical order processes. We showed that using the stepby- step approach described herein to discover processes in raw electronic health record data can be extremely effective in revealing irregular state transitions in the data and understanding clinical order information flows that are not apparent in analyzing the CDW as it is, without feature extraction.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗