Tools for Assessing Machine Learning Models' Performance in Real-World Settings
Explore the source record for details and available documents.
Martinez, Carianne↗
SEARCH · Search NASA
Results for “Performance Tools”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Turtle: A Tool for Catching Subpopulation Trends in Performance Data
Explore the source record for details and available documents.
Gray, Skyler Devan↗
Long range science scheduling for the Hubble Space Telescope
Observations with NASA's Hubble Space Telescope (HST) are scheduled with the assistance of a long-range scheduling system (SPIKE) that was developed using artificial intelligence techniques. In earlier papers, the system architecture and the constraint representation and propagation mechanisms were described. The development of high-level automated scheduling tools, including tools based on constraint satisfaction techniques and neural networks is described. The performance of these tools in scheduling HST observations is discussed.
Miller, Glenn↗
Extension of the WIPP Performance Assessment using Modern 3D Modeling Tools.
Abstract not provided.
Zeitler, Todd R.↗
Development of Remote Cutting Tools for use at ATR - 20438
As a result of years of operations of the Advanced Test Reactor (ATR) at the Idaho National Lab (INL), the ATR canal has become congested due to the storage of materials including waste consisting of highly irradiated remote handled (RH) metals. In order to support its continuing nuclear mission the ATR requires canal space to support fuel storage and future core internal change out (CIC) components. INL staff determined that the canal must be cleaned out and waste removed to efficiently utilize the valuable canal floor space. It was determined that a methodology for processing, packaging, characterizing, and removing current and future irradiated hardware from the ATR canal is needed. Orano Federal Services (OFS) and its subcontractor Babcock Services Inc. (BSI) were selected to provide remote tooling in order to safely and efficiently size reduce the material currently stored in the ATR canal. INL developed detailed requirements that established the design and performance requirements for the tooling. Orano TN and BSI had recently successfully completed similar tasks in clearing highly activated materials from spent fuel pools and canals at commercial nuclear plants. That experience directly translated into a proven approach that could be used at the ATR thereby saving time and money and lowering risks. Based on BSI's demonstrated proficiency in the design and deployment of remote size reduction specialty tooling at comparable commercial facilities, Orano Federal Services was confident that a solution to INL's requirements could be delivered. The tooling had to be designed to be compatible with the operational requirements of the ATR canal as well as the functionality needed by the facility operators that will use the equipment. Some of the major design requirements and functions of the equipment included remote operations, capability to shear highly irradiated sections of aluminum and steel tubing, pipe, and components ranging in size from 0.6 cm to 16.5 cm diameter into 1.5 m -1.8 m lengths, utilizing buoyancy compensation to assist with underwater operations, and use commercially available existing technology and components as much as practical. The design team evaluated and selected commercially available equipment that could be modified to meet the design and performance specifications. In addition, a waste sizing table (WST) was designed and fabricated to further ensure the size reduction equipment would perform as expected and give the facility operators a stable and usable remote, underwater work platform. After successful development of the tooling, a mock up test and operator training was conducted to ensure the tooling performed as expected and met the requirements of INL. (authors)
12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
Using Hygrothermal Modeling to Resolve Practical Low-Slope Roofing Issues
Within building envelope science, the use of tools to model performance of the building envelope has evolved, and designers and scientists today have a myriad of tools available. Hygrothermal modeling is one such tool and, when used correctly, offers designers and forensic investigators important information about the movement of moisture within the building envelope. This traditional use of hygrothermal modeling is maturing as models are validated with field experience, creating a confidence in the modeling tools and their predictive capabilities. Based on this confidence, application of hygrothermal modeling to a more practical application is an evolution whose time has arrived. With changes in materials and methods, the performance of roofing systems can often times no longer be based on experience alone. Roofing contractors have been confounded by the unexplained and unplanned-for presence of moisture in roofing systems. Although hygrothermal modeling can be used to explain where the moisture came from, we propose that it should be used to prevent its occurrence. Unfortunately, more than 70 % of the roofing market is in reroofing, and oftentimes, a designer is not engaged in the roofing system design. Without the expertise of a designer and given that hygrothermal modeling is not well understood by the roofing industry, its adoption as a tool that can prevent problems has not been widely seen. This paper presents an extensive summary of hygrothermal modeling performed for roofing systems and then shows how the modeling can be used to create simple resources for roofing contractors to reduce the potential for moisture condensation at critical locations within a roofing system. The systems modeled have shown a propensity for moisture accumulation, and the use of hygrothermal modeling to eliminate these problems is a practical application of a building science tool that can greatly improve the durability of roofing systems.
Desjarlais, Andre Omer↗
PC tools for project management: Programs and the state-of-the-practice
The use of microcomputer tools for NASA project management; which features are the most useful; the impact of these tools on job performance and individual style; and the prospects for new features in project management tools and related tools are addressed. High, mid, and low end PM tools are examined. The pro's and con's of the tools are assessed relative to various tasks. The strengths and weaknesses of the tools are presented through cases and demonstrations.
Bishop, Peter C.↗
Performance Evaluation of District Energy Microgrids Planning Tool for Non-Technical Users
Community Microgrids are increasingly gaining popularity worldwide for their efficiency, cost-effectiveness, and local resilience improvement. Microgrid planning tools play a crucial role in their deployment. In the process, tentative designs of the microgrid are simulated, analyzed, and optimized. Due to the complexity of the problem, planning tools must carefully balance computational efficiency while seeking the most optimal solutions. This paper investigates the impact of algorithm selection on CPU and memory utilization of a Community Microgrid planning tool that is specifically designed for non-technical users. We compare two version of the code with two alternatives for Community Microgrid planning tools: a Greedy Algorithm, and a Linear Programming Approach. This comparison examines scenarios spanning from 2 to 50 buildings. Our findings revealed a significant difference in performance between the two algorithms, underscoring the critical role of algorithm selection in optimizing the efficiency of Community Microgrid Planning Tools.
32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
Modeling and Evaluating Pilot Performance in NextGen: Review of and Recommendations Regarding Pilot Modeling Efforts, Architectures, and Validation Studies
NextGen operations are associated with a variety of changes to the national airspace system (NAS) including changes to the allocation of roles and responsibilities among operators and automation, the use of new technologies and automation, additional information presented on the flight deck, and the entire concept of operations (ConOps). In the transition to NextGen airspace, aviation and air operations designers need to consider the implications of design or system changes on human performance and the potential for error. To ensure continued safety of the NAS, it will be necessary for researchers to evaluate design concepts and potential NextGen scenarios well before implementation. One approach for such evaluations is through human performance modeling. Human performance models (HPMs) provide effective tools for predicting and evaluating operator performance in systems. HPMs offer significant advantages over empirical, human-in-the-loop testing in that (1) they allow detailed analyses of systems that have not yet been built, (2) they offer great flexibility for extensive data collection, (3) they do not require experimental participants, and thus can offer cost and time savings. HPMs differ in their ability to predict performance and safety with NextGen procedures, equipment and ConOps. Models also vary in terms of how they approach human performance (e.g., some focus on cognitive processing, others focus on discrete tasks performed by a human, while others consider perceptual processes), and in terms of their associated validation efforts. The objectives of this research effort were to support the Federal Aviation Administration (FAA) in identifying HPMs that are appropriate for predicting pilot performance in NextGen operations, to provide guidance on how to evaluate the quality of different models, and to identify gaps in pilot performance modeling research, that could guide future research opportunities. This research effort is intended to help the FAA evaluate pilot modeling efforts and select the appropriate tools for future modeling efforts to predict pilot performance in NextGen operations.
pilot human performance models↗
Reliability and Productivity Modeling for the Optimization of Separated Spacecraft Interferometers
As technological systems grow in capability, they also grow in complexity. Due to this complexity, it is no longer possible for a designer to use engineering judgement to identify the components that have the largest impact on system life cycle metrics, such as reliability, productivity, cost, and cost effectiveness. One way of identifying these key components is to build quantitative models and analysis tools that can be used to aid the designer in making high level architecture decisions. Once these key components have been identified, two main approaches to improving a system using these components exist: add redundancy or improve the reliability of the component. In reality, the most effective approach to almost any system will be some combination of these two approaches, in varying orders of magnitude for each component. Therefore, this research tries to answer the question of how to divide funds, between adding redundancy and improving the reliability of components, to most cost effectively improve the life cycle metrics of a system. While this question is relevant to any complex system, this research focuses on one type of system in particular: Separate Spacecraft Interferometers (SSI). Quantitative models are developed to analyze the key life cycle metrics of different SSI system architectures. Next, tools are developed to compare a given set of architectures in terms of total performance, by coupling different life cycle metrics together into one performance metric. Optimization tools, such as simulated annealing and genetic algorithms, are then used to search the entire design space to find the "optimal" architecture design. Sensitivity analysis tools have been developed to determine how sensitive the results of these analyses are to uncertain user defined parameters. Finally, several possibilities for the future work that could be done in this area of research are presented.
Kenny, Sean↗
Environmentally Friendly Rotorcraft Concepts: 2011-2014: US/France MoA Helicopter Aeromechanics - Task 5
Motivation: Greenhouse gas concentrations have increased well beyond their pre-industrial levels due to human activities; Aircraft emissions are becoming regulated in industrialized nations, so future rotorcraft will need to be designed for minimal environmental impact. Objectives: Determine suitable metrics to measure environmental impact of rotorcraft, focusing first on air pollution; Integrate performance metrics with rotorcraft design codes; Generate rotorcraft designs with minimal environmental impact. Outcomes: Obtain an understanding of how targeting reduced emissions affects the rotorcraft design process; Advance the capability of existing rotorcraft design tools to include environmental performance metrics. Tools: Evaluation and predesign - NDARC (NASA Design and Analysis of Rotorcraft) and CREATION (Concepts of Rotorcraft Enhanced Assessment Through Integrated Optimization Network); Flight simulation - CAMRAD II (Comprehensive Analytical Model of Rotorcraft Aerodynamics and Dynamics) and HOST (Helicopter Overall Simulation Tool).
Environmentally Friendly↗
OptiBench: An Optimization Benchmark Tool for Renewable Energy Problems
We propose a benchmark framework and visualization tool, OptiBench, for analyzing the performance of state-of-the-art optimization solvers across a variety of optimization problems in renewable energy research. Our framework is designed from the ground up in the Julia programming language and enables analysis at scale on high performance computing (HPC) systems. Our visualization tool allows effortless evaluation of optimization solver performance, robustness, and accuracy through intuitive plots, e.g., performance profiles, heat maps, and distribution plots. We have tested three benchmark suites relevant to the modeling of renewable energy systems, viz., CUTEst, PGLib-OPF, and WaterTAP water treatment optimization problems. We illustrate benchmarking of CUTEst using OptiBench on the National Renewable Energy Laboratory's (NREL) HPC Kestrel. Our findings indicate that MA57 HSL linear solver demonstrated the best overall performance for an experimental IPOPT implementation. Our work is ongoing and we intend to add support for more optimization solvers and benchmark test suites in the future.
32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
OptiBench: An Optimization Benchmark Tool for Renewable Energy Problems
We propose a benchmark framework and visualization tool, OptiBench, for analyzing the performance of state-of-the-art optimization solvers across a variety of optimization problems in renewable energy research. Our framework is designed from the ground up in the Julia programming language and enables analysis at scale on high performance computing (HPC) systems. Our visualization tool allows effortless evaluation of optimization solver performance, robustness, and accuracy through intuitive plots, e.g., performance profiles, heat maps, and distribution plots. We have tested three benchmark suites relevant to the modeling of renewable energy systems, viz., CUTEst, PGLib-OPF, and WaterTAP water treatment optimization problems. We illustrate benchmarking of CUTEst using OptiBench on the National Laboratory of the Rockies's (NLR) HPC Kestrel. Our findings indicate that MA57 HSL linear solver demonstrated the best overall performance for an experimental IPOPT implementation. Our work is ongoing and we intend to add support for more optimization solvers and benchmark test suites in the future.
97 MATHEMATICS AND COMPUTING↗
An overview of data tools for representing and managing building information and performance data
Building information modeling (BIM) has been widely adopted for representing and exchanging building data across disciplines during building design and construction. However, BIM's use in the building operation phase is limited. With the increasing deployment of low-cost sensors and meters, as well as affordable digital storage and computing technologies, growing volumes of data have been collected from buildings, their energy services systems, and occupants. Such data are crucial to help decision makers understand what, how, and when energy is consumed in buildings—a critical step to improving building performance for energy efficiency, demand flexibility, and resilience. However, practical analyses and use of the collected data are very limited due to various reasons, including poor data quality, ad-hoc representation of data, and lack of data science skills. To unlock value from building data, there is a strong need for a toolchain to curate and represent building information and performance data in common standardized terminologies and schemas, to enable interoperability between tools and applications. This study selected and reviewed 24 data tools based on common use cases of data across the building life cycle, from design to construction, commissioning, operation, and retrofits. The selected data tools are grouped into three categories: (1) data dictionary or terminology, (2) data ontology and schemas, and (3) data platforms. The data are grouped into ten typologies covering most types of data collected in buildings. This study resulted in five main findings: (1) most data representation tools can represent their intended data typologies well, such as Green Button for smart meter data and Brick schema for metadata of sensors in buildings and HVAC systems, but none of the tools cover all ten types of data; (2) there is a need for data schemas to represent the basis of design data and metadata of occupant data; (3) standard terminologies such as those defined in BEDES are only adopted in a few data tools; (4) integrating data across various stages in the building life cycle remains a challenge; and (5) most data tools were developed and maintained by different parties for different purposes, their flexibility and interoperability can be improved to support broader use cases. Finally, recommendations for future research on building data tools are provided for the data and buildings community based on the FAIR principles to make data Findable, Accessible, Interoperable, and Reusable.
32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
HPC I/O innovations in the exascale era
As high performance computing architecture evolves to deliver ever-increasing performance, the middleware tools also need to adapt in order for applications to better use these higher-performance features. Here, the Adaptable Input Output System (ADIOS), which provides scalable IO performance for exascale HPC applications is one such middleware. During the Exascale Computing Project (ECP), key portions of the ADIOS environment were adapted to respond to ongoing developments in exascale computing and the stresses and opportunities inherent in those changes. This paper examines those changes and where appropriate compares them to pre-exascale implementations.
ADIOS↗
Toward Interactive Scenario Analysis and Exploration
As Modeling and Simulation (M&S) tools have matured, their applicability and importance have increased across many national security challenges. In particular, they provide a way to test how something may behave without the need to do real world testing. However, current and future changes across several factors including capabilities, policy, and funding are driving a need for rapid response or evaluation in ways that many M&S tools cannot address. Issues around large data, computational requirements, delivery mechanisms, and analyst involvement already exist and pose significant challenges. Furthermore, rising expectations, rising input complexity, and increasing depth of analysis will only increase the difficulty of these challenges. In this study we examine whether innovations in M&S software coupled with advances in ''cloud'' computing and ''big-data'' methodologies can overcome many of these challenges. In particular, we propose a simple, horizontally-scalable distributed computing environment that could provide the foundation (i.e. ''cloud'') for next-generation M&S-based applications based on the notion of ''parallel multi-simulation''. In our context, the goal of parallel multi- simulation is to consider as many simultaneous paths of execution as possible. Therefore, with sufficient resources, the complexity is dominated by the cost of single scenario runs as opposed to the number of runs required. We show the feasibility of this architecture through a stable prototype implementation coupled with the Umbra Simulation Framework [6]. Finally, we highlight the utility through multiple novel analysis tools and by showing the performance improvement compared to existing tools.
97 MATHEMATICS AND COMPUTING↗
Multi-Rigor Agile Verification and Rapid Prototyping for Formally Verified Software
We propose a novel approach to developing formally verified systems through Multi-rigor Agile Verification. Multi-rigor Agile Verification is rooted in the hypothesis of Rigor Independence, that a system’s specification and verification architecture depend primarily on the system requirements to be verified, and they depend very little on the rigor level of the methods used to verify those requirements. Due to its iterative nature, Multi-rigor Agile Verification promises to mitigate many of the high upfront design costs experienced by formally verified systems and to deliver a better-architected, and thus better-trusted, system in the end. We then discuss the tooling needed to perform Multi-rigor Agile Verification and go in depth to build one of those tools, which directly generates executable prototype code from declarative formal specifications using the Maude rewrite-logic framework.
97 MATHEMATICS AND COMPUTING↗
A Global System for Transportation Simulation and Visualization in Emergency Evacuation Scenarios
Simulation-based studies are frequently used for evacuation planning and decision making processes. Given the transportation systems complexity and data availability, most evacuation simulation models focus on certain geographic areas. With routine improvement of OpenStreetMap road networks and LandScanTM global population distribution data, we present WWEE, a uniform system for world-wide emergency evacuation simulations. WWEE uses unified data structure for simulation inputs. It also integrates a super-node trip distribution model as the default simulation parameter to improve the system computational performance. Two levels of visualization tools are implemented for evacuation performance analysis, including link-based macroscopic visualization and vehicle-based microscopic visualization. For left-hand and right-hand traffic patterns in different countries, the authors propose a mirror technique to experiment with both scenarios without significantly changing traffic simulation models. Ten cities in US, Europe, Middle East, and Asia are modeled for demonstration. With default traffic simulation models for fast and easy-to-use evacuation estimation and visualization, WWEE also retains the capability of interactive operation for users to adopt customized traffic simulation models. For the first time, WWEE provides a unified platform for global evacuation researchers to estimate and visualize their strategies performance of transportation systems under evacuation scenarios.
Emergency Evacuation↗