Search NASA⌕ Search

SEARCH · Search NASA

Results for “Heuristic Evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Heuristic Evaluation Methods Applied to a Predictive Maintenance Chatbot

The need for an accessible iterative approach for evaluating prospective artificial intelligence (AI)/ML based technologies in the nuclear industry is needed, given the nature of algorithms and rapid advancements. This paper explores existing heuristic design principles for user-centered design and evaluates them based on their relevancy and usefulness for evaluating AI/ ML based technologies. Researchers at the Idaho National Laboratory (INL) have developed a machine learning software application called VIsualization for PrEdictive maintenance Recommendation (VIPER), which is used to help users understand and engage with the tool to learn more about work orders, data used, predictive maintenance, and machine learning (ML) algorithms. Early user research studies used to access VIPER’s technology readiness level have occurred; however, there is room for further improvement of the software through heuristic evaluations along with other methods and user testing. This work describes the applicability of heuristic evaluation methods and cognitive walkthroughs to help ensure human readiness for prospective AI/ ML based applications, using VIPER as a candidate use case. This work supports industry in ensuring that prospective AI/ML based technologies are usable and useful for plant personnel at nuclear power plants, ultimately leading to their safe, reliable, and efficient use.

99 - GENERAL AND MISCELLANEOUS↗
DOE OSTI2025

Heuristic Evaluation Methods Applied to a Predictive Maintenance Chatbot

The need for an accessible iterative approach for evaluating prospective artificial intelligence (AI)/ML based technologies in the nuclear industry is needed, given the nature of algorithms and rapid advancements. This paper explores existing heuristic design principles for user-centered design and evaluates them based on their relevancy and usefulness for evaluating AI/ ML based technologies. Researchers at the Idaho National Laboratory (INL) have developed a machine learning software application called VIsualization for PrEdictive maintenance Recommendation (VIPER), which is used to help users understand and engage with the tool to learn more about work orders, data used, predictive maintenance, and machine learning (ML) algorithms. Early user research studies used to access VIPER?s technology readiness level have occurred; however, there is room for further improvement of the software through heuristic evaluations along with other methods and user testing. This work describes the applicability of heuristic evaluation methods and cognitive walkthroughs to help ensure human readiness for prospective AI/ ML based applications, using VIPER as a candidate use case. This work supports industry in ensuring that prospective AI/ML based technologies are usable and useful for plant personnel at nuclear power plants, ultimately leading to their safe, reliable, and efficient use. PowerPoint for conference that was reviewed in PRS and LRS PRS/CON-25-05379 and INL/CON-25-82946

99 - GENERAL AND MISCELLANEOUS↗
DOE OSTI2025

HPDF Data Catalog and Lakehouse Demo Heuristic Evaluation and Roadshow Feedback Reports

The December 2025 roadshow gathered rapid feedback from community members on the proof of concept AmSC/HPDF data lakehouse and catalog solution 1. This provided the opportunity to demonstrate our current state of progress and gather input on workflows and AI agents. We recognize that the proofs of concept interfaces (OpenMetadata and Goose) are not intended for our end users so we have captured feedback to keep in mind as additional technical and conceptual design work is undertaken.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2010

Evaluating Heuristics for Planning Effective and Efficient Inspections

A significant body of knowledge concerning software inspection practice indicates that the value of inspections varies widely both within and across organizations. Inspection effectiveness and efficiency can be measured in numerous ways, and may be affected by a variety of factors such as Inspection planning, the type of software, the developing organization, and many others. In the early 1990's, NASA formulated heuristics for inspection planning based on best practices and early NASA inspection data. Over the intervening years, the body of data from NASA inspections has grown. This paper describes a multi-faceted exploratory analysis performed on this · data to elicit lessons learned in general about conducting inspections and to recommend improvements to the existing heuristics. The contributions of our results include support for modifying some of the original inspection heuristics (e.g. Increasing the recommended page rate), evidence that Inspection planners must choose between efficiency and effectiveness, as a good tradeoff between them may not exist, and Identification of small subsets of inspections for which new inspection heuristics are needed. Most Importantly, this work illustrates the value of collecting rich data on software Inspections, and using it to gain insight into, and Improve, inspection practice.

Shull, Forrest J.↗
DOE OSTI2024

Improving the User Interface of the DeepLynx Data Warehouse

DeepLynx is an open-source ontology-based data warehouse created by INL to support the creation and life cycle of digital engineering projects, with a particular emphasis on digital twins [1]. Digital twins are systems that represent physical assets and process in a real-time digital environment [1]. Most well-known commercial data warehouses use Graphical User Interfaces (GUIs) for users to interact with their systems [3]. Limited publications have addressed the design of these interfaces and understanding of their target users. The current users and development team acknowledge the need to improve the current UI, not just for aesthetics but to improve functionality and workflow of DeepLynx. Traditional data warehouse users are developers, data scientists and business analysts [2]. DeepLynx users have a vast range of experience using data warehouses, and diverse roles, including engineers, scientists and management positions. Because there is a broader audience of target users for DeepLynx than a typical data warehouse, it is essential that DeepLynx has a useable and intuitive user interface. To achieve this the team performed human-computer interaction methods, including a Heuristic Evaluation of current UI using Neilsen’s Usability Heuristic, create personas based on current users by designing a user survey, data analysis and develop of personas. Followed by a redesign of the UI following using Neilsen’s Usability Heuristic and Norman’s Principles of Interactive Design in industry standard software Figma. Lastly a Heuristic Evaluation of new UI design, using Neilsen’s Usability Heuristic and User testing of redesign UI and have a group of users complete a Thinking Aloud Test of the new UI. Preliminary results of the Heuristic Evaluation of current UI arise issue with Consistency and Standards, Visibility of System Status, Match System and Real World and Recognition Rather than Recall. These issues were addressed in the proposed redesign by applying Neilsen’s Usability Heuristic and Norman’s Principles of Interactive Design. Next steps include formalized list of lessons learned and design implications for future publications.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1997

Evaluating a Web-Based Interface for Internet Telemedicine

The objective is to introduce the usability engineering methodology, heuristic evaluation, to the design and development of a web-based telemedicine system. Using a set of usability criteria, or heuristics, one evaluator examined the Spacebridge to Russia web-site for usability problems. Thirty-four usability problems were found in this preliminary study and all were assigned a severity rating. The value of heuristic analysis in the iterative design of a system is shown because the problems can be fixed before deployment of a system and the problems are of a different nature than those found by actual users of the system. It was therefore determined that there is potential value of heuristic evaluation paired with user testing as a strategy for optimal system performance design.

Lathan, Corinna E.↗
DOE OSTI2021

Usability Evaluation of the Innovation Portal and Integrated Capability Analysis Platform

The usability evaluation of the ICAP and IP tools was conducted as part of the United States (U.S.) Department of Energy (DOE) Light Water Reactor Sustainability (LWRS) ION effort with two evaluators for the Idaho National Laboratory. Many nuclear operations in the United States are facing challenges of increased business, operations, and maintenance costs due to subsidized renewables and natural gas. The commercial NPP industry has looked to other industries to develop methods such as Integrated Operations (IO) to use available technologies to reduce operating costs. One concept for reducing costs that has been developed is the Integrated Operations for Nuclear (ION). A capability Stack model was developed as part of the ION effort to break down information related to People, Technology, Processes, and Governance (PTPG). The Innovation Portal (IP) and Integrated Capability Analysis Platform (ICAP) were created from the Capability Stack model as part of the ION effort to help utilities reduce O&M costs by identifying work reduction opportunities. Two heuristic evaluations and a cognitive walkthrough of the ICAP and IP tools were completed to identify usability issues to further develop the tools for use in the ION effort. Usability issues identified were rated by priority levels and across both tools. The priority levels identified were high (5), medium (11), and low (10). Recommendation were made to help reduce usability issues of the systems and help increase the user response rate of using the systems once they are implemented for utilities. Future work on designs for the IP tool and additional heuristic evaluations and external testing are being developed to continue refinement of the ICAP and IP tools for the ION effort.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
NASA NTRS1990

Applications of different design methodologies in navigation systems and development at JPL

The NASA/JPL deep space navigation system consists of a complex array of measurement systems, data processing systems, and support facilities, with components located both on the ground and on-board interplanetary spacecraft. From its beginings nearly 30 years ago, this system has steadily evolved and grown to meet the demands for ever-increasing navigation accuracy placed on it by a succession of unmanned planetary missions. Principal characteristics of this system are its capabilities and great complexity. Three examples in the design and development of interplanetary space navigation systems are examined in order to make a brief assessment of the usefulness of three basic design theories, known as normative, rational, and heuristic. Evaluation of the examples indicates that a heuristic approach, coupled with rational-based mathematical and computational analysis methods, is used most often in problems such as orbit determination strategy development and mission navigation system design, while normative methods have seen only limited use is such applications as the development of large software systems and in the design of certain operational navigation subsystems.

Thurman, S. W.↗
NASA NTRS1991

Learning process mapping heuristics under stochastic sampling overheads

A statistical method was developed previously for improving process mapping heuristics. The method systematically explores the space of possible heuristics under a specified time constraint. Its goal is to get the best possible heuristics while trading between the solution quality of the process mapping heuristics and their execution time. The statistical selection method is extended to take into consideration the variations in the amount of time used to evaluate heuristics on a problem instance. The improvement in performance is presented using the more realistic assumption along with some methods that alleviate the additional complexity.

Ieumwananonthachai, Arthur↗
NASA NTRS2007

Evaluation of Hands-Free Devices for Space Habitat Maintenance Procedures

Currently, International Space Station (ISS) crews use a laptop computer to display procedures for performing onboard maintenance tasks. This approach has been determined to be suboptimal. A heuristic evaluation and two studies have been completed to test commercial off-the-shelf (COTS) "near-eye" heads up displays (HUDs) for support of these types of maintenance tasks. In both studies, subjects worked through electronic procedures to perform simple maintenance tasks. As a result of the Phase I study, three HUDs were down-selected to one. In the Phase II study, the HUD was compared against two other electronic display devices - a laptop computer and an e-book reader. Results suggested that adjustability and stability of the HUD display were the most significant acceptability factors to consider for near-eye displays. The Phase II study uncovered a number of advantages and disadvantages of the HUD relative to the laptop and e-book reader for interacting with electronic procedures.

Hoffman, R. B.↗
NASA NTRS2024

Usability Evaluation of NASA TOPS Open Science 101

This report presents a usability evaluation of NASA’s Open Science 101 (OS101) interface. It offers design recommendations for improving the user experience (UX) and user interface (UI) of the initial self-paced OS101 release. The author initially conducted an independent heuristic evaluation, followed by an analysis of written user feedback to corroborate findings. Overall, the usability evaluation found that the OS101 interface offers a linear and interactive UX, with areas for improvement across five major, three minor, and two cosmetic aspects of usability.

Human-Computer Interaction↗
NASA NTRS1993

Efficient partitioning and assignment on programs for multiprocessor execution

The general problem studied is that of segmenting or partitioning programs for distribution across a multiprocessor system. Efficient partitioning and the assignment of program elements are of great importance since the time consumed in this overhead activity may easily dominate the computation, effectively eliminating any gains made by the use of the parallelism. In this study, the partitioning of sequentially structured programs (written in FORTRAN) is evaluated. Heuristics, developed for similar applications are examined. Finally, a model for queueing networks with finite queues is developed which may be used to analyze multiprocessor system architectures with a shared memory approach to the problem of partitioning. The properties of sequentially written programs form obstacles to large scale (at the procedure or subroutine level) parallelization. Data dependencies of even the minutest nature, reflecting the sequential development of the program, severely limit parallelism. The design of heuristic algorithms is tied to the experience gained in the parallel splitting. Parallelism obtained through the physical separation of data has seen some success, especially at the data element level. Data parallelism on a grander scale requires models that accurately reflect the effects of blocking caused by finite queues. A model for the approximation of the performance of finite queueing networks is developed. This model makes use of the decomposition approach combined with the efficiency of product form solutions.

Standley, Hilda M.↗
NASA NTRS1994

DTS: Building custom, intelligent schedulers

DTS is a decision-theoretic scheduler, built on top of a flexible toolkit -- this paper focuses on how the toolkit might be reused in future NASA mission schedulers. The toolkit includes a user-customizable scheduling interface, and a 'Just-For-You' optimization engine. The customizable interface is built on two metaphors: objects and dynamic graphs. Objects help to structure problem specifications and related data, while dynamic graphs simplify the specification of graphical schedule editors (such as Gantt charts). The interface can be used with any 'back-end' scheduler, through dynamically-loaded code, interprocess communication, or a shared database. The 'Just-For-You' optimization engine includes user-specific utility functions, automatically compiled heuristic evaluations, and a postprocessing facility for enforcing scheduling policies. The optimization engine is based on BPS, the Bayesian Problem-Solver (1,2), which introduced a similar approach to solving single-agent and adversarial graph search problems.

Hansson, Othar↗
NASA NTRS2004

Innovative Imagery System for Enhanced Habitability Onboard ISS: Desired Features and Possible Hardware Applications

With the advent of the ISS and the experience of Russian, European, and US crewmembers on Mir, the importance of the psychological element in long duration missions is increasingly recognized. An integrated imagery system or Magic Window System could enhance the habitability, performance, and productivity for long term stays in space. Because this is type of system is a new concept for space, functional and technical requirements need to be determined. As part of a three-year project, the functional and technical requirements for an Imagery System onboard the International Space Station (ISS) have been explored. Valuable information was gathered from a survey completed by participants that had been in analog environments (remote/isolated) such as Antarctica, Aquarius, ISS crewmember debriefs, and crew support meetings to identify key functions desired for an integrated Magic Window System. Exercise and medical care activities were identified as areas that could benefit from such a system. It was determined that for exercise, it was worth exploring the concept of displaying a dynamic screen that changes as the crewmember's speed changes while showing physiological measures in a combined display. In terms of enhancing the interfaces for medical care activities, the Magic Window System could show video clips along side procedures for just-in-time training scenarios through a heads-up display. In addition, the portability, usability, and reliability were stressed as important considerations for an integrated system of technologies or Magic Window System. In addition, a review of state-of-the-art screens and other existing technologies such as tablet PCs and Personal Digital Assistants (PDAs) was conducted and contributed to defining technical requirements and feasibility of systems. Some heuristic evaluations of large displays and PDAs were conducted. Finally, feasibility for implementation onboard ISS has been considered. Currently, specific headset units are undergoing usability testing. The outcome of these activities will be valuable to determine the best candidates for an integrated system that could accommodate different needs depending on task.

Whitmore, Mihriban↗
NASA NTRS2020

CyberGAN: Generating High-fidelity Cybersecurity Data With Generative Adversarial Networks

Machine learning for cyber defense offers the promise of detecting adversarial activity against the ground data systems managing critical space assets. A fundamental challenge facing machine learning research in cybersecurity is the lack of high-fidelity, shareable datasets for robust evaluation and testing of machine learning-based solutions. High-fidelity, real-world datasets are necessary for reliable benchmarking of nominal system behavior and malicious activity. Unfortunately, such realistic datasets of both nominal and adversarial activity are rarely shared publicly by data owners due to security and privacy concerns. Besides, the available adversarial data is sparse, which makes training models on malicious activity much harder. This situation has impeded and continues to impede the research and successful adoption of machine learning methods for cyber defense. Researchers have dealt with this problem by generating data within a low-fidelity lab environment, using classified and thus unshareable datasets, or downloading low-fidelity public datasets made available by others. We propose an innovative solution to the problem by employing machine learning methods to generate high-fidelity data. Specifically, we propose the use of Generative Adversarial Networks (GANs) to generate high-fidelity data for cybersecurity purposes. GANs have found successful image processing and natural language applications, but have not yet been investigated for cyber data generation. Our proposed approach first involves training the `discriminator' network of the GAN with a sample of real-world data consisting of malicious and nominal samples. We then use the `generator' network to generate new high-fidelity data samples consisting of an appropriate mix of malicious and nominal activity. We demonstrate applications of our architecture by generating high-fidelity cybersecurity data containing both malicious and nominal samples. We thoroughly evaluate the fidelity of our generated data using heuristics and evaluate its usefulness for machine learning applications using three different datasets. Overall, our approach results in high-fidelity, shareable datasets.

Zhang, Yuening↗
NASA NTRS1997

Automatic Generation of Heuristics for Scheduling

This paper presents a technique, called GenH, that automatically generates search heuristics for scheduling problems. The impetus for developing this technique is the growing consensus that heuristics encode advice that is, at best, useful in solving most, or typical, problem instances, and, at worst, useful in solving only a narrowly defined set of instances. In either case, heuristic problem solvers, to be broadly applicable, should have a means of automatically adjusting to the idiosyncrasies of each problem instance. GenH generates a search heuristic for a given problem instance by hill-climbing in the space of possible multi-attribute heuristics, where the evaluation of a candidate heuristic is based on the quality of the solution found under its guidance. We present empirical results obtained by applying GenH to the real world problem of telescope observation scheduling. These results demonstrate that GenH is a simple and effective way of improving the performance of an heuristic scheduler.

Morris, Robert A.↗
NASA NTRS1990

DEPEND - A design environment for prediction and evaluation of system dependability

The development of DEPEND, an integrated simulation environment for the design and dependability analysis of fault-tolerant systems, is described. DEPEND models both hardware and software components at a functional level, and allows automatic failure injection to assess system performance and reliability. It relieves the user of the work needed to inject failures, maintain statistics, and output reports. The automatic failure injection scheme is geared toward evaluating a system under high stress (workload) conditions. The failures that are injected can affect both hardware and software components. To illustrate the capability of the simulator, a distributed system which employs a prediction-based, dynamic load-balancing heuristic is evaluated. Experiments were conducted to determine the impact of failures on system performance and to identify the failures to which the system is especially susceptible.

Goswami, Kumar K.↗
DOE OSTI2020

Evaluation of low-exergy heating and cooling systems and topology optimization for deep energy savings at the urban district level

District energy systems have the potential to achieve deep energy savings by leveraging the density and diversity of loads in urban districts. However, planning and adoption of district thermal energy systems is hindered by the analytical burden and high infrastructure costs. It is hypothesized that network topology optimization would enable wider adoption of advanced (ambient temperature) district thermal energy systems, resulting in energy savings. In this study, energy modeling is used to compare the energy performance of “conventional” and “advanced” district thermal energy systems at the urban district level, and a partial exhaustive search is used to evaluate a heuristic for the topology optimization problem. For the prototypical district considered, advanced district thermal energy systems mated with low-exergy building heating and cooling systems achieved a source energy use intensity that was 49% lower than that of conventional systems. The minimal spanning tree heuristic was demonstrated to be effective for the network topology optimization problem in the context of a prototypical district, and contributes to mitigating the problem’s computational complexity. The work presented in this paper demonstrates the potential of advanced district thermal energy systems to achieve deep energy savings, and advances to addressing barriers to their adoption through topology optimization.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗