SEARCH · Search NASA

Results for “user-centered evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

12 records

All sources NASA NTRS DOE OSTI

DOE OSTI2017

User-Centered Evaluation of Visual Analytics

Visual analytics systems are becoming very popular. More domains now use interactive visualizations to analyze the ever-increasing amount and heterogeneity of data. More novel visualizations are being developed for more tasks and users. We need to ensure that these systems can be evaluated to determine that they are both useful and usable. A user-centered evaluation for visual analytics needs to be developed for these systems. While many of the typical human-computer interaction (HCI) evaluation methodologies can be applied as is, others will need modification. Additionally, new functionality in visual analytics systems needs new evaluation methodologies. There is a difference between usability evaluations and user-centered evaluations. Usability looks at the efficiency, effectiveness, and user satisfaction of users carrying out tasks with software applications. User-centered evaluation looks more specifically at the utility provided to the users by the software. This is reflected in the evaluations done and in the metrics used. In the visual analytics domain this is very challenging as users are most likely experts in a particular domain, the tasks they do are often not well defined, the software they use needs to support large amounts of different kinds of data, and often the tasks last for months. These difficulties are discussed more in the section on User-centered Evaluation. Our goal is to provide a discussion of user-centered evaluation practices for visual analytics, including existing practices that can be carried out and new methodologies and metrics that need to be developed and agreed upon by the visual analytics community. The material provided here should be of use for both researchers and practitioners in the field of visual analytics. Researchers and practitioners in HCI and interested in visual analytics will find this information useful as well as a discussion on changes that need to be made to current HCI practices to make them more suitable to visual analytics. A history of analysis and analysis techniques and problems is provided as well as an introduction to user-centered evaluation and various evaluation techniques for readers from different disciplines. The understanding of these techniques is imperative if we wish to support analysis in the visual analytics software we develop. Currently the evaluations that are conducted and published for visual analytics software are very informal and consist mainly of comments from users or potential users. Our goal is to help researchers in visual analytics to conduct more formal user-centered evaluations. While these are time-consuming and expensive to carryout, the outcomes of these studies will have a defining impact on the field of visual analytics and help point the direction for future features and visualizations to incorporate. While many researchers view work in user-centered evaluation as a less-than-exciting area to work, the opposite is true. First of all, the goal is user-centered evaluation is to help visual analytics software developers, researchers, and designers improve their solutions and discover creative ways to better accommodate their users. Working with the users is extremely rewarding as well. While we use the term “users” in almost all situations there are a wide variety of users that all need to be accommodated. Moreover, the domains that use visual analytics are varied and expanding. Just understanding the complexities of a number of these domains is exciting. Researchers are trying out different visualizations and interactions as well. And of course, the size and variety of data are expanding rapidly. User-centered evaluation in this context is rapidly changing. There are no standard processes and metrics and thus those of us working on user-centered evaluation must be creative in our work with both the users and with the researchers and developers.

intelligence analysis↗

DOE OSTI2016

Visual Analytics 101

This course will introduce the field of Visual Analytics to HCI researchers and practitioners highlighting the contributions they can make to this field. Topics will include a definition of visual analytics along with examples of current systems, types of tasks and end users, issues in defining user requirements, design of visualizations and interactions, guidelines and heuristics, the current state of user-centered evaluations, and metrics for evaluation. We encourage designers, HCI researchers, and HCI practitioners to attend to learn how their skills can contribute to advancing the state of the art of visual analytics

design of visualizations and interactions↗

DOE OSTI2025

Heuristic Evaluation Methods Applied to a Predictive Maintenance Chatbot

The need for an accessible iterative approach for evaluating prospective artificial intelligence (AI)/ML based technologies in the nuclear industry is needed, given the nature of algorithms and rapid advancements. This paper explores existing heuristic design principles for user-centered design and evaluates them based on their relevancy and usefulness for evaluating AI/ ML based technologies. Researchers at the Idaho National Laboratory (INL) have developed a machine learning software application called VIsualization for PrEdictive maintenance Recommendation (VIPER), which is used to help users understand and engage with the tool to learn more about work orders, data used, predictive maintenance, and machine learning (ML) algorithms. Early user research studies used to access VIPER’s technology readiness level have occurred; however, there is room for further improvement of the software through heuristic evaluations along with other methods and user testing. This work describes the applicability of heuristic evaluation methods and cognitive walkthroughs to help ensure human readiness for prospective AI/ ML based applications, using VIPER as a candidate use case. This work supports industry in ensuring that prospective AI/ML based technologies are usable and useful for plant personnel at nuclear power plants, ultimately leading to their safe, reliable, and efficient use.

99 - GENERAL AND MISCELLANEOUS↗

DOE OSTI2025

Heuristic Evaluation Methods Applied to a Predictive Maintenance Chatbot

The need for an accessible iterative approach for evaluating prospective artificial intelligence (AI)/ML based technologies in the nuclear industry is needed, given the nature of algorithms and rapid advancements. This paper explores existing heuristic design principles for user-centered design and evaluates them based on their relevancy and usefulness for evaluating AI/ ML based technologies. Researchers at the Idaho National Laboratory (INL) have developed a machine learning software application called VIsualization for PrEdictive maintenance Recommendation (VIPER), which is used to help users understand and engage with the tool to learn more about work orders, data used, predictive maintenance, and machine learning (ML) algorithms. Early user research studies used to access VIPER?s technology readiness level have occurred; however, there is room for further improvement of the software through heuristic evaluations along with other methods and user testing. This work describes the applicability of heuristic evaluation methods and cognitive walkthroughs to help ensure human readiness for prospective AI/ ML based applications, using VIPER as a candidate use case. This work supports industry in ensuring that prospective AI/ML based technologies are usable and useful for plant personnel at nuclear power plants, ultimately leading to their safe, reliable, and efficient use. PowerPoint for conference that was reviewed in PRS and LRS PRS/CON-25-05379 and INL/CON-25-82946

99 - GENERAL AND MISCELLANEOUS↗

DOE OSTI2018

Advanced Laboratory and Field Arrays: Novel Design and Assessment Methodologies for Wave Energy Converter Design (Wave-SPARC) (Task 8)

This task focused on advancing the mission of the Wave-SPARC project by (1) establishing a user-centered Set-Based Design methodology to enable industrial partners to design higher performance WECs, (2) assessing the means through which WEC concepts and WEC design processes are evaluated by employing and potentially improving Wave-SPARC’s Technological Performance Level (TPL) metric, and (3) clarifying how the Set-Based Design approach to WEC design would be applicable for devices of different scales and/or maritime markets.

13 HYDRO ENERGY↗

DOE OSTI2024

Human readiness levels and Human Views as tools for user-centered design

The Human Readiness Level (HRL) scale is a simple nine-level scale that brings structure and consistency to the real-world application of user-centered design. It enables multidisciplinary consideration of human-focused elements during the system development process. Use of the standardized set of questions comprising the HRL scale results in a single human readiness number that communicates system readiness for human use. The Human Views (HVs) are part of an architecture framework that provides a repository for human-focused system information that can be used during system development to support the evaluation of HRL levels. Here, this paper illustrates how HRLs and HVs can be used in combination to support user-centered design processes. A real-world example for a U.S. Army software modernization program is described to demonstrate application of HRLs and HVs in the context of user-centered design.

42 ENGINEERING↗

DOE OSTI2022

A Prospective Design Method for Nuclear Power: The Evaluation, Requirements, and Goals Outline for Nuclear (ERGON) Method

Human factors researchers at Idaho National Laboratory (INL) have worked on projects spanning control room modernization, operator support systems, visualization design, and novel system creation. These projects demonstrated the need for an explicit design method for nuclear power. Human factors teams found a high standard in the Human Factors Engineering Program Review Model (NUREG-0711) and needed a design methodology which could be successful in gaining approval. Previous work has been synthesized as the Evaluation, Requirements, and Goals Outline for Nuclear (ERGON) method here. Design tasks are broken into four phases: Context and Orientation, Human Factors Review, Prototyping and Evaluation, Iteration and Improvement. ERGON is intended as a flexible and direct design method for many applications in nuclear power. ERGON has been vetted through collaborative research and development with nuclear utilities and as such, the ERGON method can assist utilities to achieve approval from a NUREG-0711 summative evaluation for HSI implementations.

42 ENGINEERING↗

DOE OSTI2025

Immersive Scientific Visualization of Molten-Salt Reactor Waste Characteristics Using Virtual Reality

Immersive visualization is changing how we explore, communicate, and understand complex scientific systems. In nuclear energy, an area in which data are often multidimensional, time-dependent, and difficult to interpret, virtual reality (VR) represents a powerful and intuitive informational medium. This work introduces a VR-based platform that visualizes the post-shutdown behavior and waste management lifecycle of molten-salt reactors (MSRs), a next-generation reactor type with unique operational and safety characteristics. The platform, built in Unity, is streamed on the Meta Quest 3 headset. It transforms high-fidelity simulation data into an interactive, immersive experience. Users can explore time-dependent reactor characteristics such as nuclide decay, which is a key factor for evaluating reactor waste strategies. The datasets were generated using the MOOSE (Multiphysics Object-Oriented Simulation Environment) framework and then processed through ParaView scripting for smooth integration into Unity. From a visualization standpoint, the platform emphasizes spatial storytelling, temporal exploration, and user-centered interaction. Users can navigate 3D reactor geometries, slice through volumetric data, and manipulate time to observe how physical phenomena evolve. Real-scale rendering and embodied interaction make the experience feel tangible. The interface is designed to be accessible, even to those without nuclear or simulation expertise. This lowers the barrier for stakeholders, policymakers, and the general public, while still supporting expert analysis and collaborative decision-making. This work shows how immersive visualization can function as both a scientific tool and a communication interface. By integrating simulation, processing, and visualization into a cohesive workflow, we offer a scalable framework for immersive scientific storytelling. The modular design supports future extensions to other reactor types and lifecycle stages, from shutdown to long-term storage, making the platform adaptable for both research and outreach.

99 - GENERAL AND MISCELLANEOUS↗

DOE OSTI2020

A User-Centered Design Study in Scientific Visualization Targeting Domain Experts

The development of usable visualization solutions is essential for ensuring both their adoption and effectiveness. User-centered design principles, which involve users throughout the entire development process, have been shown to be effective in numerous information visualization endeavors. We describe how we applied these principles in scientific visualization over a two year collaboration to develop a hybrid in situ/post hoc solution tailored towards combustion researcher needs. Lastly, we examine the importance of user-centered design and lessons learned over the design process in an effort to aid others seeking to develop effective scientific visualization solutions.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

Thinking Beyond the Default User: The Impact of Gender, Stereotypes, and Modality on Interpretation of User Needs

Throughout the mechanical design process, designers, the majority of whom are men, often fail to consider the needs of women, resulting in consequences ranging from inconvenience to increased risk of serious injury or death. Although these biases are well studied in other fields of research, the mechanical design field lacks formal investigation into this phenomenon. In this study, engineering students (n = 301) took a survey in which they read a Persona describing a student makerspace user and a Walkthrough describing the user’s interaction with the makerspace while completing a project. During the Walkthrough, the user encountered various obstacles or Pain Points. Participants were asked to recall and evaluate the Pain Points that the user encountered and then evaluated their perceptions of the makerspace and user. Furthermore, the independent variables under investigation were the gender of the user Persona (woman, gender-neutral, or man), the Walkthrough room case (crafting or woodworking makerspace), and the modality of the Persona and Walkthrough (text- or audio-based). Results showed that participants from the Text-based modality were better able to recall Pain Points compared to participants from the Audio-based modality. Pain Points were assessed as more severe when they impacted women users, potentially stemming from protective paternalism. In addition to finding that the gender of a user impacted the way a task environment was perceived, results confirmed the presence of androcentrism, or “default man” assumptions, in the way designers view end users of unknown gender. Promisingly, providing user Persona information in an audio modality significantly reduced this bias compared to text-based modalities, indicating that providing richer detail in user personas has the capability to reduce gender bias in designers.

cognitive-based design↗

DOE OSTI2023

Measuring Equality in Machine Learning Security Defenses: A Case Study in Speech Recognition

Over the past decade, the machine learning security community has developed a myriad of defenses for evasion attacks. An understudied question in that community is: for whom do these defenses defend? This work considers common approaches to defending learned systems and how security defenses result in performance inequities across different sub-populations. We outline appropriate parity metrics for analysis and begin to answer this question through empirical results of the fairness implications of machine learning security methods. We find that many methods that have been proposed can cause direct harm, like false rejection and unequal benefits from robustness training. The framework we propose for measuring defense equality can be applied to robustly trained models, preprocessing-based defenses, and rejection methods. We identify a set of datasets with a user-centered application and a reasonable computational cost suitable for case studies in measuring the equality of defenses. In our case study of speech command recognition, we show how such adversarial training and augmentation have non-equal but complex protections for social subgroups across gender, accent, and age in relation to user coverage. We present a comparison of equality between two rejection-based defenses: randomized smoothing and neural rejection, finding randomized smoothing more equitable due to the sampling mechanism for minority groups. This represents the first work examining the disparity in the adversarial robustness in the speech domain and the fairness evaluation of rejection-based defenses.

• Artificial intelligence (AI) / machine learning ↗

DOE OSTI2024

Advanced Laboratory and Field Arrays (ALFA)/Lab Collaboration Project (LCP) for Marine Energy (Final Scientific/Technical Report)

The objective of the Advanced Laboratory and Field Arrays (ALFA) project was to reduce the Levelized Cost of Energy (LCOE) of Marine and Hydrokinetic (MHK) energy by leveraging research, development, and testing capabilities at Oregon State University, University of Washington, and the University of Alaska, Fairbanks. ALFA is a project within the Pacific Marine Energy Center (PMEC; formerly NNMREC), a multi-institution entity with a diverse funding base that focuses on research and development for marine renewables. The ALFA project aimed to accelerate the development of next-generation arrays of wave energy conversion (WEC) and tidal energy conversion (TEC) devices through a suite of field-focused R&D activities spanning a broad range of strategic opportunity areas identified in the Funding Opportunity Announcement: • Device and/or array operation and maintenance (O&M) logistics development; • High-fidelity resource characterization and/or modeling technique development and validation; • Array-specific component technology development (e.g. moorings and foundations, transmission, and other offshore grid components); • Array performance testing and evaluation; and • Novel cost-effective environmental monitoring techniques and instrumentation testing and evaluation. The objective of the Lab Collaboration Project (LCP) was to accelerate the development of next-generation marine energy conversion systems. The LCP aimed to achieve these project objectives in collaboration with the national laboratories by: • Developing concept generation and assessment tools; • Improving access to existing testing resources; • Validating collision risk models between fish and turbines; and • Advancing analysis and simulation capabilities for wave-WEC interactions and PTO analysis in nonlinear ocean waves. The ALFA portion of the project was comprised of six overarching technical tasks: • Task 1: Debris Modeling, Detection and Mitigation; • Task 2: Autonomous Monitoring & Intervention; • Task 3: Resource Characterization for Extreme Conditions; • Task 4: Robust Models for Design of Offshore Anchoring and Mooring Systems; • Task 5: Performance Enhancement for Marine Energy Converter (MEC) Arrays; and • Task 6: Evaluating Sampling Techniques for MHK Biological Monitoring. The LCP was divided into four overarching technical tasks: • Task 7: Project Management and Reporting • Task 8: Novel Design and Assessment Methodologies for Wave Energy Converter Design (Wave- SPARC) • Task 9: Testing Access for Commercial Marine Renewable Energy Technology Developers • Task 10: Quantifying Collision Risk for Fish and Turbines • Task 11: Nonlinear Ocean Waves and PTO Control Strategy Each ALFA/LCP task listed above functioned as a separate and discreet project. A final Technical Report was written for each individual task and these reports were uploaded to OSTI, after receiving DOE approval. The following document is a compilation of each of these final, approved reports arranged as individual chapters.

13 HYDRO ENERGY↗