Search NASA⌕ Search

SEARCH · Search NASA

Results for “user study”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

What Do Our Users Want? Perspectives on Understanding and Meeting User Needs for Multi-Mission Data Services

The Sun-Earth Connection Active Archive (SECAA) project of NASA's Space Physics Data Facility operates a range of unique and heavily used multi-mission data services in support of the large-scale science objectives of the Great Observatory, including services such as CDAWeb, the CDAWeb Plus client, SSCWeb, OMNIweb and the CDF data format. In developing and operating these services, we have encountered and continue to struggle with a wide range of issues such as balancing scope and functionality with simplicity and ease of use, understanding the effectiveness of our choices and identifying areas most important for further improvement. In this paper, we will review our key services and then discuss some of our observations and new approaches to understanding and meeting user data service requirements. Some observations are obvious but may still have substantial implications; e.g. functionality without information content is of little user interest, which has led to our recent emphasis on development of web services interfaces, so the content and functionality we already serve is readily and fully available as a building block for new services. Some observations require careful design and tradeoffs; e.g. users will complain when they are offered interfaces with limited options but users are also easily intimidated and become lost when offered extensive options for customization. Some observations remain highly challenging; e.g. a comprehensive multi-mission, multi-source view of all data and services available easily produces a daunting list, but a more selective view can easily lead users to overlook available and relevant data. It is often difficult to obtain and meaningfully interpret measures of true productive usage and overall user satisfaction, even with a variety of techniques including statistics, citations, case studies, user feedback and advisory committees. Most of these issues will apply to and may even be more acute for distributed implementation architectures.

McGuire, Robert E.↗

How Do Visual Explanations Foster End Users' Appropriate Trust in Machine Learning?

We investigated the effects of different visual explanations on users' trust in machine learning classification. We proposed three forms of visual explanations of a classification based on identifying relevant training instances. We conducted a user study to evaluate these visual explanations as well as a no explanation condition. We measured users' trust of a classifier, quantified the effects of these three forms of explanations, and assessed the changes in users' trust. We found that participants trust a classifier appropriately when an explanation is available. The combination of human, classification algorithm and understandable explanation makes better decisions than the classifier and human alone.This work advances the state-of-the-art closer to building trust-able machine learning models and informs the design and appropriate use of automated systems.

Yang, Fumeng↗

Understanding Interactive and Reproducible Computing With Jupyter Tools at Facilities

Increasingly Jupyter tools are being adopted and incorporated into High Performance Computing (HPC) and scientific user facilities. Adopting Jupyter tools enables more interactive and reproducible computational work at facilities across data life cycles. As the volume, variety, and scope of data grow, scientists need to be able to analyze and share results in user friendly ways. Human-centered research highlights design challenges around computational notebooks, and our qualitative user study shifts focus to better characterize how Jupyter tools are being used in HPC and science user facilities today. We conducted twenty-nine interviews, and obtained 103 survey responses from NERSC Jupyter users, to better understand the increasing role of interactive computing tools in DOE sponsored scientific work. We examine a range of issues that emerge using and supporting Jupyter in HPC ecosystems, including: how Jupyter is being used by scientists in HPC and user facility ecosystems; how facilities are purposefully supporting Jupyter in their ecosystems; feedback NERSC users have about the facility’s deployment, and, discuss features NERSC indicated would be helpful. We offer a variety of takeaways for staff supporting Jupyter at facilities, Project Jupyter and related open source communities, and funding agencies supporting interactive computing work.

97 MATHEMATICS AND COMPUTING↗

Design and evaluation of alphabetic and numeric input methods for virtual reality

In today’s virtual reality (VR), users have various ways to influence their VR experience, including alphanumeric input. While typing characters and numbers is straightforward on desktop computers, it presents challenges and opportunities in head-mounted display VR due to specific interaction methods and a lack of real-world visual stimuli. Addressing these open questions, our work implements and evaluates ten approaches to alphabetic and numeric inputs in VR. Here, we describe the design motivation behind these input methods and evaluate them in a user study with 40 participants divided into groups for alphabetic and numeric keyboards. This comparison investigates each method’s performance and user interactions. Our findings suggest that different input methods significantly impact words per minute and error rates, and that certain keyboard designs may receive better subjective evaluations despite poorer objective performance.

97 MATHEMATICS AND COMPUTING↗

PRODEM: An Annual Series of Summer DEMs (2019 through 2022) of the Marginal Areas of the Greenland Ice Sheet

Surface topography across the marginal zone of the Greenland Ice Sheet is constantly evolving in response to changing weather, season, climate, and ice dynamics. However, current digital elevation models (DEMs) for the ice sheet are usually based on data from a multi-year period, thus obscuring these changes over time. Here we present four 500 m resolution summer DEMs (PRODEMs) of the Greenland Ice Sheet marginal zone for 2019 through 2022. The PRODEMs cover the marginal zone from the ice edge to 50 km inland, hence capturing all Greenland outlet glaciers. Each PRODEM is based on data fusion of CryoSat-2 radar altimetry and ICESat-2 laser altimetry using regionally varying kriging of elevation anomalies relative to ArcticDEM. The PRODEMs are validated using leave-one-out cross-validation, and PRODEM19 is further validated against an external data set, showcasing their ability to correctly represent surface elevations within the associated spatially varying prediction uncertainties. We observe a general lowering of surface elevations during the 4-year PRODEM period, but the spatial pattern of change is highly complex and with annual changes superimposed. The PRODEMs enable detailed studies of the marginal ice sheet elevation changes. With their high spatio-temporal resolution, the PRODEMs will be of value to a wide range of researchers and users studying ice sheet dynamics and monitoring how the ice sheet responds to changing environmental conditions. PRODEMs from summer 2019 through 2022 are available at https://doi.org/10.22008/FK2/52WWHG (Winstrup, 2024), and we plan to annually update the product henceforth.

Digital Elevation Models↗

Technology Transfer Challenges: A Case Study of User-Centered Design in NASA's Systems Engineering Culture

The Upper Stage (US) section of the National Aeronautics and Space Administration's (NASA) Ares I rocket will require internal access platforms for maintenance tasks performed by humans inside the vehicle. Tasks will occur during expensive critical path operations at Kennedy Space Center (KSC) including vehicle stacking and launch preparation activities. Platforms must be translated through a small human access hatch, installed in an enclosed worksite environment, support the weight of ground operators and be removed before flight - and their design must minimize additional vehicle mass at attachment points. This paper describes the application of a user-centered conceptual design process and the unique challenges encountered within NASA's systems engineering culture focused on requirements and "heritage hardware". The NASA design team at Marshall Space Flight Center (MSFC) initiated the user-centered design process by studying heritage internal access kits and proposing new design concepts during brainstorming sessions. Simultaneously, they partnered with the Technology Transfer/Innovative Partnerships Program to research inflatable structures and dynamic scaffolding solutions that could enable ground operator access. While this creative, technology-oriented exploration was encouraged by upper management, some design stakeholders consistently opposed ideas utilizing novel, untested equipment. Subsequent collaboration with an engineering consulting firm improved the technical credibility of several options, however, there was continued resistance from team members focused on meeting system requirements with pre-certified hardware. After a six-month idea-generating phase, an intensive six-week effort produced viable design concepts that justified additional vehicle mass while optimizing the human factors of platform installation and use. Although these selected final concepts closely resemble heritage internal access platforms, challenges from the application of the user-centered process provided valuable lessons for improving future collaborative conceptual design efforts.

Quick, Jason↗

Active Learning Meets Foundation Models: Fast Remote Sensing Data Annotation for Object Detection

Object detection in remote sensing demands extensive, high-quality annotations—a process that is both labor-intensive and time-consuming. In this work, we introduce a real-time active learning and semi-automated labeling framework that leverages foundation models to streamline dataset annotation for object detection in remote sensing imagery. For example, by integrating a Segment Anything Model (SAM), our approach generates mask-based bounding boxes that serve as the basis for dual sampling: (a) uncertainty estimation to pinpoint challenging samples, and (b) diversity assessment to ensure broad data coverage. Furthermore, our Dynamic Box Switching Module (DBS) addresses the well-known cold start problem for object detection models by replacing its suboptimal initial predictions with SAM-derived masks, thereby enhancing early-stage localization accuracy. Extensive evaluations on multiple remote sensing datasets plus a real-world user study, demonstrate that our framework not only reduces annotation effort, but also significantly boosts detection performance compared to traditional active learning sampling methods. The code for training and the user interface will be made available.

Burges, Marvin [ORNL] (ORCID:0000000312690769)↗

7 Key Challenges for Visualization in Cyber Network Defense

In this paper we present seven challenges, informed by two user studies, to be considered when developing a visualization for cyber security purposes. Cyber security visualizations must go beyond isolated solutions and “pretty picture” visualizations in order to make impact to users. We provide an example prototype that addresses the challenges with a description of how they are met. Our aim is to assist in increasing utility and adoption rates for visualization capabilities in cyber security.

Cyber security, visualization, network defense↗

NASA STI modernization plan

The NASA Scientific and Technical Information (STI) modernization plan is described as a phased approach to a shared network environment. The plan is based on end user needs articulated in continuing user studies in addition to technical requirements. The modernization phases are outlined in a chronology covering the 1993-1997 time frame. Various challenges and constraints are discussed. Emphasis is placed on sustaining user involvement and meeting user needs in measurable ways as the development of the system architecture proceeds.

Kaye, Karen↗

Experts’ Familiarity Versus Optimality of Visualization Design: How Familiarity Affects Perceived and Objective Task Performance

Experts, cutting across domains like biology, climate science, cyber, energy, etc., frequently use visualizations as the principal medium for making analytical judgments or for presenting the results of their analysis. However, scientists are often skeptical about adopting new visualization methods over familiar ones, although the latter might be perceptually sub-optimal. This is due to the use of the familiarity heuristic, where the perceived cognitive ease in processing familiar representations of information leads scientists to undermine the effect of visualization best practices. Recent studies have shown that this often results in a discrepancy between scientists’ perceived and actual performance quality. It has also been shown that in some cases, participatory design sessions and qualitative and quantitative user studies are able to mitigate the effects of such bias. In this paper, we discuss the potential causes and effects of familiarity related biases with examples from recent studies and reflect on the related research questions.

visualization, climate science, cognitive bias, fa↗

Exploiting Task Tolerances in Mimicry-based Telemanipulation

We explore task tolerances, i.e., allowable position or rotation inaccuracy, as an important resource to facilitate smooth and effective telemanipulation. Task tolerances provide a robot flexibility to generate smooth and feasible motions; however, in teleoperation, this flexibility may make the user’s control less direct. In this work, we implemented a telema nipulation system that allows a robot to autonomously adjust its configuration within task tolerances. We conducted a user study comparing a telemanipulation paradigm that exploits task tolerances (functional mimicry) to a paradigm that requires the robot to exactly mimic its human operator (exact mimicry), and assess how the choice in paradigm shapes user experience and task performance. Our results show that autonomous adjustments within task tolerances can lead to performance improvements without sacrificing perceived control of the robot. Additionally, we find that users perceive the robot to be more under control, predictable, fluent, and trustworthy in functional mimicry than in exact mimicry.

42 ENGINEERING↗

Online Interactive Platform for COVID-19 Literature Visual Analytics: Platform Development Study

Background Papers on COVID-19 are being published at a high rate and concern many different topics. Innovative tools are needed to aid researchers to find patterns in this vast amount of literature to identify subsets of interest in an automated fashion. Objective We present a new online software resource with a friendly user interface that allows users to query and interact with visual representations of relationships between publications. Methods We publicly released an application called PLATIPUS (Publication Literature Analysis and Text Interaction Platform for User Studies) that allows researchers to interact with literature supplied by COVIDScholar via a visual analytics platform. This tool contains standard filtering capabilities based on authors, journals, high-level categories, and various research-specific details via natural language processing and dozens of customizable visualizations that dynamically update from a researcher’s query. Results PLATIPUS is available online and currently links to over 100,000 publications and is still growing. This application has the potential to transform how COVID-19 researchers use public literature to enable their research. Conclusions The PLATIPUS application provides the end user with a variety of ways to search, filter, and visualize over 100,00 COVID-19 publications.</:p>

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Computer program user's manual for advanced general aviation propeller study

A user's manual is presented for a computer program for predicting the performance (static, flight, and reverse), noise, weight and cost of propellers for advanced general aviation aircraft of the 1980 time period. Complete listings of this computer program with detailed instructions and samples of input and output are included.

Worobel, R.↗

A User-Centered Design Study in Scientific Visualization Targeting Domain Experts

The development of usable visualization solutions is essential for ensuring both their adoption and effectiveness. User-centered design principles, which involve users throughout the entire development process, have been shown to be effective in numerous information visualization endeavors. We describe how we applied these principles in scientific visualization over a two year collaboration to develop a hybrid in situ/post hoc solution tailored towards combustion researcher needs. Lastly, we examine the importance of user-centered design and lessons learned over the design process in an effort to aid others seeking to develop effective scientific visualization solutions.

97 MATHEMATICS AND COMPUTING↗

Demonstration and Evaluation of Explainable and Trustworthy Predictive Technology for Condition-based Maintenance

The domestic nuclear power plant (NPP) fleet has historically relied on labor-intensive and time-consuming predictive maintenance (PdM) programs, thus driving up operation and maintenance (O&M) costs to achieve high-capacity factors. Artificial intelligence (AI) and machine-learning (ML) can help simplify complex problems such as diagnosing equipment degradation to enable more effective decision-making efforts. The benefits of AI will be felt through more efficient plant O&M, improved work processes, and better integration of people and technology. Together, these benefits hold the promise to make nuclear power more sustainable by reducing O&M costs while improving employee engagement. While AI and ML technologies hold significant promise for the nuclear industry, there are challenges or barriers to their adoption. Explainability and trustworthiness of AI are two salient challenges that need to be addressed for wider deployment of these technologies in NPPs. This research focuses specifically on addressing the explainability and trustworthiness of AI technologies to advance the human, technical, and organization (HTO) readiness levels in adopting a risk-informed PdM strategy at commercial NPPs. In addition, this approach can be adapted to enhance the acceptability of AI in other nuclear applications with a few application-specific modifications. The technical approach ensuring wider adoption of AI technologies was developed by Idaho National Laboratory (INL)—in collaboration with Public Service Enterprise Group (PSEG), Nuclear, LLC—by utilizing the circulating water system (CWS) at two PSEG-owned plant sites for demonstration. Focused user studies were performed in collaboration with subject matter experts (SMEs) from PSEG and other nuclear domains to enhance human and organization readiness by building trust in AI-informed technologies. VIsualization for PrEdictive maintenance Recommendation (VIPER)—a Battelle Energy Alliance, LLC, copyrighted software—was developed and expanded to provide a user-centric visualization by incorporating inputs from the collaborating utility, human factors engineering guidelines, and data analysts. The VIPER software enables users, who may be unfamiliar with ML in general, to be interactively engaged by asking technical questions about PdM, work orders, diagnosis results and their confidence levels, the kind of data being used, and the types of ML algorithms employed. This interactive engagement enhances explainability and builds trust. One of the enabling accomplishments was the integration of large language models (LLMs), both text-based and vision-based, in the VIPER software.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

SAGE Intrusion Detection System: Sensitivity Analysis Guided Explainability for Machine Learning.

This report details the results of a three-fold investigation of sensitivity analysis (SA) for machine learning (ML) explainability (MLE): (1) the mathematical assessment of the fidelity of an explanation with respect to a learned ML model, (2) quantifying the trustworthiness of a prediction, and (3) the impact of MLE on the efficiency of end-users through multiple users studies. We focused on the cybersecurity domain as the data is inherently non-intuitive. As ML is being using in an increasing number of domains, including domains where being wrong can elicit high consequences, MLE has been proposed as a means of generating trust in a learned ML models by end users. However, little analysis has been performed to determine if the explanations accurately represent the target model and they themselves should be trusted beyond subjective inspection. Current state-of-the-art MLE techniques only provide a list of important features based on heuristic measures and/or make certain assumptions about the data and the model which are not representative of the real-world data and models. Further, most are designed without considering the usefulness by an end-user in a broader context. To address these issues, we present a notion of explanation fidelity based on Shapley values from cooperative game theory. We find that all of the investigated MLE explainability methods produce explanations that are incongruent with the ML model that is being explained. This is because they make critical assumptions about feature independence and linear feature interactions for computational reasons. We also find that in deployed, explanations are rarely used due to a variety of reason including that there are several other tools which are trusted more than the explanations and there is little incentive to use the explanations. In the cases when the explanations are used, we found that there is the danger that explanations persuade the end users to wrongly accept false positives and false negatives. However, ML model developers and maintainers find the explanations more useful to help ensure that the ML model does not have obvious biases. In light of these findings, we suggest a number of future directions including developing MLE methods that directly model non-linear model interactions and including design principles that take into account the usefulness of explanations to the end user. We also augment explanations with a set of trustworthiness measures that measure geometric aspects of the data to determine if the model output should be trusted.

97 MATHEMATICS AND COMPUTING↗

The Impact of Streaming Data on Sensemaking with Mixed-Initative Visual Analytics

Visual data analysis helps people gain insights into data via interactive visualizations. People generate and test hypotheses and questions about data in context of the domain. This process can generally be referred to as sensemaking. Much of the work on studying sensemaking (and creating visual analytic techniques in support of it) has been focused on static datasets. However, how do the cognitive processes of sensemaking change when data are changing? Further, what implication for design does this create for mixed-initiative visual analytics systems? This paper presents the results of a user study analyzing the impact of streaming data on sensemaking. To perform this study, we developed a mixed-initiative visual analytic prototype, the Streaming Canvas, that affords the analysis of streaming text data. We compare the sensemaking process of people using this tool for a static and streaming dataset. We present the results of this study and discuss the implications on future visual analytic systems that combine machine learning and interactive visualization to help people make sense of streaming data.

visualization↗