Results for “User evaluation” — Page 10

DOE OSTI2021

A Visual Designer of Layer‐wise Relevance Propagation Models

Abstract Layer‐wise Relevance Propagation (LRP) is an emerging and widely‐used method for interpreting the prediction results of convolutional neural networks (CNN). LRP developers often select and employ different relevance backpropagation rules and parameters, to compute relevance scores on input images. However, there exists no obvious solution to define a “best” LRP model. A satisfied model is highly reliant on pertinent images and designers' goals. We develop a visual model designer, named as VisLRPDesigner, to overcome the challenges in the design and use of LRP models. Various LRP rules are unified into an integrated framework with an intuitive workflow of parameter setup. VisLRPDesigner thus allows users to interactively configure and compare LRP models. It also facilitates relevance‐based visual analysis with two important functions: relevance‐based pixel flipping and neuron ablation. Several use cases illustrate the benefits of VisLRPDesigner. The usability and limitation of the visual designer is evaluated by LRP users.

Huang, Xinyi↗

DOE OSTI2021

A Visual Designer of Layer-wise Relevance Propagation Models

Layer-wise Relevance Propagation (LRP) is an emerging and widely-used method for interpreting the prediction results of convolutional neural networks (CNN). LRP developers often select and employ different relevance backpropagation rules and parameters, to compute relevance scores on input images. However, there exists no obvious solution to define a "best" LRP model. A satisfied model is highly reliant on pertinent images and designers' goals. We develop a visual model designer, named as VisLRPDesigner, to overcome the challenges in the design and use of LRP models. Various LRP rules are unified into an integrated framework with an intuitive workflow of parameter setup. VisLRPDesigner thus allows users to interactively configure and compare LRP models. It also facilitates relevance-based visual analysis with two important functions: relevance-based pixel flipping and neuron ablation. Several use cases illustrate the benefits of VisLRPDesigner. The usability and limitation of the visual designer is evaluated by LRP users.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

From Simple Labels to Time-Use Integrations: Supporting the Spectrum of Qualitative Travel Behavior Data

In transportation research, applications of travel behavior data collection are context-specific and require different types of qualitative inputs. These inputs can be viewed as spanning a spectrum of user burden and data quality, from simple trip labels to complex time-use surveys. However, each currently active smartphone-based travel diary platform appears to only support one type of qualitative input, and the effort required for customization is unclear. In this paper, we characterize the spectrum by defining four canonical use cases: (i) trip labels, (ii) trip questionnaire, (iii) counterfactual trips, and (iv) time-use surveys. We then outline a mechanism for supporting configurable user inputs on the same underlying smartphone-based sensing mechanism and demonstrate that it can support all the use cases without any code changes. We further demonstrate that the flexible data model that underpins this mechanism can enable real-time monitoring and analysis. Finally, we evaluate per-user data collection and engagement metrics for large-scale deployments of three canonical use cases, spanning 10 programs, 435 users, and 251,041 trips, and a maximum duration of 800 days. Future efforts may support additional use cases through an expanded configuration and provide greater insight into user engagement. We hope that these insights enable the research community to look at qualitative inputs through a new lens and experiment with novel use cases to fill in the spectrum.

ADVANCED PROPULSION SYSTEMS,POWER TRANSMISSION AND↗

DOE OSTI2020

Guide for Joint Participation in a DOE Complex Wide SAVY-4000 Surveillance Program

This document describes LANL Storage Program best practices when working with other User Sites to implement DOE M441.1-1 Manual requirements through the use of the SAVY nuclear material packaging system. It is a Manual directive for Field Element Managers to review and approve the SAVY-4000 container series design (with supporting technical basis) as a packaging system for use at their respective sites. This review and approval incorporates an evaluation of the User Site methodology for SAVY surveillance and the process for documenting its nuclear materials storage program. This document aims to assist the Field Element Mangers and User Sites in the processes necessary to credit LANL SAVY surveillance for key elements of the User Site surveillance plan.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

NASA NTRS1994

NASA tech brief evaluations

A major step in transferring technology is to disseminate information about new developments to the appropriate sector(s). A useful vehicle for transferring technology from the government sector to industry has been demonstrated with the use of periodical and journal announcements to highlight technological achievements which may meet the needs of industries other than the one who developed the innovation. To meet this end, NASA has very successfully pursued the goal of identifying technical innovations through the national circulation publication; NASA Tech Briefs. At one time the Technology Utilization Offices of the various centers coordinated the selection of appropriate technologies through a common channel. In recent years, each NASA field center has undertaken the task of evaluating submittals for Tech Brief publication independently of the others. The University of Alabama in Huntsville was selected to assist MSFC in evaluating technology developed under the various programs managed by the NASA center for publication in the NASA Tech Briefs journal. The primary motivation for the NASA Tech Briefs publication is to bring to the attention of industry the various NASA technologies which, in general, have been developed for a specific aerospace requirement, but has application in other areas. Since there are a number of applications outside of NASA that can benefit from innovative concepts developed within the MSPC programs, the ability to transfer technology to other sectors is very high. In most cases, the innovator(s) are not always knowledgeable about other industries which might potentially benefit from their innovation. The evaluation process can therefore contribute to the list of potential users through a knowledgeable evaluator.

Workman, Gary L.↗

DOE OSTI2025

RAG for FLAG: AI Assistance for a Physics Code

Artificial intelligence (AI) has quickly become an important tool in scientific research, where significant efforts are underway to develop tools that will expedite the research process. One area of particular impact is scientific software, which can be particularly complex, and therefore time consuming to learn and use effectively. AI assistants are increasingly helping to streamline the process by performing tasks such as interactively answering user questions or suggesting solutions. Los Alamos National Laboratory (LANL) develops several advanced scientific codes, such as FLAG, which can be used to run multiphysics simulations. With this study, our goal was to develop an AI assistant for FLAG that could help make the process of understanding the software and running physics simulations more efficient. To develop an AI assistant for FLAG, we used a method called retrieval-augmented generation (RAG), which is a technique that uses information from relevant data sources to enhance the accuracy of large language models (LLMs). We used the FLAG user manual and other FLAG documentation as the knowledge base for the RAG system. When a user provides a query, RAG retrieves relevant sections from the knowledge base in response, then uses those excerpts to generate grounded and contextually rich answers. We found that our AI assistant was able to provide context aware answers and source references to user queries. To evaluate performance, we developed a set of 40 benchmark questions and compared the accuracy of the responses to those of two standard LLMs without retrieval. Our AI assistant significantly outperformed the standard LLMs at answering FLAG-related questions, with an 82.5% accuracy rate, compared to 47.5% for both of the standard LLMs. This has the potential to make the process of learning and using FLAG much easier, especially for new users. Ultimately, it supports LANL’s broader mission by empowering scientists and engineers to focus more on discovery and analysis rather than on navigating complex software systems.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1989

Intelligent hypertext systems for aerospace engineering applications

This paper is a progress report on the utilization of AI technology for assisting users locating and understanding technical information in manuals used for planning and conducting wind tunnel test. The specific goal is to create an Intelligent Hypertext System (IHS) for wind tunnel testing which combines the computerized manual in the form of hypertext and an advisory system that stores experts' knowledge and experiences. A prototype IHS for conducting transonic wind tunnel testing has been constructed with limited knowledge base. The prototype is being evaluated by potential users.

Lo, Ching F.↗

NASA NTRSDate not supplied

Lowering Barriers to Science and Space Weather Research at the Community Coordinated Modeling Center (CCMC)

The Space Weather and Heliophysics research and modeling community has been pushing the limits of our ability to understand and predict space weather events. The Community Coordinated Modeling Center (CCMC, https://ccmc.gsfc.nasa.gov) supports the community by providing a convenient collaborative platform hosting space weather models, model simulation data, curated datasets of solar events, and associated value-added services. Using these services, researchers and other end-users may exercise, evaluate, and intercompare contributed models, triage designated R2O models, as well as collaborate on a continuously updated archive of model run results. We will focus on CCMC’s ongoing commitment to the principles and guidelines of the Open Science initiative. Particularly, we will discuss our work towards making our services more transparent and our library of model simulations more accessible, open, and reproducible. We will introduce our recent tools for data discovery and correlative analysis designed to further increase the value of the user-generated data and metadata. We will also present our recent work on making heliophysical models more accessible and open to the community, particularly through simplified user experience and expert domain support. We will report on our progress in establishing an inter-center infrastructure with the ESA Virtual Space Weather Modelling Centre (VSWMC), designed to cross organizational boundaries and provide streamlined access to a joint palette of the models.

space weather↗

NASA NTRS2018

NASA Electronic Parts and Packaging (NEPP) Field Programmable Gate Array (FPGA) Single Event Effects (SEE) Test Guideline Update

The following are updated or new subjects added to the FPGA SEE Test Guidelines manual: academic versus mission specific device evaluation, single event latch-up (SEL) test and analysis, SEE response visibility enhancement during radiation testing, mitigation evaluation (embedded and user-implemented), unreliable design and its affects to SEE Data, testing flushable architectures versus non-flushable architectures, intellectual property core (IP Core) test and evaluation (addresses embedded and user-inserted), heavy-ion energy and linear energy transfer (LET) selection, proton versus heavy-ion testing, fault injection, mean fluence to failure analysis, and mission specific system-level single event upset (SEU) response prediction. Most sections within the guidelines manual provide information regarding best practices for test structure and test system development. The scope of this manual addresses academic versus mission specific device evaluation and visibility enhancement in IP Core testing.

Test guidelines↗

NASA NTRS2018

NASA Electronic Parts and Packaging Field Programmable Gate Array Single Event Effects Test Guideline Update

The following are updated or new subjects added to the FPGA SEE Test Guidelines manual: academic versus mission specific device evaluation, single event latch-up (SEL) test and analysis, SEE response visibility enhancement during radiation testing, mitigation evaluation (embedded and user-implemented), unreliable design and its affects to SEE Data, testing flushable architectures versus non-flushable architectures, intellectual property core (IP Core) test and evaluation (addresses embedded and user-inserted), heavy-ion energy and linear energy transfer (LET) selection, proton versus heavy-ion testing, fault injection, mean fluence to failure analysis, and mission specific system-level single event upset (SEU) response prediction. Most sections within the guidelines manual provide information regarding best practices for test structure and test system development. The scope of this manual addresses academic versus mission specific device evaluation and visibility enhancement in IP Core testing.

Field Programmable Gate Array (FPGA)↗

NASA NTRS2020

Evaluating Flood Forecasting System Performance in Cambodia

Every year, Cambodia experiences flooding as a result of monsoon rains and typhoons. Flood forecasting systems are designed to enable people to mitigate economic and social impacts from these events. However, in order for forecasts to be used effectively, an assessment of their accuracy is needed. This study demonstrates the performance of regional and global flood forecasting systems over the 2019 flood season. To do this, we assess the flood forecast accuracy at different forecast lead times and gauge locations in Cambodia. We then compare the flood forecast performance to satellite-based flood maps produced by the Hydrological Remote Sensing Analysis of Floods (HYDRAFloods) tool currently being co-developed by SERVIR-Mekong in collaboration with the Myanmar Department of Disaster Management. This assessment of the flood forecasting systems’ performance and comparison to flood extents helps (1) provide valuable information to forecasters and disaster managers as they make improvements to their models, and (2) provides support to forecast users as they evaluate the strengths and weaknesses of different systems for taking action.

Nauman, Claire↗

DOE OSTI2024

Thinking Beyond the Default User: The Impact of Gender, Stereotypes, and Modality on Interpretation of User Needs

Throughout the mechanical design process, designers, the majority of whom are men, often fail to consider the needs of women, resulting in consequences ranging from inconvenience to increased risk of serious injury or death. Although these biases are well studied in other fields of research, the mechanical design field lacks formal investigation into this phenomenon. In this study, engineering students (n = 301) took a survey in which they read a Persona describing a student makerspace user and a Walkthrough describing the user’s interaction with the makerspace while completing a project. During the Walkthrough, the user encountered various obstacles or Pain Points. Participants were asked to recall and evaluate the Pain Points that the user encountered and then evaluated their perceptions of the makerspace and user. Furthermore, the independent variables under investigation were the gender of the user Persona (woman, gender-neutral, or man), the Walkthrough room case (crafting or woodworking makerspace), and the modality of the Persona and Walkthrough (text- or audio-based). Results showed that participants from the Text-based modality were better able to recall Pain Points compared to participants from the Audio-based modality. Pain Points were assessed as more severe when they impacted women users, potentially stemming from protective paternalism. In addition to finding that the gender of a user impacted the way a task environment was perceived, results confirmed the presence of androcentrism, or “default man” assumptions, in the way designers view end users of unknown gender. Promisingly, providing user Persona information in an audio modality significantly reduced this bias compared to text-based modalities, indicating that providing richer detail in user personas has the capability to reduce gender bias in designers.

cognitive-based design↗

NASA NTRS1994

Hypervelocity Impact Test Facility: A gun for hire

An affordable technique has been developed to duplicate the types of impacts observed on spacecraft, including the Shuttle, by use of a certified Hypervelocity Impact Facility (HIF) which propels particulates using capacitor driven electric gun techniques. The fully operational facility provides a flux of particles in the 10-100 micron diameter range with a velocity distribution covering the space debris and interplanetary dust particle environment. HIF measurements of particle size, composition, impact angle and velocity distribution indicate that such parameters can be controlled in a specified, tailored test designed for or by the user. Unique diagnostics enable researchers to fully describe the impact for evaluating the 'targets' under full power or load. Users regularly evaluate space hardware, including solar cells, coatings, and materials, exposing selected portions of space-qualified items to a wide range of impact events and environmental conditions. Benefits include corroboration of data obtained from impact events, flight simulation of designs, accelerated aging of systems, and development of manufacturing techniques.

Johnson, Calvin R.↗

NASA NTRS2019

Evaluating Flood Forecasting System Performance in Cambodia During the 2019 Flood Season

Every year, Cambodia experiences both flash and prolonged riverine flooding as a result of monsoon rains and typhoons. Floods occur frequently in various parts of the region, and result in significant economic cost. Flood forecasting systems are designed to mitigate economic and social impacts, enabling people to prepare for extreme events. However, in order for forecasts to be used effectively, an assessment of the available forecasting systems is needed. In addition, following the onset of flooding, satellite imagery is used to generate flood maps to aid in response efforts. This study demonstrates the performance of regional and global flood forecasting systems over the 2019 flood season. To do this, we assess the flood forecast accuracy at different forecast lead times through a series of forecast verification metrics at gauge locations in Cambodia. We then compare the flood forecast performance to Sentinel 1 flood maps produced by the Hydrological Remote Sensing Analysis of Floods (HYDRAFloods) tool currently being co-developed by SERVIR-Mekong in collaboration with the Myanmar Department of Disaster Management. This assessment of the flood forecasting systems’ performance and comparison to generated flood maps helps provide context to forecasters and disaster managers as they make improvements to their models. Additionally, these results provide support to forecast users as they evaluate the strengths and weaknesses of different systems for taking action.

Nauman, Claire M.↗

NASA NTRS2023

Usability Evaluation Technical Report: A Usability Evaluation of the Communication Tool Prototype Prepared in Adobe XD for High Density Vertiplex (HDV) Team

Purpose: As cities get more crowded, the roadway infrastructure cannot keep up with the travel demands. Aviation can be a solution. Organizations supporting NASA’s Urban Air Mobility (UAM) concept are conducting studies on feasible concepts of operations for the new air traffic management system required to implement UAM. NASA’s High Density Vertiplex (HDV) team conducts studies using both live flights and simulations. Significance: The goal of this study is to evaluate the effectiveness of a communication tool developed for this team to enhance team situation awareness. Specifically, the study aims to determine if the communication tool is effective in allowing NASA-Langley and NASA-Ames team members playing different roles to understand what others are doing so that completed steps can be communicated to the whole team. Methods: This study asks participants to conduct four distinct scenarios using an Adobe XD prototype. Participants included five members of the target audience, who each received up to 15 minutes of training on how to use the Adobe XD prototype. The participants were NASA-Ames or NASA-Langley researchers, developers or tech leads who had 0 to 15 years of experience. Their ages ranged from 20 to 40 years old, and they were of both genders. After training, each participant was asked to do 4 tasks of varying difficulty, using the prototype testbed. After each task they reported on task difficulty and after all tasks were completed they were asked to fill out the System Usability Survey (SUS). Performance will be assessed through task completion time and accuracy rating. Results: User perception was evaluated by the SUS which showed a mean score over the six participants of 84. All the participants were above the acceptable range which was 70 and nobody was in the marginal usability zone between 50 and 70. In addition, most of the participants did well in terms of time on task, success rate and task difficulty on all the tasks except for Task 3. Participants had two minutes to do the task correctly to succeed at the task, otherwise it was marked as a failure. The geomean for Task 1 for completion time was 22.20; Task 2 was 58.28; Task 3 was 84.68 and Task 4 was 19.61. In Task 3, three out of six people failed this task but all five scored this with an average score 4.2 out of 5 for self-rating on task difficulty.

human factors↗

NASA NTRSDate not supplied

Examining Performance and Usability of Situation Awareness Communication Tool for NASA's High Density Vertiplex (HDV) Team

Purpose: As cities get more crowded, the roadway infrastructure cannot keep up with the travel demands. Aviation can be a solution. Organizations supporting NASA’s Urban Air Mobility (UAM) concept are conducting studies on feasible concepts of operations for the new air traffic management system required to implement UAM. NASA’s High Density Vertiplex (HDV) team conducts studies using both live flights and simulations. Significance: The goal of this study is to evaluate the effectiveness of a communication tool developed for this team to enhance team situation awareness. Specifically, the study aims to determine if the communication tool is effective in allowing NASA-Langley and NASA-Ames team members playing different roles to understand what others are doing so that completed steps can be communicated to the whole team. Methods: This study asks participants to conduct four distinct scenarios using an Adobe XD prototype. Participants included five members of the target audience, who each received up to 15 minutes of training on how to use the Adobe XD prototype. The participants were NASA-Ames or NASA-Langley researchers, developers or tech leads who had 0 to 15 years of experience. Their ages ranged from 20 to 40 years old, and they were of both genders. After training, each participant was asked to do 4 tasks of varying difficulty, using the prototype testbed. After each task they reported on task difficulty and after all tasks were completed they were asked to fill out the System Usability Survey (SUS). Performance will be assessed through task completion time and accuracy rating. Results: User perception was evaluated by the SUS which showed a mean score over the six participants of 84. All the participants were above the acceptable range which was 70 and nobody was in the marginal usability zone between 50 and 70. In addition, most of the participants did well in terms of time on task, success rate and task difficulty on all the tasks except for Task 3. Participants had two minutes to do the task correctly to succeed at the task, otherwise it was marked as a failure. The geomean for Task 1 for completion time was 22.20; Task 2 was 58.28; Task 3 was 84.68 and Task 4 was 19.61. In Task 3, three out of six people failed this task but all five scored this with an average score 4.2 out of 5 for self-rating on task difficulty.

human factors↗

DOE OSTI2024

Coupling thermal energy storage with a thermally anisotropic building envelope for building demand-side management across various US climate conditions

Here, the thermally anisotropic building envelope (TABE) is a novel active building envelope that enhances energy efficiency and thermal comfort in buildings by transferring heat and cold between building envelopes and hydronic loops. When coupled with thermal energy storage (TES) units, the TABE + TES enables the storage of both heat and cold energy captured by the TABE roof or exterior walls. This stored energy can be later released by the TABE floor for indoor heating and cooling, benefiting both the grid and the end user. This paper evaluates the merits of TABE + TES for building demand-side management across various US climate conditions, focusing on peak load shaving, annual energy savings, and cost savings under time-of-use (TOU) electric rate schedules. Simulations were conducted by integrating time-of-day–informed, rule-based control strategies in MATLAB, TABE components and TES units in COMSOL Multiphysics, and whole-building energy analysis in EnergyPlus. A case study using the US Department of Energy’s prototype single-family detached house model in Birmingham, Alabama; Los Angeles, California; Oak Ridge, Tennessee; and Denver, Colorado, showed that the TABE + TES system achieved (1) 70 % peak load shaving in Los Angeles and Denver and 20 % in Birmingham and Oak Ridge; (2) significant peak electricity savings of 351–497 kWh, reducing peak energy consumption by 38 %–78 %; and (3) annual heating cost savings of 0.79 $\$$/m2–1.17 $\$$/m 2 and cooling cost savings of 0.60 $\$$/m 2 –1.17 $\$$/m 2 using a normal utility rate or low-TOU rate. The benefits of employing the TABE + TES system are even more significant under high TOU rates.

25 ENERGY STORAGE↗

DOE OSTI2020

Additive and Subtractive Manufacturing Augmented Reality Interface (ASMARI)

Industry 4.0 represents the next evolution of industrial technology. Some of the key characteristics of Industry 4.0 include augmented reality and additive manufacturing. In support of this evolution, researchers at Oak Ridge National Lab created a prototype augmented reality (AR) interface, ASMARI. This prototype acts as a common interface for both industrial additive and subtractive machines. Additionally, in the changing industrial workspace of Industry 4.0, the role of humans will also change. This interface focuses on increasing productivity and situational awareness, while providing new interaction modalities for interacting with hardware. The prototype was also evaluated by multiple users who provided positive feedback on the system. Ultimately, the prototype represents a first step towards significant integration between augmented reality and additive manufacturing.

Borish, Michael↗