Engineering topics

Blaha, Leslie M.

Publications and source records attributed to Blaha, Leslie M..

12 records

DOE OSTI2021

Adaptive Cognitive Mechanisms to Maintain Calibrated Trust and Reliance in Automation

Trust calibration for a human–machine team is the process by which a human adjusts their expectations of the automation’s reliability and trustworthiness; adaptive support for trust calibration is needed to engender appropriate reliance on automation. Herein, we leverage an instance-based learning ACT-R cognitive model of decisions to obtain and rely on an automated assistant for visual search in a UAV interface. This cognitive model matches well with the human predictive power statistics measuring reliance decisions; we obtain from the model an internal estimate of automation reliability that mirrors human subjective ratings. The model is able to predict the effect of various potential disruptions, such as environmental changes or particular classes of adversarial intrusions on human trust in automation. Finally, we consider the use of model predictions to improve automation transparency that account for human cognitive biases in order to optimize the bidirectional interaction between human and machine through supporting trust calibration. The implications of our findings for the design of reliable and trustworthy automation are discussed.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2019

Evaluation and Validation Approaches for Simulation of Social Behavior: Challenges and Opportunities

While models of social systems have grown in scope to represent increasingly complex phenomena, efforts to validate these models against observations suffer from a number of challenges that include the lack of shared approaches, the use of single-dimensional evaluation criteria, and the difficulty in evaluating causal mechanisms. Moreover, current evaluation approaches suffer from sensitivity of the existing measurements to initial conditions and model assumptions, and failure to account for uncertainty. These effects lead to the lack of generalizability to new contexts, inability to replicate the validation approach for new models, and difficulties in interpretation of validation results. These challenges can be addressed by building a set of diverse and robust evaluation metrics and by developing best practices for comparative, reproducible validation of varied social simulation approaches against real-world data. Herein, we outline several best practices for data-drive model validation, such as defining evaluation strategies consistent with the intended use case and defining broad comparison criteria to highlight both model strengths and weaknesses. Leveraging common validation practices will enable robust analysis of simulation performance across diverse social scenarios and modeling approaches.

Saldanha, Emily G.↗

DOE OSTI2018

Unsupervised Parsing of Gaze Data with a Beta-process Vector Auto-regressive Hidden Markov Model

The first stage of analyzing eye-tracking data is commonly to code the data into sequences of fixations and saccades. This process is usually automated using simple, predetermined rules for classifying ranges of the time series into events, such as “if the dispersion of gaze samples is lower than the threshold, then code as a fixation; otherwise code as a saccade.” More recent approaches incorporate additional eye-movement categories in automated parsing algorithms, particularly glissades, by using time-varying, data-driven thresholds. We describe an alternative approach using the beta-process auto-regressive hidden Markov model (BP-AR-HMM). The BP-AR-HMM offers two main advantages over existing frameworks. First, it provides a statistical model for eye movement classification rather than a single estimate. Second, the BP-AR-HMM uses a latent process to model the number and nature of the types of eye-movements and hence is not constrained to predetermined categories. We applied the BP-AR-HMM both to high-sampling rate gaze data from Andersson, Larsson, Holmqvist, Stridh, and Nyström (2016) and to low-sampling rate data from the DIEM project (Mital, Smith, Hill, & Henderson, 2011). Driven by the data properties, the BP-AR-HMM identified clusters of samples resembling saccades, fixations and smooth pursuit, and a category that approximately corresponds to glissades (as well as blinks, errors, etc.). The BP-AR-HMM serves as an effective algorithm for data-driven event parsing alone or as an initial step in exploring the characteristics of gaze data sets.

eye tracking, fixation, fixation, saccade, smooth ↗

DOE OSTI2018

Four Perspectives on Human Bias in Visual Analytics

Mixed-initiative visual analytic systems steer analytical models and adapt views by making inferences from users' behavioral patterns with the system. Because such systems rely on incorporating implicit and explicit user feedback, they are particularly susceptible to the injection and propagation of human biases. To ultimately guard against the potentially negative effects of systems biased by human users, we must first qualify what we mean by the term bias. Thus, in this paper we describe four different perspectives on human bias that are particularly relevant to visual analytics. We discuss the interplay of human and computer system biases, particularly their roles in mixed-initiative systems. Given that the term bias is used to describe several different concepts, our goal is to encourage mindful consideration of which perspective on bias researchers take in their work to facilitate a common language in research and development efforts going forward.

visual analytics, mixed-initiative systems, cognit↗

DOE OSTI2018

Evaluation of Visualization Heuristics

Heuristics have been used since the early 1990's in the Human Computer Interaction (HCI) domain, for evaluating user interfaces. Most notably, heuristics developed by Nielsen and Molich (1990) have been used by the HCI community. Forsell and Johansson (2010) used the same methodology as Nielsen and Molich and developed a set of 10 heuristics for information visualization. Currently, neither the information visualization or the visual analytics community has adopted these heuristics. However, several studies have been conducted using heuristics. Hearst et al. (2015) found that using the heuristics for visualizations along with asking questions about the data resulted in complementary results as the questions about the data could compensate for heuristics that were difficult to understand and apply. Heli Väätäjä et al. (2016) found that heuristics related to interaction, veracity, and aesthetics needed to be added to the Forsell and Johansson (2010) set. They also noted that a lack of domain knowledge caused the evaluators issues in conducting a heuristic evaluation. We conducted a controlled experiment to determine what factors might impact the expert evaluators' capability to conduct a heuristic evaluation of a static information visualization. In particular, we wanted to understand how their familiarity with the visualizations and their experience in conducting heuristic evaluations impacted their evaluations of visualizations. We recruited 10 participants who were domain experts in the visualization field each having experience with one of more of the following: user experience design or research, user interface development, and human computer interaction research. We used a heuristic set proposed by Forsell and Johansson. Participants were also asked to complete a demographic survey in which they rated their experience with each visualization type generally, their experience conducting heuristic evaluations, their experience using each visualization type to extract information, and their experience developing and creating each visualization type. The participants were shown a series of 5 common visualizations (Scatterplots, Sunbursts, Tree Maps, Parallel Sets, and Area Graphs) all displaying the VAST 2008 Mini Challenge Two- Migrant Boats data . Participants were given a suggested analytical question related to each visualization to keep in mind as they applied the 10 heuristics to each visualization. They were asked if they could find any usability problems in the visualizations and which heuristic(s) helped them to identify these problems. The participants were asked to rate how relevant each heuristic was for evaluating each visualization along with rating how confident they felt in their ability to apply the heuristic to evaluate each visualization. At the conclusion of the study, participants provided qualitative feedback on the usability issues of each visualization. Additionally, we collected eye tracking data on 6 of the 10 participants. We will be using this data to determine how long participants looked at the various visualizations and what parts of the visualizations they focused on. We had two hypotheses for this empirical evaluation: 1.The Forsell and Johansson heuristics of Orientation and help, Data set reduction, and Prompting are less useful for static visualizations because they require some interaction in order to test completely. 2.Participants that do not have experience with heuristic evaluation will provide lower evaluation scores for the heuristics compared to those that have experience. Related to hypothesis 1, as our study used only static visualizations, interaction-related usability challenges should be harder to identify. This should result in lower usefulness ratings for those heuristics. Preliminary results indicate that participants found it more difficult to apply heuristics to unfamiliar visualizations.

Williams, Ryan G.↗

DOE OSTI2018

Improving Automation Transparency: Addressing Some of Machine Learning’s Unique Challenges

Abstract. A variety of factors can affect one’s reliance on an automated aid. Some of these factors include one’s per-ception of the system’s trustworthiness, such as perceived reliability of the system or one’s ability to understand the system’s underlying reasoning. A mismatch between the operator’s perception and the true capabilities and charac-teristics of the system can lead to inappropriate reliance on the tool. This improper use of the system can manifest as either underutilization of the technology or complacency resulting from over-trusting the system. Increasing an au-tomated tool’s transparency is one approach that enables the operator to more appropriately rely on the technology. Transparent automated systems provide additional infor-mation that allows the user to see the system’s intent and understand its underlying processes and capabilities. Sev-eral researchers have developed frameworks to support the design of more transparent automation. However, these frameworks may not fully consider the particular challeng-es to transparency design introduced by automation that leverages machine learning. Like all automation, these systems can benefit from transparency. However, artificial intelligence poses new challenges that must be considered when designing for transparency. Unique considerations must be made in terms of the type, and amount or level of transparency information conveyed to the user.

Transparency, Machine Learning, Automation↗

DOE OSTI2017

Toward a Visualization-Supported Workflow for Cyber Alert Management using Threat Models and Human-Centered Design

Cyber network analysts follow complex processes in their investigations of potential threats to their network. Much research is dedicated to providing automated tool support in the effort to make their tasks more efficient, accurate, and timely. This tool support comes in a variety of implementations from machine learning algorithms that monitor streams of data to visual analytic environments for exploring rich and noisy data sets. Cyber analysts, however, often speak of a need for tools which help them merge the data they already have and help them establish appropriate baselines against which to compare potential anomalies. Furthermore, existing threat models that cyber analysts regularly use to structure their investigation are not often leveraged in support tools. We report on our work with cyber analysts to understand they analytic process and how one such model, the MITRE ATT&CK Matrix [32], is used to structure their analytic thinking. We present our efforts to map specific data needed by analysts into the threat model to inform our eventual visualization designs. We examine data mapping for gaps where the threat model is under-supported by either data or tools. We discuss these gaps as potential design spaces for future research efforts. We also discuss the design of a prototype tool that combines machine-learning and visualization components to support cyber analysts working with this threat model.

Cyber Security↗

DOE OSTI2017

Bias by default? A means for a priori interface measurement

Systems have biases. Their interfaces naturally guide a user toward specific patterns of action. For example, modern word-processors and spreadsheets are both capable of taking word wrapping, checking spelling, storing tables, and calculating formulas. You could write a paper in a spreadsheet or could do simple business modeling in a word-processor. However, their interfaces naturally communicate which function they are designed for. Visual analytic interfaces also have biases. In this paper, we outline why simple Markov models are a plausible tool for investigating that bias and how they might be applied. We also discuss some anticipated difficulties in such modeling and touch briefly on what some Markov model extensions might provide.

Cottam, Joseph A.↗

DOE OSTI2017

Comparing capacity coefficient and dual task assessment of visual multitasking workload

Capacity coefficient analysis could offer a theoretically grounded alternative approach to subjective measures and dual task assessment of cognitive workload. Workload capacity or workload efficiency is a human information processing modeling construct defined as the amount of information that can be processed by the visual cognitive system given a specified of amount of time. In this paper, I explore the relationship between capacity coefficient analysis of workload efficiency and dual task response time measures. To capture multitasking performance, I examine how the relatively simple assumptions underlying the capacity construct generalize beyond the single visual decision making tasks. The fundamental tools for measuring workload efficiency are the integrated hazard and reverse hazard functions of response times, which are defined by log transforms of the response time distribution. These functions are used in the capacity coefficient analysis to provide a functional assessment of the amount of work completed by the cognitive system over the entire range of response times. For the study of visual multitasking, capacity coefficient analysis enables a comparison of visual information throughput as the number of tasks increases from one to two to any number of simultaneous tasks. I illustrate the use of capacity coefficients for visual multitasking on sample data from dynamic multitasking in the modified Multi-attribute Task Battery.

capacity coefficient↗

DOE OSTI2017

CHISSL: A Human-Machine Collaboration Space for Unsupervised Learning

We developed CHISSL, a human-machine interface that utilizes supervised machine learning in an unsupervised context to help the user group unlabeled instances by her own mental model. The user primarily interacts via correction (moving a misplaced instance into its correct group) or confirmation (accepting that an instance is placed in its correct group). Concurrent with the user's interactions, CHISSL trains a classification model guided by the user's grouping of the data. It then predicts the group of unlabeled instances and arranges some of these alongside the instances manually organized by the user. We hypothesize that this mode of human and machine collaboration is more effective than Active Learning, wherein the machine decides for itself which instances should be labeled by the user. We found supporting evidence for this hypothesis in a pilot study where we applied CHISSL to organize a collection of handwritten digits.

Active Learning↗

DOE OSTI2017

Interface Metaphors for Interactive Machine Learning

To promote more interactive and dynamic machine learn- ing, we revisit the notion of user-interface metaphors. User-interface metaphors provide intuitive constructs for supporting user needs through interface design elements. A user-interface metaphor provides a visual or action pattern that leverages a user’s knowledge of another domain. Metaphors suggest both the visual representations that should be used in a display as well as the interactions that should be afforded to the user. We argue that user-interface metaphors can also offer a method of extracting interaction-based user feedback for use in machine learning. Metaphors offer indirect, context-based information that can be used in addition to explicit user inputs, such as user-provided labels. Implicit information from user interactions with metaphors can augment explicit user input for active learning paradigms. Or it might be leveraged in systems where explicit user inputs are more challenging to obtain. Each interaction with the metaphor provides an opportunity to gather data and learn. We argue this approach is especially important in streaming applications, where we desire machine learning systems that can adapt to dynamic, changing data.

Interactive machine learning↗

DOE OSTI2016

Storyline Visualizations of Eye Tracking of Movie Viewing

Storyline visualizations offer an approach that promises to capture the spatio-temporal characteristics of individual observers and simultaneously illustrate emerging group behaviors. We develop a visual analytics approach to parsing, aligning, and clustering fixation sequences from eye tracking data. Visualization of the results captures the similarities and differences across a group of observers performing a common task. We apply our storyline approach to visualize gaze patterns of people watching dynamic movie clips. Storylines mitigate some of the shortcomings of existent spatio-temporal visualization techniques and, importantly, continue to highlight individual observer behavioral dynamics.

dynamic time warping↗