Search NASA⌕ Search

Engineering topics

Huang, Zhuanyi

Publications and source records attributed to Huang, Zhuanyi.

Machine learning model explanation apparatus and methods

Explanation apparatus and methods are described. In one aspect, an explanation apparatus includes processing circuity configured to access a source instance which has been classified by a machine learning model; create associations of the source instance with a plurality of training instances; and process the associations of the source instance and the training instances to identify a first subset of the training instances which have less relevance to the classification decision of the source instance by the machine learning model compared with a second subset of the training instances; and an interface configured to communicate information to a user, and wherein the processing circuitry is configured to control the user interface to communicate the second subset of the training instances to the user as evidence to explain the classification of the source instance by the machine learning model.

Arendt, Dustin↗

How Do Visual Explanations Foster End Users' Appropriate Trust in Machine Learning?

We investigated the effects of different visual explanations on users' trust in machine learning classification. We proposed three forms of visual explanations of a classification based on identifying relevant training instances. We conducted a user study to evaluate these visual explanations as well as a no explanation condition. We measured users' trust of a classifier, quantified the effects of these three forms of explanations, and assessed the changes in users' trust. We found that participants trust a classifier appropriately when an explanation is available. The combination of human, classification algorithm and understandable explanation makes better decisions than the classifier and human alone.This work advances the state-of-the-art closer to building trust-able machine learning models and informs the design and appropriate use of automated systems.

Yang, Fumeng↗

Parallel Embeddings: a Visualization Technique for Contrasting Learned Representations

We introduce ``Parallel Embeddings'', a new technique that generalizes the classical Parallel Coordinates visualization technique from a sequence of features within a single matrix to a sequence of multiple feature matrices. This is accomplished by separately embedding each matrix in the sequence into a 2-D frame, juxtaposing these 2-D representations, then connecting points between frames. We outline how this visualization can be used for several ``model comparison'' tasks in machine learning. We compare user performance with Parallel Embeddings to a baseline using the TensorFlow Embedding Projector for estimating model accuracy and understanding conceptual model differences. We found that users were more accurate with Parallel Embeddings for model comparison and that users learned how to use Parallel Embeddings more quickly than TensorFlow Embedding Projector. Furthermore, users' analytical process using Embedding Projector was positively affected by using Parallel Embeddings beforehand.

Model comparison, machine learing, machine learnin↗

Explaining Multimodal Deceptive News Prediction Models

Social media plays a valuable role in rapid news dissemination, but it also serves as a vehicle to propagate unverified information. For example, news shared on Facebook or Twitter may actually contain disinformation, propaganda, hoaxes, conspiracies, clickbait or satire. This paper presents an in-depth analysis of the behavior of suspicious news classification models including error analysis and prediction confidence. We consider five deep learning architectures that leverage combinations of text, linguistic and image input signals from tweets. The behavior of these models is analyzed across four suspicious news prediction tasks. Our findings include that models leveraging only the text of tweets outperform those leveraging only the image (by 3-13% absolute in F-measure), and that models that combine image and text signals with linguistic cues e.g., biased and subjective language markers can, but do not always, perform even better. Finally, our main contribution is a series of analyses, in which we characterize text and image traits of our classes of suspicious news and analyze patterns of errors made by the various models to inform the design of future deceptive news prediction models.

deep learning, social media, deception↗

Evaluation of Visualization Heuristics

Heuristics have been used since the early 1990's in the Human Computer Interaction (HCI) domain, for evaluating user interfaces. Most notably, heuristics developed by Nielsen and Molich (1990) have been used by the HCI community. Forsell and Johansson (2010) used the same methodology as Nielsen and Molich and developed a set of 10 heuristics for information visualization. Currently, neither the information visualization or the visual analytics community has adopted these heuristics. However, several studies have been conducted using heuristics. Hearst et al. (2015) found that using the heuristics for visualizations along with asking questions about the data resulted in complementary results as the questions about the data could compensate for heuristics that were difficult to understand and apply. Heli Väätäjä et al. (2016) found that heuristics related to interaction, veracity, and aesthetics needed to be added to the Forsell and Johansson (2010) set. They also noted that a lack of domain knowledge caused the evaluators issues in conducting a heuristic evaluation. We conducted a controlled experiment to determine what factors might impact the expert evaluators' capability to conduct a heuristic evaluation of a static information visualization. In particular, we wanted to understand how their familiarity with the visualizations and their experience in conducting heuristic evaluations impacted their evaluations of visualizations. We recruited 10 participants who were domain experts in the visualization field each having experience with one of more of the following: user experience design or research, user interface development, and human computer interaction research. We used a heuristic set proposed by Forsell and Johansson. Participants were also asked to complete a demographic survey in which they rated their experience with each visualization type generally, their experience conducting heuristic evaluations, their experience using each visualization type to extract information, and their experience developing and creating each visualization type. The participants were shown a series of 5 common visualizations (Scatterplots, Sunbursts, Tree Maps, Parallel Sets, and Area Graphs) all displaying the VAST 2008 Mini Challenge Two- Migrant Boats data . Participants were given a suggested analytical question related to each visualization to keep in mind as they applied the 10 heuristics to each visualization. They were asked if they could find any usability problems in the visualizations and which heuristic(s) helped them to identify these problems. The participants were asked to rate how relevant each heuristic was for evaluating each visualization along with rating how confident they felt in their ability to apply the heuristic to evaluate each visualization. At the conclusion of the study, participants provided qualitative feedback on the usability issues of each visualization. Additionally, we collected eye tracking data on 6 of the 10 participants. We will be using this data to determine how long participants looked at the various visualizations and what parts of the visualizations they focused on. We had two hypotheses for this empirical evaluation: 1.The Forsell and Johansson heuristics of Orientation and help, Data set reduction, and Prompting are less useful for static visualizations because they require some interaction in order to test completely. 2.Participants that do not have experience with heuristic evaluation will provide lower evaluation scores for the heuristics compared to those that have experience. Related to hypothesis 1, as our study used only static visualizations, interaction-related usability challenges should be harder to identify. This should result in lower usefulness ratings for those heuristics. Preliminary results indicate that participants found it more difficult to apply heuristics to unfamiliar visualizations.

Williams, Ryan G.↗

Sensalert: A Real-time Group and Individual Health Tracking Application

Sensalert is a wearable health tracking application that enables monitoring of a target population’s health status in real-time. The application integrates health monitoring parameters from wearable sensors, e.g., temperature and heart rate, with relevant environmental parameters, e.g., weather and topography, and calculates the corresponding physiological strain index for each point in time. The objective of this application is to provide early warning, support decision making, and increase situational awareness by displaying comprehensive information and providing alerts at the right time and place. This application is the first of its kind being developed for integration into the Defense Threat Reduction Agencies Bio Surveillance Ecosystem (BSVE). We present the user requirements and design principles for developing this type of application. We hope this work helps and inspires future research in the area of wearable technologies for surveillance.

Huang, Zhuanyi↗

Wearable Sensor Application for Integrated Early Warning and Health Surveillance

Objective: The Wearable Sensor Application developed by Pacific Northwest National Laboratory (PNNL) provides an early warning system for stressors to individual and group health using physiologic and environmental indicators. The application integrates health monitoring parameters from wearable sensors, e.g., temperature and heart rate, with relevant environmental parameters, e.g., weather and landscape data, and calculates the corresponding physiological strain index. The information is presented to the analyst in a group and individual view with real-time alerting of abnormal health parameters. This application is the first of its kind being developed for integration into the Defense Threat Reduction Agency's Biosurveillance Ecosystem (BSVE).Introduction: Wearable devices are a low cost, minimally invasive way to monitor health. Sensor data provides real-time physiological indictors of an individual’s health status without the requirement of health care professionals or facilities. Information gleamed from wearable sensors can be used to better understand physiological stressors and prodromal symptoms. In addition, this data can be used to monitor individuals that are in high risk of health-related problems.However, raw data from wearable sensors can be overwhelming to process and laborious to monitor for an individual and, even more so, for a group of individuals. Often specific combination of ranges of sensor readings are indicative of changes to health status and need to be evaluated together or used to calculate specific signal parameters. In addition, the environment surrounding the individual needs to be considered when interpreting the data. To address these issues, PNNL has developed an application that collects, analyzes, and integrates wearable sensor data with geographic landscape and weather information to provide a real-time early alert and situational awareness tool for monitoring the health of groups and individuals.Methods: The prototype application described here was a product of PNNL’s BSVE Application Development Competition. The final product that will be deployed in the BSVE is currently under development by PNNL and will vary slightly in the exact design and architecture described.Data. Wearable sensor data was collected from the Rim2Rim (R2R) Watch Study of individuals hiking the Grand Canyon in Arizona [1]. Weather information was obtained from nearby weather stations and mapping features were derived from Google Maps.Calculations. A physiological Strain Index (PSI) was calculated using core temperature estimates derived through a Kalman Filter approach and heart rate [2,3].Application. The prototype backend application development was based in Python with a MongoDB. The front-end development was built using a scalable architecture and modular approach with components in React and D3.Results: A prototype application was developed this past summer through the PNNL BSVE App Competition (Fig 1). The application was aimed at visualizing wearable sensor data from the Grand Canyon R2R hike dataset. Simulated real-time analysis was used to calculate health status of individuals hiking based on measured physiological parameters and to alert to individuals with signs of physiologic health stress. Visualization tools were incorporated to enable sensor data for individuals and the group to be viewed simultaneously along with pertinent weather, geographic, and elevation data.Many features described in the prototype application will be incorporated into the final BSVE application. The key changes will be 1) the ability to select given time periods for viewing historical data as well as the real-time data collection, 2) environmental data and map view will come from BSVE internal data sources, and 3) the alerts will provide more information and have their own page for reviewing.Conclusions: The Wearable Sensor Application developed by PNNL for integration into the BSVE provides an early warning system for individual and group health using physiologic and environmental parameters. The application highlights health status from wearable sensors and relevant environmental parameters while monitoring a calculated physiological strain index. With this tool, an analyst can easily monitor the health of individuals and groups with the aid of real-time alerting tool for early detection of abnormal health parameters.

Early Warning↗

Design Science Methodology Applied to a Chemical Surveillance Tool

Public health surveillance systems gain significant benefits from integrating existing early incident detection systems,supported by closed data sources, with open source data.However, identifying potential alerting incidents relies on finding accurate, reliable sources and presenting the high volume of data in a way that increases analysts work efficiency; a challenge for any system that leverages open source data. In this paper, we present the design concept and the applied design science research methodology of ChemVeillance, a chemical analyst surveillance system.Our work portrays a system design and approach that translates theoretical methodology into practice creating a powerful surveillance system built for specific use cases.Researchers, designers, developers, and related professionals in the health surveillance community can build upon the principles and methodology described here to enhance and broaden current surveillance systems leading to improved situational awareness based on a robust integrated early warning system.

Huang, Zhuanyi↗