Search NASA⌕ Search

Engineering topics

Grinstein, Georges

Publications and source records attributed to Grinstein, Georges.

VAST Challenge 2014: The Kronos Incident

The 2014 IEEE Visual Analytics Science and Technology (VAST) Challenge presented researchers with a single fictitious scenario: the disappearance of staff members of the GASTech oil and gas company on location on the island of Kronos. A group named the Protectors of Kronos (POK) was the prime suspect in the disappearance. Three mini-challenges and a grand challenge were offered. Mini-challenge 1 included multiple types of text data for participants to provide a timeline of key events and characterize the POK, mini-challenge 2 focused on individuals’ movement and financial data for participants to provide patterns of daily life, and mini-challenge 3 featured real-time streaming social media and emergency service data for participants to provide hostage and kidnapper information. The grand challenge asked the participants to integrate results and generate a synopsis of events. The VAST Challenge received 73 submissions from 13 countries

visual analytics, visualization, evaluation, VAST ↗

The VAST Challenge: History, Scope, and Outcomes: An introduction to the Special Issue

Visual analytics aims to facilitate human insight from complex data via a combination of visual representations, interaction techniques, and supporting algorithms. To create new tools and techniques that achieve this goal requires that researchers have an understanding of analytical questions to be addressed, data that illustrates the complexities and ambiguities found in realistic analytic settings, and methods for evaluating whether the plausible insights are gained through use of the new methods. However, researchers do not, generally speaking, have access to analysts who can articulate their problems or operational data that is used for analysis. To fill this gap, the Visual Analytics Science and Technology (VAST) Challenge has been held annually since 2006. The VAST Challenge provides an opportunity for researchers to experiment with realistic but not real problems, using realistic synthetic data with known events embedded. Since its inception, the VAST Challenge has evolved along with the visual analytics research community to pose more complex challenges, ranging from text analysis to video analysis to large scale network log analysis. The seven years of the VAST Challenge have seen advancements in research and development, education, evaluation, and in the challenge process itself. This special issue of Information Visualization highlights some of the noteworthy advancements in each of these areas. Some of these papers focus on important research questions related to the challenge itself, and other papers focus on innovative research that has been shaped by participation in the challenge. This paper describes the VAST Challenge process and benefits in detail. It also provides an introduction to and context for the remaining papers in the issue.

visual analytics, sensemaking, evaluation, crowdso↗

Evaluation of Visual Analytics Environments: The Road to the Visual Analytics Science and Technology Challenge Evaluation Methodology

The evaluation of visual analytics environments was a topic in Illuminating the Path [Thomas 2005] as a critical aspect of moving research into practice. For a thorough understanding of the utility of the systems available, evaluation not only involves assessing the visualizations, interactions or data processing algorithms themselves, but also the complex processes that a tool is meant to support (such as exploratory data analysis and reasoning, communication through visualization, or collaborative data analysis [Lam 2012; Carpendale 2007]). Researchers and practitioners in the field have long identified many of the challenges faced when planning, conducting, and executing an evaluation of a visualization tool or system [Plaisant 2004]. Evaluation is needed to verify that algorithms and software systems work correctly and that they represent improvements over the current infrastructure. Additionally to effectively transfer new software into a working environment, it is necessary to ensure that the software has utility for the end-users and that the software can be incorporated into the end-user’s infrastructure and work practices. Evaluation test beds require datasets, tasks, metrics and evaluation methodologies. As noted in [Thomas 2005] it is difficult and expensive for any one researcher to setup an evaluation test bed so in many cases evaluation is setup for communities of researchers or for various research projects or programs. Examples of successful community evaluations can be found [Chinchor 1993; Voorhees 2007; FRGC 2012]. As visual analytics environments are intended to facilitate the work of human analysts, one aspect of evaluation needs to focus on the utility of the software to the end-user. This requires representative users, representative tasks, and metrics that measure the utility to the end-user. This is even more difficult as now one aspect of the test methodology is access to representative end-users to participate in the evaluation. In many cases the sensitive nature of data and tasks and difficult access to busy analysts puts even more of a burden on researchers to complete this type of evaluation. User-centered design goes beyond evaluation and starts with the user [Beyer 1997, Shneiderman 2009]. Having some knowledge of the type of data, tasks, and work practices helps researchers and developers know the correct paths to pursue in their work. When access to the end-users is problematic at best and impossible at worst, user-centered design becomes difficult. Researchers are unlikely to go to work on the type of problems faced by inaccessible users. Commercial vendors have difficulties evaluating and improving their products when they cannot observe real users working with their products. In well-established fields such as web site design or office software design, user-interface guidelines have been developed based on the results of empirical studies or the experience of experts. Guidelines can speed up the design process and replace some of the need for observation of actual users [heuristics review references]. In 2006 when the visual analytics community was initially getting organized, no such guidelines existed. Therefore, we were faced with the problem of developing an evaluation framework for the field of visual analytics that would provide representative situations and datasets, representative tasks and utility metrics, and finally a test methodology which would include a surrogate for representative users, increase interest in conducting research in the field, and provide sufficient feedback to the researchers so that they could improve their systems.

evaluation↗

A Reflection on Seven Years of the VAST Challenge

We describe the evolution of the IEEE Visual Analytics Science and Technology (VAST) Challenge from its origin in 2006 to present (2012). The VAST Challenge has provided an opportunity for visual analytics researchers to test their innovative thought on approaching problems in a wide range of subject domains against realistic datasets and problem scenarios. Over time, the Challenge has changed to correspond to the needs of researchers and users. We describe those changes and the impacts they have had on topics selected, data and questions offered, submissions received, and the Challenge format

visual analytics, evaluation↗

VAST 2010 Challenge: Arms Dealings and Pandemics

The 5th VAST Challenge consisted of three mini-challenges that involved both intelligence analysis and bioinformatics. Teams could solve one, two or all three mini-challenges and assess the overall situation to enter the Grand Challenge. Mini-challenge one involved text reports about people and events giving information about arms dealers, situations in various countries and linkages between different countries. Mini-challenge two involved hospital admission and death records from various countries providing information about the spread of a world wide pandemic. Mini-challenge three involved genetic data to be used to identify the origin of the pandemic and the most dangerous viral mutations. The Grand Challenge was to determine how these various mini-challenges were connected. As always the goal was to analyze the data and provide novel interactive visualizations useful in the analytic process. We received 58 submissions in total and gave 15 awards.

59 BASIC BIOLOGICAL SCIENCES↗

VAST 2009 Challenge: An Insider Threat

The 4th year of the VAST Challenge offered participating teams the opportunity to analyze three mini challenges and to combine the data and analyses and enter the Grand challenge. In this paper we provide a description of the data collections used, the overall scenario and the specific questions asked in the mini challenges. We also discuss the infrastructure developed to evaluate accuracy measures in this year’s social network mini challenge. We provide a few examples of visualizations that merited awards. As in previous year, The VAST 2009 Challenge was held in conjunction with the 2009 IEEE VAST Symposium.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗

VAST 2008 Challenge: Introducing Mini-Challenges

Visual analytics experts realize that one effective way to push the field forward and to develop metrics for measuring the performance of various visual analytics components is to hold an annual competition. In its third year, the authors restructured the contest format used in 2006 and 2007 to reduce the barriers to participation and offered four mini-challenges and a Grand Challenge. The VAST 2008 Challenge was held in conjunction with the 2008 IEEE VAST Symposium. Participants were to use visual analytic tools to explore four heterogeneous data collections to analyze the activities of a fictitious, controversial movement. Questions asked in the Grand Challenge required the participants to analyze and synthesize data from all four data sets. In this paper we give a brief overview of the data sets, the tasks, the participation, the judging, and the results.

97 MATHEMATICS AND COMPUTING↗