A Temporal Order Modeling Approach to Human Action Recognition from Multimodal Sensor Data
Explore the source record for details and available documents.
Jun Ye↗
SEARCH · Search NASA
Results for “Action Recognition Model”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
A model of EcoRII restriction endonuclease action: the active complex is most likely formed by one protein subunit and one DNA recognition site
To elucidate the mechanism of interaction of restriction endonuclease EcoRII with DNA, we studied by native gel electrophoresis the binding of this endonuclease to a set of synthetic DNA-duplexes containing the modified or canonical recognition sequence 5'-d(CCA/TGG)-3'. All binding substrate or substrate analogues tested could be divided into two major groups: (i) duplexes that, at the interaction with endonuclease EcoRII, form two types of stable complexes on native gel in the absence of Mg2+ cofactor; (ii) duplexes that form only one type of complex, observed both in the presence and absence of Mg2+. Unlike the latter, duplexes under the first group can be hydrolyzed by endonuclease. Data obtained suggest that the active complex is most likely formed by one protein subunit and one DNA recognition sequence. A model of EcoRII endonuclease action is presented.
Oligodeoxyribonucleotides/chemistry/metabolism↗
Plan recognition and generalization in command languages with application to telerobotics
A method for pragmatic inference as a necessary accompaniment to command languages is proposed. The approach taken focuses on the modeling and recognition of the human operator's intent, which relates sequences of domain actions ('plans') to changes in some model of the task environment. The salient feature of this module is that it captures some of the physical and linguistic contextual aspects of an instruction. This provides a basis for generalization and reinterpretation of the instruction in different task environments. The theoretical development is founded on previous work in computational linguistics and some recent models in the theory of action and intention. To illustrate these ideas, an experimental command language to a telerobot is implemented. The program consists of three different components: a robot graphic simulation, the command language itself, and the domain-independent pragmatic inference module. Examples of task instruction processes are provided to demonstrate the benefits of this approach.
Yared, Wael I.↗
Computer simulation and evaluation of edge detection algorithms and their application to automatic path selection
The construction of an autonomous roving vehicle requires the development of complex data-acquisition and processing systems, which determine the path along which the vehicle travels. Thus, a vehicle must possess algorithms which can (1) reliably detect obstacles by processing sensor data, (2) maintain a constantly updated model of its surroundings, and (3) direct its immediate actions to further a long range plan. The first function consisted of obstacle recognition. Obstacles may be identified by the use of edge detection techniques. Therefore, the Kalman Filter was implemented as part of a large scale computer simulation of the Mars Rover. The second function consisted of modeling the environment. The obstacle must be reconstructed from its edges, and the vast amount of data must be organized in a readily retrievable form. Therefore, a Terrain Modeller was developed which assembled and maintained a rectangular grid map of the planet. The third function consisted of directing the vehicle's actions.
Longendorfer, B. A.↗
Stress and Aeronautical Team Decision Making: Strengthening the Weak Links
A model that characterizes pilots'decision making in flight will be presented. Elements of the model that appear most vulnerable to stress will be examined in light of accidents and incidents. The model includes two major components: Situation assessment and choice of a course of action. While based on Klein's Recognition-Primed Decision Making, it is tailored to the aviation environment which includes certain features that may be common to other domains: Primarily, aviation is highly proceduralized and options are generally well known. What appears to make decisions difficult are ambiguity, time pressure, and risk. In addition, decisions must often be made while carrying out the standard procedures of flight, including checklists, review of approach plates, standard briefings, and communication with air traffic controllers or cabin crew. The effects of stressors on decision making by pilots with varying levels of expertise will be explored, along with strategies for strengthening the weak links.
Orasanu, Judith↗
A vision architecture for the extravehicular activity retriever
The Extravehicular Activity Retriever (EVAR) is a robotic device currently being developed by the Automation and Robotics Division at the NASA Johnson Space Center to support activities in the neighborhood of the Space Shuttle or Space Station Freedom. As the name implies, the Retriever's primary function will be to provide the capability to retrieve tools, equipment or other objects which have become detached from the spacecraft, but it will also be able to rescue a crew member who may have become inadvertently de-tethered. Later goals will include cooperative operations between a crew member and the Retriever such as fetching a tool that is required for servicing or maintenance operations. This report documents a preliminary design for a Vision System Planner (VSP) for the EVAR that is capable of achieving visual objectives provided to it by a high level task planner. Typical commands which the task planner might issue to the VSP relate to object recognition, object location determination, and obstacle detection. Upon receiving a command from the task planner, the VSP then plans a sequence of actions to achieve the specified objective using a model-based reasoning approach. This sequence may involve choosing an appropriate sensor, selecting an algorithm to process the data, reorienting the sensor, adjusting the effective resolution of the image using lens zooming capability, and/or requesting the task planner to reposition the EVAR to obtain a different view of the object. An initial version of the Vision System Planner which realizes the above capabilities using simulated images has been implemented and tested. The remaining sections describe the architecture and capabilities of the VSP and its relationship to the high level task planner. In addition, typical plans that are generated to achieve visual goals for various scenarios will be discussed. Specific topics to be addressed will include object search strategies, repositioning of the EVAR to improve the quality of information obtained from the sensors, complementary usage of the sensors and redundant capabilities.
Magee, Michael↗
A vision system planner for increasing the autonomy of the Extravehicular Activity Helper/Retriever
The Extravehicular Activity Retriever (EVAR) is a robotic device currently being developed by the Automation and Robotics Division at the NASA Johnson Space Center to support activities in the neighborhood of the Space Shuttle or Space Station Freedom. As the name implies, the Retriever's primary function will be to provide the capability to retrieve tools and equipment or other objects which have become detached from the spacecraft, but it will also be able to rescue a crew member who may have become inadvertently de-tethered. Later goals will include cooperative operations between a crew member and the Retriever such as fetching a tool that is required for servicing or maintenance operations. This paper documents a preliminary design for a Vision System Planner (VSP) for the EVAR that is capable of achieving visual objectives provided to it by a high level task planner. Typical commands which the task planner might issue to the VSP relate to object recognition, object location determination, and obstacle detection. Upon receiving a command from the task planner, the VSP then plans a sequence of actions to achieve the specified objective using a model-based reasoning approach. This sequence may involve choosing an appropriate sensor, selecting an algorithm to process the data, reorienting the sensor, adjusting the effective resolution of the image using lens zooming capability, and/or requesting the task planner to reposition the EVAR to obtain a different view of the object. An initial version of the Vision System Planner which realizes the above capabilities using simulated images has been implemented and tested. The remaining sections describe the architecture and capabilities of the VSP and its relationship to the high level task planner. In addition, typical plans that are generated to achieve visual goals for various scenarios are discussed. Specific topics to be addressed will include object search strategies, repositioning of the EVAR to improve the quality of information obtained from the sensors, and complementary usage of the sensors and redundant capabilities.
Magee, Michael↗
Incorporating Speech Recognition into a Natural User Interface
The Augmented/ Virtual Reality (AVR) Lab has been working to study the applicability of recent virtual and augmented reality hardware and software to KSC operations. This includes the Oculus Rift, HTC Vive, Microsoft HoloLens, and Unity game engine. My project in this lab is to integrate voice recognition and voice commands into an easy to modify system that can be added to an existing portion of a Natural User Interface (NUI). A NUI is an intuitive and simple to use interface incorporating visual, touch, and speech recognition. The inclusion of speech recognition capability will allow users to perform actions or make inquiries using only their voice. The simplicity of needing only to speak to control an on-screen object or enact some digital action means that any user can quickly become accustomed to using this system. Multiple programs were tested for use in a speech command and recognition system. Sphinx4 translates speech to text using a Hidden Markov Model (HMM) based Language Model, an Acoustic Model, and a word Dictionary running on Java. PocketSphinx had similar functionality to Sphinx4 but instead ran on C. However, neither of these programs were ideal as building a Java or C wrapper slowed performance. The most ideal speech recognition system tested was the Unity Engine Grammar Recognizer. A Context Free Grammar (CFG) structure is written in an XML file to specify the structure of phrases and words that will be recognized by Unity Grammar Recognizer. Using Speech Recognition Grammar Specification (SRGS) 1.0 makes modifying the recognized combinations of words and phrases very simple and quick to do. With SRGS 1.0, semantic information can also be added to the XML file, which allows for even more control over how spoken words and phrases are interpreted by Unity. Additionally, using a CFG with SRGS 1.0 produces a Finite State Machine (FSM) functionality limiting the potential for incorrectly heard words or phrases. The purpose of my project was to investigate options for a Speech Recognition System. To that end I attempted to integrate Sphinx4 into a user interface. Sphinx4 had great accuracy and is the only free program able to perform offline speech dictation. However it had a limited dictionary of words that could be recognized, single syllable words were almost impossible for it to hear, and since it ran on Java it could not be integrated into the Unity based NUI. PocketSphinx ran much faster than Sphinx4 which would've made it ideal as a plugin to the Unity NUI, unfortunately creating a C# wrapper for the C code made the program unusable with Unity due to the wrapper slowing code execution and class files becoming unreachable. Unity Grammar Recognizer is the ideal speech recognition interface, it is flexible in recognizing multiple variations of the same command. It is also the most accurate program in recognizing speech due to using an XML grammar to specify speech structure instead of relying solely on a Dictionary and Language model. The Unity Grammar Recognizer will be used with the NUI for these reasons as well as being written in C# which further simplifies the incorporation.
Chapa, Nicholas↗
Gesture Based Control and EMG Decomposition
This paper presents two probabilistic developments for use with Electromyograms (EMG). First described is a new-electric interface for virtual device control based on gesture recognition. The second development is a Bayesian method for decomposing EMG into individual motor unit action potentials. This more complex technique will then allow for higher resolution in separating muscle groups for gesture recognition. All examples presented rely upon sampling EMG data from a subject's forearm. The gesture based recognition uses pattern recognition software that has been trained to identify gestures from among a given set of gestures. The pattern recognition software consists of hidden Markov models which are used to recognize the gestures as they are being performed in real-time from moving averages of EMG. Two experiments were conducted to examine the feasibility of this interface technology. The first replicated a virtual joystick interface, and the second replicated a keyboard. Moving averages of EMG do not provide easy distinction between fine muscle groups. To better distinguish between different fine motor skill muscle groups we present a Bayesian algorithm to separate surface EMG into representative motor unit action potentials. The algorithm is based upon differential Variable Component Analysis (dVCA) [l], [2] which was originally developed for Electroencephalograms. The algorithm uses a simple forward model representing a mixture of motor unit action potentials as seen across multiple channels. The parameters of this model are iteratively optimized for each component. Results are presented on both synthetic and experimental EMG data. The synthetic case has additive white noise and is compared with known components. The experimental EMG data was obtained using a custom linear electrode array designed for this study.
Wheeler, Kevin R.↗
How far away is plug 'n' play? Assessing the near-term potential of sonification and auditory display
The commercial music industry offers a broad range of plug 'n' play hardware and software scaled to music professionals and scaled to a broad consumer market. The principles of sound synthesis utilized in these products are relevant to application in virtual environments (VE). However, the closed architectures used in commercial music synthesizers are prohibitive to low-level control during real-time rendering, and the algorithms and sounds themselves are not standardized from product to product. To bring sound into VE requires a new generation of open architectures designed for human-controlled performance from interfaces embedded in immersive environments. This presentation addresses the state of the sonic arts in scientific computing and VE, analyzes research challenges facing sound computation, and offers suggestions regarding tools we might expect to become available during the next few years. A list of classes of audio functionality in VE includes sonification -- the use of sound to represent data from numerical models; 3D auditory display (spatialization and localization, also called externalization); navigation cues for positional orientation and for finding items or regions inside large spaces; voice recognition for controlling the computer; external communications between users in different spaces; and feedback to the user concerning his own actions or the state of the application interface. To effectively convey this considerable variety of signals, we apply principles of acoustic design to ensure the messages are neither confusing nor competing. We approach the design of auditory experience through a comprehensive structure for messages, and message interplay we refer to as an Automated Sound Environment. Our research addresses real-time sound synthesis, real-time signal processing and localization, interactive control of high-dimensional systems, and synchronization of sound and graphics.
Bargar, Robin↗
Acting to gain information
This report is concerned with agents that act to gain information. In previous work, we developed agent models combining qualitative modeling with real-time control. That work, however, focused primarily on actions that affect physical states of the environment. The current study extends that work by explicitly considering problems of active information-gathering and by exploring specialized aspects of information-gathering in computational perception, learning, and language. In our theoretical investigations, we analyzed agents into their perceptual and action components and identified these with elements of a state-machine model of control. The mathematical properties of each was developed in isolation and interactions were then studied. We considered the complexity dimension and the uncertainty dimension and related these to intelligent-agent design issues. We also explored active information gathering in visual processing. Working within the active vision paradigm, we developed a concept of 'minimal meaningful measurements' suitable for demand-driven vision. We then developed and tested an architecture for ongoing recognition and interpretation of visual information. In the area of information gathering through learning, we explored techniques for coping with combinatorial complexity. We also explored information gathering through explicit linguistic action by considering the nature of conversational rules, coordination, and situated communication behavior.
Rosenchein, Stanley J.↗
Topics in inference and decision-making with partial knowledge
Two essential elements needed in the process of inference and decision-making are prior probabilities and likelihood functions. When both of these components are known accurately and precisely, the Bayesian approach provides a consistent and coherent solution to the problems of inference and decision-making. In many situations, however, either one or both of the above components may not be known, or at least may not be known precisely. This problem of partial knowledge about prior probabilities and likelihood functions is addressed. There are at least two ways to cope with this lack of precise knowledge: robust methods, and interval-valued methods. First, ways of modeling imprecision and indeterminacies in prior probabilities and likelihood functions are examined; then how imprecision in the above components carries over to the posterior probabilities is examined. Finally, the problem of decision making with imprecise posterior probabilities and the consequences of such actions are addressed. Application areas where the above problems may occur are in statistical pattern recognition problems, for example, the problem of classification of high-dimensional multispectral remote sensing image data.
Safavian, S. Rasoul↗
Comprehensive Environmental Informatics System (CEIS) Integrating Crew and Vehicle Environmental Health
Integrated Vehicle Health Management (IVHM) systems have been pursued as highly integrated systems that include smart sensors, diagnostic and prognostics software for assessments of real-time and life-cycle vehicle health information. Inclusive to such a system is the requirement to monitor the environmental health within the vehicle and the occupants of the vehicle. In this regard an enterprise approach to informatics is used to develop a methodology entitled, Comprehensive Environmental Informatics System (CEIS). The hardware and software technologies integrated into this system will be embedded in the vehicle subsystems, and maintenance operations, to provide both real-time and life-cycle health information of the environment within the vehicle cabin and of its occupants. This comprehensive information database will enable informed decision making and logistics management. One key element of the CEIS is interoperability for data acquisition and archive between environment and human system monitoring. With comprehensive components the data acquired in this system will use model based reasoning systems for subsystem and system level managers, advanced on-board and ground-based mission and maintenance planners to assess system functionality. Knowledge databases of the vehicle health state will be continuously updated and reported for critical failure modes, and routinely updated and reported for life cycle condition trending. Sufficient intelligence, including evidence-based engineering practices which are analogous to evidencebased medicine practices, will be included in the CEIS to result in more rapid recognition of off-nominal operation to enable quicker corrective actions. This will result from better information (rather than just data) for improved crew/operator situational awareness, which will produce significant vehicle and crew safety improvements, as well as increasing the chance for mission success, future mission planning as well as training. Other benefits include improved reliability, increase safety in operations and cost of operations. The cost benefits stem from significantly reduced processing and operations manpower, predictive maintenance for systems and subjects. The improvements in vehicle functionality and cost will result from increased prognostic and diagnostic capability due to the detailed total human exploration system health knowledge from CEIS. A collateral benefit is that there will be closer observations of the vehicle occupants as wrist watch sized devices are worn for continuous health monitoring. Additional database acquisition will stem from activities in countermeasure practices to ensure peak performance capability by occupants of the vehicle. The CEIS will provide data from advanced sensing technologies and informatics modeling which will be useful in problem troubleshooting, and improving NASA s awareness of systems during operation.
Nall, Mark E.↗
Impasse-Driven Tutoring for Reactive Skill Acquisition
We introduce a new approach to intelligent tutoring in performance-oriented training environments based on a method called situated plan attribution. The aim of this method is to provide contextualized tutoring for procedural tasks requiring reactive, goal-oriented skills. We use the term plan attribution instead of plan recognition because it does not assume that the problem solver is consciously executing plan. We avoid some of the pitfalls of other popularly used methods, i.e., model tracing and procedure net grammars, by selectively using an expert cognitive model to generate advice after detecting a problem solving impasse. The tutor attributes a set of plans to the student based on a task description. Each action is evaluated with respect to: the student's attributed plans, its actual effects on the training device, and the contextualized goals associated with the plans.
Johnson, W. Lewis↗
CATS-based Air Traffic Controller Agents
This report describes intelligent agents that function as air traffic controllers. Each agent controls traffic in a single sector in real time; agents controlling traffic in adjoining sectors can coordinate to manage an arrival flow across a given meter fix. The purpose of this research is threefold. First, it seeks to study the design of agents for controlling complex systems. In particular, it investigates agent planning and reactive control functionality in a dynamic environment in which a variety perceptual and decision making skills play a central role. It examines how heuristic rules can be applied to model planning and decision making skills, rather than attempting to apply optimization methods. Thus, the research attempts to develop intelligent agents that provide an approximation of human air traffic controller behavior that, while not based on an explicit cognitive model, does produce task performance consistent with the way human air traffic controllers operate. Second, this research sought to extend previous research on using the Crew Activity Tracking System (CATS) as the basis for intelligent agents. The agents use a high-level model of air traffic controller activities to structure the control task. To execute an activity in the CATS model, according to the current task context, the agents reference a 'skill library' and 'control rules' that in turn execute the pattern recognition, planning, and decision-making required to perform the activity. Applying the skills enables the agents to modify their representation of the current control situation (i.e., the 'flick' or 'picture'). The updated representation supports the next activity in a cycle of action that, taken as a whole, simulates air traffic controller behavior. A third, practical motivation for this research is to use intelligent agents to support evaluation of new air traffic control (ATC) methods to support new Air Traffic Management (ATM) concepts. Current approaches that use large, human-in-the-loop simulations are unquestionably valuable for this purpose, but pose considerable logistical, fiscal, and experimental control problems. First, data analysis is extremely complicated, owing simply to the large number of participants and data sources in such simulations. In addition, experienced human air traffic controllers working adjacent sectors tend to flexibly adapt to the evolving control problem - potentially shifting to other strategies than those under investigation. In addition, their performance is tightly coupled to the control interface, which in the development phase may support some concepts and supporting strategies better than others. A simple shift in strategy by one controller can change the character of a particular traffic scenario dramatically, which makes experimental comparison of ATC performance under different traffic scenarios difficult. Training a given team of controllers on operations under a new ATM concept for a sufficient period of time could avert such difficulties, but instituting an adequate training program is expensive and logistically difficult.
Callantine, Todd J.↗
Human Research Program (HRP) Exploration Medical Capability (ExMC) Standing Review Panel (SRP)
The SRP believes strongly that regularly performed in-flight crew assessments are needed in order to identify a change in health status before a medical condition becomes clinically apparent. It is this early recognition in change that constitutes the foundation of the "occupational health model" expounded in the HRP Requirements Document as a key component of the HRP risk mitigation strategy that will enable its objective of "prevention and mitigation of human health and performance risks". A regular crew status examination of physiological and clinical performance is needed. This can be accomplished through instrumented monitoring of routine embedded tasks. The SRP recommends addition of a new gap to address this action under Category 3.0 Mitigate the Risk. This new gap is closely associated with Task 4.19 which addresses the lack of adequate biomedical monitoring capabilities for performing periodic clinical status evaluations and contingency medical monitoring. A corollary to these gaps is the critical emphasis on preventive medicine, not only during pre- and post-flight phases of a mission as is the current practice, but continued into the in-flight phases of exploration class missions.
Cintron, Nitza↗
Generating Control Commands From Gestures Sensed by EMG
An effort is under way to develop noninvasive neuro-electric interfaces through which human operators could control systems as diverse as simple mechanical devices, computers, aircraft, and even spacecraft. The basic idea is to use electrodes on the surface of the skin to acquire electromyographic (EMG) signals associated with gestures, digitize and process the EMG signals to recognize the gestures, and generate digital commands to perform the actions signified by the gestures. In an experimental prototype of such an interface, the EMG signals associated with hand gestures are acquired by use of several pairs of electrodes mounted in sleeves on a subject s forearm (see figure). The EMG signals are sampled and digitized. The resulting time-series data are fed as input to pattern-recognition software that has been trained to distinguish gestures from a given gesture set. The software implements, among other things, hidden Markov models, which are used to recognize the gestures as they are being performed in real time. Thus far, two experiments have been performed on the prototype interface to demonstrate feasibility: an experiment in synthesizing the output of a joystick and an experiment in synthesizing the output of a computer or typewriter keyboard. In the joystick experiment, the EMG signals were processed into joystick commands for a realistic flight simulator for an airplane. The acting pilot reached out into the air, grabbed an imaginary joystick, and pretended to manipulate the joystick to achieve left and right banks and up and down pitches of the simulated airplane. In the keyboard experiment, the subject pretended to type on a numerical keypad, and the EMG signals were processed into keystrokes. The results of the experiments demonstrate the basic feasibility of this method while indicating the need for further research to reduce the incidence of errors (including confusion among gestures). Topics that must be addressed include the numbers and arrangements of electrodes needed to acquire sufficient data; refinements in the acquisition, filtering, and digitization of EMG signals; and methods of training the pattern- recognition software. The joystick and keyboard simulations were chosen for the initial experiments because they are familiar to many computer users. It is anticipated that, ultimately, interfaces would utilize EMG signals associated with movements more nearly natural than those associated with joysticks or keyboards. Future versions of the pattern-recognition software are planned to be capable of adapting to the preferences and day-today variations in EMG outputs of individual users; this capability for adaptation would also make it possible to select gestures that, to a given user, feel the most nearly natural for generating control signals for a given task (provided that there are enough properly positioned electrodes to acquire the EMG signals from the muscles involved in the gestures).
Wheeler, Kevin R.↗
Monitoring issues from a modeling perspective
Recognition that earth's climate and biogeophysical conditions are likely changing due to human activities has led to a heightened awareness of the need for improved long-term global monitoring. The present long-term measurement efforts tend to be spotty in space, inadequately calibrated in time, and internally inconsistent with respect to other instruments and measured quantities. In some cases, such as most of the biosphere, most chemicals, and much of the ocean, even a minimal monitoring program is not available. Recently, it has become painfully evident that emerging global change issues demand information and insights that the present global monitoring system simply cannot supply. This is because a monitoring system must provide much more than a statement of change at a given level of statistical confidence. It must describe changes in diverse parts of the entire earth system on regional to global scales. It must be able to provide enough input to allow an integrated physical characterization of the changes that have occurred. Finally, it must allow a separation of the observed changes into their natural and anthropogenic parts. The enormous policy significance of global change virtually guarantees an unprecedented level of scrutiny of the changes in the earth system and why they are happening. These pressures create a number of emerging challenges and opportunities. For example, they will require a growing partnership between the observational programs and the theory/modeling community. Without this partnership, the scientific community will likely fall short in the monitoring effort. The monitoring challenge before us is not to solve the problem now, but rather to set appropriate actions in motion so as to create the required framework for solution. Each individual piece needs to establish its role in the large problem and how the required interactions are to take place. Below, we emphasize some of the needs and opportunities that could and should be addressed through participation by the theoreticians and modelers in the global change monitoring effort.
Mahlman, Jerry D.↗